What makes a new technology trustworthy? It’s a short question, yet riddled with much complexity.
When magnetic resonance imaging technology was being developed in the 1970s, the primary innovator, Dr. Peter Mansfield, used himself as a guinea pig: scrambling himself into the prototype MRI scanner to test the effectiveness of his research idea that magnets could provide clear diagnostic images of internal body organs. His was deemed to be a high risk approach: colleagues and peers counselled against the potential risks (‘they are powerful magnets: you don’t know what the long term consequences could be on you’) but he persevered.
After years of development, testing and countless prototypes and a fulsome safety testing, the technology was deemed to be ready for wider use: but would people trust MRI machines to be safe? And would their results be trusted in complex medical contexts?
The answer to both eventually was a resounding yes: MRI machines have proven to be one of the safest and most effective diagnostic tools available.
But in the early days of MRI, there was strong reticence on its use: the magnetic technology had associations with nuclear science, which the general public had fears of post WWII. Because there were so few success stories as the technology hadn’t been widely trialled, there was scepticism on whether the technology worked without causing harm. The user experience was also known to be challenging: the noise from within a machine reached >100dB and claustrophobia was a common complaint.
Fast forward to today, and draw parallels with the new world of AI: much of the same fears and reticence exists. There is a widespread fear of AI: “AI will take your jobs” or “the robots are going to take over”. The technology is deemed to be new as, for most people, the AI era started when ChatGPT was released in 2021 (albeit it’s actually a technology developed by DARPA in the 1950s).
The AI user experience is also novel and developing, with the current chat-based form factor the flavour of the month. Then we have the big blocker to trust for AI and something of a hidden secret amongst data scientists: it is close to impossible to explain how the majority of AI models actually work – which is a bit shocking!!
AI developers and data scientists spend months developing their algorithms. Layer upon layer of neural networks are developed and integrated. Designed to find patterns in billions of rows of data, they frequently reach thousands of interconnecting layers with a spider-web of interconnections to enable a prediction such as whether house prices will go up or down, or how many new customers will be acquired this month via Facebook ads.
Figure 1: Deep learning model architectures: understandable and explainable?
Now try this the next time you speak to a data scientist or AI engineer: Ask them to explain how their AI system works and you’ll receive a whole host of deep technical responses that sound convincing, while in actuality they aren't as deep learning algorithms are virtually impossible to decipher and understand. Deep learning algorithms are black-box, not glass-box.
With that backdrop, how can AI be trusted?
When we think of building human trusted AI systems at Ergodic, there are several dimensions that we consider [4]:
Figure 2: Parameters of Trust [1]
Integrity elements relate to external standards, certifications and regulations which can be adhered to or acquired to rubber-stamp the level of trustworthiness of AI. Abilities pertain to specific capabilities of AI – for example, whether they are explainable or reliable under certain technical conditions.
Benevolence factors invoke external characteristics such as whether the AI acts ethically given an input or constraint, or how sustainable a model is from an energy consumption perspective.
If there is a silver bullet for building trust in AI, explainability is that bullet [5]. Explainable AI is AI that can coherently describe why an output from a model was a certain value given a specific input. Let’s take an example of an AI developed to automate decisions on automotive loans: should a customer get a loan or not and what is the maximum amount of loan to be offered, if positive.
In this context, an explainable AI model can show why a customer was offered a loan - they perhaps had a strong credit rating, valuable assets and lived in a low default neighbourhood - and why the level of the loan was set. Taking this example further, an explainable AI model could also provide the reasons why a specific loan application was rejected. This is crucial for many reasons: recent AI regulations in Europe are trending to explainability being crucial for adoption and regulatory acceptance. Further, being able to prove that a loan rejection was for say an applicant’s credit history rather than more nefarious reasons such as their gender, race or sexuality, is going to become more crucial as the deployment of AI models becomes more widespread. This aspect of fairness with respect to gender, race or other protected attributes is another tenet of increasing trust in AI. Most of the existing AI modelling approaches struggle to maintain fairness: how can they when their outputs are inherently blackbox and unexplainable.
Explainable models, by their very nature, can be designed to be fair and reduce the bias found in historical data. Outputs can be stress tested under different scenarios: for example if two automotive loan applications are submitted to a model, both with the same attributes with the exception of the gender of the applicant to prove fairness in outcome. This is important. If AI is to be trusted, it should aim to be essentially bias-free. In principle, that means reducing the reliance on historic data to drive algorithm output.
So how do we get there? We believe that a different approach is required, one that builds trust from the get go rather than attempting to add it after the fact.
How we, at Ergodic, plan to get there is a topic for another deep dive which will be coming soon! Follow us on LinkedIn and subscribe to our newsletter to receive a notification!
[1]: Nature: Trust in AI: progress, challenges, and future directions
[2]: Mckinsey: Building AI trust: The key role of explainability:
https://www.mckinsey.com/capabilities/quantumblack/our-insights/building-ai-trust-the-key-role-of-explainability
[3] HBR: AIs Trust Problem:
https://hbr.org/2024/05/ais-trust-problem
[4] Can we trust AI? An empirical investigation of trust requirements and guide to successful AI adoption.
https://doi.org/10.1108/JEIM-06-2020-0233
[5] Zolanvari M, Yang Z, Khan K, Jain R, Meskin N (2021) TRUST XAI: model-agnostic explanations for AI with a case study on IIoT security. IEEE Internet Things J 10(4):2967–2978