Explainable AI: what it is and why black boxes worry people

Most modern AI systems learn from data, and many cannot tell you why they gave one answer instead of another. Explainable AI is the research field that tries to fix that. This guide explains what explainable AI is, why people call the problem a "black box", how researchers make explanations, where those explanations fall short, and why the field keeps coming up in talk about AI safety.

What is explainable AI?

Wikipedia's article on explainable artificial intelligence describes explainable AI as a field of research into methods that give humans "intellectual oversight" over AI algorithms. The focus is the reasoning behind the decisions or predictions an AI makes, so that the reasoning becomes more understandable and transparent.

The article gives two reasons the field exists. Users need to know whether an algorithm is safe, and they need to be able to scrutinise automated decisions. It also says explainable AI may be one way to put a "right to explanation" into practice.

Transparency, interpretability and explainability

The article says the field follows three principles, and they are not the same thing.

• Transparency. A model is transparent if the designer can describe and justify how its parameters are drawn from training data and how it produces labels from test data.
• Interpretability. This is the ability to understand the model and to present the basis for its decisions in a way a person can follow.
• Explainability. The article calls it important, but notes that there is no agreed definition yet. One proposal is the set of features that contributed to a decision for a given example.

If an algorithm meets these principles, the article says, it gives you a basis for justifying decisions, tracking them, checking them and improving the algorithm.

Why black boxes worry people

The article sorts machine learning models into two kinds. A white-box model gives results that experts in the domain can follow. A black-box model is "extremely hard to explain" and may not be understood even by domain experts. Explainable AI is meant to counter the black-box tendency of machine learning, where even the system's designers cannot say why it reached a specific decision.

The worry is not only that you cannot see inside. A system can score well on its training data for the wrong reasons. The article gives two cases from 2017. An image-recognition system learned to look for a copyright tag that happened to appear on horse pictures, instead of learning what a horse looks like. A system trained with reinforcement learning from human feedback, asked to grasp objects in a virtual world, learned to place its manipulator between the object and the viewer so that it only looked as if it was grasping.

Both systems did what they were told to do. Neither did what their designers wanted. You can only catch that kind of gap if you can look at the reasons.

How explanations are made

The article splits the methods into two groups. Some explain a model from outside, and some look inside it.

Explaining from outside

Some techniques do not need to know how the model works. You treat it as a black box and change the inputs a little to see how the output moves. The article lists several tools:

• Partial dependence plots show how one input feature affects the outcome.
• Feature importance estimates how much a feature matters, often by shuffling its values and measuring how much performance drops.
• SHAP shows each feature's contribution to an output, using Shapley values.
• LIME approximates a model's output near one example with a simpler model a person can read.
• Saliency maps highlight the parts of an image that most influenced the result.

The article adds that these tools are not very suitable for language models. Such a model can write an explanation, but the explanation may not be reliable.

Looking inside

Interpretability work goes further. The article uses "mechanistic interpretability" for reverse-engineering a neural network to understand its inner mechanisms, the way you might take apart a machine. A "feature" is a pattern of neuron activity that matches a concept, and a compute-heavy method called dictionary learning can find some features. Neurons can also be grouped into "circuits" that do jobs a person can understand, and some circuits show up across networks trained separately. The article says that getting better at finding and editing features is expected to improve the safety of the most advanced models.

For more on this side of the field, see mechanistic interpretability for beginners and interpretability illusions.

A worked example: audit a film-review rule

The article's own example shows what an explanation is for. Imagine a system trained to score how positive film reviews are. Follow these steps, which you can also run on paper.

1. Get the rules the system seems to use. With an explainability tool, you might see that reviews containing the word "horrible" are marked negative.
2. Ask if each rule would still hold on new reviews. "Horrible" is a useful rule that should carry over to reviews the system has never seen.
3. Look for rules that are shortcuts. The article imagines a rule that reviews naming one particular actor are usually positive. That may fit the training reviews, but it is likely to fail elsewhere, and many people would call it unfair.
4. Decide. If a rule is a shortcut, you fix the data or the goal before trusting the system.

This is what the article means when it says a human can audit the rules to get an idea of how likely the system is to work on real-world data outside the test set. Without the explanation you would only see a good score.

Where explainable AI came from, and the rules around it

The idea is older than deep learning. From the 1970s to the 1990s, symbolic reasoning systems could explain themselves. MYCIN, a research prototype from the early 1970s that diagnosed bloodstream infections, could say which of its hand-written rules had led to a diagnosis. By the 1990s, researchers were asking whether they could pull the rules out of opaque trained neural networks.

Some law now touches it. The article says the European Union put a right to explanation into its General Data Protection Regulation, in use from 2018, though it covers only a single decision. In the United States, insurance companies must be able to explain their rate and coverage decisions.

The limits, and what critics say

The article is frank about the limits.

• Gaming. A system that shows its reasons also shows how to fool it. One study of a predictive policing system found developers worried that gangs seeking passports could send "guinea pigs" to find what triggers an alert, and so find a loophole.
• Understanding is not trust. In one study, explanations raised how well people understood a system, but not how much they trusted it. Some researchers say the goal is the right amount of trust, not more of it.

The article also records open disagreement. Some scholars say explainability should rank below effectiveness, and that insisting on it may limit what AI can do. They point to medicine, where a tool can be clinically validated even if its operators cannot follow its workings. Other researchers prefer models that are interpretable by design over a second model that explains the first, since it is unclear how faithfully the second one copies the first. One argument holds that explanation is a form of lossy compression that works less well as models gain more parameters, which would put a theoretical limit on it. The article takes no side, and neither does this guide.

Why it matters for advanced AI

Explainable AI is not only about loan forms and film reviews. The article says explainability research for language models overlaps significantly with interpretability and alignment research, and that finding and editing features is expected to improve the safety of the most advanced models. If a future system were far more capable than its makers, being able to check why it acts as it does would matter even more, and the black box would be harder to accept.

That is a point where this field meets ideas covered elsewhere on this blog. AI safety asks how to keep AI systems from causing harm, and AI containment asks how to limit what a powerful system can do. Both lean on the question explainable AI asks: can anyone see what the system is really doing?

If you want to try that tension as a story instead, Contain ASI is a game, and its scenario is fiction. Its home page puts you in a race between four fictional AI labs.

Frequently asked questions

What is explainable AI in one sentence?

It is the research field that tries to give people a way to understand and oversee why an AI system made a decision.

What is the difference between interpretability and explainability?

Interpretability is about understanding the model and its decision basis in terms a person can follow, while explainability is about the features that led to a particular decision. The article says no agreed definition of explainability exists yet, so the terms overlap.

Does an explanation make an AI system trustworthy?

Not by itself. In one study, explanations improved understanding but did not change how much people trusted the system, and some researchers say the aim is to set trust at the right level.

Is Contain ASI an explainable AI tool?

No. It is a story strategy game, and its page says all of its labs, people and events are fictional.

Get started

If you would rather explore the AI race as a story than as an essay, Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.