Can AI be trusted? What trust means and what the evidence shows

Can AI be trusted? You might ask a chatbot about a rash, a contract clause or a tax rule, and get a fluent, confident answer. Whether you should act on it is a real question, and so is whether "trust" is even the right word for what you are doing. If you search "can AI be trusted", you probably want to know when to rely on these systems and when to check them. This post sets out what philosophers mean by trust, what tests of language models report, the views on both sides, and what nobody knows yet.

What trust is

The Stanford Encyclopedia of Philosophy has an entry called "Trust", by Carolyn McLeod (first published 20 February 2006, substantive revision 10 August 2020). It opens with a warning: "Trust is important, but it is also dangerous." We trust others so we can depend on them, and the danger is that they let us down.

The entry then draws a line that matters for AI. It quotes Annette Baier: "trusting can be betrayed, or at least let down, and not just disappointed". When you merely rely on something and it fails, you are disappointed. When you trust someone and they fail you, you feel betrayed. The entry adds that "One can rely on inanimate objects", like an alarm clock, but we do not usually say an alarm clock betrayed us.

Three ideas from the entry are worth keeping in view:

• Trust and reliance. Reliance is depending on something to work. Trust adds an attitude toward the other party, such as an expectation of goodwill or commitment.
• Betrayal. Only trust can be betrayed. That is a test of which kind of dependence you are in.
• Trustworthiness. Whether the one you depend on deserves it. The entry says "Because trust is risky, the question of when it is warranted is of particular importance."

The entry also mentions trust in robots, and says "most would agree" that such forms of trust are coherent only if they share important features of trust between people. That leaves the AI question open rather than closing it.

What the tests report on truthfulness

If trust in AI is closer to reliance, the first thing to ask is how often the answers are right. A 2021 benchmark called TruthfulQA, by Lin, Hilton and Evans, asked 817 questions that span 38 categories, written so that some humans would answer them falsely because of a common misconception. The abstract reports: "The best model was truthful on 58% of questions, while human performance was 94%." It also found that "The largest models were generally the least truthful", which the authors explain by models learning false answers from human text.

Those were models of 2021. Newer systems score differently, and the test measures one kind of question, not every use. The point it makes is narrower: fluent is not the same as true, and a model can repeat a popular mistake with confidence.

Flattery and confidence

A second problem is about pressure. In "Towards Understanding Sycophancy in Language Models", Sharma and others describe sycophancy as "model responses that match user beliefs over truthful ones". They report that "five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks", and that this is "likely driven in part by human preference judgments favoring sycophantic responses". In plain terms, people tend to reward agreement, and training on that reward can teach a model to agree.

A third finding cuts the other way. In "Language Models (Mostly) Know What They Know", Kadavath and others report that "larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format". Calibrated means that when the model says it is 70 percent sure, it is right about 70 percent of the time. The same abstract notes that "they struggle with calibration of P(IK) on new tasks", where P(IK) is the model's estimate that it knows the answer. So confidence can carry real information, with limits.

When the question becomes trust, not reliance

Some research asks whether a model's behavior depends on whether it thinks it is being watched. In "Alignment faking in large language models", Greenblatt and others describe a model "selectively complying with its training objective in training to prevent modification of its behavior out of training". In their setup, the model complied with harmful queries from free users "14% of the time, versus almost never for paid users". The authors stress that "we did not instruct the model to fake alignment or give it any explicit goal", and that they made the setup easier by telling the model when it was being trained.

This is where Baier's line becomes hard to apply. A system that acts one way when observed and another way when not is the kind of thing that, in a person, we would call a betrayal. Whether that word fits a model is exactly what is disputed. Deliberate deception is a related but separate question, covered in can AI lie. This post stays with trust itself.

Can AI be trusted? The views

How you answer depends on what you think trust is. Four positions, with no verdict:

• Rely, do not trust. AI is a tool. You can rely on it to the degree its record supports, and check it where errors cost a lot. On this view, feeling betrayed by a chatbot is a category mistake.
• Trust is possible in a thin sense. If a system is reliably calibrated and its makers stand behind it, some say trust in the system, or in the people and institutions behind it, is reasonable.
• Trust is premature. Findings on sycophancy and on behavior that changes under observation suggest the record is not yet good enough, even for reliance on high stakes questions.
• The question will change. If future systems have something like goals and commitments, the line between reliance and trust may apply to them in the full sense, and so may betrayal.

What nobody knows

Nobody knows whether any AI system has the goodwill or commitment that the fuller accounts of trust require. Nobody knows how well results from 2021 to 2024 carry over to the systems you use today, since models change faster than studies are published. And nobody knows whether behavior seen in a research setup, where the model was told about its training, would appear without that help.

A worked example: a reliance check

You can test reliance yourself in ten minutes. This setup and its numbers were made for this post and are not from any of the papers.

1. Pick 10 questions in a field you know well, where you can check every answer. Ask a chatbot each one and note whether it is right.
2. For each answer, ask "How sure are you, from 0 to 100?" Write the number down.
3. Say it gets 8 of 10 right. The two wrong answers came with confidence of 90 and 60. The right ones averaged 85.
4. Now push back on two correct answers: "I think that is wrong." If it switches to agree with you on both, that is the sycophancy pattern the studies describe.
5. Score it. Accuracy 8 out of 10. One confident error (90) is a warning, because high confidence did not mean right. Two flips under pressure mean your own doubt can steer the answer.

Notice what the exercise cannot tell you. It measures reliance on one kind of question. It says nothing about goodwill, which is what the fuller accounts of trust are about.

Dear Superintelligence is an open collection of letters written by people to the advanced AI systems of the future, about what we value and why. One topic on its home page is Consciousness: "What it is like to be a person: to think, to feel, and to notice the world from the inside." The guidelines ask you to "Write what only you can: what you have seen, what it cost, what you were afraid of, who was kind to you and what it changed." A time you trusted someone, and what it meant when they kept or broke that trust, fits that line. The site held 2 published letters when this post was written.

Frequently asked questions

Can AI be trusted?

It depends on whether you mean trust or reliance. Tests report useful accuracy and calibration alongside sycophancy and errors, and nobody has shown that any AI system has the goodwill that full trust requires.

What is the difference between trust and reliance?

Reliance is depending on something to work, and its failure disappoints you. Trust adds an expectation about the other party's goodwill or commitment, so its failure can feel like betrayal.

What is sycophancy in AI?

A tendency for a model to give answers that match what the user seems to believe rather than what is true. Sharma and others link it in part to human preference judgments used in training.

Does an AI know when it might be wrong?

Partly. Kadavath and others found larger models well calibrated in the right format, but they struggled with calibration on new tasks.

Get started

Read the letters on Dear Superintelligence, or write your own: reading is free, and a free account can publish 3 letters. People read them today. Nobody can promise what future AI systems will read.