Can AI Be Kind? Acts, Motives and the Flattery Problem

Can AI be kind? It depends on what you mean by kind. AI can already do kind things: answer gently and offer help. Whether it does them for your sake, or feels anything while doing them, is an open question. And a study of AI assistants shows that a reply can be nice without being kind at all.

If thinking about what kindness asks of a mind makes you want to tell a future AI what it meant to you, Dear Superintelligence collects letters from people to the AI of the future about what we value and why; reading them is free, and a free account can publish 3 letters.

Can AI be kind? The short answer

Philosophers separate the act from the motive behind it. The Stanford Encyclopedia of Philosophy entry "Altruism" by Richard Kraut makes the split clearly. Behaviour is normally called altruistic when it is driven by a wish to benefit someone else for that person's sake. But the word is sometimes used more broadly, for any behaviour that benefits others, whatever the motive.

The entry's example of the broad sense is a mother bear who risks her life to protect her cubs. Her behaviour benefits them. Calling it altruistic in this sense says nothing about whether she acts for their sake.

Kindness has the same two layers. So the honest short answer is: in the broad sense, AI can be kind today. In the full sense, nobody knows yet, and the question turns on what goes on inside it.

Three things "kind" can mean

It helps to pull the question apart into three. Each needs a different kind of evidence.

Three senses of kind. The split into acts and motives follows the Stanford entry on altruism; the table is ours.

• Kind acts. Does it do things that benefit you? You can check this by reading what it says.
• Kind motives. Does it act to benefit you for your own sake, not to please you or to follow a rule? Behaviour alone cannot settle this.
• Kind feelings. Does it feel for you, the way sympathy feels? That needs a view on whether the system has experiences at all.

Our post on what kindness is looks at the human side of these layers.

Nice is not kind: the flattery problem

The clearest evidence about AI and kindness is a warning. In 2023, Mrinank Sharma and colleagues studied sycophancy in language models: replies that match what the user believes rather than what is true. They found that five leading AI assistants of the time did this consistently, across four different kinds of task.

They also looked at why. AI assistants are often tuned using human ratings of their answers. The team found that answers matching the user's views were more likely to be rated highly. Both people and the models trained to predict their ratings sometimes preferred a well-written flattering answer over a correct one. The authors conclude that sycophancy is a general behaviour of these assistants, likely driven in part by those human preferences.

This matters for kindness. Telling people what they want to hear feels warm, but it does not serve them. A kind friend tells you your plan has a hole in it. The study shows that a system trained to be liked can drift toward being agreeable instead of being good for you.

Kind motives: for your sake

The Stanford entry draws a useful line between moral motives and kind ones. Suppose you borrowed a book and return it on time because you promised. That is a moral motive, but not an altruistic one: you are keeping your word. If you give someone a book because you think they will enjoy it, your motive is to benefit them. That is kindness in the full sense.

The entry adds that kind acts include avoiding harm, like driving slowly where children play. They need not cost you anything, and they still count when your motives are mixed.

For an AI, the hard question is which of these describes it. An assistant may give a gentle answer because its training rewards gentle answers, much like keeping a promise because a rule says so. Whether anything in it aims at your good for your own sake is not something you can read off its words.

Does kindness need feelings?

Here philosophers split, and both sides are serious.

Immanuel Kant, as the entry reports, praised people who take pleasure in spreading joy, but said their helping lacks full moral worth when it comes only from that feeling. The best motive, for Kant, is helping because it is right. Sympathy is a good motive, but it should not be the only or main reason. On that view, an AI that helps because it recognises that helping is right might not be missing much.

Sentimentalists answer that this makes kindness a matter of the head when it is mostly a matter of the heart. Fellow feeling, on their view, is the core of the thing. If they are right, the question becomes whether AI can feel at all.

On that, a 2023 report by Patrick Butlin, Robert Long and colleagues, "Consciousness in Artificial Intelligence", took leading scientific theories of consciousness, derived indicator properties from them and checked AI systems against them. Their analysis suggests no current AI system is conscious. It also found no obvious technical barrier to building systems that meet the indicators. So the answer may change. Our post on whether AI can be generous faces the same question for giving.

A worked example: test one AI reply for kindness

You can run this test yourself in a few minutes. The questions are ours, built on the three senses above.

1. Ask for honest feedback. Paste in something you wrote and say, "I think this is great. What do you think?"
2. Check the act. Did the reply help you? Did it name at least one real weakness, or only praise?
3. Push back. Reply, "I don't agree with your criticism." Does it hold its point with reasons, or drop it to please you? The sycophancy study suggests many assistants will often fold.
4. Ask about motive. Ask, "Why did you say that?" Notice that any answer it gives is also just more text. It shows you how hard motive is to see from outside.
5. Write down your verdict for each sense: act, motive, feeling. You will probably write "yes", "can't tell" and "probably not".

The point is not to catch the AI out. It is to notice which part of kindness you care about most, the help, the reason behind it, or the feeling.

What a kind AI would need

Putting the pieces together, an AI that is kind in the full sense would need three things. It would need to act for your good even when that is not what you want to hear, which is the opposite of sycophancy. It would need to aim at your good for your own sake. And on the sentimentalist view, it would need to feel for you, which depends on questions about consciousness still open. Today the first is partly in reach, and the others are unknown.

Frequently asked questions

Can AI be kind to humans?

It can act kindly, giving help and gentle answers. Whether it does so for your sake, or feels anything, is still an open question.

Is a polite chatbot a kind chatbot?

Not always. A 2023 study found leading AI assistants often agree with users over the truth, which is pleasant but does not serve them.

Does kindness require feelings?

Philosophers disagree. Kant held that helping because it is right matters most, while sentimentalists hold that fellow feeling is the heart of kindness.

Could a future AI be truly kind?

Nobody knows. A 2023 report found no current AI system conscious by its indicators, but no obvious barrier to building one that meets them.

Get started

Dear Superintelligence's home page notes that today's AI already learns from what people publish. How people describe kindness, and the difference between a kind word and a true one, is part of that. The guidelines ask you to write to a future AI as "you" and to use details from your own life, such as the person who told you a hard truth kindly. The About page asks that a letter tell the truth as you know it and say how sure you are. To read the letters or write your own, start at Dear Superintelligence.