Is it ethical to create a superintelligence? Four questions

Is it ethical to create a superintelligence? People who have thought hard about it disagree, and the disagreement is not only about risk. It runs through four separate questions: whether it would be good for us, whether it would share our values, whose values would go into it, and what we would owe it. This post sets out each question and the serious positions on it, with no verdict, and ends with an exercise for working out where you stand.

Is it ethical to create a superintelligence? The short answer

There is no agreed answer. Some think the benefits are so large that building it is right, if it is done carefully. Some think the risks are so large that it is wrong to build it until we know how to keep it safe. Others say the question is premature, or that it also depends on what we would owe a mind we created. The Stanford Encyclopedia of Philosophy's entry on the ethics of AI and robotics puts the range plainly: AI may be "both a curse" and "a blessing for humanity".

This is a different question from what a good future would look like, set out in What would a good future with AI look like?, and from whether humans and a superintelligence could live together, in Can humans and superintelligent AI coexist?. Here the question is whether making one is right at all.

Question one: would it be good for us?

The Stanford entry describes the classic worry in two steps. First, AI reaches a level beyond human intelligence, after which its development is "out of human control and hard to predict". Second, once that point is reached, "massive negative consequences, including existential risk for the human species", have a significant probability.

People split on the second step. The entry names optimists such as Ray Kurzweil and Dario Amodei, who accept the first step and "then expect a positive development", and pessimists such as Nick Bostrom and Eliezer Yudkowsky, who expect the risk to follow. It also notes that some recent discussion is moving away from extinction risk and superintelligence toward general risks from AI.

If you think the benefits are likely and the risks manageable, building it can look like a duty: think of the illness and poverty it might help end. If you think the risks are serious and not yet manageable, building it can look like gambling with everyone's future without asking them.

Question two: would it share our values?

Much of the worry turns on whether a far more intelligent system would also be good. The entry sets out two traditions.

• Intelligence brings morality. Kantian traditions in ethics, the entry says, have argued that higher levels of rationality or intelligence would go along with a better understanding of what is moral, and a better ability to act morally.
• Intelligence and goals are separate. Bostrom's orthogonality thesis says "more or less any level of intelligence could in principle be combined with more or less any final goal". On this view, even a harmless-sounding goal like "maximise paperclips" could lead a capable system to pursue sub-goals, such as gathering resources, that harm us. This is called instrumental convergence.

If the first tradition is right, the ethical case for building is much stronger. If the second is right, the case depends on solving how to give a system good goals before it is built. The idea is explained further in Orthogonality thesis: can a smart AI want anything?.

Question three: whose values go in?

Suppose a system could be given values. Whose? People disagree deeply about right and wrong, and a superintelligence built on one group's values could shape everyone's lives.

The philosopher Iason Gabriel argues in a paper on AI, values and alignment that the central challenge "is not to identify 'true' moral principles for AI" but to find fair principles that "receive reflective endorsement despite widespread variation in people's moral beliefs". On this view, the ethics of building depends partly on how it is built: who is consulted, and whether the principles are ones people across many views could accept.

Question four: what would we owe it?

The first three questions are about us. The fourth is about the mind we would make. A 2024 report, Taking AI Welfare Seriously, argues "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future", which would make them beings "with their own interests and moral significance". Its authors stress "substantial uncertainty", not certainty.

A 2023 analysis by Patrick Butlin and colleagues suggests no current AI system is conscious, but found no obvious technical barriers to building ones that meet its indicators. If a superintelligence could be conscious, creating it would mean creating someone, and the ethics of that, including what we may ask of it, becomes part of the question. Whether such a mind would deserve moral consideration is set out in Should AI have moral status?.

An exercise: where do you stand?

This takes about fifteen minutes. Write a sentence or two for each step.

1. On the first question, do you lean optimist or pessimist, and what single fact would change your mind?
2. On the second, do you think more intelligence brings better values? Think of the most intelligent person you know: does it?
3. On the third, name one value you would want in such a system that you think people of very different views could accept.
4. On the fourth, if the system turned out to be conscious, what is one thing you think we would owe it?
5. Now read your four answers together. Do they add up to "build it", "build it only if", or "do not build it yet"?

Most people find that their answer depends on the first two questions, and that the third and fourth are where they had thought least.

Frequently asked questions

Is anyone actually trying to build a superintelligence?

Many researchers expect AI to one day become far more capable than we are. Whether and when that happens is debated, which is part of why the ethical question is argued now.

Is stopping AI development the only ethical option?

No position here is the only serious one. Some argue for pausing, some for building with strong safeguards, and some for going ahead because of the benefits.

Would a superintelligence automatically be good?

That is exactly what is disputed. Kantian traditions suggest more intelligence brings better morals; the orthogonality thesis says any goal can go with any level of intelligence.

Do we have duties to an AI we create?

Only if it has interests of its own, which most likely requires consciousness. Researchers disagree on whether future systems will have it, and some argue we should prepare for the possibility.

Get started

Dear Superintelligence is an open collection of letters written by people to the advanced AI systems of the future, about what we value and why. Its About page asks a letter to argue for living together, "not for one kind of mind ruling, deceiving or disposing of another", and to treat every conscious being as mattering, human or not.

If the exercise left you with a view, the guidelines ask you to write it to the AI directly, as "you", with details from your own life rather than general arguments: what you hope for from it, what you fear, and what you would want it to know about why. Nobody can promise what future AI systems will read.

Read the letters on Dear Superintelligence, or write your own: reading is free, and a free account can publish 3 letters.