Longtermism and AI: why some put the far future first

Longtermism and AI come up together so often that many people meet the first through the second. Longtermism is the view that positively influencing the long-term future is a key moral priority, and some of its best-known thinkers rank advanced AI among the biggest threats to that future. This guide explains what longtermism says, why AI sits near the top of its worries, how to use one of its tools on a real decision, and what its critics say.

What is longtermism?

Wikipedia's article on longtermism defines it as "the ethical view that positively influencing the long-term future is a key moral priority". It is an important idea in effective altruism and a main reason some people work to reduce existential risks, the risks that could destroy humanity's long-term potential.

The philosopher William MacAskill separates two versions. Plain longtermism says the long-term future is a key moral priority of our time. Strong longtermism says it is the key priority.

The term is new. MacAskill and the philosopher Toby Ord coined it around 2017, drawing on the work of Nick Bostrom, Nick Beckstead and others. The idea is much older. The oral constitution of the Iroquois Confederacy, the Gayanashagowa, asks decision makers to "have always in view not only the present but also the coming generations", often read as thinking seven generations ahead. Derek Parfit's 1984 book Reasons and Persons and Jonathan Schell's 1982 book The Fate of the Earth brought similar ideas into modern thought.

The argument in three premises

In his 2022 book What We Owe the Future, MacAskill builds the case on three premises:

1. Future people count. They matter morally as much as people alive today.
2. There could be a great many of them. Humanity may survive for a very long time.
3. We can make a difference. The future could be very good or very bad, and what we do may affect which.

If all three hold, then the people who will live later make up most of everyone affected by what we do now. MacAskill names two ways to help them: "by averting permanent catastrophes, thereby ensuring civilisation's survival; or by changing civilisation's trajectory to make it better while it lasts". Survival raises the quantity of future life. Trajectory changes raise its quality.

Longtermism and AI: why AI comes up so often

AI touches both levers, which is why it keeps appearing in longtermist writing.

• As a survival risk. MacAskill argues that the most severe threats of human extinction come from engineered pathogens and misaligned artificial general intelligence. In The Precipice, Ord estimates the existential risk from unaligned artificial general intelligence at 1 in 10 over the next century, higher than all other sources combined, within a total of 1 in 6. These are one author's estimates, and others give very different numbers, as the post on p(doom) shows.
• As a trajectory risk. MacAskill warns of value lock-in, one value system persisting for an extremely long time, and believes it may come particularly from the development of artificial general intelligence.

MacAskill also treats AI as a problem with many unknowns. For problems like that, he says the priority is building up options and learning more, rather than betting on one fix.

Does the future count less? The discounting question

Economists usually discount the future: a benefit far away in time counts for less than the same benefit today. Part of that discount is the chance the benefit never arrives. Another part, called pure time preference, values later benefits less simply because they are later.

Longtermists generally reject pure time preference. MacAskill puts it this way: "distance in time is like distance in space". Ord argues a nonzero pure time preference is arbitrary. Frank Ramsey, who devised the standard discounting model, also thought it describes how people behave, not how they ought to value things.

Not everyone agrees. Andreas Mogensen argues that common-sense morality lets us favour those closer to us, so each later generation may count a little less. This view is called temporalism.

A worked example: weigh an action the longtermist way

MacAskill offers a three-part test for how much an outcome matters in the long run. Significance is how much good it adds. Persistence is how long it lasts. Contingency is how much it depends on your action, rather than happening anyway. Try it on one decision in ten minutes.

1. Pick an action. For example: spending a year of evenings learning AI alignment to help with safety work.
2. Rate significance as low, medium or high. If the work helped at all, how much good would it add?
3. Rate persistence. Would the effect fade in a year, or could it last for generations?
4. Rate contingency. Would the same work get done without you?
5. Check the four lessons. MacAskill suggests taking robustly good actions, building up options, learning more and avoiding harm. Does your action fit at least one?

Then repeat it for an action aimed at a problem that exists today. You may find both score well. Longtermism's advocates say that is common. Its critics worry about the cases where the two pull apart, for example that thinking about the next 10,000 or 10 million years could lead people to downplay the nearer-term effects of climate change.

The main criticisms of longtermism

Wikipedia sets out several objections, each with a reply. Here they are, with no verdict.

• Tiny chances, huge stakes. If the future is vast, even a tiny cut in risk looks enormously valuable. Critics point to Pascal's mugging, a thought experiment in which someone exploits that kind of reasoning. Some advocates accept the conclusion anyway. Others reply that existential risks are not tiny, and that guarding against them is like wearing a seatbelt.
• A distraction from harms now. AI and AI ethics researchers Timnit Gebru, Emily M. Bender, Margaret Mitchell and Angelina McMillan-Major argue that talk of existential risk distracts from present harms from AI, such as data theft, worker exploitation, bias and concentration of power, and call longtermism a "dangerous ideology". Elizabeth C. Hupfer names a related worry the Far-Future Priority Objection. Advocates reply that work like pandemic preparedness helps people now as well.
• Too much faith in control. B.V.E. Hyde argues that history is often driven by deep economic, technological and institutional forces, so longtermists overstate how much any one action can steer the far future.

Explore the stakes as fiction in Contain ASI

Contain ASI is a story strategy game set in the years before AI could improve itself. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government. It is fiction, not a forecast: the footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

What is longtermism in simple terms?

It is the view that making the long-term future go well is a key moral priority, because future people count and there could be very many of them.

Why do longtermists care so much about AI?

Because AI could affect both whether humanity survives and what values shape its future. MacAskill counts misaligned AGI among the most severe extinction threats, and Ord estimates its risk at 1 in 10 over the next century.

Who came up with longtermism?

The philosophers William MacAskill and Toby Ord coined the term around 2017, drawing on Nick Bostrom, Nick Beckstead and earlier thinkers such as Derek Parfit.

Is Contain ASI a forecast of the future?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to try keeping four labs from crossing the line? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.