21 posts
Differential technological development is the idea that the order in which technologies arrive matters as much as how fast they arrive. Instead of trying to stop progress, you slow down the dangerous technologies and…
A global catastrophic risk is a possible event that could seriously harm human well-being across the whole world, endangering or even destroying modern civilization. The term covers pandemics, nuclear war, asteroid…
Human Compatible by Stuart Russell is a 2019 book arguing that the way AI is usually built, around fixed goals that machines chase as hard as they can, could become dangerous as machines get more capable. Its full title…
An AI singleton is a single artificial intelligence that has become the one decision-maker for the whole world, with nothing left that can challenge it. The idea comes from Nick Bostrom, who coined the word singleton for…
The Precipice by Toby Ord is a 2020 book arguing that humanity is living through a uniquely dangerous period, with risks it has never faced before. Its full title is The Precipice: Existential Risk and the Future of…
The vulnerable world hypothesis is the idea that some future invention could destroy civilization by default, unless people take extraordinary measures first. The philosopher Nick Bostrom set it out in a 2018 working…
Superintelligence by Nick Bostrom is a 2014 book about the risk of smarter-than-human AI. Its full title is Superintelligence: Paths, Dangers, Strategies, and its claim is simple to state: a machine far smarter than us…
Longtermism and AI come up together so often that many people meet the first through the second. Longtermism is the view that positively influencing the long-term future is a key moral priority, and some of its…
An s-risk, short for a risk of astronomical suffering, is a risk of a future with much more suffering than all that has occurred on Earth so far. Most talk about AI risk asks whether humanity survives. S-risk asks a…
Value lock-in is the risk that one set of values gets fixed in place for a very long time, mistakes and all, so that people can no longer change their minds. It is one of the less obvious worries about advanced AI. The…
An AI takeover is the idea that artificial intelligence systems could one day override human decisions and end up, in practice, running things. Most people picture robots with guns. The researchers who worry about it…
P(doom) is the number people in AI safety give when they are asked how likely it is that artificial intelligence ends in catastrophe for humanity. It is usually given as one short figure, such as "my p(doom) is 10%".…
Existential risk from AI is the worry that advanced AI, and especially superintelligence, could lead to human extinction or another catastrophe humanity could never recover from. Serious researchers argue for it, and…
The paperclip maximizer is a thought experiment about an AI given one simple goal: make as many paperclips as possible. Pushed far enough, with no limits and no care for anything else, that harmless goal ends with the…
An AI race is a competition between groups to build the most powerful AI first. The groups can be companies, research teams or whole countries. Researchers worry about it less because of who might win and more because of…
The orthogonality thesis is the idea that how smart a mind is and what it wants are two separate things. A very intelligent AI could, in principle, pursue almost any goal, including one that looks pointless or harmful to…
The debate over hard takeoff vs soft takeoff is about one question: if AI ever reaches human level, how fast would it climb past us? In a hard takeoff, the climb takes days or months. In a soft takeoff, it takes years or…
AI containment means limiting what an AI system can see and do, so that people keep the power to watch it, correct it and switch it off. Researchers also call it capability control or AI confinement. It is one of the…
Artificial superintelligence, often shortened to ASI, means an AI that would be far smarter than the best human minds in almost every area, not just at one task. It does not exist. It is a hypothesis, and one that…
An intelligence explosion is the idea that a machine smart enough to design machines could build a smarter one, which could build a smarter one still, until the result leaves human intelligence far behind. It is one of…
Recursive self-improvement is the idea of an AI system that rewrites and tests its own code, so that each new version is better at improving the next one. It sits at the center of many debates about superintelligence,…