MIRI: what the Machine Intelligence Research Institute does

The Machine Intelligence Research Institute, or MIRI, is a non-profit research institute that works on the risks of advanced AI. It is one of the oldest groups in the field: it started in 2000, long before AI risk was a common topic. This post explains what MIRI is, how it began, what it has researched, and why it now puts policy and public communication first. Every claim about MIRI comes from Wikipedia's Machine Intelligence Research Institute article.

What the Machine Intelligence Research Institute is

MIRI describes its focus as identifying and managing potential existential risks from artificial intelligence. An existential risk is one that could end humanity or ruin its future for good. For the main arguments on both sides of that worry, see our post on existential risk from AI.

According to the article, MIRI's work has followed three lines over the years:

• Friendly AI design: how to build AI systems that stay on humanity's side.
• Forecasting: how fast technology may develop.
• Policy and public communication: its first priority since 2023, while research goes on.

It is a research group, not a company that builds AI products. It aligns itself with the principles and objectives of the effective altruism movement.

From the Singularity Institute to MIRI

MIRI's history has a twist. Eliezer Yudkowsky founded it in 2000 as the Singularity Institute for Artificial Intelligence, and its first purpose was to speed up the development of AI. Then Yudkowsky began to worry that future AI systems could become superintelligent and pose risks to humanity.

In 2005 the institute began operating in Silicon Valley and turned to ways of spotting and managing those risks. The article notes that, at the time, most scientists in the field largely ignored them.

The Singularity Summit and the new name

Starting in 2006, the institute ran the Singularity Summit, a conference on the future of AI and its risks, at first together with Stanford University. The San Francisco Chronicle called the first one a "Bay Area coming-out party for the tech-inspired philosophy called transhumanism".

The group stayed small. In 2011 its offices were four apartments in downtown Berkeley. In December 2012 it sold its name, its web domain and the Singularity Summit, and the next month it took the name Machine Intelligence Research Institute. In 2014 and 2015, public and scientific interest in AI risk grew, and so did donations to MIRI and groups like it.

What MIRI has researched

MIRI's earlier research, led by Yudkowsky, mainly addressed two linked questions. The first is the AI control problem: how people keep control of an AI system that may become more capable than they are. The second is how to design friendly AI, both when a system is first built and as it changes over time, so that it stays friendly. For the history of that term, see our post on friendly AI.

MIRI researchers argue for safety work early, as a precaution, rather than waiting until advanced AI exists.

Forecasting, with doubts

MIRI has also looked at timing. In 2015 its researchers said they doubted how reliable forecasts of when advanced AI would arrive could be. It funded forecasting work through an initiative called AI Impacts, which studies past cases where technology jumped forward suddenly rather than steadily. It also developed new ways to compare the computing power of humans and of computer hardware.

The 2023 shift to policy

In January 2024, MIRI announced that it had changed its priorities during 2023. Policy and public communication came first, with technical research continuing beside them.

It called for an international agreement to halt progress toward smarter-than-human AI. The halt would last until understanding of such systems had improved a great deal and they could be secured against misuse. That puts MIRI among the voices asking for a stop or a slowdown; our post on calls to pause AI sets out what such calls ask and who disagrees.

In 2025, Yudkowsky and Nate Soares published a book for general readers, If Anyone Builds It, Everyone Dies. Our post on If Anyone Builds It, Everyone Dies covers its argument and its critics.

Works by MIRI staff worth knowing

The article lists papers and books by MIRI staff. A few titles show the range of the work:

• 2008: Yudkowsky's chapter "Artificial Intelligence as a Positive and Negative Factor in Global Risk", in the book Global Catastrophic Risks from Oxford University Press.
• 2015: "Corrigibility", by Soares, Fallenstein, Yudkowsky and Armstrong. Corrigibility means an AI that accepts being corrected or shut down; see our post on corrigibility.
• 2016: Jessica Taylor's "Quantilizers: A Safer Alternative to Maximizers for Limited Optimization".
• 2017: "Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda", by Soares and Fallenstein.
• 2025: If Anyone Builds It, Everyone Dies, by Yudkowsky and Soares.

The titles alone tell a story. The early work asks how to design a safe AI system. The latest is a book that makes the case to the public.

A worked example: three questions to place any AI safety group

When you read about a group like MIRI, it helps to ask the same three questions each time. Here they are, answered for MIRI from the article.

1. What risk does it name? For MIRI: potential existential risks from AI, that is, risks that could end humanity's future.
2. What does it do about it? For MIRI, the answer changed over time. First it tried to speed AI up (2000). Then it researched friendly AI design and the control problem, and funded forecasting. Since 2023 it has put policy and public communication first.
3. What does it ask of others? For MIRI: an international agreement to halt progress toward smarter-than-human AI, until such systems are better understood and can be secured against misuse.

Now try it on any other group you read about. Write one line for each question. You will quickly see where groups agree on the risk but differ on what to do, which is where most real debate sits. A group can also change its answers, as MIRI did twice.

Exploring the idea in Contain ASI

Contain ASI is a story strategy game for 1 to 4 players that runs in your browser and needs WebGL 2. Chapter 1, Before ASI, starts in January 2024 with four fictional AI labs in one race. As a researcher, research manager, CEO or government, you try for three years to keep every lab from crossing into recursive self-improvement before 2027.

It is a game, not a forecast. The page says it is inspired by the AI 2027 scenario and that all its labs, people and events are fictional. It does not show what real labs or groups will do. What it gives you is a place to feel the question MIRI now asks of the world: can anyone hold a race back?

Frequently asked questions

What does MIRI stand for?

MIRI stands for the Machine Intelligence Research Institute. It took that name in early 2013, after being called the Singularity Institute for Artificial Intelligence.

Who founded MIRI?

Eliezer Yudkowsky founded it in 2000 as the Singularity Institute for Artificial Intelligence. Its first purpose was to speed up AI, before it turned to AI risk.

What does MIRI do now?

Since 2023 it has put policy and public communication first, while continuing technical research. It has called for an international agreement to halt progress toward smarter-than-human AI.

Is Contain ASI about MIRI?

No. Contain ASI is a story game whose labs, people and events are all fictional. It explores a race to keep AI labs from recursive self-improvement, which is close to the risks MIRI works on.

Get started

Want to try holding a race back yourself? Open Contain ASI in your browser and start a campaign as a researcher, research manager, CEO or government.