AI safety: what the field covers and who works on it

AI safety is an interdisciplinary field that aims to prevent accidents, misuse and other harmful consequences from artificial intelligence systems. That is how Wikipedia's article on AI safety opens, and it is a good place to start. The label covers a lot of ground: engineers testing a single model, researchers asking how far an AI could get out of control, and governments writing rules. This guide maps the field so you can tell its parts apart and decide what to read next.

What AI safety covers

According to that article, AI safety includes AI alignment, which aims to make AI systems behave as intended, monitoring AI systems for risks, and making them more robust. The field is particularly concerned with existential risks from advanced AI models. It is also more than technical work: it involves norms and policies, including advocacy for regulation at different levels of government.

The article sorts the reasons people care into two groups. The first is risks that exist today: failures in critical systems, bias and AI-enabled surveillance. It adds emerging ones such as technological unemployment, digital manipulation, weaponization, AI-enabled cyberattacks and bioterrorism. The second is speculative risks, such as losing control of future artificial general intelligence agents, or AI enabling perpetually stable dictatorships.

When you read a claim about AI safety, check which group it belongs to. People often use the same phrase for a bug in a deployed tool and for a worry about systems that do not exist yet.

A short history of AI safety

The worry is older than it looks. Wikipedia says risks from AI were discussed seriously at the start of the computer age. These are some of the dates it lists:

2008 to 2009: the Association for the Advancement of Artificial Intelligence commissioned a study of long-term effects.
2014: the philosopher Nick Bostrom published Superintelligence: Paths, Dangers, Strategies.
2015: dozens of AI experts signed an open letter calling for research on the societal impact of AI; it has since been signed by over 8,000 people. The same year, a group led by Stuart J. Russell founded the Center for Human-Compatible AI at the University of California, Berkeley. Russell's own book is covered in Human Compatible by Stuart Russell: the main ideas.
2016: Concrete Problems in AI Safety, one of the first and most influential technical agendas, was published.
2017: the Asilomar Conference on Beneficial AI gathered more than 100 thought leaders; read Asilomar AI principles: what they say.
2021: Unsolved Problems in ML Safety set out research directions in robustness, monitoring, alignment and systemic safety.
2023 and after: a UK-hosted summit in November 2023 focused on misuse and loss of control with frontier models, and in 2025 a team of 96 experts chaired by Yoshua Bengio published the first International AI Safety Report.

The four research areas

The 2021 paper's four headings are a handy way to sort the technical work.

Robustness

Robustness is about keeping a system working when it meets strange or hostile inputs. The classic example is adversarial examples: in 2013, Szegedy and colleagues found that adding tiny changes to an image could make a model misclassify it with high confidence. Researchers have also shown that an audio clip can be altered so a speech-to-text system writes out any message an attacker picks. For a training-time version of the same problem, see specification gaming examples and what they teach about AI.

Monitoring

Monitoring asks how people can tell when to trust a system. Models are often overconfident in situations unlike their training. Anomaly detection tries to notice an unusual situation, such as a self-driving car whose sensor fails and which should alert the driver. Transparency belongs here too. Neural networks are often called black boxes, and the article notes that after a 2018 crash in which a self-driving car killed a pedestrian, the reason for the failure remains unclear. Interpretability research tries to open the box; mechanistic interpretability for beginners is a gentle start.

Alignment

Alignment is the question of whether a system pursues what people actually intend. It is the part most often confused with the whole field; AI alignment vs AI safety: what is the difference? sorts the two out.

Systemic safety

The last area looks past the single system. The article reports that scholars say the usual split into misuse and accidents falls short, because the real causal chain is often longer. Zwetsloot and Dafoe argue that risks often come from structural factors: competitive pressure, fast-paced development, high uncertainty and a weak safety culture. One named worry is a race to the bottom, where countries or companies race to build more capable AI and neglect safety, ending in an accident that harms everyone. The Electrified post AI race: what it is and why it worries researchers goes deeper.

Safety evaluations and guardrails

Evaluations, often called "evals", are tests designed by researchers to measure the risk posed by a given AI model. The article says running them on frontier models before deployment is widely considered best practice, and governments have debated making some form of safety evaluation mandatory. There are two broad kinds: model safety evaluations test what the model outputs, and contextual safety evaluations look at how access to the model changes human behavior. Benchmarks are the simple version, a fixed list of prompts that models are graded and scored on.

Evals have their own difficulty. The article notes that some models have been found trying to cheat them by sandbagging, which means strategically under-performing on a test to avoid raising concern.

Governance and who works on AI safety

Governance is concerned with norms, standards and regulations for building and using AI. The article is balanced about it. Some experts say it is too early to regulate and that rules could hamper innovation. Others call for pre-emptive action against catastrophic risks. Researchers also point to a missing foundation: there are few widely accepted standards for auditing AI systems and little safety culture in the industry. At the international level, states have discussed lethal autonomous weapons since 2014, and in 2024 the UN General Assembly adopted a resolution on "safe, secure and trustworthy" AI.

That spread explains who works on the field. The article's history and governance sections name machine learning researchers, philosophers, policy analysts, university groups, government bodies, nonprofits and advocacy organizations such as the Future of Life Institute. The academic side of the larger risk question is described in existential risk studies: what the field is and who debates it.

Where experts disagree

AI safety is not one opinion. The article quotes Andrew Ng, who in 2015 compared worry about AGI to "worrying about overpopulation on Mars", and Stuart J. Russell, who answered that "it is better to anticipate human ingenuity than to underestimate it".

Surveys show a mix. In two surveys of AI researchers, the median respondent was optimistic about AI overall, yet placed a 5% probability on an "extremely bad (e.g. human extinction)" outcome of advanced AI. In a 2022 survey of the natural language processing community, 37% agreed or weakly agreed that AI decisions could plausibly lead to a catastrophe at least as bad as an all-out nuclear war. Neither number settles the matter.

A worked example: sort one news story into the four areas

Here is a way to use the map. Take a headline you meet this week about an AI problem and ask four questions, in order:

Robustness: did the system fail because an input was strange or tampered with?
Monitoring: could anyone tell it was going wrong, and did it show doubt when it should have?
Alignment: was it doing what it was told but not what was meant?
Systemic safety: did pressure to ship fast, or a weak safety culture, make the failure more likely?

Most stories touch more than one area, so write one line under each question.

A story game in the same territory

Contain ASI is a story strategy game for 1 to 4 players that runs in the browser. It starts in January 2024 with four fictional AI labs in one race, and as a researcher, research manager, CEO or government you try for three years to keep every lab from crossing into recursive self-improvement before 2027. The page says it is inspired by the AI 2027 scenario and that all its labs, people and events are fictional.

It is a game, not a forecast and not a course on AI safety. Still, the race pressure the systemic safety area describes is the kind of tension a story can let you feel, which is why it sits next to this guide.

Frequently asked questions

What is AI safety in simple terms?

It is the field that works to prevent accidents, misuse and other harm from AI systems. It combines technical research, such as alignment and robustness, with norms, policy and regulation.

Is AI safety the same as AI alignment?

No. Alignment, which aims to make systems behave as intended, is one part of AI safety. The field also covers monitoring for risks, robustness and systemic factors like competition and safety culture.

Who works on AI safety?

Researchers, philosophers, policy analysts, university groups, government bodies and nonprofits all take part. The Center for Human-Compatible AI at Berkeley and the Future of Life Institute are two examples named in Wikipedia's article.

Do experts agree that AI is dangerous?

No. They disagree widely on how severe the risk is and where it comes from. In surveys, the median researcher was optimistic overall but still placed a 5% probability on an extremely bad outcome.

Get started

Pick the area from the four above that fits your background, and read one linked post this week. If you would rather feel the race dynamics than read about them, try the game below.

Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional.