Lethal autonomous weapons and the AI risk debate

Lethal autonomous weapons are military drones or robots that can search for and engage targets on their own, based on programmed constraints and descriptions, without a person making each decision. Wikipedia's article on lethal autonomous weapons notes that most military drones and robots today are not truly autonomous, because they still need human control. This post explains why the term is so hard to define, how much human control people argue a weapon must keep, and the main positions in the debate, set out side by side with no verdict.

If you want to think through how states and labs might handle AI under pressure, Contain ASI is a browser story strategy game for 1 to 4 players where you can play a researcher, research manager, CEO or government and try to keep four fictional AI labs from crossing into recursive self-improvement before 2027.

What lethal autonomous weapons are

The article uses the short forms LAWS, for lethal autonomous weapons systems, and AWS, for autonomous weapons systems. They can take the form of aerial, underwater or ground vehicles with no crew, and most of the systems alleged to be autonomous so far are aerial drones.

The word "autonomous" is the hard part. Wikipedia says it is ambiguous, and its meaning varies a lot between scholars, states and organizations. There is no definition all states agree on. Some definitions in the article:

• The International Committee of the Red Cross describes autonomous weapons as weapons that can track, target and engage fully on their own, without needing human control.
• The UN Group of Governmental Experts on lethal autonomous weapons systems describes one as a combination of weapons and technical parts that together "can identify, select, and engage a target with force, without intervention by a human being operating the system".
• Some states take a narrow view: only a system able to change its own rules of operation counts.
• Scholars such as Peter Asaro and Mark Gubrud take a wide view: any weapon that can release lethal force without a human supervisor's operation, decision or confirmation is autonomous.

Despite those differences, a wide range of states and groups, including the Red Cross, Human Rights Watch and Stop Killer Robots, agree on the core trait: once activated, the weapon can select and engage targets without human input at that moment. Some observers also object to the word "lethal", arguing that lethality comes from how a weapon is used.

Autonomy is older than AI

The article points out that some broad definitions include land mines and naval mines, simple weapons triggered automatically that have been used for centuries. Radar-guided close-in defense guns that protect ships have been in use since the 1970s. They can identify and attack incoming missiles, rockets, artillery fire, aircraft and boats according to criteria a human operator sets. Similar protection systems exist for tanks, and many missile defense systems also target on their own.

The article explains why: the need for a very fast response has kept a "human in the loop" out of these defensive systems. It also notes that the meaning of autonomy in weapons has been blurred by commentators who use it as if it meant artificial intelligence. A weapon can be autonomous without AI, and AI can be used in a weapon that a person still controls.

How much human control?

A 2012 Human Rights Watch report by B. Docherty set out three degrees of human control:

• Human in the loop: a human must start the weapon's action, so the weapon is not fully autonomous.
• Human on the loop: a human supervises the operation and may abort or override it.
• Human out of the loop: no human action is involved, and the weapon is fully autonomous.

More recent talks use the phrase "meaningful human control". The elements suggested as necessary include that humans have enough information about the weapon's abilities and the situation to comply with international humanitarian law, that they can use their judgment to the extent that law requires, and that they can limit the types of tasks and targets. The UN expert group's report says that control and human judgment over these weapons are needed to keep to the rules of distinction, proportionality and precautions in attack.

The arguments against

Several lines of criticism run through the article.

• Telling people apart. Stuart Russell, professor of computer science at the University of California, Berkeley, has said he sees these weapons as unethical and inhumane, because it is hard for them to distinguish combatants from non-combatants. Some economists and legal scholars ask whether they would break the principles of distinction and proportionality in international humanitarian law.
• Who is responsible. The philosopher Robert Sparrow argues that autonomous weapons are causally but not morally responsible, like child soldiers, so atrocities could happen with no fitting subject to hold responsible.
• The threshold for war. PAX For Peace, one of the founding groups of the Campaign to Stop Killer Robots, warns that fully automated weapons could lower the threshold for going to war by taking soldiers off the battlefield and distancing the public from war, and make democratic control of war harder.
• An arms race. In July 2015 over 1,000 experts in artificial intelligence signed a letter warning of an AI arms race and calling for a ban on autonomous weapons. It was presented at the International Joint Conference on Artificial Intelligence in Buenos Aires.

The arguments for regulation, not a ban

The article also gives the other side.

• Nathan Wood notes that in many cases specific autonomous systems can reliably tell military targets from protected people and objects, and argues that in some circumstances they are likely to be "superior, both morally and legally, to conventional alternatives".
• The article says it is doubtful that the humanitarian law concern can be an argument against weapons that do not break that law.
• Thomas Simpson and Vincent Müller argue that these weapons may make it easier to record who gave which command, which bears on the responsibility question.
• Another approach regulates use instead of banning. Arms control for military AI would likely need new international norms with technical specifications, active monitoring, expert-to-expert diplomacy and a legal and political verification process.

Where the talks stand

States have discussed these weapons under the Convention on Certain Conventional Weapons since 2014, and in 2016 its states parties set up an open-ended Group of Governmental Experts to carry on. The talks cover humanitarian law, accountability, possible bans and rules, and how much human control is needed. The Campaign to Stop Killer Robots formed in 2013, and the Holy See has called for a ban several times on weapons whose critical functions, such as picking and engaging targets, are under a machine's control. In March 2019 most governments at a UN meeting on the subject favoured a ban, while a minority opposed one. On 22 December 2023 the UN General Assembly adopted a resolution supporting international discussion of these concerns, by 152 votes to 4, with 11 abstentions.

Try it: place a system on the control scale

You can use Docherty's three degrees and the Holy See's test for meaningful human control to judge any system you read about. The Holy See's test asks three things: is there human supervision at least able to call off a mission, are all courses of action well known to the operator, and is the setting perfectly bounded and known?

1. A land mine. No one decides at the moment it fires: human out of the loop. It fails the Holy See's first test, since nothing can call it off once laid.
2. A ship's radar-guided defense gun. An operator sets the criteria and can switch it off, but it fires faster than a person could approve: closest to human on the loop.
3. A remotely piloted drone. A person chooses each strike: human in the loop.
4. Your own case. Take a system from the news, write which degree it fits, then answer the three questions. Note where you are unsure; that uncertainty is often the point the real talks turn on.

Frequently asked questions

What is a lethal autonomous weapon?

A military drone or robot that, once activated, can search for and engage targets without human input at that moment. There is no definition all states agree on.

Are lethal autonomous weapons banned?

The article cites a 2021 research report that found no domestic or international legal bans on developing or using them. States have discussed possible bans and rules under the Convention on Certain Conventional Weapons since 2014.

What does human in the loop mean?

It means a human must start the weapon's action. On the loop means a human supervises and can stop it; out of the loop means no human acts at all.

What is meaningful human control?

It is the idea that humans must keep enough information, judgment and power to limit tasks and targets so that a weapon's use complies with international humanitarian law.

Is Contain ASI a forecast of the future of AI?

No. Contain ASI is a game, and its page says all its labs, people and events are fictional. It is inspired by the AI 2027 scenario, and it is a story, not a forecast.

Get started with Contain ASI

The autonomous weapons debate is about how much control people keep over machines that act fast. In Contain ASI you face a related question as fiction: playing a researcher, research manager, CEO or government, you try for three years to keep four fictional AI labs from reaching recursive self-improvement before 2027. It runs in the browser and needs WebGL 2.

Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional.