Facebook Twitter LinkedIn Email App

March 2020 Online Exclusive Article

A Call for Dialogue with the Opponents of Lethal Autonomous Weapon Systems

Lt. Col. Richard T. Muster, U.S. Air Force

Download the PDF depuy

Protest leaders organize a worldwide campaign to stop development of autonomous intelligence lethal weapons on 22 April 2013 in London.

Lethal autonomous weapon systems (LAWS) will be critical to twenty-first-century warfare. Nation-states adversarial to the United States are developing them now. LAWS have significant advantages in speed, complexity management, machine-human teaming, performing in electromagnetically denied environments, and in standoff capability. Yet the Silicon Valley industrial base and others in academia have serious qualms about developing weaponized artificial intelligence (AI).


The Problem

Organizations such as Campaign to Stop Killer Robots and the Future of Humanity Institute have called for a complete international ban of all LAWS.1 The former is a lobbying organization that prompted UN Secretary-General António Guterres to call for a prohibition of LAWS. The resulting list of signatories to this call resulted in a quorum that included Algeria, Columbia, Djibouti, Ghana, Venezuela, and the Holy See (i.e., no one with skin in the game). The latter was created by Nick Bostrom, the Oxford University philosopher of Superintelligence: Paths, Dangers, Strategies.2 However, an international ban on LAWS would be completely unenforceable (think Intermediate-Range Nuclear Forces Treaty). Moreover, if the United States unilaterally decided to declare against LAWS, it would be tantamount to a one-sided disarmament that would yield future battlefields to America’s adversaries.

The Solution

In the absence of an international or a unilateral ban on LAWS, the Department of Defense (DOD) listened to Silicon Valley, academia, and other concerned organizations and established a dialogue that addressed those concerns. The DOD inaugurated the Defense Innovation Board, which began holding listening sessions with industry and academia in 2019 regarding the ethical use of AI for war and has released a set of governing principles.3 The DOD also created the Joint Artificial Intelligence Center to work with industry.4 However, the United States should do more. This fundamental question must be considered: Can human morality be made somehow portable, such as by going out with an AI weapon and constraining it to abide by “ideal” moral behavior?

The Concerns

In July 2015, at the International Joint Conference on Artificial Intelligence, a body of academics and scientists published an open letter opposing autonomous weapons.5 The letter cited a range of ethical issues with LAWS and called for an international ban.

First, the academics and scientists worried that if LAWS targeted humans, there might be some difficulty in assigning culpability for violating the laws of war. However, military doctrine and regulation are clear on this issue; the commander who fields LAWS would be responsible for its actions in the same way he or she is responsible for any actions taken by his or her subordinates.6 An AI-driven weapon system should act in accordance with the commander’s intent, and it must still follow rules of engagement, which derive from the Laws of Armed Conflict and national policy and provide a moral code for the conduct of war.

Left Quote

An AI-driven weapon system should act in accordance with the commander’s intent, and it must still follow rules of engagement, which derive from the Laws of Armed Conflict and national policy and provide a moral code for the conduct of war.

Right Quote

Second, there was worry that LAWS would lower the threshold for going to battle because it lowered the potential for friendly casualties. However, LAWS might do the opposite. LAWS may instead deter combat as many other weapon systems do. For instance, the development in the west of fifth generation fighter aircraft did not lead to increased western incursion into Chinese airspace because it made attack easier. It merely led to the latter’s development of anti-access/area denial defense strategies leading to a stable equilibrium. This concern is not trivial, but it deserves better study before becoming a decision point for policy.

Third, there were concerns that LAWS would be “ideal for tasks such as assassinations, destabilizing nations, subduing populations, and selectively killing a particular ethnic group.”7 Assassination, destabilizing nations, subduing populations, and selective killing do regrettably occur sometimes in armed conflict. However, human imperfection is the cause of violations of the Laws of Armed Conflict, and machine intelligence that is governed by commander’s intent and the Laws of Armed Conflict will restrict such abuses.

Fourth, there were worries that “if any big military power goes ahead with the construction and employment of LAWS, then AI will become the Kalashnikov of tomorrow, because everybody’s going to have them.”8.That worry may depend on the situation at hand. If LAWS is an expensive and technically advanced weapon, then not everyone will have them. Making the claim that “everybody’s going to have them” is like claiming that intercontinental ballistic nuclear missiles with multiple, independently targeted reentry vehicles (MIRVs) will become the Kalashnikovs of tomorrow. Extremely technical and costly weapon systems like MIRVs defy mass production and proliferation. Conversely, if a LAWS is inexpensive and easy to duplicate, it will probably be akin to the toy-like drones depicted in the YouTube short film Slaughterbots. This 2017 social media production was created by Stuart Russell, who is a Future of Life Institute advisory board member and a Berkeley University computer science professor.9 This film depicts quadcopters with cameras, feature recognition software, and explosive shaped charges that swarmed and killed masses of people. Weapons such as those depicted in Slaughterbots could be countered with extremely low-tech countermeasures. Noted policy expert Paul Scharre, author of Army of None: Autonomous Weapons and the Future of War, noted in his op-ed repudiation of the video that such weapons would be easy to guard against.10 He said, “There would be a run on the chicken wire market.”11 Not to mention, shotguns. Essentially, cheap LAWS would be easy to defend against.

A Russian Uran-9, an armed robot, is displayed during a parade rehearsal 6 May 2018 in Moscow.

In addition to the concerns of the International Joint Conference on AI, there have been worries rooted in popular science fiction that are not rooted in science. As mentioned above, Bostrom opined that AI could grow beyond human control.12 Author Louis Del Monte takes this notion to a finale in his book Genius Weapons: Artificial Intelligence, Autonomous Weaponry, and the Future of Warfare, worrying that weapons connected to AI could break away from human control by becoming recursively superintelligent.13 Both authors conflate artificial narrow intelligence (ANI) with artificial general intelligence (AGI). ANI, the only operationally effective form of AI currently, solves narrow problems, such as Google’s AlphaGo defeating Ke Jie in the game Go.14 This is the kind of AI under discussion, and it could never become superintelligent. Russell worries that many ANIs develop by “making progress towards AGI” in his recently published book Human Compatible: Artificial Intelligence and the Problem of Control.15.But he provides an ample repudiation for this assertion a few pages later when he reflects that the game AlphaGo, if given the task of reaching a distant star system, could never do so by generating millions of Go moves. AlphaGo is essentially a black box only capable of generating extremely narrow plays for the game Go. So too with the proposed AI weapon system. A weapon designed to find, fix, track, target, engage, and assess an enemy combatant will not have the sophistication or complexity to improve itself to the point of becoming sentient. And as previously stated, no one in the Pentagon is talking about plugging AGI (whenever it becomes technically feasible) into weapons.

Machines Can Make Faster Decisions than Humans

Daniel Kahneman, author of Thinking, Fast and Slow and a renowned critic of primate thinking, popularized the notion of System 1 and System 2 modes of operation in human cognition.16 System 1 is a way of thinking and making decisions that is fast but sloppy and prone to errors. In other words, it is usually “good enough.” This kind of decision-making has been enshrined in modern military thought since Col. John Boyd proposed the OODA loop: observe, orient, decide, act. This kind of decision-making kept its inventor Col. Boyd alive while flying against North Korean MiG 15s during the Korean War. The System 2 mode of operation, on the other hand, is slower than System 1 but more accurate. It is more akin to the military decision-making process (MDMP), which can take weeks or months and is used by military planners to formulate theater campaign plans. During the heat of battle, commanders or troops are obliged to rely on System 1 OODA loops in order to react quickly. This is because an F-22 air superiority fighter cannot conduct the MDMP to decide how to evade a missile launched by a Russian S-400 integrated air defense system. A combatant must react quickly, but quick reactions are often sloppy. But it may be argued that AI could perform System 2 computation in the same time a human could perform System 1 computation on any given military response. Witness the deep strategic thinking and decision-making of Google’s AlphaGo.17

Machines Can Have Better Judgment than Humans

Much has been written about human cognitive fallacies. For instance, some of humanity subscribe to outrageous pseudoscience in order to cement group membership; think #FlatEarth and #VeganCats. Also, humans are not very good at statistical thinking, as evidenced by the chronic misapplication of probability and expectation in lotteries, surveys, and insurance. People see patterns and connections where none exist, and people make up stories to explain what they do not understand. This in turn leads others to believe in gods, demons, fairies, alien abductions, and government conspiracies. People sometimes think myopically and make judgments without deep consideration or collecting enough data. Lecturer Michael Shermer points out that humans are prone to type I, or false-positive, statistical errors.18 For instance, humanity’s hominid ancestors came to favor the assumption that rustling in the grass was most likely a lion, and therefore, they should run away from any instance of rustling grass. If humanity’s ancestors were wrong, who cared? There was no lion waiting there to eat the hapless individual who witnessed the rustling grass. If humanity’s ancestors were prone to type II statistical errors, or false negatives, they might assume the rustling in the grass was just the wind, and one day a lion would eat them. Any such hominids were selected for extinction. The modern corollary is that warfighters, assuming they do not run away like the type I hominid ancestors might have, may shoot first and ask questions later. AI-driven target recognition could collect data, compare it to specific target folders, and act—or not act—before any human was even sure which bit of grass was rustling.

Left Quote

A weapon designed to find, fix, track, target, engage, and assess an enemy combatant will not have the sophistication or complexity to improve itself to the point of becoming sentient.

Right Quote

Machines Can be More Moral than Humans

Famous Renaissance writer on statecraft and war Niccolò Machiavelli wrote in his book Florentine Histories (1532),

For while God and Nature have set man’s fortunes within his reach, these are to be won by violence rather than by industry, and by evil arts rather than by good. Whence it comes that men devour one another, and the weakest goes to the wall.19

It could be argued that Machiavelli was a bit of a pessimist. But the lesson should not be lost on anyone in modern times. Humans can be and often are complicated, unpredictable, and sometimes lean toward acts of villainy. Examples of this from modern science include the Milgram Experiment and the Stanford Prison Experiment, where people under certain conditions expressed “genuine sadistic tendencies.”20 The point here is the following consideration: What makes anyone think that sensors and weapons, coupled with AI, would be less moral than human morality? If morality is a function of high executive cognition, then so too is immorality. Wasn’t Adolf Eichmann human? Pol Pot? Suharto? If I was presented with a human and a nonhuman entity and asked to choose which of the two was less likely to harbor evil, I would choose the robot every time. If I was asked which of the two would be less self-serving, ideological, cynical, racist, corruptible, sadistic, murderous, or tyrannical, I would choose the robot. Moreover, in his book Moral Machines: Teaching Robots Right from Wrong, Wendell Wallach posits that

computer intelligence is built on a logical platform free from desires, drives and goals other than those that engineers design into the system. Human cognitive faculties evolved from and develop alongside an instinctive emotional platform directed at survival and procreation.21

People are asked to oversee the weapon system with the (questionable) better angels of their nature, based as they are on survival and procreation. Machines, on the other hand, will behave as they are programmed, within the bounds of the international laws of armed conflict and as codified in the commander’s intent.

A Marine with Kilo Company, 3rd Battalion, 4th Marine Regiment, poses with a drone during Urban Advanced Naval Technology Exercise 2018 at Camp Pendleton, California

Commander’s Intent

A thought experiment: imagine that an Air Force pilot’s brain was read and duplicated exactly into an artificial intelligence. Imagine that the duplication was so complete that the AI knew everything the pilot knew. The AI retained all of the education the pilot gained from his or her time at the Air Force Academy through the Air Force War College. It loves the pilot’s spouse and children. Moreover, just like the pilot, it believes in Jesus, leans Republican, is slightly homophobic, loves single malt Scotch, prefers blondes, bets (and loses) heavily at the track, has a pathological aversion to custard, and harbors an ambition for a general’s star. Now imagine that specific AI was sent to war. Also imagine that the original pilot, the flesh and blood one, was held accountable for all the AI’s actions. As long as the machine was refreshed regularly to sync it with the pilot so that it could benefit from newly learned experiences, one could expect two things: that the pilot would be agreeable to this arrangement, and that reasonable people would agree that the AI would discharge the duties of a U.S. military officer, including moral duties, as well as those of the flesh-and-blood pilot. I propose that no great theory of mind is necessary in order to govern autonomous warfighting machines. Essentially, that pilot’s agency is somehow placed in charge of the machine’s actions and the pilot is always responsible for those actions.

Left Quote

Military personnel cannot be trained to abdicate their military positions when they experience a decline in their cognitive or moral faculties, but machines can.

Right Quote

The approach of command delegation is not that uncommon in the military. The U.S. Navy, for example, employs a command-and-control style called command by negation. In this model, fleet commanders, while underway, will exercise fully delegated command of their maritime formations in anticipation that communications with their mainland headquarters might be denied or degraded. In that event, they would wield the authority of the mainland commanders who would only negate a maritime commander’s actions if it were necessary. The U.S. Navy takes this a step further through its command and control of the AEGIS missile defense system. As described by Scharre in Army of None, AEGIS has four settings of varying autonomy that range from full human command and control to almost none except by negation.22 The latter is necessary if the battle is deemed too fast-moving for the human mind to participate in. Yet, the commander is still responsible for the AEGIS weapon by dint of the weapon control status the commander directs for the machine before any enemy engagement. The commander always provides commander’s intent to the machine (in actuality, network of machines) and is always responsible for the machine’s behavior. Consider what would happen if commander’s intent was separated into two discrete functions: to achieve a military objective and to restrain the robot as provided by the Laws of Armed Conflict.

Toward a Machine Morality

Machines can be engineered to stop before they fail. It is an engineering strategy called fail-safe. This, along with other engineering strategies like elegant degradation, fail-secure, and fault-tolerance provides performance envelopes for machines that do not exist in humans. As found in the DOD AI Principles,

DoD has a long-established history of test and evaluation (T&E) and verification and validation (V&V) of its systems. In this respect, AI systems ought not be any different. In high-risk areas, such as with nuclear weapons, there also exist additional programs for authorization, safety, and reliability (e.g., system surety).23

Military personnel cannot be trained to abdicate their military positions when they experience a decline in their cognitive or moral faculties, but machines can. One idea central to modern computing is the universal Turing machine, posited by Alan Turing, one of the great computational pioneers.24 The idea was that a machine can read and write bits on an unbounded length of tape such that it would be capable of computing any computable thing. Cloud computing services, the personal computers in contemporary offices, and in smart phones are all considered to be Turing machines or Turing complete. Concomitant to Turing’s machine was something called an oracle.25 This was generally intended to be a human operator who could provide the Turing machine with guidance in operation. In particular, the oracle provided input to the Turing machine that it could not generate itself. Relative to this discussion, a LAWS could be provisioned with two AI brains. The first would be a mission AI programmed to achieve a military objective through acts of violence. The second AI would be an ethics AI (an oracle) loaded with the rules of engagement that could subsume the mission AI if it moved to violate rules of engagement. This approach has been recently borne out by generative adversarial networks (GANs). Ian Goodfellow, now a researcher on the Google Brain team, developed a way to employ an AI oracle to guide another AI to create hyper-realistic imagery. A recent article on Goodfellow states,

The magic of GANs lies in the rivalry between the two neural nets … The first one, known as the generator, is charged with producing artificial outputs, such as photos or handwriting, that are as realistic as possible. The second, known as the discriminator, compares these with genuine images from the original data set and tries to determine which are real and which are fake. On the basis of those results, the generator adjusts its parameters for creating new images.26

It is not impossible to imagine at this point a Turing oracle acting as an ethics AI that regulates the mission AI to adhere to the rules of engagement.

“Robots can be unnerving if they begin to resemble human beings or excel at human activities.”


I began this article by stating that LAWS must happen if the United States is to remain militarily competitive in the future. I also stated that rather than argue with the opponents of LAWS, there must be a conversation to resolve their concerns. It is easy to understand the ready revulsion that many people feel toward the idea of AI. The members of Campaign to Stop Killer Robots and the Future of Humanity Institute are not stupid people. Rather, they number among some of the greatest minds of science. What is more, the Army needs to attract the participation of at least some of them. But I believe these great minds have been excessively unnerved by the inherent unnaturalness of weaponized robots. The fault, I believe, rests with what robotics professor Masahiro Mori called the uncanny valley.27 This idea posits that robots can be unnerving if they begin to resemble human beings or excel at human activities. When they are designed to kill, robots can be even more unsettling. Also, it is easy to imagine that apocalyptic scenarios from popular culture like the films Transcendence would seize upon the minds of such people as Stephen Hawking, Max Telmark, Stuart Russell, and Frank Wilczek who worried in their joint article “Transcending Complacency on Superintelligent Machines” that humanity could become enslaved by science fiction super-intelligence.28 But these luminaries have either conflated artificial general intelligence with artificial narrow intelligence or exaggerated the closeness between the two. I have laid out in this article one possible avenue for humanity to explore that contains two caveats: that general artificial intelligence never be hooked up to a weapon but only to narrow artificial intelligence, which is the black box Russell described, and that a Turing oracle (ethics AI) enforces rules of engagement by constraining the computer that carries out the mission (mission AI) the same way a GAN discriminator constrains a GAN generator. There are certainly other avenues that we could also explore. If humanity does not discover any creative moral solutions at all, then it risks becoming victims of a far less generous adversary.

The views expressed here are the author’s and do not reflect those of the U.S. government or any part thereof.


  1. Campaign to Stop Killer Robots, 10 June 2019, accessed 23 January 2020, https://www.stopkillerrobots.org; Toby Walsh, ”Autonomous Weapons: An Open Letter From AI & Robotics Researchers,” Future of Life Institute, 28 July 2015, accessed 23 January 2020, https://futureoflife.org/open-letter-autonomous-weapons/?cn-reloaded=1.
  2. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014); “About FHI,” Future of Humanity Institute, accessed 24 January 2020, https://www.fhi.ox.ac.uk/about-fhi/.
  3. Defense Innovation Board, AI Principles: Recommendations on the Ethical Use of Artificial Intelligence by the Department of Defense (Washington, DC: Department of Defense, 31 October 2020), 8, accessed 23 January 2020, https://media.defense.gov/2019/Oct/31/2002204458/-1/-1/0/DIB_AI_PRINCIPLES_PRIMARY_DOCUMENT.PDF.
  4. Sydney J. Freedberg Jr., “Joint Artificial Intelligence Center Created under DoD CIO,” Breaking Defense, 29 June 2018, accessed 23 January 2020, https://breakingdefense.com/2018/06/joint-artificial-intelligence-center-created-under-dod-cio/.
  5. Toby Walsh, “Autonomous Weapons: An Open Letter from AI & Robotics Researchers,” Future of Life Institute, 28 July 2015, accessed 6 March 2020, https://futureoflife.org/open-letter-autonomous-weapons/?cn-reloaded=1.
  6. Field Manual 6-27, The Commander’s Handbook on the Law of Land Warfare (Washington, DC: U.S. Government Publishing Office [GPO], 7 August 2019), 8-1, 8-5; Office of General Counsel, Department of Defense, Department of Defense Law of War Manual (Washington, DC: U.S. GPO, June 2015, updated December 2016), para. 18.23.3.
  7. Walsh, “An Open Letter from AI & Robotics Researchers."
  8. Ibid.
  9. “‘Slaughterbot’ Autonomous Killer Drones,” YouTube video, posted by “Ralph’ster,” 17 November 2017, 7:39, https://www.youtube.com/watch?v=ecClODh4zYk.
  10. Paul Scharre, Army of None: Autonomous Weapons and the Future of War (New York: W. W. Norton, 2018).
  11. Paul Scharre, “Why You Shouldn’t Fear ‘Slaughterbots,’” Center for a New American Security, 22 December 2017, accessed 23 January 2020, https://www.cnas.org/publications/commentary/why-you-shouldnt-fear-slaughterbots.
  12. Bostrom, Superintelligence.
  13. Louis A. Del Monte, Genius Weapons: Artificial Intelligence, Autonomous Weaponry, and the Future of Warfare (Amherst, NY: Prometheus, 2018).
  14. Paul Mozur, “Google’s AlphaGo Defeats Chinese Go Master in Win for A.I.,” New York Times (website), 23 May 2017, accessed 28 January 2020, https://www.nytimes.com/2017/05/23/business/google-deepmind-alphago-go-champion-defeat.html.
  15. Stuart J. Russell, Human Compatible: Artificial Intelligence and the Problem of Control (London: Penguin Random House, 2019), 37–49.
  16. Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2011), 39–49.
  17. Mozur, “Google’s AlphaGo Defeats Chinese Go Master in Win for A.I.”
  18. Michael Shermer, “The Pattern Behind Self-Deception,” TED, February 2010, accessed 23 January 2020, https://www.ted.com/talks/michael_shermer_the_pattern_behind_self_deception?language=en.
  19. Niccoló Machiavelli, Florentine Histories, Book III (Los Angeles: HardPress, 2014), 190.
  20. Stanley Milgram, “Behavioral Study of Obedience,” Journal of Abnormal and Social Psychology 67, no. 4 (October 1963): 371–78; Philip George Zimbardo, The Stanford Prison Experiment: A Simulation Study of Psychology of Imprisonment (Palo Alto, CA: Stanford University, 1971).
  21. Wendell Wallach, Moral Machines: Teaching Robots Right from Wrong (Oxford, UK: Oxford University Press 2010), 142.
  22. Scharre, Army of None, 194.
  23. Defense Innovation Board, AI Principles.
  24. Robert I. Soare, “Turing Oracle Machines, Online Computing, and Three Displacements in Computability Theory,” Annals of Pure and Applied Logic 160, no. 3 (2009): 368–99.
  25. Ibid.
  26. Martin Giles, .The GANfather: The Man Who’s Given Machines the Gift of Imagination,” MIT Technology Review (website), 21 February 2018, accessed 23 January 2020, https://www.technologyreview.com/s/610253/the-ganfather-the-man-whos-given-machines-the-gift-of-imagination/.
  27. Norri Kageki, “An Uncanny Mind: Masahiro Mori on the Uncanny Valley and Beyond,” IEEE Spectrum, 12 June 2012, accessed 23 January 2020, https://spectrum.ieee.org/automaton/robotics/humanoids/an-uncanny-mind-masahiro-mori-on-the-uncanny-valley.
  28. Stephen Hawking et al., “Transcending Complacency on Superintelligent Machines,” Huffington Post, 19 June 2014, accessed 16 December 2019, https://www.huffpost.com/entry/artificial-intelligence_b_5174265.

US ISSN 0026-4148


Lt. Col. Richard Muster, U.S. Air Force, is a cyber warfare officer at the 16th Air Force in Joint Base San Antonio-Lackland, Texas. He has a master’s degree in computer engineering from the Air Force Institute of Technology with an emphasis in artificial intelligence and machine learning algorithms. He has a second master’s degree in military operational arts and sciences from the Air Command and Staff College with an emphasis in multi-domain operational strategy. He has deployed twice to Afghanistan.

Back to Top