The following is an excerpt from “Lethal Autonomy: The Future of Warfare Whether We Like It Or Not,” by former Air Force Secretary Frank Kendall. The book, published by Knox Press, is available to preorder here.
As a humanitarian and a human rights attorney, I’m all for effective arms control, but effective means meaningful and enforceable. Given that lethal autonomy is essentially with us now, as many examples show, I don’t see how an effective arms control regime is possible.
One has to begin with definitions of the things one wants to ban or constrain. I don’t know how to draw a line between the supposedly acceptable types of systems that already exist and those that would be banned. Trying to distinguish defensive (good) from offensive (bad) weapons doesn’t work when an autonomous air defense system can engage a civilian airliner. Most weapons can be used both defensively and offensively; killing is killing. Trying to distinguish degrees of autonomy is problematic as there is no obvious place to draw a well-defined line between permissible and impermissible technologies or capabilities.
The problem here is that there is a great appetite for technologies that automate functions like target detection, enemy intent inference, and optimum engagement planning. These capabilities are continuums with no sharp boundaries. There is a sharp boundary between preparing to engage and actually pulling the trigger, but that boundary is technically trivial and crossing it would be easy to conceal.
One possibility might be to ban all unmanned systems from carrying weapons. This approach would ban existing lethal drones, which one could consider, but it would not solve the problem of the risks associated with lethal autonomy. A system which includes a human operator can still operate in an autonomous mode. It would also tend to ban existing cruise missiles and other fire-and-forget munitions systems with autonomous seeker functionality, something which I expect no major power would accept. I’d be very willing to engage in a conversation aimed at solving this problem, but I’m not optimistic.
Assuming one could draw a line, there is still the problem of enforcement. The difference between full lethal autonomy and human control is some mechanism by which a human decision must be made to permit an engagement. Think of this as a trigger. In modern electronics systems, a trigger is a switch, which is more likely to be digital than mechanical. It is also likely to be remote from the lethal mechanism itself and the decision communicated through a digital link of some kind. I haven’t heard anyone suggest that it isn’t acceptable to automate everything but the crucial engagement decision. As a result, “cheating” on a ban on lethal autonomy is simplicity itself; somewhere in the weapon system include the capacity to close that switch autonomously.
The only way to prevent this from being done covertly is to have extremely detailed design information and extremely intrusive inspection regimes. Even if nations were willing to permit this, which they are not, it would be next to impossible to implement.
I would also question whether restricting the availability of lethal autonomy through international law is even wise, assuming it could be done effectively. Is there some good reason why we’d prefer that humans kill and be killed in war as opposed to machines assuming these functions? Over time many unpleasant functions humans performed have been turned over to machines. This includes the mind-numbingly repetitive and those involving brute force. A lot of human progress has been about relieving humans of functions that can be done less painfully and more efficiently by machines—freeing humans to engage in other pursuits.
It may sound fanciful, but if we have to resolve our conflicts through competitive violence, why not let our machines do that for us? This might strike some as ridiculous, but what is really ridiculous to me is that we still have a preference for sending humans out to kill each other to resolve our differences.
I believe there is a better and more pragmatic way to constrain human rights abuses than to ban lethal autonomy; it is to hold the responsible people accountable, regardless of the tools they employ. We have reasonably well-defined laws governing use of force now. The law of armed conflict may not be enforced consistently, but it does provide established and agreed limits to the brutality of war and the application of violence. There is no reason why the people who unleash indiscriminate autonomous killers on innocent victims, including neutrals and noncombatants, cannot be held accountable for their behavior. At the end of the day, we want to constrain the behaviors that we find objectionable. I believe we can still do that without banning autonomous weapons.
It would certainly take some effort by legal theoreticians and others to determine who in the chain of control of an autonomous lethal system to hold accountable and what the standards for violations should be for specific acts. I would offer that a good place to start is by enhancing the standards we apply today.
Currently, we accept that some “collateral damage” is inevitable in war. We ban collateral damage that is not a military necessity, intended to inflict terror on civilian populations, or intended to achieve genocide or ethnic cleansing. If we are going to permit autonomous machines to make decisions about lethal engagement, shouldn’t we raise the bar for when that’s acceptable? In any event, the minimum is the standard we employ today, and there is no reason to relax it.
So, who should be held responsible? This is a matter that is very standard in criminal law and personal injury cases; determining intent or degree of negligence required for liability. Any person or country that uses an autonomous lethal system should have a legal duty to ensure that system will not violate the law of armed conflict. This means that the designers, the testers, the acquirers, the fielders, the military operators, and the political decision-makers, all have a duty to ensure that a fielded system is compliant with established rules governing operational behaviors. In some cases, strict liability could be implemented.
It might be useful to also require that a specific designated official and organization be responsible for testing and verifying compliance, analogous perhaps to air worthiness certification today. For egregious violations—the intentional wholesale slaughter of innocents—the humans who directed or even permitted this use would be responsible, just as they are today.
As a last point on this topic, I don’t think the development of lethal autonomy can be stopped or even slowed appreciably. Most of the enabling technologies will proceed for independent reasons. The military advantages of employing lethal autonomous systems will become increasingly apparent to all. As I look at just ongoing US military service and DARPA efforts in this area, it is clear advances that will take us further into the world of lethal autonomy are well underway and advancing rapidly. Some nation-states and groups will not hesitate to embrace this opportunity, regardless of any ethical concerns.
I’ve heard representatives of several liberal democracies state publicly they will pursue the relevant technologies because of the fear less principled nations will do so. Almost daily I read open literature accounts about new unmanned lethal systems or improvements in the relevant areas of artificial intelligence—especially pattern (target) recognition. This genie will not be put back in the bottle.
If one can get past the objections to lethal autonomy, it opens up a fascinating range of options, including tactical behaviors that would certainly not be doctrinal for manned systems. Specifically, it is generally permissible, if the cost is not prohibitive, to sacrifice unmanned systems, singly or in groups, for a tactical or operational advantage.
My late friend, retired Army Major General Bob Scales, liked to talk about how a fundamental function of infantrymen, particularly in counterinsurgency campaigns, is to walk down a trail with a rifle toward the enemy in order to draw fire and expose the enemy to attack. That’s an oversimplification, but there is some truth in it. I once convened a group of general officers in the Pentagon to try exploring ways to end this paradigm. We didn’t get very far, but the option of using relatively inexpensive and expendable unmanned systems for this purpose is very attractive to me. The Ukraine conflict has shown us that it isn’t just attractive, it’s required.