The use of artificial intelligence in military and national security assessments can have serious consequences. In September, a report was released indicating that the United States had come close to military conflict with China due to an incorrect assessment of a Chinese vessel. An analysis conducted with the help of an AI chatbot mistakenly concluded that this vessel was carrying nuclear materials. This error was serious enough that by the time it was identified, American aircraft were on standby for intervention.
Serious Risks of AI Assessments
Extensive discussions arose regarding how this dangerous error occurred. However, what is concerning is the speed at which AI-based assessments may move through the command chain and become regarded as credible information. While we discuss the impact of a super-intelligent AI on humanity, it is clear that this technology will soon enter the national security crisis room.
AI can process large volumes of security information faster than any human team and identify patterns and anomalies in the data. But the question arises as to what impact this will have on the President's judgment. Based on my experiences studying decision-making in crises and information assessment, the real danger is that AI can reinforce leaders' biases and return its analyses to them in the form of prior assumptions.
Indirect Analyses and the Danger of Artificial Consensus
We often talk about AI as if these systems are completely independent and can act as a logical intelligent system. However, the reality is that humans created these models and trained them based on data generated by themselves. The result is that AI outputs are far from impartial.
Human decision-making flaws are well-known. The President and his advisors must always be wary of confirmation bias, which is our tendency to interpret evidence based on prior beliefs. In this context, there is an even greater risk in using AI. Presidential decisions involve multiple judgments that reasonable individuals can disagree on. Simplifying these judgments to a probabilistic assessment creates more ambiguity than clarity.
For example, on the night of May 1, 2011, then-President Barack Obama gathered with his advisors to assess information indicating that Osama bin Laden was hiding in a location in Abbottabad, Pakistan. If this information was correct, he would send American forces there. However, his advisors, all looking at the same data, provided individual estimates of the likelihood of bin Laden's presence at that location ranging from 30 to 95 percent.
Now imagine that a powerful AI system was present in that room and combined these assessments into a 63.7 percent probability of bin Laden's presence. We might achieve greater accuracy, but what is lost? This number cannot tell the President why there are such significant differences in the assessments and what assumptions have been formed based on them.
If the problems arising from human decision-making are well-known, there are also strategies to counter them. We can design our systems to act as benevolent rivals and train our decision-makers to treat them as such rivals. Furthermore, we should ask better questions and carefully examine our evidence and assumptions. This will lead to better decisions and reduce the risks arising from human biases.
Read more: Stephen Millar appointed as new Chief of Staff at the White House · NATO seeks to turn data into a defensive advantage
