A Chatbot Nearly Triggered a U.S. Military Operation

An erroneous intelligence report generated by a chatbot led U.S. aircraft to take off before a last-minute cancellation. The incident raises alarms about the military uses of AI, balancing the promise of accelerated decision-making against the risk of hallucinations in the absence of adequate supervision.

Potentially Lethal Decisions Require Safeguards, According to Jake Steckler

Jake Steckler, a researcher at GovAI and a former U.S. Army officer, believes it is essential for the military to be aware of the uncertainties inherent in language models. He considers this vigilance particularly important for any decision that could lead to the use of force, such as targeting, intelligence analysis, or operational planning, as these choices can have life-or-death consequences. Steckler argues that the incident should prompt a strengthening of safeguards surrounding AI, rather than a retreat from its use. According to him, these tools can be beneficial in appropriate contexts and with suitable safeguards. He warns that an adoption focused solely on speed could lead to incidents that might erode military trust in these systems, ultimately slowing their adoption. More broadly, military officials and external experts are concerned that errors produced by AI could ascend the chain of command without being questioned. The speed of AI can indeed foster hallucinations if human oversight is insufficient.

How an Automated Summary Produced False Intelligence

According to CNN, the source of the false information originated from an analyst at the Special Operations Command who used an artificial intelligence chatbot to merge data from open sources with classified signal intelligence. The chatbot produced an erroneous interpretation of the cargo manifest of a ship. Subsequently, the analyst used this tool again to present these incorrect conclusions in the form of an official summary, which then circulated through command channels. During the conflict with Iran, it was suggested that the ship was carrying components intended for a military nuclear program.

Flights Already Launched, Last-Minute Cancellation, and Pentagon Ambitions

CNN reports that in the spring, U.S. military aircraft were already in the air when the discovery of the original error was made. The operation, which targeted a Chinese ship based on intelligence generated by a chatbot, was canceled at the last minute, narrowly avoiding a potential confrontation with China. This case comes as the U.S. armed forces seek to adopt artificial intelligence to accelerate decision-making and maintain their superiority over China. The Pentagon has presented AI as a crucial lever to reduce the time of the kill chain so that commanders can act at the right moment.