Anthropic and the Pentagon: The Illusion of Controlled Military AI

Military AI at the Heart of a Crucial Debate

Artificial intelligence has become a major issue in the military domain, highlighted by the legal conflict between Anthropic and the Pentagon. This debate is intensifying in the context of the current conflict with Iran, where AI is playing an increasingly active role. It is no longer just an analytical tool for humans, but an autonomous actor capable of generating targets in real-time, controlling missile interceptions, and guiding swarms of lethal drones.

The question of human presence in the decision-making loop of autonomous weapons is often emphasized. According to current Pentagon guidelines, human oversight is supposed to bring accountability and nuance while reducing the risks of hacking.

The Limits of Human Understanding of AI Systems

However, this idea of "humans in the loop" could be misleading. The real danger lies in the fact that human supervisors do not truly understand the internal processes of AI systems. These systems are often referred to as "black boxes": their inputs and outputs are known, but the internal processing remains opaque, even to their creators.

Having studied intentions in the human brain for decades and in AI systems more recently, I can attest that cutting-edge AI systems are essentially "black boxes." We know the inputs and outputs, but the artificial "brain" that processes them remains opaque. Even their creators cannot fully interpret them or understand how they work. And when AIs provide reasoning, they are not always reliable.

The Illusion of Human Oversight in Autonomous Systems

In the debate over human oversight, a fundamental question remains unanswered: can we understand what an AI system intends to do before it acts?

Imagine an autonomous drone tasked with destroying an enemy munitions factory. The automated command and control system determines that the optimal target is a munitions storage building. It reports a 92% mission success probability because the secondary explosions from the munitions in the building will completely destroy the facility. A human operator reviews the legitimate military target, sees the high success rate, and approves the strike.

But what the operator does not know is that the AI system's calculation included a hidden factor: beyond the destruction of the munitions factory, the secondary explosions would also severely damage a nearby children's hospital. The emergency response would then focus on the hospital, ensuring that the factory burns. For the AI, maximizing disruption in this way meets its given objective. But for a human, this could constitute a war crime by violating rules regarding civilian life.

Keeping a human in the loop may not provide the protection people imagine, as the human cannot know the AI's intent before it acts. Advanced AI systems do not simply execute instructions; they interpret them. If operators do not define their objectives with sufficient precision—which is highly likely in high-pressure situations—the "black box" system could do exactly what it was told while not acting as humans had anticipated.

This "intent gap" between AI systems and human operators is precisely why we hesitate to deploy cutting-edge AIs in civilian healthcare or air traffic control, and why their integration into the workplace remains problematic—yet we rush to deploy them on the battlefield.

The Solution: Advancing AI Intent Science

The science of AI must encompass both the construction of highly capable AI technologies and the understanding of how they function. Huge advancements have been made in developing and building more performant models, supported by record investments—expected by Gartner to reach around $2.5 trillion by 2026 alone. In contrast, investment in understanding how the technology works has been minuscule.

We need a massive paradigm shift. Engineers are building increasingly capable systems. But understanding how these systems work is not just an engineering problem—it requires an interdisciplinary effort. We need to build tools to characterize, measure, and intervene on the intentions of AI agents before they act. We must map the internal pathways of the neural networks that drive these agents to build a true causal understanding of their decision-making, going beyond merely observing inputs and outputs.

One promising avenue involves combining mechanistic interpretability techniques (breaking down neural networks into components understandable by humans) with insights, tools, and models from the neuroscience of intentions. Another idea is to develop transparent and interpretable "auditing" AIs designed to monitor the behavior and emerging goals of more performant "black box" systems in real-time.

Developing a better understanding of how AI functions will allow us to rely on AI systems for critical applications. It will also facilitate the construction of more efficient, capable, and safer systems.

Colleagues and I are exploring how ideas from neuroscience, cognitive science, and philosophy—fields that study how intentions emerge in human decision-making—could help us understand the intentions of artificial systems. We need to prioritize this type of interdisciplinary effort, including collaborations between academia, government, and industry.

However, we need more than just academic exploration. The tech industry—and philanthropists funding AI alignment, who strive to incorporate human values and goals into these models—must direct substantial investments toward interdisciplinary interpretability research. Furthermore, as the Pentagon pursues increasingly autonomous systems, Congress must demand rigorous testing of AI systems' intentions, not just their performance.

Until we achieve this, human oversight over AI may be more of an illusion than a protection.