Google DeepMind: $10 Million to Prevent AI Agent Chaos

Google DeepMind and the Risks of Interacting AI Agents

Google DeepMind has recently decided to fund research on the potential dangers associated with the interaction of millions of online artificial intelligence (AI) agents. Rohin Shah, head of research on the safety and alignment of artificial general intelligence (AGI) at Google DeepMind, has expressed concerns about the emergence of new risks. These risks are linked to the mass market arrival of agents capable of performing tasks without human supervision and following the instructions of other agents.

To address these concerns, Google DeepMind, which showcased agent-based tools at Google I/O last month, has partnered with several organizations to create a $10 million fund. This fund is intended to encourage researchers to study the behavior of multi-agent systems and find ways to prevent dangerous scenarios. Partners include Schmidt Sciences, a philanthropic foundation created by Eric and Wendy Schmidt, ARIA, the UK agency dedicated to ambitious projects, the Cooperative AI Foundation, a UK-based nonprofit research organization, and Google.org, Google's charitable arm.

An Investment to Anticipate Threats

Rohin Shah and James Fox, who leads the Science of Trustworthy AI program at Schmidt Sciences, explained their expectations regarding the use of this $10 million. Although this amount may seem modest compared to Google DeepMind's research budgets, the goal is to stimulate academic research outside of tech companies. Shah emphasizes that the academic world has the capacity to project far into the future and conduct work that is not prioritized in industrial labs.

Shah also mentioned that there is currently no dedicated research field for multi-agent safety, but it would be desirable to create one. The concern lies in the fact that the increase in the number of AI agents could lead to a tipping point where imagined scenarios become real. Shah compares this to humanity, where institutions can accomplish things that no individual could achieve alone.

Potential Risks of AI Agents

Shah believes we have a few months before agents are deployed in sufficient numbers for potential risks to become a real concern. He aims to anticipate this moment. The risks envisioned by Shah and Fox include amplified versions of problems already present on the Internet, such as scams, prompt injections (where an AI agent receives malicious instructions and becomes autonomous malware), and other forms of cyberattacks. Shah explains that the idea is to examine what humans are currently doing and imagine the agent version of those actions.

James Fox emphasizes the importance of protecting the digital common good, essential for the functioning of society, against total anarchy. When asked if they were considering more pessimistic catastrophic scenarios, such as a widespread economic collapse, Shah replied that this was not expected in the short term, joking that it might happen a bit later.

Simulations and Academic Research

Shah and Fox believe that the only way to understand what might happen when many multi-agent systems interact is to conduct realistic simulations. They want researchers to immerse AI agents in controlled environments to study their behavior. Fox specifies that it is impossible to predict what will happen by studying individual agents or small groups of agents in isolation. The complexity lies in the large number of simultaneous interactions.

Some researchers, including a team from Google DeepMind, have suggested that artificial general intelligence could emerge not from a single super-intelligent model, but from a kind of collective consciousness of agents, where the capabilities of the whole exceed the sum of its parts.

A Call for Caution in AI Development

Google DeepMind is not the only company warning about the risks associated with the technology it develops. Recently, Anthropic released guidelines for the deployment of AI agents based on a cybersecurity approach known as "zero trust," which assumes that a computer system is vulnerable and that a breach will occur.

Refael Angel, co-founder and CTO of Akeyless, a cybersecurity company based in Tel Aviv, highlights the importance of understanding the new risks introduced by agent-based systems. Angel explains that every security approach in the past assumed that the machine was software written by a human, performing fixed tasks. An agent, on the other hand, breaks these assumptions by reasoning, improvising, and potentially being misled by a simple phrase buried in a document.

Angel welcomes the funding call launched by Google DeepMind but warns that security researchers might overlook current issues in favor of more exotic hypothetical scenarios. Fox notes that risks that were hypothetical a few years ago are now very real, emphasizing that the future has arrived more quickly than expected.