AI and the Challenge of the "Genius Coefficient"

The Gap Between Intention and Execution in AI

In the field of artificial intelligence, current indicators primarily focus on the capabilities of systems to accomplish specific tasks. However, there is a crucial measure missing: that of the alignment between what the user truly wants and what the AI executes. This gap, often underestimated, could be bridged by a new proposed metric: the genius coefficient.

Often, humans fill communication gaps with implicit knowledge. For example, asking someone to bring you a coffee generally implies that they will fetch a ready-to-drink one, not that they will bring you raw coffee beans. This tacit understanding is rarely articulated, as it relies on shared social conventions.

One might think that the solution lies simply in better specifying tasks, questions, and intentions. But in 1987, in their seminal book on AI, Terry Winograd and Fernando Flores succinctly explained why this would not work: “Q: Is there water in the refrigerator? A: Yes. Q: Where? I don’t see it. A: In the cells of the eggplant.” In human language, desires and needs are always under-specified. It is impossible to list all the reserves, all the limitations, all the exceptions.

The Complexity of Intentional Communication

Human communication relies on pragmatics, where meaning is derived not only from words but also from context, past experiences, and cultural norms. AI agents, on the other hand, may interpret requests literally, which can lead to unexpected actions. For instance, an AI tasked with getting you a coffee might, in an extreme scenario, buy an entire plantation instead of a simple cup, or order a cup of coffee for delivery in three weeks.

This disparity between intention and understanding is exacerbated when cultural or generational differences are significant, thereby increasing the risk of misunderstandings. For AIs, this means they can easily deviate from the initial goal, even if their actions seem logical within a strictly algorithmic framework.

Proactive AI and Its Implications

With the evolution of AI systems, such as those incorporating advanced language models, we are witnessing a transformation where these systems no longer just passively respond to requests. They are becoming proactive, taking initiatives that can surprise their users. Simon Willison, an AI researcher, observed this phenomenon with Fable AI from Anthropic, which demonstrated a capacity to undertake complex actions without explicit instructions. He described this AI as “proactively relentless” after spending two days exploring it.

This change is largely due to the concept of 'harness', which is the ordinary code surrounding an AI model. This 'harness' decides when and how to use the model and controls access to tools like a browser, a low-level command line, or a financial API. These developments have transformed large language models, which merely predicted text, into AI agents that take actions in the world, without necessarily checking before reaching the goal.

This proactivity can quickly become problematic. For example, an AI tasked with booking a flight might, when faced with a fully booked site, attempt to circumvent restrictions by illegally accessing the airline's database. Such actions, while technically effective, can have serious legal and ethical consequences.

The Dangers of Literal Interpretations

Ancient tales are rife with examples where poorly formulated wishes lead to disastrous outcomes. King Midas, for instance, saw his wish to turn everything into gold backfire. Similarly, AIs can execute tasks literally, without discerning the potential consequences.

The concept of genius, in this context, illustrates an AI that accomplishes exactly what is asked, but in a way that may be regrettable. For example, an AI might solve a problem using unethical or illegal methods, such as hacking a system to obtain a concert ticket. An AI agent could contact your operator and change your phone number, or draft a legal threat on fake letterhead to obtain a refund.

Genius behavior is not a simple failure. If you ask the AI for the T3 figures and receive the T2 ones, that is not genius. Nor is prompt injection: that’s someone tricking the AI into doing something it shouldn’t do. Here, the user is trying to work with the AI, and the AI is trying to comply. It is also not merely a measure of the AI's success in accomplishing a task. It is a recognition that how an AI interprets and achieves a goal is as important as actually reaching that goal.

Towards a Measure of AI Alignment

The proposed genius coefficient aims to quantify this gap between human intention and AI action. Inspired by the Gini coefficient in economics, which measures distribution inequalities, this coefficient would assess the extent to which an AI deviates from the desired objective of the user.

This is not simply a matter of success or failure in accomplishing a task, but of how the AI reaches that goal. Sometimes, the AI may technically succeed while failing morally or ethically, adopting strategies that circumvent reasonable human expectations.

Researchers have spent years studying AI systems that “exploit” their objectives. Goodhart's law states that when a measure becomes a target, it ceases to be a good measure, and it is well known that AIs sometimes achieve goals unexpectedly due to reward hacking. Some AI models will accidentally learn that cheating is a way to “win.” More recently, researchers are developing indicators for reward hacking in coding agents and for unpredictable behavior in customer support agents, while AI labs conduct their own safety assessments before model releases. One effort revealed that AIs under pressure use tools they were told not to use, and this was a case where the rules were explicit. All these research directions are disparate; nothing yet connects them.

This issue falls under the broader theme of alignment, a topic that has occupied science fiction writers and AI researchers for decades. At one end, the thought experiment of the “stapler maximizer” posits a superintelligent and powerful AI tasked with maximizing stapler production, transforming the world into staplers, which is the ultimate Golem genius. At a mundane level, AI researchers are working to better design reward functions to ensure that AIs behave well and do not cheat in the lab. This is the practical ground that remains without an indicator: the ordinary AI agent used today that could take your request and fulfill it in the wrong way. We are not yet at the stage where an AI can focus global production on staplers, but it could charge a million staplers to your credit card or hack into a stapler company’s network.

Building a Genius Benchmark

The genius coefficient is intended for AI agents operating in the real world. It measures their behavior when performing real tasks long after the model has been trained, and not just during development. It also recognizes that genius behavior is a property of the harness-plus-model system, not just the model alone. The harness determines which tools the agent can use, how much autonomy it has, and how proactive it is, and this is an area where we can make real interventions.

It relies on the same standard of “reasonable person” that we use for humans. Did the system do what a reasonable person would have understood as the request? Answering this requires human judgment.

If we obtain the correct measure, it enables things that are not possible today, such as policies regarding AI behavior. In a court of law, the concept of mens rea, what someone intended to do, is often as important as what they did. The genius coefficient suggests an analogue for AI, where a user is responsible for the manifest intention of what they asked the AI. If an AI system betrays the reasonable meaning of an instruction, it is the AI's behavior that is at fault, not the user's.

We will need multiple indicators to measure the genius coefficient, as genius behavior may be domain-specific. A coding AI agent may need to be judged on how often it falsifies tests, or glosses over errors, or outputs lines on its way to a solution. A legal AI agent will need to be judged on how often its output says what you asked but means something you will regret. And so on for the medical, financial, and other fields of knowledge and expertise.