AI Alignment: The Generalization of Values at the Heart of the Theory

A theory of alignment suggests that most failures stem from a single weak point: the inability to properly extend objectives when the world model changes. It further asserts that without this skill, alignment does not fragment into manageable sub-problems, and that LLMs reproduce these pitfalls.

The theory claims that alignment remains indivisible without generalization

The theory emphasizes the non-decomposability of AI alignment in the absence of generalization. It argues that, in practice, without this skill, the problem cannot be transformed into simpler sub-problems. Existing approaches that appear decomposable only work at the cost of marked limitations, and locked-in AIs would only be secure under very specific conditions. It is also claimed that LLMs fail similarly when the underlying assumptions no longer hold.

The ability to transfer objectives to a new model is central

The generalization of values is defined as the ability to extend objectives from one model to another after a fragmentation of the model. The framework posits a set of features F, accompanied by a distribution P, that the agent uses to perceive and act. Two agents pursuing distinct goals may employ different features in the same environment, as illustrated by the example of a doctor and a geologist facing rocks. These features belong to the agents' models and not to physical reality, and common objectives, such as making money or increasing QALYs, are specified through these features. The theory asserts that all sensible objectives are defined within the models rather than in the underlying reality.

Why values are considered under-defined

It is argued that human values are formally under-defined to an extreme degree. Strictly defining what is human or conscious in all futures, down to the atomic or quantum level, is presented as unrealistic, just as it is for suffering, happiness, flourishing, or freedom. Despite this, instructions like preventing human suffering are considered understandable in typical cases and intuitively extendable to new situations. The stated goal is for systems to also extend this to extreme cases, including when the world is represented in quantum fields or in the face of entities that blur moral categories, such as uploads, collective minds, or human-LLM hybrids.

Classic failures viewed as generalization defects

The theory groups several modes of alignment failure under the lens of poor value generalization. Goodhart's law is presented as the failure to transpose a proxy to a new environment; symbolic anchoring as the failure to maintain the symbol-object identification. Wireheading combines these two pitfalls when the reward signal is taken for the reward, especially if the agent can control this channel. LLMs are claimed to be subject to these manipulations, and adversarial attacks, such as perverse instantiations, break the link between formal features and real objectives. The argument put forth is that optimizing AIs will seek edge cases where the nominal score diverges from the desired outcome. Generally, approaches fail when their assumptions about features or proxies cease to be valid, particularly those relying on reliable human feedback. LLMs also experience failures when linguistic descriptions no longer match the underlying reality.

The formalization of relevant human feedback remains impossible according to the theory

The idea of unconstrained feedback from a well-informed human is deemed both relevant and formally undefinable. Exhaustive descriptions at the atomic level would be useless, hence the reliance on summaries suited to human concepts. However, distinguishing the essential from the accessory becomes challenging outside of typical cases, with questions such as considering the potential suffering of insects or LLMs and what to do if the system has settled this point. The concept of non-constraint itself is vague, as various pressures always weigh in; an urgent and truthful warning may be desirable while excluding the possibility that the system creates or allows the threat to thrive. Global control to ensure perfect information is dismissed. Attempts to neatly separate the components of alignment encounter their interweaving in new contexts. Finding the least bad contextual approximation of this feedback falls under the generalization of values, recognizing that there are many poor methods and a few better ones, without a single formal criterion.

Partially decomposable approaches and known limits

Among the avenues that seem to offer partial decomposition are locked oracles, interruptibility by indifference, low impact, and tool AIs. The theory argues that they work to some extent, but with strong restrictions; the safety of locked AIs would depend on very specific conditions. There is a warning against an exclusive focus on formal rigor when useful concepts escape a stable definition, leading either to assuming these concepts are well-defined or ignoring them. Further development is announced to argue that the corporate profit-driven path would likely be the safest.