LLMs and Their Hallucinations: An Inevitable Structural Challenge

LLMs and Their Hallucinations: An Inevitable Structural Challenge

A Characteristic of the Architecture

The hallucinations observed in large language models (LLMs) do not stem from poor data quality or inadequate training. Contrary to what one might think, these errors cannot be simply corrected by more extensive human reinforcement learning (RLHF), improved filtering, or an extended context window. These hallucinations are, in fact, an intrinsic characteristic of the systems, related to how they are optimized.

For several months, this position has been defended, although some researchers, focused on improving retrieval and alignment techniques, prefer to consider more optimistic solutions. However, this argument overlooks a crucial aspect: the internal geometry of the models. To fully understand the phenomenon, it is essential to examine what happens inside the model when it confidently generates an incorrect response. This involves analyzing the internal trajectory of the representation, layer by layer, from input to output.

What the Residual Flow Knows Before the Model Lies

To explore this phenomenon, an experiment was designed using a factual prompt, where a transformer should normally retrieve a memorized association. This prompt is tested under two conditions: one where the model provides the correct answer and another where it produces a hallucination. By following the trajectory of the residual flow, or internal representation vector, through the network, the aim is to determine whether the divergences in trajectory are due to a lack of information or another factor.

The internal state of the model at each layer can be visualized as a point in a high-dimensional space. As the model processes a prompt, this point moves, tracing a path. The experiment measures whether the path taken during a correct response and that during a hallucination diverge simply because one path is shorter, or because they orient differently while covering the same distance.

The results show that the paths have the same length but point in different directions. Figure 1 illustrates two trajectories starting from the same origin, covering the same distance, but ending at different points in space: one towards the correct answer, the other towards an error.

The Engagement Ratio: Where Suppression Becomes Visible

The study introduces a new metric, the engagement ratio κ, which measures how much of the model's probability mass is directed towards or away from the correct token at each layer.

In a correct processing scenario, κ increases monotonically across the network, indicating that the model gradually commits to the correct answer. This is the expected behavior of a system retrieving a learned association.

However, in the case of a hallucination, κ does not remain stable, which would indicate a simple retrieval failure. Instead, κ collapses, reaching a significantly lower minimum than its starting value before slightly recovering in the final layers. For example, in the LLaMA-2 13B and Mistral 7B models, κ drops to 0.08, with p-values below 10⁻¹⁰⁰, highlighting the magnitude of this effect.

What Happens?

The model does not simply fail to find the correct answer. It actively shifts the probability mass away from the correct token in the same layers where it should be moving it towards it in a correct condition. This failure is essentially an override.

The model has indeed encoded the correct answer, making the collapse of κ significant. If the model simply lacked the relevant association, we would observe a flat or noisy trajectory. Instead, the trajectory starts in the right direction but then turns. The correct token accumulates probability in the early layers, then loses it in the intermediate layers, where it should increase in a correct condition.

The article precisely establishes the phenomenon but leaves the why open. The most plausible interpretation is competition among token predictions in a given context, generating its own pressure.

Practical Implications

For those developing large-scale production systems, the conclusion is clear: it is necessary to implement a domain-specific monitor, trained on representative data from that domain. The idea of a universal detector is not supported by current evidence.

What Geometry Cannot Correct

The documented override mechanism is not a bug to be fixed. It directly stems from the objective function used to train LLMs. Predicting the next token in discrete sequences does not allow the model to prioritize factual accuracy over contextual coherence. The training signal cannot make this distinction. The model learns to be fluent, which is remarkable, but when fluency and accuracy do not coincide, fluency prevails.

To identify the specific circuits that implement this suppression and determine if they can be modified, further research is needed, including large-scale activation patching and circuit-level analysis. Several research groups are already working on these issues.