Brief IA
›

RoPE Encodes Positions at the Heart of LLM Attention

Research·Tom Levy·

RoPE Encodes Positions at the Heart of LLM Attention

RoPE Encodes Positions at the Heart of LLM Attention
Key Takeaways
1RoPE emerged in 2021 with RoFormer.
2GPT-2 has 1,024 learned positional vectors.
3Sinusoidal encoding does not require learned positional parameters.
Why it matters : these methods provide models with information about the order of tokens that attention alone does not supply.

RoPE, which emerged in 2021 with RoFormer, has become the primary method for representing positions in modern large language models. Its principle differs from that of the learned vectors of GPT-2: positional information is incorporated within the attention mechanism.

RoPE Rotates Query and Key Vectors

With RoPE, the initial representations of tokens do not receive positional additions. It is the so-called query and key vectors that undergo a rotation, determined by the token's position in the sequence.

This operation occurs in every attention layer. Previous methods provided positional information at the time of constructing the initial representation of tokens.

GPT-2 Learns Its Positions, Sinusoidal Encoding Calculates Them

In GPT-2, each position has its own vector, learned during training according to a principle similar to that of word representations. This vector is added to that of the token: for instance, "fox" receives a different addition at positions 3 and 7, resulting in a different representation for attention. However, GPT-2 only has 1,024 learned positional vectors: none correspond to position 1,025. With this method, the context length must therefore be fixed before training.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

In 2017, the initial work on transformers presented another solution: calculating positional representations using sine and cosine functions. No positional parameters need to be learned in this case. The different pairs of features follow oscillations of distinct frequencies.

Attention Alone Does Not Detect Order

At the core of transformers, attention determines what is relevant for a token by examining other tokens. Its calculation uses dot products between vectors, without having an indication of their order in the sequence on its own.

Positional information provides this missing reference. Whether learned, calculated by sinusoidal functions, or integrated through rotation, it allows the model to take into account the order of words and better grasp linguistic nuances.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.