Brief IA
›

Llama 3.1 Drops the Original Transformer Encoder

Research·Tom Levy·

Llama 3.1 Drops the Original Transformer Encoder

Llama 3.1 Drops the Original Transformer Encoder
Key Takeaways
1Llama 3.1 does not have a separate text encoder from the initial Transformer.
2A theoretical example with 8,192 tokens would require 4 GiB of cache, excluding the model weight.
3FlashAttention maintains mathematically exact dense attention.
Why it matters : the differences relate to both the learning task, context storage, and computation execution.

Llama 3.1 no longer features a separate encoder or one of the attention blocks from the Transformer introduced in 2017. The adaptations also concern memory, token positions, and the execution of calculations, without abandoning the principles of the Transformer.

Context and Calculation Cost in Memory

In a theoretical example, a model with 32 layers, 32 query heads, and 128 numbers per head would require 4 GiB of cache for a single sequence. This scenario assumes 8,192 tokens of context, two bytes per stored number, and, for each query head, its own key and value heads. Model parameters and other execution memory requirements are additional.

This storage is particularly useful in scenarios where a document is inserted into a conversation and then queried multiple times while retaining previous details. To consult previous tokens, attention leverages keys and values: the KV cache can retain them during generation rather than recalculating them for each new token. Each attention layer has its own cache, the content of which increases as the sequence grows.

Data movement constitutes another expense. A simple realization of the calculation may deposit a large score matrix in GPU memory and then retrieve it for the next step; these transfers remain costly despite fast arithmetic. FlashAttention operates in tiles, progressively retaining the statistics necessary for normalization without fully storing this matrix in high-bandwidth memory.

This method changes execution, not the mathematical nature of the operation: attention remains dense and accurate. However, numerical results may vary with the order of floating-point operations.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Generation Does Without Separate Source Text

To learn to continue a text, a model receives a sequence start and predicts the next token; the accuracy of its prediction serves as the basis for learning. Books, web pages, or code provide this material, which is more abundant than the carefully paired text pairs used in translation. At scale, this exercise exposes the model to facts, grammar, writing styles, and code regularities. The initial work on GPT adopted this generative pre-training with a Transformer decoder in 2018.

The already read context and the expected continuation then belong to a single sequence. There is no longer a distinct source sentence to encode: a causal attention stack directly processes the prefix. The successive production of tokens, however, is not a novelty of this organization, as the initial decoder already operated this way.

Introduced by Attention Is All You Need, the Transformer intended for translation appears in Figure 1 with two sets: the encoder represents each source token, while the decoder consults these representations to choose the next part to write. Cross-attention ensures this link and provides access to the text to be translated as well as the translation already produced.

Another attention mechanism focuses on the ongoing output. Its mask prevents a position from consulting subsequent tokens during training; otherwise, the answer would be accessible in advance.

RoPE Encodes Token Positions Differently

The differences do not stop at the removal of components: they also concern the internals of the blocks. The order of words can change meaning, so their position must be represented. In the initial architecture, positional signals are added to the vector representations of the tokens.

With rotary positional embeddings, known as RoPE, the coordinates of queries and keys undergo a rotation related to their location instead. The difference between the rotations involves the relative position in their dot product. This mechanism is useful for long contexts, but their extension still requires appropriate training, scaling methods, and evaluation.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.