Llama 3.1 Drops the Original Transformer Encoder

Llama 3.1 no longer features a separate encoder or one of the attention blocks from the Transformer introduced in 2017. The adaptations also concern memory, token positions, and the execution of calculations, without abandoning the principles of the Transformer.
Context and Calculation Cost in Memory
In a theoretical example, a model with 32 layers, 32 query heads, and 128 numbers per head would require 4 GiB of cache for a single sequence. This scenario assumes 8,192 tokens of context, two bytes per stored number, and, for each query head, its own key and value heads. Model parameters and other execution memory requirements are additional.
This storage is particularly useful in scenarios where a document is inserted into a conversation and then queried multiple times while retaining previous details. To consult previous tokens, attention leverages keys and values: the KV cache can retain them during generation rather than recalculating them for each new token. Each attention layer has its own cache, the content of which increases as the sequence grows.
Data movement constitutes another expense. A simple realization of the calculation may deposit a large score matrix in GPU memory and then retrieve it for the next step; these transfers remain costly despite fast arithmetic. FlashAttention operates in tiles, progressively retaining the statistics necessary for normalization without fully storing this matrix in high-bandwidth memory.
This method changes execution, not the mathematical nature of the operation: attention remains dense and accurate. However, numerical results may vary with the order of floating-point operations.
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisissez votre rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
Generation Does Without Separate Source Text
To learn to continue a text, a model receives a sequence start and predicts the next token; the accuracy of its prediction serves as the basis for learning. Books, web pages, or code provide this material, which is more abundant than the carefully paired text pairs used in translation. At scale, this exercise exposes the model to facts, grammar, writing styles, and code regularities. The initial work on GPT adopted this generative pre-training with a Transformer decoder in 2018.
The already read context and the expected continuation then belong to a single sequence. There is no longer a distinct source sentence to encode: a causal attention stack directly processes the prefix. The successive production of tokens, however, is not a novelty of this organization, as the initial decoder already operated this way.
Introduced by Attention Is All You Need, the Transformer intended for translation appears in Figure 1 with two sets: the encoder represents each source token, while the decoder consults these representations to choose the next part to write. Cross-attention ensures this link and provides access to the text to be translated as well as the translation already produced.
Another attention mechanism focuses on the ongoing output. Its mask prevents a position from consulting subsequent tokens during training; otherwise, the answer would be accessible in advance.
RoPE Encodes Token Positions Differently
The differences do not stop at the removal of components: they also concern the internals of the blocks. The order of words can change meaning, so their position must be represented. In the initial architecture, positional signals are added to the vector representations of the tokens.
With rotary positional embeddings, known as RoPE, the coordinates of queries and keys undergo a rotation related to their location instead. The difference between the rotations involves the relative position in their dot product. This mechanism is useful for long contexts, but their extension still requires appropriate training, scaling methods, and evaluation.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.