Olmo Hybrid: The Revolution of Hybrid Models Against Transformers

Hybrid Models: An Emerging Trend in LLM Architectures

In the realm of open-weight language models, hybrid architectures are gaining popularity. Recently, several models have been introduced, such as Qwen 3.5, foreshadowed by Qwen3-Next, and Kimi Linear, a more compact version of Kimi K2. Nvidia also unveiled the Nemotron 3 Nano, with larger versions in the works, while IBM presented the Granite 4. These innovations mark a widespread adoption of research on hybrid architectures, and the Muon optimizer may soon follow this trend.

To understand this evolution, we must go back to December 2023, when the Mamba and Striped Hyena models sparked interest. These models challenged the necessity of full attention in language models. Although they quickly faded due to the complexity of their implementation and challenges related to open-source tools, they paved the way for reflections on the limits of large-scale models. At that time, hybrid models had not yet met expectations.

The Innovation of Recurrent Neural Networks in Hybrid Models

Hybrid models stand out by integrating Recurrent Neural Network (RNN) modules with traditional attention, which made the transformer architecture famous. RNN layers allow for compressing part of the computation into a hidden state, used to predict the next token. This approach, inspired by LSTMs, reduces the quadratic computational cost of attention and can solve new problems.

The mentioned models, such as Qwen and Kimi, leverage Gated DeltaNet (GDN), while others, like Granite and Nemotron, utilize Mamba layers. The recently launched Olmo Hybrid model relies on GDN, based on extensive experimentation. This choice is based on GDN's ability to learn features that neither attention nor Mamba layers can capture.

Olmo Hybrid: A Promising Model and Its Theoretical Advances

The Olmo Hybrid model consists of 7 billion parameters and offers 3 post-training checkpoints. The first is an Instruct model, with a reasoning model in development. This model is a valuable tool for exploring hybrid models, being almost identical to the previous year's Olmo 3 7B, except for its modified architecture.

An accompanying paper details why hybrid models may outperform standard transformers. This document, although complex, is a valuable resource for understanding the advantages of hybrid models. It is a research project led by Will Merrill, who has done remarkable work.

The Impact of Hybrid Models on Efficiency and Expressiveness

Previous theoretical research has demonstrated that attention and recurrence possess complementary strengths. Combining them allows for the creation of an architecture that benefits from the advantages of both approaches. New theoretical results show that hybrid models are more powerful than the sum of their parts, capable of solving problems that neither transformers nor GDN can tackle alone.

However, this increased expressiveness does not automatically guarantee better performance for hybrid models. Rigorous studies have compared these models to transformers, demonstrating that the expressiveness of hybrid models translates into better token efficiency, in line with the observations from the pre-training of Olmo Hybrid.

Towards Wider Adoption of Hybrid Models

The results suggest that hybrid models outperform transformers, both theoretically and empirically. They offer a balance between expressiveness and parallelism, excelling on benchmarks and in long contexts. These findings could promote wider adoption of hybrid models, encouraging the research community to delve deeper into exploring this architecture.

In summary, hybrid models are distinguished by their expressiveness, their ability to learn a variety of functions, and their efficiency in deep learning. This flexibility, compared to the Bitter Lesson, allows the optimizer to work without excessive constraints. More expressive models, following optimized scaling laws, could well redefine the performance standards of language models.