Subquadratic and Manifest AI: Innovations to Transform LLMs

A Revolution Initiated by Google

In 2017, a group of artificial intelligence researchers at Google published a landmark paper titled "Attention Is All You Need." This document described a new neural network architecture called transformer, which quickly established itself as a major breakthrough for processing long sequences of data, particularly textual data.

Today, nine years later, transformers have become the backbone of all major language models (LLMs) available on the market. Justin Dangel, co-founder and CEO of the AI startup Subquadratic, emphasizes the importance of this technology by stating, "The AI industry relies on transformers. They represent one of the most significant innovations in the history of computing, having transformed our world."

However, despite their success, transformers are beginning to show certain limitations. Recent advancements in the field of LLMs, such as the development of reasoning models and their ability to simultaneously process large amounts of data, are not mere improvements on this foundational technology. They often represent workarounds aimed at addressing some of its intrinsic flaws.

An increasing number of researchers and engineers are now questioning the future of these models. While LLMs are not on the verge of disappearing, their construction is undergoing significant changes. MIT Technology Review even referred to this next generation of models as LLMs+ in its annual list of the top ten breakthroughs in AI.

The Power and Limits of Transformers

To understand the issue, we must first examine the main strength of transformers: a mechanism called dense attention. This mechanism encodes the meaning of a text into a series of numbers, comparing each word (or token) with all other words in the text through complex multiplications.

Dense attention allows for an impressively precise understanding of the text's meaning. However, as the text length increases, the number of calculations required grows exponentially. For example, processing a 10,000-word document could require 50 million multiplications, which explains why LLMs consume so much energy.

The associated costs are colossal. OpenAI expects to spend $50 billion on computing this year, according to its president Greg Brockman. Additionally, the International Energy Agency anticipates that electricity consumption in data centers will double by 2030.

Moreover, transformers struggle with certain tasks for which recent models are designed. Due to their sequential processing of text, they find it challenging to manage large amounts of information simultaneously (their context window cannot be too extensive). To accomplish more complex tasks, LLMs will need to process even larger volumes of data, such as an entire library of documents or a complete codebase.

Reasoning models, on the other hand, operate by writing notes for themselves (in a sort of notepad called chain of thought) and reviewing them, which further increases the amount of data to manage.

As a result, while LLMs become more capable, transformers are becoming a hindrance. Their initial strength is now seen as a limitation.

01: Rethinking Attention for More Efficient LLMs

An obvious solution to make LLMs faster and less costly is to modify how attention is managed. Replacing dense attention with a sparse attention mechanism, which performs calculations only on certain word associations in a text, could significantly reduce the necessary computations.

Many sparse attention mechanisms have been proposed over the years, but none have matched dense attention in terms of accuracy.

This could change with Subquadratic, a Miami-based startup that claims to have developed the first sparse attention mechanism capable of competing with the best LLMs on the market across various tasks, such as search and programming. Although this claim has raised skepticism, it is promising.

Subquadratic's model, SubQ, works by identifying important words in a text in real-time. The company claims that thousands of people have signed up for its waiting list and plans to make the model widely accessible soon.

Meanwhile, Manifest AI, a San Francisco startup, approaches the problem differently. Instead of modifying attention, it replaces it with a mechanism called power retention, which stores only the essential information for a specific task, thus avoiding data overload.

Attention mechanisms force LLMs to track everything within their context window. A sparse attention model, like SubQ, discards many individual words but retains an overall view. In contrast, power retention provides a continuous summary of the context window, removing less relevant information as new data is added.

The principle of retention has existed for a decade. Manifest AI claims to have modernized these techniques to create models capable of competing with transformer-based LLMs.

The company asserts that it is possible to transform a transformer model into a power retention model with minimal retraining. To demonstrate this, it adapted an open-source coding LLM, StarCoder, into a version using power retention, called PowerCoder. It has also released a model named Brumby, which it claims competes with certain versions of Alibaba's open-source model, Qwen.

Manifest AI hopes that its power retention technology will become the go-to solution for LLMs handling large amounts of data. The potential applications are numerous, ranging from analyzing hours of video to creating agents capable of focusing on a task for weeks, as explained by co-founder and CTO of Manifest AI, Carles Gelada, in a technology presentation video.

02: Reducing Model Size and Increasing Flexibility

Liquid AI, a startup spun out of MIT and based in Cambridge, Massachusetts, has not completely abandoned transformers. It combines them with its own technology, liquid neural networks, to create what co-founder and CEO Ramin Hasani calls LFMs (liquid foundation models).

Liquid AI's models are much smaller and consume less energy than most LLMs. The company is developing models for automotive manufacturers, such as Mercedes, that operate on small chips embedded in vehicles. Its latest models can even run on a Raspberry Pi, a low-power computer costing $50.

These models are available for free to any organization with an annual revenue of less than $10 million. They have seen great success, with nearly 34 million downloads, according to Hasani.

Liquid neural networks are inspired by the brains of worms and are an extension of convolutional networks, another type of neural network that predates transformers. The key innovation lies in a mechanism that allows a model to adapt its behavior to new information, enabling continuous learning. This is not possible with transformers, whose behavior is fixed once the model is trained.

The early models of Liquid AI were quite rudimentary but could pilot drones or drive vehicles. With LFMs, the company aims to elevate its technology to compete with mainstream LLMs. Its new models match the performance of competitors four times their size, including some versions of Alibaba's Qwen and Google's open-source LLM Gemma.

A typical LLM consists of a stack of interconnected transformers. Liquid AI's recent LFMs are hybrid models composed of 20% transformers and 80% liquid neural networks.

This ratio was determined by another AI system developed by Liquid AI, used to design all its models. "This is the core technology of our company right now," says Hasani. This designer AI explores numerous combinations of neural networks—liquid, convolutional, and more, as well as transformers—and proposes optimal designs in terms of performance and efficiency.

Hasani is convinced that transformers are just the beginning: "Your brain is a general artificial intelligence system, and it operates on 20 watts of power. How is that possible? We can be much more innovative."

03: Generating Text All at Once

Most LLMs produce their output one word at a time, which seems logical since that's how humans speak and write. However, for computers, this method is highly inefficient.

It is faster and less costly for LLMs to generate text all at once, producing entire sentences or paragraphs in one go. This is the approach taken by Inception, a startup based in Palo Alto, California, which develops LLMs using a technique called diffusion.

Diffusion is best known as the technology behind most image and video generation models. Diffusion models are trained to transform a grid of random pixels—similar to the noise of an old television—into an image. They achieve this by working on all the pixels simultaneously, determining which ones need to be altered for the noise to resemble a high-definition photo.

It turns out that this process also works for text. Inception has trained its LLMs to transform a string of random words into coherent sentences. Diffusion LLMs still use transformers to encode meaning, but by producing entire blocks of text at once, they make transformers more efficient. "You still use a large transformer model, but you can predict many tokens at the same time," explains Stefano Ermon, co-founder and CEO of Inception. "That's why these models are much faster and more cost-effective compared to what most others are building today."