Rho-1 by Reka AI Unifies Text, Image, Video, and Robots

Reka AI unveils Rho-1, an "omni" model that combines multimodal processing and robotic control within a single neural network. The architecture operates without external tools, relies on 320 H100 GPUs for training, and utilizes an inverse dynamics model to leverage online videos. This announcement follows the launch of Reka Core and is part of the research on "world models." Rho-1 has 19 billion parameters.

Rho-1 in the World Models Initiative

The release of Rho-1 is part of a research initiative in artificial intelligence aimed at developing so-called world models. In April 2024, Reka AI launched Reka Core, a multimodal language model. Reka Core was presented as having competed with GPT-4, Claude 3, and Gemini Ultra during benchmarks.

Unified Architecture and Execution Without External Tools

Rho-1 is capable of processing and producing text, images, video, as well as commands for robots within a single neural network. All modalities are represented as tokens within the same context window, unlike methods that assign each task to distinct models. The model operates without tool calls or reliance on external models. It can produce continuous video in real-time and respond to new instructions without restarting. The same weights that predict camera images are also used to control the movements of robots.

Rare Robot Data, Inverse Dynamics, and H100 Training

To compensate for the lack of dedicated training datasets for robots, Reka AI developed an inverse dynamics model that allows for control signals to be derived from standard videos sourced from the Internet. The training of Rho-1 required the use of 320 H100 GPUs over a period of approximately three months. This is a research preview published by Reka AI for an omni model with 19 billion parameters.