Bytedance: Helios Revolutionizes Real-Time AI Video

Helios: A Cutting-Edge AI Video Model

Bytedance recently unveiled Helios, an AI video generation model that stands out for its ability to produce one-minute videos at an impressive rate of 19.5 frames per second (FPS) on a single GPU. This model, based on a 14 billion parameter architecture, is not only high-performing but also has its code and weights publicly available, paving the way for widespread adoption and future innovations.

Current video generation models often struggle to exceed clips of 5 to 10 seconds, requiring several minutes for rendering. Real-time solutions for longer videos typically rely on smaller models, around 1.3 billion parameters, but these struggle to maintain acceptable quality. For example, more advanced models like Krea-RealTime-14B cap out at 6.7 FPS on an H100 GPU and suffer from drift artifact issues.

An Optimized Architecture for Performance

Helios is based on the Wan-2.1-14B model, which takes about 50 minutes to generate five seconds of video on an A100 GPU. The training process for Helios occurs in three distinct phases: Helios-Base, which focuses on architecture and drift reduction; Helios-Mid, which introduces token compression and achieves 1.05 FPS; and finally Helios-Distilled, which maximizes speed by simplifying the computation to just three steps.

In tests conducted by the developers, Helios-Distilled achieved a speed of 19.53 FPS, even surpassing some much smaller distilled models. In comparison, SANA Video Long, which has 2 billion parameters and is about seven times smaller, only reaches 13.24 FPS.

At 19.53 FPS on an H100 GPU, Helios-Distilled competes with the speed of much smaller models with 1.3 billion parameters, while other 14 billion parameter models struggle to exceed 1 FPS.

Video Quality and User Evaluation

In terms of video quality, Helios scores 6.00 for short videos containing 81 frames. The model's authors claim it outperforms all distilled models and compares favorably to most base models of this size. For long videos, Helios achieved a score of 6.94, surpassing the previous leader, Reward Forcing, which had a score of 6.88. A user study involving 200 participants confirmed these results.

Helios-Base ranks highest for the quality of long videos and performs well against significantly larger base models for short video generation.

Addressing Drift Issues

Longer generated videos are often subject to quality loss, color consistency, and content drift over time. Previous models attempted to solve this issue with complex techniques like self-forcing, where the model reintroduces its own output as input during training to reduce the gap between training and inference. Helios, on the other hand, takes a different approach.

The authors identified three typical drift patterns and proposed simpler solutions:

• Relative Position Encoding: This technique prevents the model from reaching unknown position indices in long videos, thus avoiding repetitive movements.
• First Frame Anchor: It permanently retains the first frame in memory, providing the model with a visual reference point to avoid color drift.
• Targeted Disturbance Simulation: This method makes the model more resilient to its own errors, which would otherwise accumulate over time.

A Unified Architecture for Multiple Tasks

Helios employs a unified architecture that allows for text-to-video, image-to-video, and video-to-video tasks within a single framework. The model automatically switches between tasks based on the preceding context.

• If the context is empty, the model generates from text.
• If only the last image is present, it operates as an image animator.
• If multiple images are available, it continues an existing video.

Users also have the option to change the text of the prompt during generation; a gradual fade between the old and new prompt helps avoid abrupt visual breaks.

Helios's architecture compresses historical context across three temporal scales and automatically switches between text, image, and video inputs.

The model was trained in three stages on 800,000 short video clips, each under ten seconds. The current resolution reaches 384 x 640 pixels, and flickering artifacts still appear during segment transitions. As no open benchmark exists for real-time long videos, the researchers built their own test dataset called HeliosBench with 240 prompts.

Aggressive Compression to Reduce Costs

Helios achieves its speed goals without resorting to common acceleration tricks like KV cache, sparse attention, or quantization. Instead, the model aggressively compresses input data at two levels.

• A hierarchical memory structure divides the video history into three temporal scales. Recent images benefit from lighter compression, while older images are compressed more heavily. This reduces the number of tokens to process by a factor of eight.

• A multi-step sampling process reduces the number of tokens for the segment of video currently being generated by a factor of 2.29. The initial steps operate at a lower resolution, and only later steps fill in the fine details. Together, these two techniques reduce computational costs to roughly the same level as generating a single image.

As the length of the context increases, the naive approach sees a linear increase in the number of tokens, memory usage, and inference time, while Helios remains nearly constant. The standard approach runs out of memory at a context length of 6.