OpenWorldLib Redefines World Models, Sora Excluded

A New Definition of World Models

An international research team, including members from Peking University, Kuaishou Technology, the National University of Singapore, Tsinghua University, and other institutions, has introduced an innovative framework to clarify what constitutes a "world model" in artificial intelligence. Kuaishou Technology, known for its Kling video generator, is participating in this initiative. The term "world model," often used but rarely defined consistently, is now specified: a world model must perceive its environment, interact with it, and retain memory. This definition excludes text-to-video generators like Sora, which lack these essential characteristics.

Launch of OpenWorldLib

To support this new definition, the team has launched OpenWorldLib, an open-source project that encompasses five modules. These modules cover various aspects, such as input processing, synthesis, reasoning, 3D reconstruction, and memory. This project aims to unify and evaluate world models within a coherent framework.

OpenWorldLib integrates an operator module that converts all types of inputs—text, images, sensor data—into a standardized format. The synthesis module generates images, videos, audio, and control commands. The reasoning module manages spatial, visual, and acoustic context. A representation module constructs 3D reconstructions and simulation environments. Finally, the memory module stores interaction sequences to ensure system coherence across multiple stages.

A high-level pipeline orchestrates all the modules and exposes a standardized interface, allowing researchers to compare different models and methods within the same framework, thus avoiding the need to create custom infrastructure each time.

Exclusion of Text-to-Video Generators

The researchers' article emphasizes that text-to-video generators, such as Sora from OpenAI, do not meet the criteria for a world model. Although these models can simulate physical relationships, they lack the feedback loop with the real world, which is essential according to the researchers. Industry figures, like Demis Hassabis from DeepMind, had positioned these technologies as world simulators. Google's video model Veo was mentioned by Hassabis as a step towards world models.

The researchers share the viewpoint of Yann LeCun, who asserts that text-to-video generation lacks the crucial feedback loop with the real world. They also exclude code generation, web search, and avatar video generation from the definition of a world model. Avatar videos, for example, are entertainment-focused and have little to do with understanding the physical world.

Requirements for World Models

The researchers highlight three key areas for world models: interactive video generation, multimodal reasoning, and vision-language-action. These tasks require active interaction with the environment, in contrast to the passive generation of media. 3D reconstruction and simulators are also considered crucial for testing physical rules in a controlled environment.

In interactive video generation, a model predicts the next frame based on previous frames and user inputs, responding to actions like control commands or camera movements. Multimodal reasoning encompasses the ability to understand spatial, temporal, and causal relationships from images, videos, and audio. In vision-language-action, the model converts visual inputs and voice instructions into specific movement commands for robotic arms or autonomous vehicles.

Hardware Challenges

The authors criticize the limitations of current chips, which process data inefficiently for world models. Modern processors are designed to handle individual tokens, so even when a model needs to predict entire video frames, the data is still processed token by token internally. They call for new chip architectures, moving away from the Transformer model, which currently powers almost all large AI models, to meet the specific needs of these models. Vision-language models like Bagel, using the Qwen architecture, show potential to bridge some of these gaps.

In tests on Nvidia's A800 and H200 GPUs, Hunyuan-WorldPlay demonstrated superior visual quality in interactive video generation. Nvidia's Cosmos excelled in complex interactive scenarios, while older models like Matrix-Game-2 were faster but showed notable color lag in longer sequences.

Open-Source Availability

OpenWorldLib is available as an open-source project on GitHub, providing researchers with a platform to compare and develop world models within a standardized framework.