Google DeepMind: Video Generators and Computer Vision

Google Deepmind: Video Generators and Computer Vision

Researchers at Google Deepmind have developed GenCeption, a model that reuses a pre-trained video generator for classic computer vision tasks such as depth estimation and segmentation.

The system relies on an open-source video model from Alibaba and provides results in a single forward pass, guided by textual prompts. It has been trained on a small set of synthetic videos and requires significantly less data than competing approaches.

In benchmarks, GenCeption matches established specialized models and transfers its capabilities to real sequences and untrained categories like animals. The authors claim this supports the contested idea that video generators can serve as a foundation for universal models of the world in computer vision.

An Innovative Model for State-of-the-Art Performance

A new model from Google Deepmind, called GenCeption, uses a pre-trained video generation model as a base for classic computer vision tasks. It achieves state-of-the-art performance in depth estimation, segmentation, and 3D pose estimation while requiring very little training data.

Language models have become versatile processing systems almost as a byproduct of learning to predict the next word. This task seems to require models to absorb grammar, world knowledge, and contextual relationships during training. This is the dominant explanation for the emerging capabilities of large language models.

Computer vision still lacks an equivalent training method. Specialized models dominate the field, including "Segment Anything" for segmentation and "Depth Anything" for depth estimation. Each uses its own architecture.

GenCeption generates depth maps, surface normals, segmentation masks, and pose estimates from the same video based on a textual prompt. Despite being primarily trained on synthetic data, it operates on real sequences and object classes it has never encountered during training.

A Novel Approach to Perception

GenCeption builds on Alibaba's open-source video model Wan2.1. The main change is a simplified architecture. Diffusion models typically generate videos from noise through many small steps. GenCeption instead produces a prediction in a single forward pass, making it fast enough for practical computer vision tasks.

The researchers use a straightforward method to enable a model to handle multiple tasks. GenCeption represents each output as a standard three-channel RGB image, whether the result is a depth map, a surface normal map, or a segmentation mask. It also converts camera movement into an image representation.

A textual prompt tells the model which task to perform, similar to an instruction given to a chatbot. The team adds trainable modules for tasks like 3D keypoint prediction, which do not produce images.

Training uses a single loss function for all tasks. Rather than modifying the architecture for each task, the researchers process the training data to account for task-specific requirements.

Impressive Results with Minimal Data

Most of the training data comes from a synthetic dataset containing only 7,500 videos. The team combined 800 digital human models with 200 motion sequences from a motion capture dataset. They then rendered the results in Blender with different backgrounds and camera angles. Real videos were used only for language-guided segmentation.

GenCeption approaches state-of-the-art results in many tasks while using between one-seventh and one-five-hundredth of the training data compared to established models.

According to the study, GenCeption matches or exceeds state-of-the-art results in many benchmarks despite using a single architecture for each task. Its depth estimates align with those of specialized models like DepthAnything 3. GenCeption outperforms NormalCrafter and Lotus-2 in surface normal estimation. It also surpasses Genmo and TRAM in 3D pose recognition. In complex language-guided segmentation, it matches Meta's SAM 3 combined with Gemini 3.5 Flash.

Models like D4RT and VGGT Omega have been trained on millions of videos. GenCeption achieves similar results using 7 to 500 times less data. When tested under the same conditions, pre-training on video generation also outperforms methods like V-JEPA and VideoMAE V2. The authors attribute these results to the generation task rather than data volume alone. They argue that video generation helps models learn useful representations of space and motion.

Strong Generalization but Clear Limitations

GenCeption has been trained almost exclusively on synthetic videos showing a single person at a time, yet it still functions on real videos with multiple people in the frame, as well as on unrelated categories like animals and humanoid robots. According to the study, some outputs contain more detail than the Blender renders used for training. The results can preserve a cat's whiskers and the contours of individual hair strands.

GenCeption has been trained solely on synthetic videos of people but transfers its perception capabilities to animal categories it has never encountered during training.

Joint training on all tasks hampers 3D keypoint estimation. The researchers suspect that the additional components required for this task interfere with the mechanisms learned by the base model during pre-training.

Their conclusion is that as few changes as possible should be made to the original model architecture. Processing speed also needs improvement. The smaller model takes about six seconds to process a video of 81 frames, while the larger model, which has 14 billion parameters, takes about ten seconds.

Video Generators as World Models

The authors reject the notion that video generators are merely entertainment tools. They argue that these models already contain a sort of universal "world model" that could support a foundational model for computer vision. Such a system could fulfill the same role in image processing as large language models do in text.

The validity of this framework is debated. An international research team recently proposed a uniform definition with OpenWorldLib and explicitly excluded text-to-video models because they lack real-world feedback.

Former Chief AI Scientist at Meta, Yann LeCun, went further, stating that generative video models are a dead end. V-JEPA 2 from Meta follows the alternative he has advocated, predicting abstract concepts rather than pixels. A benchmark from Tsinghua University highlights the limitations of pixel prediction. Sora 2, Seedance 2.0, and Veo 3.1 have repeatedly failed basic physics and logic tests, even when their output appeared convincing.

GenCeption uses video generation for narrower purposes. The researchers do not ask Wan2.1 to predict how the world will behave. They use it to extract features for specific tasks like depth estimation and segmentation, where these learned features can outperform specialized systems. Google Deepmind is also exploring a different approach with Genie 3, which builds interactive 3D environments to train AI agents.