Nvidia Revolutionizes Physical AI with Cosmos 3 and Alpamayo 2 Super

Nvidia took advantage of the GTC Taipei event to announce major advancements in the field of physical artificial intelligence. The tech giant introduced several models aimed at transforming the capabilities of robots, autonomous vehicles, and video systems.

Among the new offerings, the global model Cosmos 3 stands out. This is an update to Nvidia's omnimodel, capable of simultaneously processing text, images, video, ambient audio, and action data. This model is designed to generate synthetic training data, interpret complex scenes, and predict future states without requiring a physical recreation of these situations. Applications include detecting traffic anomalies in smart cities, as already done by partner Linker Vision, and generating photorealistic video sequences of rare situations. Cosmos 3 also produces digital motion data for robotic learning, as demonstrated by industrial partner Agile Robots.

Cosmos 3 utilizes a mixed transformer architecture, combining a reasoning transformer and a generation transformer. The available variants include Cosmos 3 Super for optimal quality, Nano for rapid inference, and an Edge model for real-time embedded systems. These models are accessible under the OpenMDW-1.1 license on Hugging Face and GitHub.

In parallel, Nvidia presented Alpamayo 2 Super, a model intended for Level 4 autonomous driving. With its 32 billion parameters, it enhances spatial understanding and management of rare situations. Alpamayo 2 Super introduces meta-actions such as "lane change" or "stop," and provides a chain of causality for each decision, facilitating safety documentation and regulatory review. The model is designed to be an educator, allowing manufacturers to distill smaller models for use on the Drive AGX Thor chip. Nvidia is also releasing AlpaGym, an open-source framework for closed-loop reinforcement learning in simulation, and OmniDreams, a generative model for rare traffic scenarios.

Finally, Nvidia unveiled the Isaac GR00T Reference Humanoid Robot, a reference platform for humanoid robotics research. This robot, approximately two meters tall, uses five-finger tactile hands from Sharpa and is powered by the Jetson AGX Thor T5000. It features 75 degrees of freedom and utilizes the Isaac GR00T stack, which covers teleoperation, simulation in Isaac Sim, foundational models, and the ROS middleware. Although Nvidia does not sell the robot, Unitree plans to offer the hardware by the end of 2026, with ongoing academic collaborations with prestigious institutions such as the Stanford Robotics Center and UC San Diego ARC Lab.