Nvidia and Berkeley: CaP-X Redefines AI Robotics Control

A Collaborative Framework for Evaluating AI in Robotics

Nvidia, in partnership with UC Berkeley, Stanford, and Carnegie Mellon, has developed an innovative framework called CaP-X. This open-access system systematically evaluates the ability of artificial intelligence models to control robots through self-written programs. Test results reveal that without human-designed abstractions, even the most advanced models fail to match the reliability of human programs in a single attempt. None of the twelve leading models tested can achieve this reliability.

Language Techniques Applied to Robotics

Researchers have adopted techniques from language models to apply them to robotics. Among these methods are reinforcement learning with rewards derived from physical simulations, on-the-fly computation scaling, and the use of agentic patterns such as automated debugging. These techniques help bridge the performance gap between AI models and human programs. Models fail at most robotic tasks when operating without high-level abstractions. They must then correctly combine dozens of lines of code where a single function call sufficed before.

Testing Various Models

The team tested models such as Gemini-3-Pro, GPT-5.2, Claude Opus 4.5, as well as several open-source models like Qwen3-235B and DeepSeek-V3.1 on seven robotic manipulation tasks. Performance varies significantly based on the foundational elements accessible to the models. Without pre-built commands, models must correctly combine dozens of lines of code where a single function call sufficed before, complicating their task.

Impact of Raw Images on Performance

The direct introduction of camera images into the models hampers their performance. Researchers suspect a gap in cross-modal alignment—base models are rarely trained to reason simultaneously about software code and the physical execution of the robot. An intermediate Visual Differentiation Module, which translates scenes into text, significantly improves results. This separate vision-language model first describes the scene in text, extracts relevant properties for the task, and after each execution step, reports what has changed in the image and whether the task is complete.

CaP-Agent0: A Promising Zero-Shot System

Based on these findings, researchers developed CaP-Agent0, a zero-shot system that utilizes three main components: a Visual Differentiation Module, an automatically generated function library, and parallel code generation. This system has demonstrated that it can match or surpass human-written code in several tasks. Some ideas from CaP-Agent0 trace back to Voyager, a Minecraft agent from the team around co-author Jim Fan, Nvidia's director of robotics. The model automatically breaks down a high-level task into subtasks, using programming and mathematical tools to execute the button press. Despite its total reliance on low-level elements, CaP-Agent0 matches or exceeds human-written code in four of the seven tasks. A supervising agent then synthesizes the candidates into a final solution.

Comparison with Vision-Language-Action Models

Researchers also evaluated CaP-Agent0 against trained Vision-Language-Action (VLA) models. On the LIBERO-PRO benchmark, which tests tasks with modified object positions and rephrased instructions, CaP-Agent0 achieved performance similar to that of the VLA model from Physical Intelligence, pi0.5. When task descriptions were rephrased, CaP-Agent0 proved significantly more robust, as it interprets instructions directly rather than relying on a specific training distribution.

Improving Models Through Reinforcement Learning

The CaP-X framework also includes CaP-RL, a method that uses reinforcement learning to enhance language models in robot control. A model Qwen2.5-Coder-7B saw its success rate in stacking cubes increase from 4 to 44% in simulation. On a real Franka robot, the same model achieved 76% without further tuning, as it optimizes through abstract programming interfaces rather than from camera images. Thus, the visual gap between simulation and reality is of little importance. The model is trained using reward signals from physical simulations: when the generated code produces a successful robot movement, the model receives positive feedback.

A Hybrid System for Robotic Control

The team proposes a hybrid system where coding agents manage high-level task logic and retrieval, while specialized vision-language-action policies handle detailed motor control. The entire CaP-X framework—including CaP-Gym, CaP-Bench, CaP-Agent0, and CaP-RL—is available as an open-access platform for the research community, thus offering new perspectives for the development of autonomous robotic solutions.