XDOF: The Bold Bet to Revolutionize Robot Training

A New Challenge for the AI Industry

Recently, OpenAI announced the relaunch of its robotics program, which was shut down in 2021, marking a turning point for artificial intelligence laboratories seeking to master the physical world. Training robots requires specific data, quite different from that used for language models. This data, essential for developing robotic capabilities, is currently scarce and difficult to obtain.

Unlike language models that rely on widely available texts, robots need data that reflects real physical interactions. Online videos and recordings of workers are insufficient, as they lack the precision and relevance for robotic applications.

XDOF: A Response to the Data Shortage

XDOF, a recently spotlighted startup, bets that the next major obstacle for AI lies not in models or hardware technologies, but in the collection of the necessary data to teach robots how to interact with their environment. The company is committed to building data pipelines, collection tools, and annotation systems that robotics labs and companies cannot develop on their own.

With $70 million in funding from Thrive Capital, Spark Capital, a16z, Lux, and WndrCo, XDOF is well-positioned to tackle this challenge. Philipp Wu, co-founder and CEO, reveals that the company, which employs around 60 people, is already collaborating with about twenty clients, including several renowned AI labs, although their names remain confidential.

A Race Against Time for Robotics

Wu emphasizes the urgency for labs to dive into robotics to avoid falling behind in this new frontier of AI. As a PhD student at UC Berkeley, Wu himself faced the lack of large-scale data for training robots, a major obstacle to developing effective robotic models.

Together with Fred Shentu, future CTO of XDOF, Wu worked on GELLO, an economical teleoperation system that allows a human operator to control a robotic arm to generate training data. This project has had a significant impact in the field of robotics, inspiring many researchers to adopt similar devices for data collection.

A Data Ecosystem for Robotics

In October 2024, Wu, Shentu, and Nemo Jin founded XDOF to create a data ecosystem for companies developing robotic models. Aware that simply providing data might not be enough, they also focus on data cleaning, tools, and annotation, thereby creating a beneficial feedback loop for training robots.

The company has partnered with the AI research lab at UC Berkeley to publish ABC, the largest collection of high-quality robot training data ever assembled. This collection includes 130,000 robot manipulation trajectories, 300 hours of simulation, and 100 hours of evaluations, offering researchers unprecedented access to large-scale pre-training data.

The Impact of Data on Robotics Research

David McAllister, a PhD student at Berkeley, highlights the importance of this publication for the scientific community. The published data and models enable unexpected advancements, as seen in other areas of AI such as language and image generation.

The collected data has already been used to train robots to perform specific tasks, such as folding T-shirts, flattening boxes, or loading AirPods into their cases, thus demonstrating their potential to enhance robotic capabilities.

A Three-Step Strategy for Data Collection

XDOF plans to adopt a three-tiered approach to data collection. The most valuable level involves teleoperation data collected from real robots in deployment. Next come teleoperated robots collecting more general data, similar to GELLO. Finally, "egocentric" data will be gathered by humans performing everyday tasks, for which XDOF plans to develop its own wearable sensors.

Wu stresses the importance of selecting the right hardware from the outset, as the quality of the collected data heavily depends on the equipment used. Poor initial design can lead to unexpected issues in the data, thereby affecting the performance of training algorithms.

A Labor-Intensive Business Model

To achieve its goals, XDOF plans to recruit and train numerous teleoperators and egocentric data operators around the world. This model, while labor-intensive, addresses a crucial question: why don't large labs produce this data themselves?

Wu explains that large-scale data production requires considerable infrastructure, such as warehouses spanning hundreds of thousands of square feet and hundreds of robots. Maintaining these robots, calibrating them, and training operators demand a concentration, capital, and operational scale that few AI labs can afford, thus justifying the outsourcing of these tasks to companies like XDOF.

The Ambition Behind the Name XDOF

The name XDOF is a play on the term "degrees of freedom" in robotics, which describes the number of independent movements a robot can make. For example, a human arm, from shoulder to wrist, has seven degrees of freedom, while the latest robot from Figure AI has 30. The "X" in XDOF symbolizes the company's ambition: to offer unlimited degrees of freedom, thereby pushing the boundaries of what robots can achieve.