Encord and Zander Labs: Physical AI through Brain Waves

A New Frontier for Physical AI: The Use of Brain Waves

In a warehouse located in San Leandro, California, an innovative project is taking shape. Encord, a company specializing in developing data tools for training AI models, has embarked on an intriguing experiment. Andrew Ceja, one of Encord's robotic trainers, manipulates a Jenga tower while wearing a headset equipped with sensors. These sensors, designed to measure his brain waves, track his mental activity as he performs this task. This technology allows for the collection of valuable data on the cognitive processes involved in physical tasks, thus offering a new dimension to robot training.

This device is not just a futuristic gadget; it represents a potential breakthrough in the collection of training data for robots. Encord, in collaboration with Zander Labs, a German startup specializing in neuroscience, is exploring how brain activity can enrich the datasets used to train robotic models. The idea is to capture mental states such as error, intention, or surprise, in order to create more relevant and effective data. This approach could transform how robots learn to interact with their environment, enabling them to understand and anticipate human actions with greater precision.

Encord's work with Zander Labs is currently in the trial phase. The goal is to create an initial dataset labeled by brain waves. This data will then be integrated into clients' robotic models to assess whether it truly enhances performance. Only after this evaluation will a decision be made regarding the scale to which this technology should be developed. This trial phase is crucial for determining the viability of using brain waves in training physical AIs and could pave the way for broader applications across various industrial sectors.

The Challenge of Scarce Physical Data

Encord and other similar startups face a major challenge: the scarcity of physical training data in the real world. Unlike language models that rely on the abundance of text available online, robots require specific data that is often difficult to obtain. Therefore, Encord has decided not to settle for managing existing data but to create the data that is lacking. This proactive approach aims to bridge the gap between the current capabilities of robots and their future potential by providing the necessary data for more comprehensive and nuanced training.

Lucas Gehrke, a neuroscientist at Zander Labs, explains that brain activity observed during specific tasks can provide valuable insights to model designers. This information helps determine the moments when models need to perform at their best, thereby optimizing robot training. By better understanding human mental processes, robots could be programmed to react more intuitively and effectively, thus enhancing their utility in complex and dynamic environments.

Strategies to Overcome the Data Bottleneck

Vineeth Velmurugan, head of robotic learning at Encord, emphasizes that the real obstacle to advancing physical AI lies in the lack of data. Formerly at OpenAI and Berkshire Grey, Velmurugan joined Encord to strengthen the team dedicated to data creation. The company was initially founded to help annotate and evaluate machine vision models but quickly evolved into producing training data. This evolution reflects a growing awareness of the importance of both the quality and quantity of data for the development of physical AI.

Large language models (LLMs) have been built on a vast amount of text available on the Internet. However, for robots, obtaining comparable data in terms of quality and quantity is a challenge. Velmurugan estimates that a dataset five times larger than YouTube's video corpus would be needed to achieve a similar level of efficiency. This scale highlights the magnitude of the challenge faced by robotics companies and justifies the investment in creating new data sources.

The Importance of Egocentric and Muscular Data

To meet these needs, Encord is turning to two main sources: egocentric videos and data collected by remotely operated robots. Egocentric videos, captured by workers equipped with cameras, are often supplemented by other camera angles and metrics. Encord is also using its San Leandro facility to experiment with new modalities, such as brain waves. This approach allows for the collection of rich and varied data, essential for training robots on complex tasks.

During a demonstration, pilots used leader-follower devices to create data on tasks such as pouring coffee or stacking poker chips. These tasks, while simple, are crucial for training robots to perform precise manipulations. By replicating human movements with high fidelity, robots can be trained to accomplish tasks with increased precision and efficiency, which is essential for their integration into diverse work environments.

Encord is also developing a new data modality using sensors attached to the forearm to detect electrical signals in the muscles. These sensors enable the creation of a 3D representation of hand position, providing a more comprehensive understanding for models. By capturing the subtleties of human movements, this technology could significantly enhance robots' ability to interact with their environment in a more natural and intuitive manner.

Annotation and the Cost of Training Data

Encord's datasets are meticulously annotated with detailed physical descriptions, making them extremely valuable for training models. Velmurugan estimates that these dense annotations are worth a hundred times more than lower-quality egocentric data, even though they cost twenty times more to produce. This cost difference underscores the importance of investing in high-quality data for the development of physical AI.

However, cost remains a major obstacle. While textual data can be extracted for free from the Internet, generating physical data requires considerable resources. This presents an economic challenge for building physical AI models. The necessity to manufacture this data, rather than simply collecting it, fundamentally changes the economics of training robotic models.

Towards a Better Understanding of Physical AI Models

Despite these challenges, Velmurugan is optimistic about the progress being made. Encord, thanks to its unique position in the industry, can observe which data techniques are gaining popularity and efficiency. This perspective allows the company to stay at the forefront of innovation in training data. By being at the crossroads of numerous robotics companies, Encord has a valuable overview that enables it to identify emerging trends and adapt its strategies accordingly.

Encord's pilots, such as Sofia Infante and Andrew Ceja, play a crucial role in this process. Their work involves developing the building blocks of neural networks, a complex yet essential task for the future of physical AI. Ceja, a former waste management employee, appreciates the daily challenge of training robots, a task that, according to him, always offers something new to explore. This diversity and constant innovation are at the heart of Encord's mission, which aims to push the boundaries of what robots can achieve.