Open-H-Embodiment: Revolutionizing Surgical Robotics with AI

Open-H-Embodiment: A Collaborative Initiative for AI in Healthcare

The Open-H-Embodiment initiative stands out for its ambition to create an open and shared database for AI in surgical robotics and ultrasound. This ambitious project is led by prominent figures such as Prof. Axel Krieger from Johns Hopkins, Prof. Nassir Navab from the Technical University of Munich, and Dr. Mahdi Azizian from NVIDIA. Currently, it brings together 35 international organizations.

The goal is to establish the first large-scale dataset dedicated to physical AI in healthcare robotics. This dataset includes 778 hours of training data covering surgical robotics, ultrasound autonomy, and colonoscopy. It encompasses simulations, bench exercises such as suturing, as well as real clinical procedures. The data is collected using commercial robots like those from CMR Surgical, Rob Surgical, Tuodao, as well as research robots such as dVRK, Franka, and Kuka. Two new open-source models have been released after being trained on this data.

GR00T-H: A Visual Language Model for Surgery

The GR00T-H model represents a major development in the field of surgical robotics. Trained on approximately 600 hours of data from Open-H-Embodiment, it is the first policy model for surgical tasks. GR00T-H is a derivative of the Vision-Language-Action (VLA) model series Isaac GR00T N. Based on NVIDIA's open-source ecosystem, it utilizes the visual language model Cosmos Reason 2 2B.

Technical Innovations of GR00T-H

• Unique embodiment projectors: A unique MLP adapts the kinematics of each robot to a common action space.
• State dropout (100%): Proprioceptive inputs are ignored during inference, thereby improving real-world outcomes.
• Relative EEF actions: A common end-effector action space is used to overcome inconsistencies.
• Task prompt metadata: Instrument names and control indices are directly integrated into task prompts.

GR00T-H has demonstrated its ability to perform complete suturing in the SutureBot benchmark, proving its dexterity over extended periods.

Cosmos-H-Surgical-Simulator: An Advanced Simulator for Surgery

The Cosmos-H-Surgical-Simulator is a foundational global model designed for surgical robotics. It overcomes the limitations of traditional simulators by generating plausible surgical videos from kinematic actions. Traditional simulators often fail due to the complexities of the real world, such as soft tissues, reflections, blood, and smoke.

Key Features of the Simulator

• Bridging the Sim-to-Real gap: Utilizing NVIDIA Cosmos Predict 2.5 2B, it creates realistic surgical videos.
• Efficiency gains: For 600 deployments, it only took 40 minutes in simulation, compared to 2 days with traditional methods.
• WFM as a physical simulator: Learns tissue deformation and tool interaction from the data.
• Synthetic data generation: Produces synthetic video-action pairs to enrich datasets.

The model has been refined on the Open-H-Embodiment dataset, using 9 robot embodiments and 32 datasets. This process mobilized 64 A100 GPUs over 10,000 GPU hours, leveraging a 44-dimensional action space.

Towards Reasoned Autonomy in Surgical Robotics

The Open-H-Embodiment initiative aims to evolve towards reasoning-capable autonomy, akin to a ChatGPT moment for surgical robotics. The goal of version 2 is to go beyond mere perceptual control to achieve reasoned autonomy, where systems can explain, plan, and adapt during lengthy procedures. This involves expanding the dataset with reasoning-ready data, including annotated task traces. Community engagement is crucial for this objective, and contributions are encouraged via the Open-H Github Repo.

Resources and Access

For those interested in exploring these advancements, several resources are available:

• Open-H-Embodiment: HF Dataset / Github Repo
• NVIDIA Isaac GR00T-H Model: HF Model / GR00T-H Github Repo
• NVIDIA Cosmos-H-Surgical-Simulator: HF Model / Github Repo
• Cosmos Cookbook: Step-by-step guide to building your own WFM
• Hugging Face: Discover new open Cosmos models and datasets.