Microsoft adds remote inference to its physical AI toolchain

Key Takeaways

1Microsoft releases a remote inference capability in its open-source Physical AI Toolchain

2The company measures slowdowns of up to 383% in mapping/planning on lightweight GPUs, a 30% decrease in obstacle detection, and a 50% drop in accuracy for VLA models

3Offloading to the edge or cloud improves performance and autonomy, with tools based on Kubernetes and integration with Azure, NVIDIA, simulators, LeRobot, and ROS2

4Examples are provided for SO-101 and UR10e

Why it matters : Offloading inference allows for the use of larger AI models and increases the autonomy of robots, addressing the limitations of embedded GPUs.

Microsoft highlights offloading inference for robotics, as a complement or alternative to embedded GPUs. The company has released a dedicated capability in its Physical AI Toolchain and documented performance and battery life gains measured on mobile manipulation workloads.

Battery Life and Tools: Offloaded Inference Capability Available

Microsoft indicates that larger embedded GPUs, such as the Jetson Thor, could lead to battery consumption increases of up to 160%, even on large robots. The company recently published its Physical AI Toolchain and added a physical AI inference offloading capability for robots. This version includes example projects for offloading inference from a SO-101 and a UR10e. Microsoft states that it has already tested this feature in numerous real-world use cases and is making the source code available, inviting the community to provide feedback.

Measured Performance: Slowdowns with Lightweight GPUs, Gains from Offloading

The evaluation conducted shows that some smaller GPUs could not run the entire mobile manipulation stack. On GPUs with sufficient memory, mapping and planning exhibited slowdowns of up to 383% compared to an A100, which limited capabilities in dynamic environments. Navigation experienced a 30% decrease in rapid obstacle detection with lighter GPUs. VLA models were not massively slowed down, but the degradation was sufficient to reduce their accuracy by 50%. According to Microsoft, embedded GPUs limited performance, while offloading inference to an on-site or cloud GPU improved operations, task success rates, and the use of larger models, while helping robots respond better in real-world environments. Offloading also improved response time, accuracy, battery life, and cost.

Network Trade-offs: Latency, Bandwidth, and Resources to Balance

Offloading inference involves a trade-off between performance, latency, network bandwidth, and the availability of GPU resources. The models were evaluated on configurations ranging from embedded to edge and cloud. A specific comparison studied the effect on battery life by replacing an embedded GPU with a Raspberry Pi 5 and sending all data to the offloaded GPU.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Context: Why Embedded Shows Its Limits

The predominant model confines inference to the robot's GPU, with high-level planning potentially done in the cloud while execution remains local. As the size and sophistication of models increase, the constraints of embedded systems become more pronounced: energy consumption, reduced battery life, costs, weight, and limitations against recent models. Microsoft believes that offloading is essential for operating in open environments with large models and long battery lives, and suggests that replacing embedded computation with lightweight hardware and remote inference can significantly improve robot endurance.

Methodology and Scope: Mobile Manipulation and Three Capabilities

Recent research focuses on mobile robotic manipulation, with canonical tasks such as locating waste in a kitchen, picking it up, and disposing of it. Three key capabilities were evaluated: semantic mapping and planning, navigation, and manipulation. The precise details of the test hardware are described in a separate technical report.

Toolchain: Kubernetes, Simulators, and Azure–NVIDIA Integration

To automate offloading, Microsoft has developed a set of tools that orchestrate inference between edge and cloud, with Kubernetes serving as a common abstraction between embedded, edge, and cloud computing. The tooling offers automatic containerization driven by declarative specifications and container distribution via Kubernetes policies, with integration into robotic simulators, LeRobot, and ROS2. The Physical AI Toolchain is presented as a production-ready open-source framework, integrating Azure services with NVIDIA's physical AI stack. These components aim to meet the requirements of physical AI exposed to open environments, interacting with humans and robots, and accommodating diverse embodiments, complementing hardware advancements and models.