NVIDIA XR AI: Open Source AI Agents on AR Glasses

NVIDIA XR AI: Open Source AI Agents for AR Glasses

NVIDIA has launched in public beta XR AI, an open source library for creating AI agents on AR glasses and XR headsets. The goal is to develop assistants capable of seeing what the operator sees and guiding them in real-time, hands-free.

The News in 3 Points

• NVIDIA has launched XR AI, an open source library for creating AI agents on AR glasses and XR headsets.

• This technology aims to provide contextual assistance in real-time, particularly in environments like factories and operating rooms.

• Companies like Siemens and laboratories at Stanford are already testing XR AI to enhance workflows and scientific research.

Connected glasses, a long-held dream in tech, may finally find their concrete use case. Not in the living room, but in the field: in operating rooms, factories, and research labs focused on cell therapy. NVIDIA has just made available in public beta NVIDIA XR AI, an open source library that allows developers to connect video, audio, and sensor streams from an XR device to AI models, delivering contextual assistance in real-time. Specifically, an agent that sees what you see, understands what you say, queries your enterprise systems, and responds without you having to take your hands off what you are doing.

NVIDIA XR AI: A Modular Architecture Designed for the Field

The library is structured around four blocks. It first ingests raw signals from the XR device: images, audio, depth, and position data. It then connects the agent to third-party tools and services via the MCP protocol, allowing it to query enterprise databases, digital twins, or maintenance management systems. For reasoning, the platform relies on NVIDIA's Nemotron and Cosmos Reason models, with multi-agent orchestration provided by the NeMo Agent Toolkit.

What’s interesting about this architecture is the attention given to latency. Video pixels remain in shared memory; only lightweight metadata circulates through the system, and images are transmitted to the models only when a task truly requires it. For AR glasses worn throughout a full workday, this kind of detail is significant. The infrastructure can run in the cloud, in a datacenter, or at the edge, on DGX Spark, DGX Station, or RTX PRO systems.

Use Cases Already Being Tested at Siemens and Stanford Labs

The initial deployments give an idea of the ambition. Siemens is exploring, in a research context, how a maintenance engineer wearing lightweight glasses could query an agent about a programmable logic controller failure and receive real-time instructions by cross-referencing data from the digital twin and the factory's automation workflows.

On the scientific research side, the startup Rana has integrated XR AI into its LabOS system to guide researchers in cell therapy and CRISPR genomic editing at Cong Lab at Stanford and Wang Lab at Princeton, providing step-by-step assistance without them having to take their eyes off their procedures.

UPMC (University of Pittsburgh Medical Center) has presented an integration for the operating room: the agent is designed not to obstruct the surgeon's field of vision, surfacing only useful information without creating visual distractions. LabOS is compatible with Meta, Rokid, and VITURE glasses. The latter even showcased at AWE 2026 "Helix," its first AI safety glasses built on XR AI. On the hardware front, Qualcomm has concurrently announced the Snapdragon Reality Elite chip, boasting +60% GPU performance, +30% CPU performance, and 20% more battery life compared to the previous generation, to equip the next generation of AR glasses with embedded intelligence.

NVIDIA Builds a Complete Software Stack

NVIDIA is no longer just selling GPUs to train models. The company is methodically building a complete software stack, from sensor to reasoning, to establish itself as the reference infrastructure for agentic AI in the physical world. XR AI fits into the same logic as its Avatar Cloud Engine or its Omniverse ecosystem: providing developers with a reusable foundation rather than a finished product, thereby capturing value at every layer of the chain.

The library is open source and available on GitHub, lowering the barrier to entry for developers, but the path from a convincing prototype to a certified tool for an operating room is long. NVIDIA's investment strategy in the AI ecosystem suggests that the firm is playing on multiple fronts simultaneously, and XR AI is just one more piece of the puzzle.

Agentic AI that perceives the physical world in real-time is no longer just a laboratory demonstration: it is beginning to integrate into real workflows, even if deployments are still experimental. What is at stake here goes beyond the mere announcement of a library: it is a battle to define who will control the software infrastructure for agents in the physical world.