NVIDIA and Siemens: AI Revolutionizes Ultrasound

Ultrasound is one of the most common medical imaging methods, valued for its safety, real-time capability, portability, and low cost. Traditionally, ultrasound images are created from a manual reconstruction pipeline that compresses the raw measurements from sensors into a final image. This process often relies on simplifying assumptions about physics, such as a constant speed of sound throughout the body.

In the era of artificial intelligence and foundational models, a question arises: is it possible to surpass the traditional beamforming pipeline, learn directly from the raw data of ultrasound sensors, and leverage the information that is typically lost during reconstruction? NVIDIA and Siemens Healthineers have collaborated to address this question, resulting in the creation of a reconstruction model named NV-Raw2Insights-US.

At the heart of ultrasound, it is not an image but sound. Clinicians see on the screen a reconstructed image from millions of tiny echoes returned by the body. However, in this reconstruction process, a significant portion of the original signal, the richness of how sound actually travels through tissues, is simplified or lost.

An Innovative Approach

The approach taken by NVIDIA and Siemens begins earlier in the process. Instead of working from finished images, NV-Raw2Insights-US learns directly from the raw signals captured by the ultrasound probe, the closest representation of how sound actually interacts with the body. This allows the model to "listen" more attentively and understand how each patient uniquely shapes these sound waves. This vision of an end-to-end AI for ultrasound imaging is a first step toward this ambition. This class of models is called Raw2Insights.

In this initial Raw2Insights application, the system estimates the speed of sound for adaptive image focusing. The result is a system capable of generating a customized speed of sound map for each patient and using it to correct the image in real-time. What once required complex and time-consuming calculations is now performed in a single pass of AI. This is the transition from raw data of ultrasound channels to actionable insights: an AI system that not only processes ultrasound images but actively understands and adapts to the physics of each patient.

Cutting-Edge Technology

Typically, raw data from ultrasound channels is not easily accessible on clinical-grade ultrasound scanners due to their high bandwidth. The Holoscan Sensor Bridge (HSB) is an open-source FPGA IP developed by NVIDIA that enables high-bandwidth, low-latency data transfer to the GPU via RDMA over Converged Ethernet. An Altera Agilex-7 FPGA development kit paired with NVIDIA's Holoscan Sensor Bridge allows for streaming raw data from the ultrasound channels from the DisplayPort outputs of an ACUSON Sequoia ultrasound scanner. This technology is called Data over DisplayPort.

The NVIDIA HSB then aggregates the data and transmits it via Ethernet to NVIDIA IGX for data collection and AI inference. This demonstrates how modern high-performance computing capabilities can be integrated into existing scanner architectures using high-bandwidth DisplayPort outputs.

Deployment and Capabilities

We are deploying NV-Raw2Insights-US using NVIDIA Holoscan, an AI sensor processing platform designed for real-time, high-performance workloads on systems such as NVIDIA IGX Thor and NVIDIA DGX Spark.

Once the data is in GPU memory, NV-Raw2Insights-US executes accelerated inference on a Blackwell-class GPU, producing a patient-specific speed of sound estimate. This estimate is sent back to the ultrasound scanner, allowing for improved focusing in the live imaging flow.

This demonstration architecture offers flexibility in both development and deployment:

• Software-only integration: The acceleration of existing medical devices by NVIDIA is possible with software-only modifications using Data over DisplayPort.

• Software-defined ultrasound: This software-defined approach allows for continuous improvement through software updates.

• Modular expansion: With raw data from ultrasound channels already in GPU memory, new AI models can be seamlessly integrated.

Towards Native AI Imaging

By shifting ultrasound intelligence from traditional algorithms to an AI-driven Raw2Insights pipeline, we open a scalable path toward native AI imaging. By learning directly from the raw data of ultrasound channels rather than reconstructed images, NV-Raw2Insights-US reduces errors introduced by traditional assumptions and effectively tailors imaging for each patient.

This architecture not only enhances image clarity today but also establishes a modular foundation for the next generation of AI-powered diagnostic systems.