Amazon Trainium: the chip that challenges Nvidia and attracts OpenAI

⚡

Key Takeaways

1Amazon has invested $50 billion with OpenAI, strengthening its partnership with the Trainium chip.

2AWS provides 2 gigawatts of Trainium capacity to OpenAI, despite strong demand from Anthropic and Bedrock.

3Amazon's Trainium3 chips offer performance comparable to Nvidia GPUs at a 50% reduced cost.

💡Why it matters — Amazon is positioning itself as a key player in the AI chip market, threatening Nvidia's near-monopoly.

Exclusive Visit to Amazon's Trainium Lab

Shortly after Andy Jassy, CEO of Amazon, announced a colossal $50 billion investment from AWS with OpenAI, Amazon opened the doors to its chip development lab for a private tour. This lab is at the heart of this ambitious agreement, primarily funded by Amazon itself.

Industry experts are closely monitoring Amazon's Trainium chip, developed in this lab, for its potential implications on low-cost AI inference. This chip could also help reduce Nvidia's near-monopoly in the market.

My guides for this visit were Kristopher King, lab director, Mark Carroll, engineering director, and Doron Aronson, head of public relations for the team that organized this tour.

AWS has been the primary cloud platform for Anthropic since the inception of the AI lab, a relationship that has endured despite the addition of Microsoft as a cloud partner and Amazon's growing collaboration with OpenAI.

The agreement with OpenAI positions AWS as the exclusive provider of OpenAI's new agent builder, Frontier. This project could become a major component of OpenAI's operations if agents reach the scale that Silicon Valley anticipates. However, the Financial Times reported that Microsoft may view this agreement with Amazon as a violation of its own agreement with OpenAI, which stipulates that Microsoft has access to all OpenAI models and technologies.

Why AWS Attracts OpenAI

As part of this agreement, Amazon has committed to providing OpenAI with 2 gigawatts of Trainium computing capacity. This commitment is all the more impressive given that Anthropic and Amazon's Bedrock service are already consuming Trainium chips at a rate exceeding Amazon's production capacity.

Currently, 1.4 million Trainium chips are deployed across three generations, and Anthropic's Claude model operates on over a million Trainium2 chips, according to the company.

While Trainium was initially designed to accelerate and reduce the cost of training models, it is now optimized for inference. Inference, which is the process by which an AI model generates responses, is currently the main performance bottleneck in the industry.

Trainium vs. Nvidia

In addition to offering an alternative to Nvidia's GPUs, which are often out of stock and difficult to acquire, Amazon claims that its new chips, operating on the Trn3 UltraServers, cost up to 50% less to run for comparable performance compared to using traditional cloud servers.

With the launch of Trainium3 in December, the AWS team has also developed new Neuron switches. Mark Carroll asserts that this combination is transformative.

The switches allow each Trainium3 chip to communicate with all other chips in a mesh configuration, thereby reducing latency. "That's why Trainium3 breaks all records," particularly in terms of "price per power," he stated.

Amazon's Ambitions Beyond Chips

Amazon's ambitions extend beyond the chips themselves. The company is also designing the server that houses these chips. In addition to the networking components, this team has designed Nitro, a hardware-software combination that provides virtualization technology, allowing multiple software instances to run separately on the same server. They have also developed cutting-edge liquid cooling technology and the server chassis that house this equipment.

All of this aims to control costs and improve performance.

AWS Chip Lab in Austin

Amazon's custom chip design unit was established when the cloud giant acquired the Israeli chip designer Annapurna Labs in January 2015 for approximately $350 million. This team now has over 10 years of experience in chip design for AWS.

The lab is located in a modern, chrome-windowed building in the upscale Domain neighborhood of Austin, a pedestrian area filled with shops and restaurants, sometimes referred to as Austin's Silicon Valley.

The lab, equivalent in size to two large conference rooms, is a noisy industrial space due to the fans on the equipment. It resembles a mix between a high school technology class and a Hollywood set for a high-end lab, except the engineers are dressed in jeans rather than lab coats.

The Magic of "Bring-Up"

"Bring-up" is the moment when the chip is powered on for the first time to verify that it works as intended. The team even filmed part of the bring-up of Trainium3 and posted it on YouTube.

For Trainium3, the prototype chip was initially air-cooled, like previous versions. The current chip is now liquid-cooled, which offers energy advantages and has been a true engineering feat.

The lab also contains custom and commercial tools to test and analyze chip-related issues.

The "sleds" are the trays that house the Trainium AI chips, Graviton CPU chips, as well as support cards and components. By stacking them on a rack with the networking component, also custom-designed by this team, you get the systems that are at the core of Anthropic's Claude.