An Exclusive Tour Of Amazon’s Trainium Lab, The Chip That’s Won Over Anthropic, Openai, Even Apple 

Shortly aft Amazon CEO Andy Jassy announced AWS’s groundbreaking $50 cardinal finance deal pinch OpenAI, Amazon invited maine connected a backstage circuit of nan spot improvement laboratory astatine nan bosom of nan deal, astatine (mostly*) its ain expense. 

Industry experts are watching Amazon’s Trainium chip, created astatine that facility, for its implications for lower-cost AI conclusion and, potentially, a dent successful Nvidia’s adjacent monopoly.  

Curious, I agreed to go.  

My circuit guides for nan time were nan lab’s director, Kristopher King (pictured beneath right) and head of engineering Mark Carroll (below left), arsenic good arsenic nan team’s PR personification who arranged nan visit, Doron Aronson (pictured pinch yours genuinely later successful nan story). 

AWS Chip laboratory leaders Mark Carroll and Kristopher King.Image Credits:TechCrunch/Julie Bort

AWS has been Anthropic’s awesome unreality level since nan AI lab’s early days — a narration important capable to past Anthropic later adding Microsoft arsenic a unreality partner arsenic well, and Amazon’s increasing business pinch OpenAI.

The OpenAI woody makes AWS nan exclusive supplier of nan exemplary maker’s caller AI supplier builder, Frontier, which could go an important portion of OpenAI’s business if agents go arsenic large arsenic Silicon Valley thinks they will. We’ll spot if that exclusivity stands precisely arsenic announced. The Financial Times reported this week that Microsoft whitethorn judge OpenAI’s woody pinch Amazon violates its ain woody pinch OpenAI, namely pinch Redmond getting entree to each of OpenAI’s models and tech.

What makes AWS truthful appealing to OpenAI? As portion of this deal, nan unreality elephantine has agreed to proviso OpenAI pinch 2 gigawatts of Trainium computing capacity. This is simply a elephantine commitment, fixed that Anthropic and Amazon’s ain Bedrock work are already consuming Trainium chips faster than Amazon tin nutrient them. 

There are 1.4 cardinal Trainium chips deployed crossed each 3 generations, and Anthropic’s Claude runs connected complete 1 cardinal of nan Trainium2 chips deployed, nan institution said.

It’s worthy noting that while Trainium was primitively geared toward faster, cheaper exemplary training (a bigger privilege a mates of years ago), it’s now tuned and utilized for conclusion arsenic well. Inference — nan process of really moving an AI exemplary to make responses — is presently nan biggest capacity bottleneck successful nan industry. 

Case successful point: Trainium2 handles nan mostly of nan conclusion postulation connected Amazon’s Bedrock service, which supports nan building of AI applications by Amazon’s galore endeavor customers and allows nan apps to usage aggregate models.

“Our customer guidelines is conscionable expanding arsenic accelerated arsenic we tin get capacity retired there,” King said. “Bedrock could beryllium arsenic large arsenic EC2 1 day,” he added, referring to AWS’s behemoth compute unreality service. 

Amazon’s Trainium3 chip.Image Credits:Amazon

Trainium vs. Nvidia

Beyond offering an replacement to Nvidia’s backlogged, hard-to-acquire GPUs, Amazon says its caller chips moving connected its caller specialty Trn3 UltraServers costs up to 50% little to tally for comparable capacity than utilizing classical unreality servers. 

Along pinch Trainium3, released successful December, this AWS squad besides built caller Neuron switches, and Carroll says that combo is transformative.

“What that gives america is thing huge,” Carroll said. The switches let each Trainium3 spot to talk to each different spot successful a mesh configuration, reducing latency. “That’s why Trainium3 is breaking each kinds of records,” peculiarly successful “price per power,” he said. 

When trillions of tokens a time are involved, specified improvements adhd up.  

In fact, Amazon’s spot squad was lauded by Apple successful 2024. In a uncommon infinitesimal of openness for nan secretive company, Apple’s head of AI publically described really it utilized different of nan team’s chips — Graviton, a low-power, ARM-based server CPU and nan first breakout spot this squad designed. Apple besides lauded Inferentia — a spot specifically designed for conclusion — and gave a motion to Trainium, which was caller astatine nan time. 

These chips correspond nan classical Amazon playbook: See what group want to buy, past build an in-house replacement that competes connected price. 

The drawback for chips, historically, has been switching costs. Applications written for Nvidia’s chips must beryllium re-architected to activity pinch others — a time-consuming process that discourages developers from switching.

But nan AWS spot squad proudly told maine that Trainium now supports PyTorch, a celebrated unfastened root model for building AI models. That includes galore of nan ones hosted connected Hugging Face, a immense room wherever developers stock unfastened root models.

The transition, Carroll told me, requires “basically a one-line change, and past recompile, and past tally connected Trainium.” In different words, Amazon is attempting to spot distant astatine Nvidia’s marketplace power wherever possible.

AWS has besides this period announced a partnership pinch Cerebras Systems, integrating that company’s conclusion spot connected servers moving Trainium for what Amazon promises will beryllium superpowered, low-latency AI performance. 

But Amazon’s ambitions spell beyond nan chips themselves. It besides designs nan server that hosts nan chips. Besides nan networking components, this squad has designed “Nitro,” a hardware-software combo that provides virtualization tech (which allows galore instances of package to tally separately connected nan aforesaid server); caller state-of-the-art liquid cooling technology; and nan server sleds (pictured below) that big this gear. 

All of that is to power costs and performance. 

AWS Austin spot laboratory tour, sled pinch components.Image Credits:TechCrunch/Julie Bort

Working 24/7 connected nan “bring-up” 

Amazon’s civilization chip-designing portion was calved erstwhile nan unreality elephantine bought Israeli spot designer Annapurna Labs successful January 2015 for astir $350 million. So this squad has now had much than 10 years designing chips for AWS. The portion has retained its Annapurna roots and sanction — its logo is everyplace successful nan office. 

This spot laboratory is located successful a shiny, chrome-windowed building successful Austin’s upscale “The Domain” district, a walkable area filled pinch shops and restaurants that’s sometimes called Austin’s Silicon Valley. 

The offices person your classical tech firm vibe: desks successful cubicles, gathering spots, and convention rooms. But tucked distant astatine nan backmost of a precocious level successful nan building is nan existent lab, pinch sweeping views of nan city.  

The shelving-filled lab, astir nan size of 2 ample convention rooms, is simply a noisy business abstraction acknowledgment to nan fans connected nan equipment. It looks for illustration a transverse betwixt a precocious schoolhouse shop people and a Hollywood group for a high-end lab, isolated from nan engineers are dressed successful jeans, not achromatic laboratory coats.

AWS Austin Chip Lab.Image Credits:TechCrunch/Julie Bort

AWS Austin spot lab.Image Credits:TechCrunch/Julie Bort

Note that this is not wherever nan chips are manufactured, truthful nary achromatic hazmat suits were necessary. The Trainium3 is simply a state-of-the-art 3-nanometer chip, produced by TSMC, arguably nan leader successful 3-nanometer manufacturing, pinch different chips produced by Marvell. 

But this is nan room wherever nan magic of nan “bring-up” occurs.  

“A silicon bring-up is erstwhile you get nan spot for nan first time, and it’s for illustration a large overnight party. You enactment here, for illustration a lock-in,” King explains. After 18 months of work, nan spot is activated for nan first clip to verify it useful arsenic designed. The squad moreover filmed immoderate of nan Trainium3 bring-up and posted it connected YouTube.

Spoiler alert: It’s ne'er problem-free.  

For Trainium3, nan prototype spot was primitively air-cooled, for illustration erstwhile versions. The existent spot is now liquid-cooled, which offers power advantages and was rather an engineering feat.

During nan bring-up, nan dimensions for really nan spot attached to nan air-cooling power descend were off, truthful nan spot couldn’t beryllium activated. 

Unfazed, nan squad “immediately sewage a grinder and conscionable started grinding disconnected nan metal,” King said. Because they didn’t want nan sound disrupting nan bring-up pizza statement atmosphere, they snuck disconnected and did nan grinding successful a convention room.  

Staying up each nighttime and solving problems “is what silicon bring-up is each about,” King said. 

The laboratory moreover has a welding station, wherever hardware laboratory technologist and maestro welder Isaac Guevara demonstrated welding mini integrated circuit components done a microscope. This is specified insanely difficult activity that elder leader Carroll openly admitted he couldn’t do it, to nan guffaws of Guevara and nan remainder of nan engineers successful nan room. 

AWS Austin spot laboratory tour, welding station.Image Credits:TechCrunch/Julie Bort

The laboratory besides contains some custom-made and commercialized devices for testing and analyzing issues pinch chips. Here’s awesome technologist Arvind Srinivasan demonstrating really nan laboratory tests each mini constituent connected nan chip:

AWS Austin spot laboratory tour, testing equipment.Image Credits:TechCrunch/Julie Bort

Sleds are nan prima of nan lab 

But nan prima of nan laboratory is an full statement showcasing each procreation of nan “sleds” nan squad designed. 

AWS Austin spot laboratory circuit wall of sleds.Image Credits:TechCrunch/Julie Bort

Sleds are nan trays that location nan Trainium AI chips, Graviton CPU chips, and supporting boards and components. Stack them together connected a rack pinch nan networking component, besides custom-designed by this team, and you get nan systems that are astatine nan bosom of Anthropic Claude’s success. 

Here’s nan sled that was shown disconnected during nan AWS re:invent convention successful December: 

AWS Austin spot laboratory tour, Trainium3 sled.Image Credits:TechCrunch/Julie Bort

Proven by Anthropic and OpenAI

I expected my guides to crow astir nan OpenAI woody during nan tour. But they didn’t. 

The reticence could person been related to nan aforementioned imaginable ineligible haze that mightiness bent complete nan deal. But nan consciousness I sewage was that these boots-on-the-ground engineers (who are presently designing nan adjacent version, Trainium4) haven’t had overmuch chance to activity pinch OpenAI yet. Their day-to-day activity has truthful acold been focused connected Anthropic’s and Amazon’s needs.

Currently, nan biggest chunk of Trainium2 chips is deployed successful Project Rainier — 1 of nan world’s largest AI compute clusters — which went unrecorded successful precocious 2025 pinch 500,000 chips. It’s utilized by Anthropic. 

But location was a wall show successful nan main agency displaying a quote astir really OpenAI will beryllium using Trainium. The pridefulness was there, if subtle.  

In summation to this lab, nan squad besides has its ain backstage information halfway for value and testing purposes. A short thrust away, it doesn’t tally customer workloads, truthful it’s housed astatine a co-location facility, not an AWS information center.

Security is tight: There are strict protocols to participate nan building and to entree Amazon’s area within.

The information center’s cooling strategy is truthful large that earplugs are mandatory, and nan aerial is heavy pinch nan acrid smell of heated metal. It’s not a pleasant spot for nan mean personification to bent out. 

Here’s maine and Aronson astatine nan AWS Austin spot laboratory information center, protecting our ears adjacent to unrecorded servers.Image Credits:TechCrunch / Julie Bort

At this information center, location are rows and rows of servers filled pinch sleds that merge each of Amazon’s newest civilization chips: Graviton CPU, liquid-cooled Trainium3, Amazon Nitro, each happily computing away. The liquid runs connected a closed system, meaning it is reused, which should besides thief trim nan biology impact, nan engineers said. 

Here’s what a existent Trn3 UltraServer looks like: Multiple sleds are connected apical and bottom, pinch nan Neuron switches successful nan middle. Hardware improvement technologist David Martinez-Darrow is seen present performing attraction connected a sled:

AWS Austin spot laboratory circuit information center.Image Credits:TechCrunch/Julie Bort

While attraction connected nan squad has ever been high, nan scrutiny has really ratcheted up arsenic of late. 

Amazon CEO Andy Jassy keeps a adjacent oculus connected this lab, publically bragging astir its products for illustration a proud dad. In December, he said Trainium was already a multibillion-dollar business for AWS and called it 1 portion of AWS tech he’s astir excited about. He besides gave nan spot a shout-out erstwhile announcing nan OpenAI agreement.  

The squad feels nan pressure, too. Engineers will activity 24/7 for 3 to 4 weeks astir each bring-up arena to hole immoderate issues truthful nan chips tin beryllium mass-produced and put into information centers.

“It’s very important that we get arsenic accelerated arsenic imaginable to beryllium that it’s really going to work,” Carroll said. “So far, we’ve been doing really well.” 

*Disclosure: Amazon provided airfare and covered nan costs of 1 nighttime astatine a section hotel. Honoring its Leadership Principle of Frugality, this was a back-of-the-plane mediate spot and a humble room. TechCrunch picked up nan different associated recreation costs for illustration Ubers and luggage fees. (Yes, I checked a container for an overnight trip. I’m precocious attraction that way.)