All posts

Blog

Meta opens up about Prometheus, the Ohio supercluster that trained its Muse AI models

FindAmNow
MetaMuseAI infrastructuredata centresNvidiasupercomputing

Meta has detailed Prometheus, its gigawatt-scale AI supercluster in New Albany, Ohio. The completed first phase, with hundreds of thousands of Nvidia GPUs, trained the Muse models, and Meta is scaling toward about 1 million GPUs. It reports a 99% cut in interruptions and over 85% training goodput; the figures are Meta's own.

Meta has published a detailed look inside Prometheus, the giant AI computer it is building in New Albany, Ohio, in the United States. In a post on its engineering blog on 19 October 2026, Meta said the first phase is complete, that it was used to train the company's Muse family of AI models, including Muse Wonder, and that the site is being expanded toward roughly 1 million GPUs.

What Prometheus is

A supercluster is many computer clusters joined so they can work on one training job. Meta says Prometheus already holds "hundreds of thousands of GPUs" spread across several buildings, from ordinary data centre halls to quickly built temporary structures, all linked by fibre and "functioning as a single machine". Two years ago, Meta says, the same site was a conventional data centre serving web traffic.

The company describes it as a "gigawatt-scale" system, meaning its power demand is measured in the billions of watts, comparable to a large power station.

The hardware

Meta runs two generations of Nvidia equipment at Prometheus, both built around Nvidia's NVL72 racks, where 72 GPUs are wired closely together:

  • Catalina uses Nvidia's GB200 chips, spread across two racks of 36 GPUs each.
  • Clemente uses the newer GB300 chips, with 288 GB of memory per GPU and all 72 GPUs in a single rack.
  • San Miguel, based on Nvidia's next-generation Vera Rubin chips, is due to join. Meta expects 10% to 20% better performance per watt than GB300.

Training traffic runs on its own Ethernet network, separate from the network carrying ordinary service traffic. Each building's network can link up to 20,736 GPUs, and a wider network joins buildings between 2 km and more than 100 km apart. Meta says running training across those distances needed special work because of delays caused by the speed of light.

Keeping the machine running

The most unusual part of the post is how openly Meta discusses failures. With so many parts, one fault can stall a whole training job. Meta says that in mid-2025, early in the project, it saw nearly 1,000 interruptions per 8,000 GPUs every day. It says a year of work with Nvidia cut that by 99%.

According to Meta:

  • Training runs for Muse Wonder achieved more than 85% "goodput" (the share of time spent on useful work) for the main pre-training stage and more than 90% for later reinforcement-learning stages.
  • In each 72-GPU rack, only 64 GPUs do the work. The other eight wait as spares, ready to take over if one fails. About 2% of whole racks are also held in reserve.
  • Firmware fixes can be pushed to more than 100,000 GPUs within a few hours.

These figures come from Meta and have not been independently audited.

Power and water

A single large training job can swing the site's power use by 30% to 40%, or tens of megawatts in seconds, Meta says. A system it calls Dynamo Dimmer gently caps GPU power as the site nears its electrical limit, backed by batteries. Newer buildings use closed-loop liquid cooling, which Meta says is the most water-efficient option. Meta also says it pays the full cost of its energy and matches its use with new clean and renewable power, a claim that cannot be checked from the post alone.

Why it matters

Building leading AI models now depends as much on electricity, networking and reliability engineering as on clever software. Big companies rarely share failure rates or spare-capacity numbers, so Meta's post gives a rare view of what running AI at this scale involves. Meta says the lessons are shaping Hyperion, an even larger supercluster it is designing in Richland Parish, Louisiana. For countries such as Nigeria, the post is also a reminder of how much power and infrastructure frontier AI training now demands.

Sources

Source