Washington: Power Outage Exposes Data Center Flaws

A Fallen Power Line Exposes a Growing Problem for AI Data Centers

This week, an unusual incident occurred near Washington, DC, when a power line fell, causing an unexpected disruption to the electrical grid. Normally, the grid would have stabilized within seconds, but this time it took more than 10 minutes to regain stability. The cause? Over 3 gigawatts of data centers ceased consuming power almost simultaneously.

The event led to a voltage spike on the PJM grid, which stretches from Northern Virginia to Chicago. Data was collected by Ting Labs, a startup specializing in managing IoT sensors installed on residential power outlets. Although the incident did not cause a power outage, it resulted in flickering lights across the region, illustrating the potential impact of data centers on the electrical grid.

Northern Virginia, located within the PJM territory, is known for hosting the highest concentration of data centers in the world. Ricardo de Azevedo, CTO at ON.Energy, compared this event to a "canary in the coal mine," emphasizing that such incidents involving large loads are becoming increasingly frequent.

A Recurring and Growing Issue

This incident recalls a similar event that occurred two years ago on the same PJM grid and could be a harbinger of larger disruptions in the future. PJM Interconnection, which manages the grids from New Jersey to Illinois, serves 67 million customers, making it the largest grid operator in the United States.

When the power line fell, data centers immediately switched to their backup power, resulting in a 3.1 gigawatts load reduction in just 30 seconds. The grid attempted to stabilize, but other loads continued to drop. At the peak of the incident, the PJM grid had a surplus of 3.49 gigawatts of electricity, and it took 11 minutes for the situation to normalize. The disconnected data centers represented about 3% of the total demand on PJM at that time, according to Reuters.

While 3% may seem insignificant, the electrical grid must maintain a nearly perfect balance between supply and demand. Imbalances can lead to voltage fluctuations, which the grid and connected devices can tolerate only up to a certain point. Beyond that, safety systems trigger, causing disconnections.

The Response from Data Centers

In response to the fluctuation caused by the fallen power line, data centers in Northern Virginia switched to backup power, thereby removing their load from the grid. This process amplified the initial drop in supply, leading to an even greater decrease in demand and causing flickering lights.

Most data centers make decisions in a fraction of a second. Those that disconnected this week were no exception. Ali Zain Banatwala, senior market modeling specialist at the Independent Electricity System Operator, highlighted the need to find a way for these loads to disconnect or reconnect sequentially, allowing grid operators to develop more robust procedures.

Potential Solutions

One solution could be to design data centers capable of absorbing disturbances without disconnecting. ON.Energy, an innovative startup, is working on a product to help data centers and the grid manage such events. The company has developed an uninterruptible power supply for an entire campus of data centers, covering not only servers but also coolers and other equipment.

ON.Energy's system hides the data center behind a bank of batteries connected to sophisticated energy conversion equipment. Thus, the grid perceives only a stable load, avoiding individual peaks and troughs. This system allows data centers to adjust computing loads, including AI training, without disrupting the grid.

Towards Better Management of Fluctuations

Most importantly, this system enables data centers to absorb power fluctuations from the grid. Rather than disconnecting, ON.Energy's system can use excess power to charge its batteries and provide energy to servers when the flow decreases. It can track grid variations within milliseconds, thus avoiding the voltage drops or spikes that caused the recent issue for PJM.

ON.Energy is in the process of installing a total of 3 gigawatts of its systems across four different data center campuses, according to de Azevedo.

Measures by Grid Operators

Grid operators have also become aware of the issue. For example, ERCOT plans to require that large loads like data centers "ride through" disturbances, according to de Azevedo.

However, time is of the essence. The massive disconnection this week was twice as significant as a similar event in 2024, when 60 data centers disconnected simultaneously, removing 1.5 gigawatts of load from the grid. At that time, data centers represented about 6% of PJM's load, according to Synapse Energy Economics. By 2040, they are expected to represent 24%. If the problem is not resolved quickly, the situation could become much more serious.