Zillow and the AI Failure: A $7.8 Billion Lesson

Lesson in $9 Billion

In November 2021, Zillow admitted during a meeting that their AI had caused financial losses on 7,000 homes. They had developed an algorithm to buy and resell houses, but this algorithm spent two years overpaying for everything in Phoenix, Atlanta, and a dozen other markets. When the company realized the situation, it had to record a loss of $304 million, lay off a quarter of its staff, and completely shut down its iBuying business. The market capitalization dropped by about $7.8 billion in just a few days.

The CEO blamed unpredictability. Data scientists faced criticism on Twitter. Post-mortem analyses all said the same thing in different words. The forecasting model worked well until the world changed, and no one noticed.

You’re Already Making Predictions

Take a moment. Look out the window and guess if it will rain in two hours. You have an answer, perhaps even a level of confidence. Congratulations, you just executed a forecasting model in your head.

Your phone does this for you hundreds of times a day. Maps predict traffic. Spotify predicts the next track. Your weather app predicts the rest of the week. Even your body makes predictions. It knows when you’ll be hungry at 7 PM because you usually are at that time.

The technical term for this is time series forecasting, and the mathematics is older than your grandparents. The first serious model dates back to the 1920s: Yule's autoregressive equations. We have refined this for a century. We are not exactly working with new tools.

The Three Ingredients of Every Noisy Series

Practically every time series you encounter is made up of three overlapping elements. Once you can name these three layers, you’ve already passed the first interview question. Every series is a noisy stack.

• Trend: The slow drift. Do the numbers generally increase over the months? Decrease? Are they stable? The trend is the long story.

• Seasonality: The repetitive heartbeat. Pumpkin spice sales peak every October. Gym memberships hit their maximum in the first week of January and drop by February 15.

• Residual: The leftover movement. Everything that remains after accounting for the trend and season. A surprise storm. A tweet from a celebrity. A holiday Monday that fell at the wrong time.

Stationarity: The Boring Superpower

A series is stationary when its statistical properties stop drifting over time. The mean remains stable. The variance doesn’t swell. How today correlates with yesterday doesn’t change.

Why does this matter? Because most classical models, especially the ARIMA family, assume your data is stationary before processing it. If you provide them with a wandering series, they will learn nonsensical things. That’s pretty much what happened to Zillow. Their model was trained on fundamentally non-stationary real estate markets and continued to extrapolate an upward trend that the world had stopped agreeing with.

Differencing: The Working Adhesive

Take today’s value. Subtract yesterday’s value. You just differentiated a series. That’s it. That’s the whole technique.

Sometimes, a single round isn’t enough. The series still drifts because the rate of change itself is increasing. In that case, you need to differentiate again. Twice. We call this the order of integration, and it’s the d in ARIMA(p, d, q). Most real series require d to be equal to 0, 1, or 2.

Autocorrelation: The Ghost of Yesterday Haunting Today

A time series is not a set of independent samples. Today is haunted by yesterday. Sometimes by last week. Autocorrelation is simply a sophisticated term for measuring how much.

Two tools you will see everywhere: ACF and PACF. They look similar but are not identical.

• ACF (Autocorrelation Function): For each lag k, how correlated is today with the value k steps ago?

• PACF (Partial Autocorrelation Function): The same idea, but at each lag k, it removes the influence of all lags between today and k steps back.

Decomposition: Taking Apart the Dish

Two ways to decompose a series. The choice depends on a single question: does the seasonal swing increase as the trend grows?

• Additive: ( y_t = \text{trend} + \text{season} + \text{residual} ). Use this when seasonal variations remain roughly the same size, regardless of where the trend stands.

• Multiplicative: ( y_t = \text{trend} \times \text{season} \times \text{residual} ). Use this when seasonal variations increase with the trend.