# How Netflix Built a Data Canary to Catch Bad Catalog Data
*Netflix's Data Canary validates new catalog data with real production traffic, detects corruption in minutes, and blocks bad data before publication.*
By [Rohit Lakhotia](https://scaleengineer.com/authors/rohit-lakhotia)
Published: 2026-10-12
Canonical: https://scaleengineer.com/blog/how-netflix-built-a-data-canary-to-catch-bad-catalog-data
---
Every time you open Netflix and hit **Play**, a lot of data has to be correct behind the scenes\.

Which titles are available? Where can they be watched? Can a particular title be played? Which artwork and metadata should be shown?

All of this information comes from **catalog metadata**, and Netflix continuously transforms and distributes it across its infrastructure\. Now imagine something goes wrong with that data\. Not because someone deployed bad code, Not because a configuration was changed\. **The data itself is simply wrong\.**

That's exactly what happened at Netflix\.

During a previous incident, a manual mitigation accidentally corrupted a data feed, leaving it empty for a subset of titles\. The result was immediate: missing metadata prevented manifest generation, which led to catalog service failures and playback issues\.

And there was a problem with the way Netflix was protecting itself\. Its existing **code canary** system didn't catch the issue\. Why? Because there was no code change to test\. **The data had changed\. **That raised an interesting engineering question:

If we canary our code before releasing it to production, why can't we do the same for data?

That's the problem Netflix set out to solve with its **Data Canary**, a system that validates new catalog data using production traffic, detects problems within minutes, and blocks bad data before it reaches members\.

## Why Traditional Canarying Wasn't Enough

Netflix already had sophisticated canary deployment systems for validating code changes\. But catalog metadata follows a different lifecycle\. Multiple upstream sources continuously provide data\. That data gets transformed, and the resulting catalog is then distributed across Netflix's infrastructure\. Each upstream source had its own validation, but corruption could still appear in the **final transformed output**\.

That created an important gap\. Netflix wasn't just interested in knowing whether the input data was valid\. It needed to know: **Is the final catalog state that customers will consume actually correct?**

And there was another constraint\. The catalog pipeline operates at a high velocity, with new data cycles happening frequently\. Netflix's existing canary analysis tools typically needed **30–60 minutes** to reach statistical confidence\. That was too slow\.

Netflix needed to detect a problem, decide whether the new data was safe, and block its publication **within a single data cycle**\. So the challenge became simple but important: **Netflix needed to validate every new catalog version quickly enough to catch regressions, while keeping the number of customers exposed to potentially bad data extremely small\.**

The system therefore had to do three things at once: **validate the new data, detect meaningful regressions quickly, and block publication before the issue could affect a large number of customers\. **But how could Netflix do all of this using real production traffic without putting a large number of users at risk?

## The Idea: Treat Data Like a Deployment

Netflix's solution was to create a dedicated **Data Canary Orchestrator**\. The basic idea is similar to a code canary\. Instead of immediately sending a new catalog version everywhere, Netflix creates a controlled environment where the new data can be compared against a known\-good version\.

The architecture has three important pieces:

1. A dedicated orchestrator
2. A permanent baseline cluster
3. A permanent canary cluster

The **baseline cluster** continuously serves the latest production catalog version\. The **canary cluster** receives the new catalog version that needs to be validated\.

The orchestrator coordinates the entire process\. When a new catalog version arrives, it first makes sure that both clusters are healthy and properly synchronized\. It then triggers a chaos experiment to send traffic through the two versions and compare their behavior\.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/8396f79d-7fe0-472f-996f-5b5e31d9ead3/image.png?t=1790366593)

Source: [Netflix’s blog](https://netflixtechblog.com/the-data-canary-how-netflix-validates-catalog-metadata-18b699d58e36)

The important detail is that the baseline and canary clusters are **permanent** parts of the setup\. That gives Netflix a consistent environment for comparing the existing catalog version against the new one\.

## Why Netflix Needed Real Production Traffic

At this point, you might wonder: **Why not simply replay traffic against the new data? **Netflix initially considered using shadow traffic for validation\. But shadow traffic wasn't enough\. It could replay requests against the catalog metadata service, but it couldn't reproduce the complete playback lifecycle across the multiple services and domains involved\.

And that mattered because catalog corruption doesn't necessarily show up as an obvious error inside the catalog service\. A request could technically succeed while the resulting catalog data still causes problems further downstream\. So Netflix decided that **real production traffic was essential** for detecting actual customer impact\.

But there was an obvious problem: You don't want to discover that your new data is broken by sending it to millions of customers\. So Netflix needed to combine **production traffic with a very small blast radius**\.

## The 10\-Minute Challenge

The Data Canary had to make a decision quickly\. Traditional canary analysis could take 30–60 minutes, but Netflix wanted the entire validation process to fit within roughly **10 minutes**\. That meant the system couldn't simply collect metrics for a long time and perform a detailed analysis afterward\. It had to identify meaningful signals almost immediately\. This led to several changes in how Netflix used its existing chaos platform\.

### Custom Thresholds

Netflix worked with its Resilience team to customize the thresholds used by the chaos experiments\. The standard thresholds were too conservative for this particular use case because the data canary had a much tighter time constraint\.

### Test Different Client Types

Netflix's catalog service serves multiple types of clients, and those clients can have different traffic patterns and downstream dependencies\. So Netflix ran separate experiments for major client types\. Interestingly, the traffic path handling **playback requests** consistently identified failures the fastest\.

That gave Netflix a much more useful signal than simply testing every client in the same way\.

## The Clever Part: Measure What the Customer Actually Experiences

One of the most interesting decisions was the metric Netflix chose to monitor\. You might naturally think about things like:

- latency
- error rates
- request failures

But Netflix found that these weren't necessarily the best indicators of catalog corruption\. Instead, it focused on **Starts Per Second \(SPS\)** essentially, actual customer playback attempts\.

Why? Because the goal wasn't simply to determine whether the catalog service was technically healthy\. The goal was to determine whether customers could actually start watching content\. A catalog data problem might not produce an obvious application error at the metadata service, but it could still prevent playback\.

So behavioral metrics were more useful than purely technical metrics for this problem\. SPS gave Netflix a much more direct signal of whether the new catalog version was actually affecting customers\.

## Don't Wait for the Analysis to Finish

So instead of collecting all the experiment data and analyzing it afterward, the system **streams metrics in real time**\. If a regression is detected, the experiment is immediately aborted\. This means Netflix trades some statistical confidence for speed\.

For this particular use case, that trade\-off was intentional\. The system has tight thresholds and a clear behavioral signal, making rapid detection more valuable than waiting for the longer analysis window used by traditional canaries\.

So instead of waiting for the experiment to finish and analyzing everything afterward, Netflix evaluated the metrics **in real time**\. If the system detected a regression, it immediately aborted the experiment\. The objective wasn't to produce a perfect statistical analysis after an hour\. It was to **catch a potentially damaging catalog change before it could be published widely**\.

Netflix was willing to trade some statistical confidence for speed because the system had clear signals and tight thresholds\. For this use case, detecting a serious regression quickly mattered more than waiting longer for additional confidence\.

## Production Makes the Edge Cases Hard

Once Netflix started running these experiments regularly, another set of problems appeared\. The system itself had to be production\-hardened\. For example, what happens if the orchestrator gets redeployed while an experiment is already running? It can't simply forget about the experiment\. The new orchestrator instance needs to detect ongoing experiments and continue polling them\.

Then there's another problem: **leader election**\. During a deployment, multiple orchestrator instances might temporarily be running at the same time\. Without safeguards, multiple instances could trigger the same experiment for a single catalog version\. Netflix therefore added protection to ensure that only one experiment is triggered for each version announcement\.

Version synchronization was another concern\. Different clients can consume catalog data at different cadences, so Netflix tracks version state to ensure that the baseline and canary clusters are aligned before an experiment begins\.

## How Do You Test a System Designed to Catch Failures?

Netflix didn't want to simply assume the Data Canary worked\. They deliberately broke the data\. The team ran controlled experiments where they corrupted catalog data, including denylisting high\-profile titles, to simulate realistic data corruption scenarios\. These experiments were run as proactive incidents during business hours, with product operations teams available if needed\.

But the team still had to control the blast radius\. Only approximately **0\.2% of global traffic** was routed through the validation flow\. That was enough to generate meaningful production signal while keeping the experiment contained\. And the results showed that the approach worked\.

### The Results

The Data Canary detected issues in approximately **2\.5–4 minutes**, depending on the client type\. The canary and baseline showed a **10× error differential**, providing a clear signal when the catalog was corrupted\. Most importantly, the publishing workflow was automatically blocked when a regression was detected\.

The experiments validated the complete loop: corrupted data could be introduced into the canary environment, exposed to real production traffic, detected through behavioral signals, and automatically blocked before publication\.

The experiments also revealed something important: **different client traffic patterns detected failures at different speeds**\. This meant Netflix had to tune its thresholds based on the **magnitude of impact it wanted the system to catch**\.

This was an important part of making the Data Canary reliable\. The goal wasn't simply to detect any difference between the baseline and canary, but to identify **meaningful regressions quickly enough to prevent problematic data from being published**\.

## The Bigger Idea: Data Deserves the Same Treatment as Code

The most interesting takeaway isn't really the specific architecture\. It's the principle behind it\. Software teams have spent years building sophisticated systems to safely deploy and validate code\. But production systems don't only break because of bad code\.

**They can also break because of bad data\.**

Netflix's catalog metadata is continuously transformed and published, so corrupted data can directly affect the customer experience\. The Data Canary brings some of the same thinking used for code deployments to high\-velocity data pipelines\. And the pattern isn't limited to catalog metadata\. The same approach can also be useful for other systems where data changes frequently and directly affects customers\.

The question then becomes: **If your production system depends on rapidly changing data, how quickly can you detect when that data goes bad?**

The validation process can detect high\-impact catalog corruption within minutes, rather than waiting for customers to discover the problem\.

## Key Takeaways

- **Data can break production even when no code or configuration changes\.**
- Netflix's catalog metadata can be continuously transformed, making traditional code canaries insufficient for detecting certain data problems\.
- The **Data Canary** uses dedicated baseline and canary clusters to validate new catalog versions\.
- Netflix uses **real production traffic** because shadow traffic couldn't reproduce the full playback lifecycle\.
- **Starts Per Second \(SPS\)** was used as a primary behavioral signal because it more directly reflected customer impact than technical metrics such as latency or error rates\.
- Metrics are evaluated in real time, allowing the experiment to **abort immediately when a regression appears**\.
- Controlled failure injection showed detection times of **2\.5–4 minutes**, with a **10× error differential** between canary and baseline\.
- The broader lesson: **data deployments deserve the same reliability discipline as code deployments\.**

Official blog from Netflix: **[The Data Canary: How Netflix Validates Catalog Metadata](https://netflixtechblog.com/the-data-canary-how-netflix-validates-catalog-metadata-18b699d58e36)**

By now, you must have had a clear idea of,** How Netflix Built a Data Canary to Catch Bad Catalog Data? **In a nutshell, Netflix built a **Data Canary** that uses controlled production traffic to compare new catalog data against a known\-good version and automatically block corrupted data before it reaches customers\.

**Congratulations\! You've just advanced another step in your tech journey\. Keep progressing\!**