How Meta Migrated One of the World's Largest Data Ingestion Systems Without Downtime

Every day, billions of people use Meta's apps like liking posts, sending messages, watching videos, following creators, and much more. Behind every one of these actions is data. A lot of data.

Every day, Meta moves several petabytes of social graph data from one of the world's largest MySQL deployments into its data warehouse. This data powers everything from analytics dashboards and business reporting to machine learning models and product development.

For years, this process relied on a legacy data ingestion system that worked well at smaller scales. But as Meta's infrastructure continued to grow, the system started struggling to meet increasingly strict data freshness requirements.

Simply replacing it wasn't an option. Migrating thousands of production ingestion jobs meant ensuring every dataset remained correct, performance didn't regress, and the teams relying on this data never noticed the transition.

So how do you replace the engine of a system that's already running at hyperscale? Let's see how Meta pulled off one of its largest infrastructure migrations in this blog today.

Why Meta Needed a New Data Ingestion System

Before understanding the migration, let's first understand what the data ingestion system actually does.

Imagine every action performed on Facebook or Instagram like a new friendship, a comment, a reaction, or a profile update. These changes are first stored in Meta's massive MySQL infrastructure. But that raw operational data isn't where engineers build dashboards or train machine learning models.

It first needs to be copied into Meta's data warehouse, where thousands of teams can analyze it efficiently. That's the job of the data ingestion system. It continuously captures changes from production databases and delivers fresh snapshots to downstream systems that power analytics, reporting, experimentation, and AI workloads.

The legacy system worked well when Meta's infrastructure was smaller. However, as the volume of data and the number of ingestion jobs kept increasing, maintaining individual customer-owned pipelines became more difficult. At the same time, stricter expectations around data landing time meant the system had less room for delays or failures.

Rather than continuing to evolve the existing architecture, Meta redesigned the platform around a simpler, self-managed data warehouse service that could operate more reliably at hyperscale.

The Hardest Part Wasn't Building the New System

Replacing software is relatively straightforward. Replacing software that moves petabytes of production data every day is not. Meta wasn't migrating a handful of jobs, it was migrating thousands.

Each of those jobs delivered data to internal teams, dashboards, reports, or machine learning pipelines. A single incorrect migration could introduce inconsistent data, delay business decisions, or affect downstream systems.

Simply copying jobs into the new platform and hoping everything worked wasn't an option. Every migrated job had to satisfy three important conditions before Meta considered it successful:

• The data had to be identical. The new system should produce exactly the same data as the old one, verified using row counts and checksums.

• Performance couldn't get worse. Data should arrive at least as quickly as before.

• Resource usage shouldn't increase. Compute and storage consumption needed to remain comparable or ideally improve.

For business-critical datasets, Meta also worked directly with the teams depending on those tables to define additional migration requirements before moving them to production. Instead of treating migration as a one-time event, Meta designed it as a carefully controlled process where every job gradually earned its promotion to the new system.

Meta's Three-Phase Migration Strategy

To safely move thousands of ingestion jobs, Meta created a migration lifecycle with three distinct phases.

Rather than switching everything over at once, every job had to prove it was reliable before progressing to the next stage.

Source: Meta Blog

Let's look at what happened in each phase.

Phase 1: Shadow Testing

The first step was creating a shadow job. A shadow job consumed the exact same production data as the original ingestion job but wrote its output to a separate shadow table instead of the production table.

This gave Meta something incredibly valuable: A production-realistic test environment. Because both systems processed the same live data, engineers could compare their outputs without exposing users or downstream systems to any risk.

For every migrated job, Meta continuously compared:

• Row counts

• Checksums

• Compute usage

• Storage consumption

Whenever differences appeared, engineers investigated the root cause, deployed fixes to the new system, and reran the validation until both systems produced identical results.

Only after passing these checks could a job move to the next phase.

Phase 2: Reverse Shadow

Most migrations stop once the new system appears to be working. Meta went one step further. Instead of immediately retiring the legacy pipeline, it reversed the roles of the two systems. The new ingestion job now wrote data to the production table, while the old production job continued running in parallel and wrote to the shadow table instead.

This approach offered two major advantages.

First, engineers could continue comparing outputs from both systems even after the rollout, providing continuous confidence that the migration remained healthy.

Second, if a problem appeared, rolling back became much easier because the legacy pipeline was still running and already processing fresh production data.

There was no need to recreate old jobs or rebuild missing state, the previous system was already standing by.

Phase 3: Cleanup

Once both systems continued producing identical results for a sustained period, Meta removed the old shadow job. At this point, the new ingestion system became the only system responsible for delivering production data, marking the successful completion of that job's migration.

By following this gradual promotion process instead of performing a big-bang migration, Meta dramatically reduced the risk of introducing data quality issues across thousands of production ingestion jobs.

But validating data correctness was only one piece of the puzzle.

Meta also had to detect problems before downstream teams noticed them, stop incorrect data from spreading through its change data capture pipeline, and automate the migration of thousands of jobs with minimal manual effort. We'll explore those engineering solutions in the next part.

Catching Problems Before They Reached Users

Validating a job before migration was important, but Meta knew that wasn't enough. Some issues only appear after a system starts handling real production traffic. Waiting for downstream teams to report incorrect data would mean the problem had already spread.

Instead, Meta built a comprehensive data quality validation system that continuously compared the outputs of both the legacy and the new ingestion systems. For every shadow table generated by the new system, Meta compared it against the corresponding production table generated by the old system.

The comparison focused on two simple but powerful checks:

• Row Count: Did both systems produce the same number of records?

• Checksum: Even if the row counts matched, was the actual data identical?

Whenever a mismatch was detected, it wasn't just flagged and forgotten.

Meta automatically logged the discrepancy into Scuba, its internal real-time analytics platform. Every hour, a data quality analysis tool examined these logs, identified sample rows causing the mismatch, and generated detailed debugging information for engineers.

Instead of manually searching through billions of records, engineers could immediately focus on the exact rows that caused the problem.

Even after the migration was completed, this tooling continued to be used as part of Meta's release validation process, making future deployments safer as well.

Rolling Back Without Causing More Damage

Detecting bad data is only half the battle. The next challenge is preventing that bad data from spreading. Meta's ingestion pipeline is built using Change Data Capture (CDC). Rather than copying the entire database every time, CDC captures only the changes made since the previous update. This makes ingestion much faster and more efficient.

However, it also introduces an interesting challenge. If incorrect data gets written today, tomorrow's updates build on top of it. In other words, bad data can keep propagating through every future ingestion cycle until someone notices it. Meta needed a way to stop that chain reaction immediately.

Stopping Bad Data Before It Spread

During the Reverse Shadow Phase, Meta continuously monitored data quality. If a partition showed inconsistencies, it was immediately marked in its metadata as containing bad data.

What happened next depended on the type of partition. If it was a delta partition, the system simply stopped ingesting new changes and alerted engineers.

If it was a target partition, the system avoided using the corrupted data altogether. Instead, it selected an older healthy partition and merged it with newer delta changes until the issue was resolved.

Preventing Bad Data Propagation

This approach significantly reduced the impact of failures.

Instead of allowing corrupted data to silently spread across the warehouse, Meta could isolate the problem, prevent further damage, and recover much more quickly.

Migrating Thousands of Jobs Automatically

Once the migration strategy was proven on a small set of jobs, the next challenge was scale. Meta wasn't migrating dozens of jobs. It was migrating tens of thousands. Tracking each migration manually would have been nearly impossible. So the team automated almost everything.

Every ingestion job continuously reported its migration status, validation results, resource usage, and current lifecycle stage. Automation tools monitored these signals and made promotion decisions automatically. If a job met all migration criteria, it progressed to the next phase.

If new issues appeared, it could be automatically held back or even demoted to an earlier stage until the problem was resolved. Meta also built dashboards that allowed engineers to monitor both the overall migration progress and the health of individual jobs without manually inspecting thousands of pipelines.

Smart Planning Was Just as Important

Running shadow jobs consumes additional compute resources. Unfortunately, Meta couldn't afford to create shadow jobs for every ingestion pipeline simultaneously. Instead, the migration happened in carefully planned batches.

Jobs were grouped based on factors like:

• Throughput

• Business priority

• Known issues

• Special operational requirements

If a particular class of jobs was already known to have unresolved bugs, Meta simply postponed migrating similar jobs until those problems were fixed. This prevented engineers from repeatedly debugging the same issue across hundreds of pipelines.

The team also optimized expensive full database snapshots by reusing existing snapshots wherever possible instead of generating new ones for every migration. This significantly reduced migration cost while improving overall efficiency.

Key Takeaways

• Large-scale migrations are more about risk management than simply moving workloads.

• Meta validated every ingestion job through a three-phase migration strategy before fully promoting it to production.

• Running the old and new systems in parallel enabled continuous verification and fast rollbacks.

• Automated data quality checks helped detect issues long before downstream teams were affected.

• Automation played a critical role in migrating tens of thousands of jobs safely and efficiently.

• Careful batch planning and capacity management allowed Meta to complete one of its largest infrastructure migrations while minimizing operational risk.

Official blog from Meta: Migrating Data Ingestion Systems at Meta Scale

By now, you must have had a clear idea of, How Meta Migrated One of the World's Largest Data Ingestion Systems Without Downtime? In a nutshell, Meta successfully migrated one of its largest data ingestion systems by validating every migration step, continuously comparing data quality, and automating the rollout of thousands of production jobs. Instead of relying on a risky one-time migration, the company used shadow testing, automated validation, and smart rollback strategies to ensure users never noticed the transition.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!