How Zomato Reduced a 150 GB Flink State to Just 500 MB

Every time you see or click on an ad in the Zomato app, a lot happens behind the scenes. That interaction isn't just recorded for analytics. It's used to measure how well an ad campaign is performing, calculate billing for restaurant partners, and ensure advertisers are paying for the right number of impressions and clicks.

Since thousands of these interactions happen every second, this data has to be processed in real time. Even small inaccuracies can lead to incorrect billing, over-delivery of ads, or unreliable performance metrics. To power this feedback system, Zomato relied on Apache Flink.

For years, the pipeline successfully processed millions of ad events. But as the platform grew, a new problem began to surface. Some Flink jobs were maintaining more than 150 GB of state, making the system increasingly difficult to operate. Recovery became slower, checkpointing became unreliable, and occasional state loss started affecting the accuracy of the Ads platform.

Instead of trying to manage larger and larger state, Zomato redesigned the feedback loop itself. The result was impressive:

• 99% reduction in state size

• Over $3,000 in monthly infrastructure savings

• Zero downtime after migration

Let's understand how they achieved it in this blog today.

Why Real-Time Feedback Matters

Zomato's Ads platform helps restaurant partners promote their businesses inside the app. Restaurants can create different kinds of advertising campaigns depending on their business goals. Some campaigns focus on increasing visibility, while others are designed to drive more orders by targeting customers based on factors like previous transactions, cuisine preferences, or spending patterns.

To measure whether these campaigns are successful, the system continuously tracks several key metrics:

• Impressions: how many times an ad is shown.

• Clicks: how many users click on the ad.

• Conversions: actions such as placing an order after clicking the ad.

• ROI: the return advertisers receive on their ad spend.

Because these metrics directly influence billing and campaign performance, the feedback pipeline needs to remain both accurate and reliable.

How the Original Feedback Loop Worked

The original real-time feedback loop was built using Apache Flink 1.8 and implemented in Java. Whenever a user generated an ad event such as a click or an impression, the Flink job processed it in real time.

The pipeline performed two important tasks.

First, it deduplicated user interactions to ensure repeated clicks or impressions weren't counted multiple times.

Second, it aggregated campaign counts and sent the updated totals to the Ads Billing Engine every minute.

To perform deduplication, the system maintained a large collection of user IDs inside Flink's Managed State. Every time a new event arrived, the job checked whether that user interaction had already been processed. If not, the event was counted and the state was updated.

This worked well initially. But as more users interacted with ads, the amount of state that Flink had to maintain kept growing.

Old Feedback Loop Architecture :Source

The Real Reason the State Became So Large

At first glance, storing user IDs for deduplication doesn't sound like a problem. The challenge came from late events. A late event is an interaction that occurs on a user's device but reaches the server much later.

For example, a user might click on an ad and immediately close the app before the click event is uploaded. The event still carries the original timestamp, but it isn't processed until the app synchronizes again later.

To avoid counting these delayed interactions incorrectly, the original system kept user IDs in Flink's state for long periods. This allowed late events to be deduplicated correctly, but it also meant the state kept growing over time.

In some jobs, particularly those tracking impressions, the Flink state reached 150 GB. The state had become the biggest bottleneck in the entire pipeline.

Why Large State Became a Problem

The growing state size affected much more than memory usage. One major issue was checkpointing. Flink periodically creates checkpoints so jobs can recover after failures. As the state grew larger, checkpoint files also became much larger, making recovery slower and increasing the chances of restart failures.

Large state also increased memory consumption. During periods of heavy traffic, memory usage would spike, making the pipeline less stable. Jobs processing large volumes of impressions were particularly affected.

The most serious issue, however, was state loss. If the Flink state was lost during a restart or upgrade, the consequences extended beyond the streaming job itself.

Campaign counters could reset, deduplication became inaccurate, and in some cases the platform over-delivered ads. Since ad metrics are directly tied to billing, these inaccuracies affected both reporting and financial outcomes. Simply allocating more memory wouldn't solve these problems. The architecture itself needed to change.

The Key Idea: Introduce Reconciliation

Instead of depending entirely on one real-time pipeline, Zomato introduced a reconciliation mechanism.The engineering team describes reconciliation as a safety net for real-time systems.

Its role is to periodically verify processed data and recover anything that might have been missed because of job failures or restarts. Rather than replacing the streaming pipeline, reconciliation works alongside it, helping ensure the final counts remain accurate.

For Zomato's Ads platform, this became especially important. If the Flink job experienced issues, the reconciliation process could fill the gaps, preventing incorrect counts from affecting campaign performance or billing. This additional verification layer reduced the impact of failures and made the overall feedback system far more reliable.

But adding reconciliation was only one part of the solution. Zomato also redesigned the feedback loop itself, changing how counts were published, reducing the amount of state Flink needed to maintain, and modernizing the entire pipeline with Flink SQL.

Redesigning the Feedback Loop

The biggest change wasn't simply moving from Flink Java to Flink SQL. Zomato changed how the system handled real-time counts and late events.

The new design introduced two separate flows:

1. A real-time attribution flow for events that arrive on time.

2. A reconciliation flow for events that arrive late or need to be recovered.

This separation was important because the real-time pipeline no longer needed to keep user IDs for 24 hours just to handle late events.

New System Architecture :Source

Real-Time Attribution with Flink SQL

The real-time pipeline now uses Flink SQL 1.17 instead of the older Flink Java 1.8 implementation.

The job processes ad events such as clicks and impressions and produces incremental counts that are sent to Redis in the Ads billing engine. The pipeline uses a TUMBLE window to group events into fixed time intervals.

For example, if the window is 60 seconds, meaning events within each 60-second interval are grouped together for processing. The job also uses a custom User-Defined Function (UDF) called adsDedupeCount for deduplication. This UDF contains the custom logic required to ensure that user interactions aren't counted more than once.

Handling Late Events with Watermarks

Real-time systems have one unavoidable problem: events don't always arrive when they happen.

An interaction has an event time, when the user actually clicked or viewed an ad and an ingestion time, when the event reaches the Flink job and gets processed. These two times can be different.

Zomato uses watermarking to handle this difference.

For the example, the watermark interval is 60 seconds. After a window ends, Flink waits for this additional period before finalizing the results, giving slightly delayed events an opportunity to arrive.

For example, an event that occurred at 00:36 but reached the system at 01:30 can still be processed as part of the 00:00–01:00 window because it falls within the watermark period.

But an event that occurred at 00:24 and isn't ingested until 02:15 arrives too late and is discarded by the real-time pipeline. These events aren't simply lost, though.

They're handled separately by the reconciliation job.

The Trick That Reduced State by 99%

This is where the architecture change made the biggest difference. Previously, the Flink job needed to retain deduplication information for 24 hours.

With late events now handled by the reconciliation process, Zomato reduced the deduplication state's Time-to-Live (TTL) from 24 hours to just 2 hours. That dramatically reduced the amount of state the real-time Flink job had to maintain.

Instead of keeping large amounts of user interaction data around for an entire day, the system only retained what was needed for the shorter real-time processing window. The remaining late events could be picked up by reconciliation.

The Recon Job as a Safety Net

The reconciliation job runs at a fixed time interval and processes raw events from the S3 Data Lake.

Rather than sending only late events, it calculates total counts for each campaign per hour, excluding the current hour, and updates Redis in the Ads billing engine. The job only overwrites the existing billing count when the newly calculated count is greater.

It also uses a rate limiter to control resource usage. This gives the system an additional layer of protection. If the real-time Flink job goes down, the reconciliation process can recover the missing counts. Because it sends total counts rather than only late events, it also limits the impact of downtime to the fixed interval defined for reconciliation.

So instead of expecting the real-time pipeline to handle every possible failure by itself, Zomato added a second process that could periodically verify and correct the results.

Why Move to Flink SQL?

Alongside the architecture changes, Zomato upgraded from Flink 1.8 to Flink 1.17 and moved from Java-based processing to Flink SQL. The older Flink version was missing several performance improvements and bug fixes, which contributed to stability and maintenance challenges.

The move to Flink SQL also improved state management and simplified the implementation, making the code easier to maintain. The new system also made deployments more flexible. Zomato could experiment with new business logic more easily, something that had previously been difficult and time-consuming.

A Safer Migration

Zomato didn't switch the production system overnight. The migration happened in phases.

First, the new pipeline was deployed in shadow mode, running alongside the existing system. This allowed the team to compare the outputs and validate the new implementation before making it responsible for production traffic.

During this phase, the team found issues such as a timezone mismatch and fixed them before continuing. Once the new system had been validated, Zomato gradually scaled it and eventually completed the production migration with minimal disruption.

The Results

The redesign produced a major change in the system's operational profile.

The Flink state dropped from 150 GB to just 500 MB, a reduction of more than 99%.

Infrastructure costs also dropped by approximately $3,000 per month.

And since the migration, Zomato reports zero downtime for the system.

The system also moved from Flink Java 1.8 to Flink SQL 1.17, bringing performance and maintainability improvements.

The key idea wasn't simply to make Flink handle more state.

It was to avoid keeping that state in the first place by moving late-event handling into a separate reconciliation process.

Key Takeaways

• Zomato reduced Flink state from 150 GB to 500 MB by handling late events separately.

• Deduplication state TTL was reduced from 24 hours to 2 hours.

• A reconciliation job acts as a safety net for late events and Flink downtime.

• The real-time pipeline uses Flink SQL, TUMBLE windows, and a custom deduplication UDF.

• Watermarks allow slightly late events to be processed while very late events are handled by reconciliation.

• The migration from Flink 1.8 Java to Flink SQL 1.17 improved performance and maintainability.

• The redesigned system delivered $3,000 monthly infrastructure savings and zero downtime to date.

Official blog from Zomato: Eliminating Bottlenecks in Real-Time Data Streaming: A Zomato Ads Flink Journey

By now, you must have had a clear idea of, How Zomato Reduced a 150 GB Flink State to Just 500 MB? In a nutshell, Zomato redesigned its Flink Ads feedback pipeline by separating real-time processing from late-event reconciliation, cutting state from 150 GB to 500 MB.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!