Why Netflix Replaced Its Custom Batch Scheduler with Kueue

Whenever we open Netflix, everything feels instant. Recommendations appear immediately, your watch history is updated, and new content keeps showing up without delay. But behind that smooth experience are millions of batch jobs running every day, processing logs, generating recommendations, training machine learning models, and powering countless internal systems.

Managing these jobs isn't as simple as pressing "Run." With thousands of teams sharing the same compute infrastructure, Netflix has to decide which jobs should run first, how resources should be shared fairly, and how critical workloads always get the capacity they need. For years, Netflix relied on an in-house platform called Compute Managed Batch (CMB) to solve this problem.

As Kubernetes evolved, however, many of the capabilities CMB provided were already available through mature open-source projects. Rather than continuing to maintain a custom scheduling system, Netflix decided to migrate millions of batch jobs to Kueue, a Kubernetes-native job queuing system.

Let's discover, why they made this move and what they learned along the way in this blog today.

How Netflix Originally Managed Batch Jobs

Before understanding why Netflix switched to Kueue, let's first look at how its existing system worked.

Whenever an application wanted to run a batch workload, it didn't communicate directly with Kubernetes. Instead, it submitted the job to Compute Managed Batch (CMB), Netflix's internal platform for managing workloads that eventually complete, such as data processing, analytics pipelines, recommendation generation, and machine learning jobs.

Rather than placing every job into one large queue, CMB organized workloads using tenants.

You can think of a tenant as a logical grouping for jobs belonging to a particular team, platform, or application. This made it easier to manage resources independently while still sharing the same underlying compute infrastructure.

Netflix supported two types of tenants:

• Internal tenants, which acted as parent groups used to organize other tenants.

• Leaf tenants, which actually accepted jobs and maintained execution queues. Unlike internal tenants, leaf tenants couldn't have child tenants beneath them.

Source: Netflix TechBlog

This hierarchy gave different teams flexibility in organizing their workloads while allowing Netflix to manage compute resources at scale.

How Compute Resources Were Shared

Simply organizing jobs into tenants wasn't enough. Netflix also needed a way to decide how much compute capacity each tenant could use. Every tenant was associated with a capacity configuration, which defined the amount of compute resources available to it.

CMB supported two different types of capacity.

Reserved Capacity

Reserved capacity guaranteed compute resources for a tenant.

If a leaf tenant reserved resources, those resources belonged exclusively to that tenant and couldn't be reserved by anyone else. This ensured that important workloads always had the compute capacity they needed, regardless of what other teams were doing.

For internal tenants, reserved capacity was shared fairly across all of the leaf tenants within that subtree.

Shared Capacity

Alongside reserved capacity, Netflix also maintained a global pool of shared capacity.

Any tenant could temporarily use this pool whenever additional compute resources were available. This allowed workloads to burst beyond their reserved limits without permanently reserving more infrastructure.

The idea worked well, but it came with an important limitation.

Fair sharing only happened when a job entered the system. Once a workload started running, CMB wouldn't interrupt it even if a much higher-priority job arrived later. This meant lower-priority jobs could continue occupying resources while more important workloads had to wait.

Why Netflix Decided to Move Away from CMB

When CMB was originally built in 2018, there weren't many mature open-source solutions for managing large-scale batch workloads on Kubernetes.

Over the years, that changed.

The Kubernetes ecosystem introduced projects that already supported many of the capabilities Netflix had spent years building themselves, including:

• Fair sharing

• Hierarchical tenants

• Capacity management

• Priority queues

At the same time, adding new features to CMB was becoming increasingly difficult because it sat above Kubernetes rather than integrating directly with it. Features like preemption where lower-priority jobs can be paused to make room for more important workloads were much harder to build in this architecture.

Instead of continuing to expand their own scheduling platform, Netflix asked a different question:

Why maintain custom scheduling logic if Kubernetes already provides the building blocks?

Why Kueue?

Netflix evaluated several Kubernetes-native scheduling solutions before deciding on Kueue. One of the biggest reasons was that Kueue works alongside Kubernetes' default scheduler instead of replacing it.

This was important because Netflix already relied on Titus' existing scheduling profiles. Replacing the scheduler entirely could have disrupted how jobs were placed across clusters and reduced scheduling efficiency.

Kueue also supported several capabilities that aligned closely with Netflix's long-term goals:

• Multi-tenant quota management across different hardware.

• Native support for Kubernetes resources like Pods and Jobs.

• Support for higher-level workloads such as RayJob and RayCluster.

• Built-in features like preemption, topology-aware scheduling, and all-or-nothing scheduling.

Rather than building and maintaining these features internally, Netflix could adopt an actively evolving open-source project while continuing to use the scheduling infrastructure they had already invested in.

Migrating Millions of Batch Jobs

Choosing Kueue was only half the challenge. The harder part was migrating millions of production batch workloads without disrupting developers or existing applications.

Netflix established three goals before beginning the migration:

• Existing users shouldn't need to change how they submit jobs.

• There should be no regression in throughput or container launch rates.

• CMB's queuing and scheduling logic should be completely replaced by Kueue.

To achieve this, Netflix kept the user experience exactly the same. Applications continued submitting workloads through the same interfaces they had always used. The difference was entirely behind the scenes. Previously, CMB handled both queuing and scheduling.

After the migration, those responsibilities moved to Kueue, while Titus continued handling workload execution across Kubernetes clusters. A custom Kueue router directed jobs to Kueue-enabled Titus cells, allowing Netflix to adopt the new scheduling system without changing how users interacted with the platform.

Source: Netflix TechBlog

From a developer's perspective, almost nothing changed.

From Netflix's perspective, however, the scheduling layer became much simpler, more Kubernetes-native, and significantly easier to extend in the future.

But migrating workloads was only the beginning. Netflix also had to translate its existing tenant hierarchy into Kueue's resource model while introducing fair sharing and preemption changes that ultimately improved resource utilization across the platform. That's exactly what we'll explore in the next part.

Translating CMB into Kueue

Keeping the user experience unchanged was only part of the migration. Behind the scenes, Netflix still had to translate its existing scheduling model into concepts that Kueue understood.

Instead of creating an entirely new resource hierarchy, Netflix mapped its existing CMB tenants directly to Kueue resources.

Here's how the mapping worked:

• Internal tenants became Cohorts.

• Leaf tenants became a combination of ClusterQueues and LocalQueues.

• A tenant's capacity configuration was converted into resource flavors and nominal quotas.

This mapping meant Netflix could preserve the same organizational structure while replacing only the scheduling engine underneath.

Soucre: Netflix TechBlog

From the users' perspective, nothing changed. Teams continued organizing workloads exactly as before, while Kueue handled scheduling using Kubernetes-native resources.

Fair Sharing Became Smarter

One of the biggest improvements came from how compute resources were shared.

Earlier, CMB supported reserved and shared capacity, but fairness was only enforced when a job entered the queue. Once a workload started running, it continued until completion, even if a higher-priority workload arrived later.

Kueue introduced a much more flexible approach using preemption-based fair sharing.

Imagine Team A has reserved compute resources but isn't using them. Instead of leaving those resources idle, Kueue allows other teams to temporarily borrow them. If Team A later needs its reserved capacity back, Kueue can preempt lower-priority workloads and return the resources to their rightful owner.

This improves utilization without sacrificing reservation guarantees.

Kueue also supports priority-based preemption, allowing higher-priority workloads to interrupt lower-priority ones when necessary. This helps ensure business-critical jobs don't remain stuck behind less important workloads.

As a result, teams could safely submit more workloads, make better use of idle compute resources, and reduce the chances of important jobs waiting unnecessarily.

Lessons Netflix Learned

Migrating a critical production system is never just about replacing technology. Throughout the project, Netflix picked up several lessons that made the migration smoother.

• Keep the User Experience the Same

Instead of introducing a completely new API, Netflix retained the existing CMB interface while replacing the scheduling components underneath.

This significantly reduced risk because users didn't need to modify their applications or learn a new workflow. The migration became almost invisible to customers.

• Migrate the Hardest Customer First

Rather than beginning with small workloads, Netflix intentionally migrated its largest and most complex customer early in the project. Successfully handling the most demanding use case gave the team confidence that the remaining migrations would be much simpler. In fact, once the approach had been validated, the production rollout itself took only four weeks.

• Test for Production Scale Early

The default Kueue configuration wasn't designed for Netflix's workload volume. To support millions of batch jobs, the team increased configuration parameters such as QPS, Burst, and groupKindConcurrency well beyond their default values.

Instead of discovering these limitations during production, Netflix identified them early by running load tests in an environment that closely resembled its production Titus clusters.

Where Netflix Is Today

Today, Kueue is fully deployed across Netflix's production environment and manages millions of batch workloads. The migration didn't just simplify scheduling, it also laid the foundation for future improvements.

Netflix has already expanded fair sharing and preemption to improve utilization of reserved capacity. The team's experience with Kueue is also helping other internal engineering teams, including those building Kubernetes-native machine learning infrastructure, design their own scheduling and queuing systems.

Key Takeaways

• Netflix replaced years of custom batch scheduling logic with Kueue, a Kubernetes-native job queuing system.

• Instead of changing how users submitted workloads, Netflix kept the existing APIs and migrated the scheduling engine behind the scenes.

• Mapping CMB's tenant hierarchy directly to Kueue resources made the migration significantly simpler.

• Features like fair sharing and preemption improved resource utilization while ensuring higher-priority workloads received compute capacity when needed.

• One of the biggest lessons from this migration is that modern engineering isn't always about building new systems. Sometimes, adopting mature open-source solutions allows teams to simplify their architecture while focusing on solving higher-value problems.

Official blog from Netflix: How Netflix Simplified Batch Compute with Kueue

By now, you must have had a clear idea of, Why Netflix Replaced Its Custom Batch Scheduler with Kueue? In a nutshell, Netflix replaced its custom batch scheduler with Kueue, a Kubernetes-native job queuing system, to simplify scheduling and leverage built-in Kubernetes capabilities. By keeping the user experience unchanged while modernizing the scheduling layer, Netflix successfully migrated millions of batch jobs with minimal disruption.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!