How Xiaomi Optimized MiMo-V2.5 Inference to Make Long-Context AI More Efficient

AI Adoption has Outpaced Strategy. Fix It. (#AtlassianPartner)

Ad: Join Andrew Boyagi, Customer Chief Technology Officer, Atlassian and key industry thinkers Kelsey Hightower, Jennifer Riggins, and Nathen Harvey on November 9th for the 2026 State of Dev Ex webinar. It’s based on real research from over 1,500 people in the industry. And the webinar’s focused on developing a clear AI strategy that fixes friction and builds workflows that actually work for devs. Register today.

Register today →

As AI models handle longer conversations, analyze large codebases, and process images and videos, running them efficiently becomes increasingly difficult. The challenge isn't just how well a model understands a prompt. It's also how much memory it needs, how requests are scheduled, how quickly cached information can be reused, and how effectively GPUs stay busy.

Xiaomi encountered this challenge while building the inference system for its MiMo-V2.5 model family, which includes MiMo-V2.5 and MiMo-V2.5-Pro. The models combine Hybrid Sliding Window Attention, a Mixture of Experts architecture, and multimodal encoders to support long-context reasoning and processing across different types of data.

These architectural choices offer significant efficiency benefits. But Xiaomi found that realizing those benefits in production required changes throughout the inference pipeline. Let's discover in this blog today, how the team approached it.

The Problem: An Efficient Model Doesn't Guarantee Efficient Inference

To understand the challenge, let's start with how attention works in a large language model. When a model processes a prompt, it calculates relationships between tokens to understand the context. In traditional Full Attention, each token can attend to the entire preceding sequence. This helps the model capture long-range relationships, but the computational cost and memory requirements grow as the context becomes longer.

The model also maintains something called a Key-Value Cache, or KVCache. Instead of recalculating attention information for previously processed tokens every time it generates a new token, the model stores and reuses that information. This makes generation more efficient, but the cache itself consumes memory. With long conversations or repeated agent interactions, KVCache can become a major constraint on how many requests a GPU can handle simultaneously.

Xiaomi's MiMo-V2.5 architecture addresses part of this problem through Hybrid Sliding Window Attention (Hybrid SWA). Rather than using Full Attention in every layer, the architecture combines two approaches:

• Sliding Window Attention (SWA): Most layers attend only to a limited window of recent tokens, reducing computation and cache storage.

• Full Attention: A smaller number of layers retain access to the full sequence, preserving the ability to model long-range dependencies.

For MiMo-V2.5-Pro, it has 70 layers: 60 SWA layers and 10 Full Attention layers, with a sliding window of 128 tokens.

The theoretical advantage is substantial. Xiaomi estimates that Hybrid SWA can reduce attention computation and KVCache storage to roughly one-seventh of the Full Attention baseline in the scenarios analyzed.

But there was a catch.

The architecture could reduce memory requirements on paper while the serving infrastructure continued storing and moving far more cache data than necessary.

The team needed to redesign the system around the architecture, not simply deploy the model and expect the savings to appear.

1. Rebuilding KVCache Management Around Sliding Windows

The first challenge was how to store KVCache.

Full Attention layers need cache information for the full sequence, while SWA layers need to retain only the tokens within their sliding window. Xiaomi found that the existing cache implementation in its serving stack did not fully support this distinction. Its early SWA support could still store the full KVCache to maintain compatibility, limiting the expected memory savings.

The team introduced a dual-pool design.

Instead of managing all cached data in one pool, the system maintains separate pools for Full Attention and SWA. The Full Attention pool retains the sequence information it needs, while the SWA pool is sized around the sliding window and can independently evict tokens that are no longer required.

At the same time, the system presents a unified view of the sequence to components such as the scheduler and prefix cache. This allows the infrastructure to manage the two types of cache separately without forcing every component to understand the underlying storage details.

The result was stricter control over SWA cache storage, helping the system realize approximately sevenfold KVCache capacity efficiency in the architecture's intended design.

Making cache loading overlap with computation

Reducing the amount of cached data is only part of the solution. That data must also reach the GPU when it is needed. If the GPU has to wait for cache data to load before continuing computation, the saved memory can come at the expense of execution time.

Xiaomi introduced layerwise KVCache prefetching. Rather than waiting to load all the necessary cache data before computation proceeds, the system schedules cache loading alongside the model's layer-by-layer execution. This allows data transfers from host memory to overlap with GPU computation, reducing the time the GPU spends waiting for cache reads.

The broader lesson is that memory efficiency and execution efficiency must be designed together. A smaller cache helps, but the system must also move and reuse that cache effectively.

2. Making Prefix Caching Work Correctly

Caching becomes even more valuable when multiple requests share the same prompt prefix.

Consider an AI coding assistant that repeatedly works with the same codebase and system instructions. If several requests begin with identical tokens, the inference system may be able to reuse previously computed KVCache instead of processing the entire prefix again.

This is called prefix caching.

However, Sliding Window Attention introduces a complication. Two requests may share the same token sequence, but some of the cached tokens required by an SWA layer may already have been evicted because they fall outside its window.

A conventional cache lookup based on token equality alone could therefore report a match even when some of the underlying cache data is no longer valid.

Xiaomi addressed this by making its prefix cache aware of the sliding window. A cache match is considered reusable only to the extent that the required SWA cache entries remain valid. The system also tracks Full Attention and SWA cache positions separately, allowing each to follow its own eviction rules. This distinction matters: a prefix can still be logically reusable for Full Attention even when some of its SWA cache entries have been discarded.

The team also worked on keeping cache states consistent across GPU memory, host memory, and storage. These tiers have different capacities and access speeds, and asynchronous transfers can leave their cache states out of sync.

To improve reuse, Xiaomi refined how cache entries are repaired, retained, and evicted across these tiers. It also developed GCache, a distributed cache system that supports memory, disk, and remote storage, with data placement managed through consistent hashing. The aim was to retain useful cache data for longer without requiring expensive dedicated storage infrastructure. GCache was designed to run alongside GPU inference machines and use their available memory and NVMe storage.

According to Xiaomi, server-side KVCache hit rates averaged 93% under mainstream high-quality harness frameworks. For heavy users with sustained high-intensity usage, the rate reached 95% or higher.

A higher cache hit rate means more requests can reuse existing computation, reducing the amount of work required to process repeated context.

3. Scheduling Requests Based on Cache Reuse

Even with a more efficient cache, the inference system can lose time if requests are routed to machines that don't have the data they need.

Xiaomi developed LLM-Router, a dynamically scalable, stateless scheduler that uses Redis as centralized storage. The goal was to avoid cache scheduling degradation when a router instance failed or requests were routed through a different instance.

The router uses cache-affinity scheduling. It considers which worker already holds a request's prefix in cache, while also accounting for the worker's current load.

In simple terms, the scheduler tries to send a request somewhere that can reuse its existing cache, without overwhelming one machine simply because it has a popular prefix.

After deployment, this approach improved L2 cache hit rates by approximately 25% and per-node input throughput by approximately 30%.

The team also changed how waiting requests are prioritized.

Traditional First Come, First Served (FCFS) scheduling processes requests in arrival order. But a request with a high cache hit rate may require much less additional computation than an earlier request with little reusable cache. Making the cache-friendly request wait can waste an opportunity to improve throughput.

Xiaomi prioritized requests with fewer uncached tokens, while adding a wait-time penalty to reduce the risk of some requests being delayed indefinitely.

For longer requests, the approach reduced P90 Time to First Token (TTFT) by up to 30%. Shorter requests saw essentially unchanged TTFT.

TTFT measures the time between a request arriving and its first generated token becoming available. It's an important metric because users experience that initial wait before seeing a response.

4. Optimizing the Prefill and Decode Stages

LLM inference generally has two main stages.

Prefill processes the input prompt and builds the initial KVCache. Decode generates the response one token at a time, reusing the cache.

Each stage has different bottlenecks, so Xiaomi optimized them separately.

During prefill, the team adjusted its Expert Parallelism configuration. The MiMo-V2.5 models use a Mixture of Experts (MoE) architecture, in which only a subset of experts is activated for each token. How those experts are distributed across machines affects communication overhead, memory usage, and load balancing.

Because the original SWA cache implementation required more memory, it constrained how small the Expert Parallelism configuration could be. After optimizing cache storage, Xiaomi was able to reduce EP to half its original size, improving end-to-end performance by approximately 40%.

The team also grouped requests into three input-length buckets: 0 to 64K tokens, 64K to 256K tokens, and 256K to 1 million tokens. Grouping requests with similar lengths helps prevent short requests from being slowed down by much longer ones in the same batch.

Another optimization involved NUMA, a system architecture for accessing memory across CPU sockets. A Linux kernel setting conflicted with the inference framework's NUMA configuration, creating sporadic delays between GPU computation kernels. Disabling the conflicting setting improved end-to-end performance by approximately 10%.

During decode, the growing KVCache can limit how many requests fit in GPU memory at once. Xiaomi improved effective KVCache capacity by approximately five times through its SWA support and additional memory optimizations.

The team also improved Multi-Token Prediction (MTP), a technique that helps accelerate token generation. By enabling MTP support during prefill and adapting its cache handling, Xiaomi reported a 2.3× speedup for the first 128 generated tokens and a 1.5× speedup for tokens 128 through 256 in the scenarios described.

These changes show why optimizing inference requires attention to the entire request lifecycle. Improvements to one stage can be limited by memory, scheduling, or execution choices elsewhere in the pipeline.

5. Speeding Up Images, Audio, and Video

MiMo-V2.5 also supports multimodal inputs, which introduce additional work before the model can reason over the content.

Images and video frames must be downloaded, decoded, and processed into representations the model can consume. If these steps run sequentially or depend too heavily on the CPU, the GPU may sit idle while waiting for input data.

Xiaomi optimized its multimodal encoder pipeline by overlapping data transfer with inference, batching inputs from concurrent requests, and moving large-image preprocessing to the GPU.

It also parallelized video decoding. Instead of extracting frames from a long video sequentially, the system splits the frame-extraction work into chunks and processes them in parallel.

For a one-hour video, the reported processing time fell from 156 seconds to 23 seconds.

The team also improved how requests were routed to encoder instances. Using consistent hashing to send requests with the same cache key to the same encoder increased the multimodal cache hit rate by 30%.

Across the encoder service, throughput doubled from 15 to 30 queries per second. Average latency was approximately 78.39 ms before optimization and 80.28 ms afterward, while P90 latency improved from 100.76 ms to 82.94 ms. In other words, the team doubled throughput without a material regression in latency, and improved the reported P90 latency.

What Did Xiaomi Achieve?

The improvements did not come from one isolated optimization. They came from coordinating changes across attention, KVCache storage, distributed caching, request scheduling, prefill, decode, and multimodal processing.

Some of the reported results include:

• Approximately 7× KVCache capacity efficiency from the Hybrid SWA cache design.

• 93% average server-side KVCache hit rate, reaching 95% or higher for heavy users.

• Approximately 25% higher L2 cache hit rates and 30% higher per-node input throughput from cache-aware routing.

• Up to 30% lower P90 TTFT for longer requests through scheduling changes.

• Approximately 40% end-to-end performance improvement from reducing Expert Parallelism after cache optimization.

• 2.3× speedup for the first 128 generated tokens through MTP improvements.

• 2× multimodal encoder throughput, alongside a reduction in one-hour video processing time from 156 seconds to 23 seconds.

Xiaomi states that the combined engineering work helped realize the theoretical efficiency advantages of Hybrid SWA in production. It also says the resulting cost savings were passed on to users through API price reductions. The supplied article does not specify a numerical amount for those reductions.

Key Takeaways

• Architecture is only the starting point. The theoretical benefits of Hybrid SWA required corresponding changes to cache management and inference infrastructure.

• Memory efficiency affects throughput. Reducing KVCache requirements lets the system accommodate more concurrent requests and retain useful context.

• Caching needs correctness, not just capacity. Prefix reuse must account for which cached tokens are still valid, especially under sliding-window attention.

• Scheduling should consider both cache reuse and load. Routing requests intelligently can reduce unnecessary computation and waiting time.

• Multimodal inference has its own bottlenecks. Parallel decoding, GPU preprocessing, batching, and cache reuse can improve throughput beyond text-only optimizations.

Official blog from Xiaomi: Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit

By now, you must have had a clear idea of, How Xiaomi Optimized MiMo-V2.5 Inference to Make Long-Context AI More Efficient? In a nutshell, Xiaomi optimized MiMo-V2.5 inference by redesigning KVCache management, distributed caching, request scheduling, and the prefill, decode, and multimodal pipelines. The result was a production system better able to translate Hybrid SWA's theoretical efficiency into practical performance and cost savings.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!