# How LinkedIn Cut a 7-Hour Spark Pipeline Down to 3 Hours
*LinkedIn cut a 7-hour Spark pipeline to 3 hours using critical path analysis, repartitioning, broadcast joins, and Spark tuning.*
By [Rohit Lakhotia](https://scaleengineer.com/authors/rohit-lakhotia)
Published: 2026-08-31
Canonical: https://scaleengineer.com/blog/how-linkedin-cut-a-7-hour-spark-pipeline-down-to-3-hours
---
### Monitor your agents with Moyai

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/91e2cd90-bb4a-464f-86a8-d7aa6f50d3e1/Ad-creative_16x9.png?t=1787687542)

Stop finding out from your customers if your agent failed\. Start monitoring your agent traces for structural and semantic outliers and find failures before your customers do\. We provide RCA and remediation steps for your agent failures so you can build more reliable agents\. We work on your existing observability stack so no new SDK required\. Build more reliable agents with Moyai\.

[ Get Early Access](https://moyai.ai/)

Building a search engine isn't just about answering queries quickly\. Before users can search for anything, the search index itself has to be built\. And when you're powering a product like **LinkedIn Sales Navigator**, that means processing massive amounts of data every day\.

Sales Navigator helps sales professionals discover prospects, find leads, and build customer relationships\. To keep search results accurate, LinkedIn continuously processes large datasets and builds search indexes that power features like **Lead Search**, **Relationship Explorer**, **Lead Recommendations**, and several other product experiences\.

This indexing pipeline was anything but small\. It consisted of **more than 100 Spark\-based data manipulation \(DM\) jobs**, with some of the largest jobs using **around 5,000 Spark executors**\. Despite this massive infrastructure, the complete pipeline still took **6\-7 hours** to finish\.

Simply adding more compute wasn't an option because resource limits were already in place to ensure fair resource allocation across different Spark jobs\. Instead, LinkedIn had to answer a much harder question: **How do you make an already massive Spark pipeline significantly faster without simply throwing more hardware at it?**

Let's discover How LinkedIn did it in this blog today\.

# Understanding the Search Pipeline

Before looking at the optimizations, let's first understand how the search system works\. LinkedIn divides the Sales Navigator search system into **three major components**:

- **Offline Processing**
- **Nearline Processing**
- **Serving**

### Offline Processing

The offline layer is responsible for building the base search index\. It periodically reads datasets stored in **[HDFS](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)**, transforms them using large Spark jobs, and creates immutable base indexes that are later used for search\.

As part of this process, the offline pipeline also generates **watermarks**\. These timestamps indicate that all updates before a particular point in time have already been included in the generated index\.

### Nearline Processing

Of course, new profile updates, job changes, and other activities continue happening after the offline pipeline finishes\. That's where the nearline component comes in\.

It continuously processes updates that occur after the watermark and builds a **live index**, ensuring newer changes become searchable without waiting for the next offline run\.

### Serving

When a user submits a search query, it passes through several serving components before reaching the actual search indexes\. Queries are routed to the appropriate search servers, distributed across different partitions, processed by individual searchers, and finally merged into a single ranked result set before being returned to the user\.

LinkedIn also maintains **live clusters** and **dark clusters**, where responses can be compared before rolling out changes to production\.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/0ce309f5-a089-4b85-931f-68d45b39432f/image.png?t=1787307509)

While the overall architecture is important, the focus of this blog is the **offline data manipulation pipeline**, where most of the optimization work happened\.

# Why Optimizing the Pipeline Was so Difficult

At first glance, optimizing a Spark pipeline might sound straightforward\. Find the slow jobs then tune them and you’re done\! But LinkedIn's pipeline wasn't made up of just a few Spark jobs\.

It contained **more than 100 interconnected jobs**, where the output of one job often became the input of several others\. Each job itself consisted of multiple Spark stages connected through shuffle operations\. Because of these dependencies, performance problems were rarely isolated\.

If one job became slower, every downstream job waiting for its output was delayed as well\. On the other hand, making one job faster didn't always improve the total pipeline time\. In some cases, downstream jobs simply became the new bottleneck\. This made traditional job\-by\-job tuning inefficient\.

Instead of treating every Spark job equally, LinkedIn first needed to understand **which jobs actually determined the overall execution time**\.

# More Machines Weren't the Answer

One obvious way to speed up Spark workloads is to allocate more executors\. Unfortunately, that wasn't an option\. LinkedIn enforces resource limits for Spark jobs to ensure fair sharing across multiple workloads\. Many of the largest Sales Navigator jobs were already operating close to those limits\. This meant the engineering team had to improve performance **without relying on additional compute resources**\.

Another challenge came from the data itself\. Many jobs performed operations such as **unioning more than 20 datasets**, with dataset sizes ranging from just a few megabytes to **hundreds of terabytes**\.

Such large differences naturally led to uneven work distribution across Spark executors, creating another major source of performance bottlenecks\.

# Optimization Started Before Spark Tuning

Interestingly, LinkedIn didn't begin by tweaking Spark configurations\. Instead, the team first looked at the pipeline itself\. The entire data manipulation workflow can be represented as a **dependency graph**, where every node represents a Spark job and every edge represents a dependency between jobs\.

Before tuning any individual Spark application, LinkedIn first **pruned this job graph**\.

The goal was simple:

- Remove unnecessary dependencies\.
- Merge jobs whenever intermediate outputs weren't needed\.
- Eliminate unnecessary reads and writes to storage\.

One example involved a product dataset that originally passed through **three separate jobs**:

- Data preprocessing
- Data transformation
- Data postprocessing

After analyzing the dependencies, LinkedIn discovered that no other jobs depended on the intermediate outputs\. Instead of keeping three independent jobs, they merged them into a single Spark job\. This eliminated multiple storage writes and reads, reducing execution time for that part of the pipeline by **more than 30 minutes**\.

The lesson was clear\. Sometimes the biggest performance improvement doesn't come from tuning Spark, it comes from reducing unnecessary work altogether\.

# Finding the Real Bottlenecks

Even after simplifying the workflow, LinkedIn still had over a hundred Spark jobs\. Optimizing every one of them would have required enormous effort\. Instead, the team focused on the **critical path**\. The critical path is the sequence of dependent jobs that determines the total execution time of the pipeline\.

Imagine three jobs running in parallel\. If two finish in 20 minutes but one takes 95 minutes, the next dependent job can't begin until the slowest one completes\. Improving the two faster jobs won't reduce the overall pipeline duration\. Improving the slowest job will\.

By analyzing execution times alongside job dependencies, LinkedIn identified which jobs truly limited the pipeline and prioritized those for optimization\. Rather than spending weeks tuning every Spark application, engineers concentrated their efforts where they would have the greatest impact on the total execution time\.

# Repartitioning to Solve Data Skew

One of the biggest performance issues was **data skewness**\. In Spark, data is divided across multiple partitions so executors can process it in parallel\. Ideally, every executor should receive roughly the same amount of work\.

But that wasn't always happening\.

Many Sales Navigator jobs performed operations like **unioning multiple datasets** with vastly different sizes\. Some datasets were only a few megabytes, while others were hundreds of terabytes\. As a result, some executors ended up processing significantly more data than others, becoming bottlenecks for the entire job\.

LinkedIn identified skew by looking at the **Shuffle Read** metric in the Spark UI\. If only a small number of executors handled most of the shuffle data, it was a strong indication that the data wasn't evenly distributed\.

To address this, the team **repartitioned** the data using a column with high cardinality and a relatively uniform distribution\. In one of their jobs, repartitioning based on the **unique search document ID** reduced the execution time from **around 2 hours to just 30 minutes**\.

The team also discovered another issue\. Some jobs had **fewer shuffle partitions than Spark executors**, leaving many executors underutilized\. By increasing the number of shuffle partitions to better match the available executors, LinkedIn reduced another Spark job's runtime by **more than 30 minutes**\.

# Using Broadcast Joins to Reduce Network Shuffling

Another expensive operation in Spark is joining large datasets\. Normally, Spark performs a **shuffle join**, where both datasets are redistributed across the cluster so matching records end up on the same executor\. This network shuffle can become expensive, especially when dealing with massive tables\.

Instead, Spark offers another option called a **broadcast join**\. If one of the tables is relatively small, Spark can copy that table to every executor\. Each executor then performs the join locally without shuffling the larger dataset across the network\.

LinkedIn used this technique extensively\.

In one case, broadcasting a table of roughly **40 MB** reduced a Spark job's execution time from **more than one hour to around 20 minutes**\. However, broadcast joins aren't always the right choice\. Since every executor stores its own copy of the broadcast table, broadcasting very large tables increases memory usage and network traffic\.

According to LinkedIn, attempting to broadcast tables larger than **2 GB** resulted in noticeable performance degradation\. Because of this, engineers carefully considered table sizes before deciding whether to use a broadcast join\.

# Learning From Historical Job Runs

Not every optimization has to be performed manually\. LinkedIn's Spark team developed a rule\-based auto\-tuning system called **Right\-Sizing**\. Rather than relying on fixed Spark configurations, Right\-Sizing analyzes the previous **30 days** of a job's execution history and automatically adjusts settings such as executor memory and memory overhead for future runs\.

The tool also helps engineers identify recurring problems\. For example, if it repeatedly increases executor memory because of **OutOfMemory** failures, that serves as a signal that the job's memory usage should be investigated further\. Even if similar auto\-tuning tools aren't available, LinkedIn recommends using historical execution data to guide future optimization efforts\.

# Parallelizing Work with `.par`

The final optimization focused on the last stage of the data manipulation pipeline\. This stage involved a series of join operations across multiple transformed DataFrames\. Originally, these joins were executed sequentially\. Each join had to wait for the previous one to finish before the next could begin\.

To improve this, LinkedIn used Scala's `.par` operation\. Applying `.par` converts a collection into a **parallel collection**, allowing multiple join operations to be orchestrated concurrently using multiple CPU cores\. For this workload, enabling `.par` reduced the execution time by **around 30 minutes**\.

Like the other optimizations, this one also came with trade\-offs\. Because `.par` coordinates work on the Spark driver, processing a very large number of DataFrames increases orchestration overhead\. It also increases the demand for executor resources, so sufficient compute capacity is still required for the concurrent join operations\.

# The Bigger Lesson

One of the biggest takeaways from LinkedIn's work is that performance optimization isn't about applying isolated Spark tricks\. Every optimization started with understanding the pipeline itself\. The team first simplified the workflow by removing unnecessary dependencies\. Then they identified the jobs that truly limited the overall execution time\.

Only after that did they begin tuning Spark jobs using techniques like repartitioning, broadcast joins, auto\-tuning, and parallel processing\. By combining these optimizations, LinkedIn reduced the total execution time of the Sales Navigator data manipulation pipeline from **6–7 hours to around 3 hours**, allowing search indexes to be updated much more quickly\.

# Key Takeaways

- Before tuning Spark, LinkedIn first optimized the overall job graph by removing unnecessary dependencies and merging related jobs\.
- Critical path analysis helped identify the jobs that had the biggest impact on the total pipeline runtime\.
- Repartitioning reduced data skew and significantly improved executor utilization\.
- Broadcast joins minimized expensive network shuffling when joining tables of very different sizes\.
- Historical execution data powered LinkedIn's **Right\-Sizing** auto\-tuning system\.
- Parallelizing DataFrame joins using Scala's `.par` further reduced execution time\.
- Combining these optimizations reduced the Sales Navigator Spark pipeline from **6–7 hours to around 3 hours**\.

Official blog from LinkedIn: [Optimizing LinkedIn Sales Navigator’s search pipeline with Spark](https://www.linkedin.com/blog/engineering/infrastructure/optimizing-linkedin-sales-navigators-search-pipeline-with-spark)

By now, you must have had a clear idea of,** How LinkedIn Cut a 7\-Hour Spark Pipeline Down to 3 Hours? **In a nutshell, LinkedIn optimized its Spark\-based data manipulation pipeline by first identifying bottlenecks and then applying targeted techniques like repartitioning, broadcast joins, auto\-tuning, and parallel processing\. Together, these changes reduced pipeline execution time from **6–7 hours to around 3 hours**\.

**Congratulations\! You've just advanced another step in your tech journey\. Keep progressing\!**