Staff Software Engineer - Compiler & Spark Runtime

Staff Software Engineer · Staff · Full Time

US-CA-Menlo ParkUSD 236k – 339k1w ago
Apply for this role

Opens Snowflake's application page

Role

What you'll do.

Join Snowflake's Spark Runtime Engineering Team as a Staff Software Engineer to lead the design and implementation of compiler and runtime systems that translate Apache Spark workloads into efficient execution on Snowflake's cloud-native query engine. This role combines deep technical expertise in compiler infrastructure, query-plan lowering, and runtime integration with hands-on leadership responsibilities, requiring 10+ years of software engineering experience with substantial expertise in compilers, query-processing systems, or runtime infrastructure. You'll drive critical architectural decisions, mentor engineers, and deliver production systems that improve workload performance and cost-efficiency at scale.

Responsibilities

  • Lead Spark Compiler and Runtime Architecture: Design and implement critical Spark query compilation and runtime integration systems, including intermediate representations (IRs), type resolution, semantic compatibility validation, query-plan transformations, and efficient SQL code generation. Drive architectural decisions that maintain Spark API compatibility while optimizing performance on Snowflake's cloud-native query engine.
  • Own Complex Engineering Initiatives End-to-End: Take ownership of complex projects from problem definition and technical specification through architecture design, hands-on implementation, comprehensive validation, and production rollout. Establish clear success metrics and measurable engineering outcomes aligned with customer needs and business priorities.
  • Provide Technical Leadership and Mentorship: Guide engineering teams through technical collaboration, design discussions, and code reviews rather than formal management authority. Clarify engineering trade-offs, resolve technical blockers, and strengthen team capabilities through knowledge sharing and mentorship on compiler systems, runtime architecture, and production practices.
  • Drive Performance and Cost-Efficiency Improvements: Profile and optimize end-to-end workload performance through compilation-latency reduction, metadata-access improvements, and close collaboration with query-engine teams. Validate changes against representative customer workloads and track measurable improvements in compatibility, latency, reliability, and cost-efficiency metrics.
  • Establish Correctness and Compatibility Standards: Build and enforce strong correctness practices including differential testing against Apache Spark, comprehensive regression coverage, clear error handling for unsupported behavior, and robust compatibility-test frameworks that catch edge cases and semantic divergences early.
  • Drive Production Quality and Observability: Establish production quality standards for owned compiler and runtime components through comprehensive observability, incident investigation protocols, fault recovery procedures, and safe staged rollout practices. Build monitoring, logging, and alerting systems to detect and resolve production issues proactively.
  • Cross-Functional Partnership and Prioritization: Collaborate with senior technical leaders, query-engine teams, runtime teams, and Product Management to align technical priorities, identify cross-team dependencies, scope deliverable engineering milestones, and translate customer requirements into focused technical initiatives.
  • Contribute Directly to Codebase and Code Review: Maintain hands-on involvement in implementing critical components, reviewing designs and code for quality and architectural soundness, and investigating challenging correctness, performance, and reliability issues in production systems.

Qualifications

What we look for.

Technical

  • Compiler Architecture and Optimization

    Deep expertise in compiler design including intermediate representation (IR) design, compiler optimization passes, code generation strategies, and type systems. Experience implementing compiler infrastructure for data processing or database systems preferred.

  • Relational Query Processing

    Strong understanding of relational database fundamentals, query planning and optimization, logical and physical query execution, and SQL semantics. Experience with query-plan transformation, cost-based optimization, or query rewriting systems.

  • Scala, Java, or C++ Proficiency

    Production-level proficiency in at least one systems implementation language, with demonstrated ability to write performant, maintainable code and effectively review complex implementations. JVM performance tuning and GC behavior understanding highly valuable for this role.

  • Systems Programming Fundamentals

    Strong foundation in systems concepts including concurrency patterns, memory management, performance profiling and optimization, and debugging complex production issues. Experience with lock-free algorithms, thread synchronization, or memory-efficient data structures.

  • Performance Analysis and Profiling

    Demonstrated expertise in performance analysis techniques including profiling tools, bottleneck identification, latency and throughput optimization. Experience reducing compilation latency, improving metadata access patterns, or optimizing runtime execution efficiency.

Education

  • Computer Science or Related Field

    Bachelor's degree in Computer Science, Computer Engineering, or equivalent field. Advanced degree (Master's or PhD) in compiler theory, database systems, or programming languages a plus. Equivalent practical experience in systems engineering accepted.

Experience

  • 10+ Years Software Engineering Experience

    Substantial hands-on software engineering background with proven expertise in building production systems. Required experience includes significant work on compiler infrastructure, query-processing systems, runtime systems, or database engine components.

  • Technical Leadership of Complex Initiatives

    Demonstrated track record guiding engineers through complex projects and coordinating delivery without requiring formal management authority. Experience influencing technical direction, making architectural decisions, and building consensus across engineering teams.

  • Production System Design and Implementation

    Ownership of significant architectural and implementation decisions in production systems. Experience building systems used at scale, handling real-world correctness requirements, and maintaining production quality through observability and incident response.

  • Apache Spark or Query Engine Internals

    Preferred experience with Apache Spark internals, Catalyst optimizer, Spark Connect architecture, or DataFrame APIs. Understanding of how Spark translates user code to physical execution plans and integrates with underlying compute engines.

  • Data Systems Production Hardening

    Experience technically leading data systems through customer adoption, production hardening, and staged release processes. Background building compatibility-test frameworks, differential testing, performance benchmarks, or production observability for data systems.

Skills

Required

  • Compiler Design and Implementation

    Architecture and implementation of compiler systems, including IR design, optimization passes, code generation, and type system implementation for production systems.

  • Scala and JVM Ecosystem

    Production-level proficiency in Scala programming, JVM performance characteristics, garbage collection behavior, and runtime optimization techniques commonly used in data processing frameworks.

  • Query Processing and Optimization

    Deep understanding of relational algebra, query planning, logical to physical query transformation, cost-based optimization, and SQL semantics and execution.

  • Systems Performance Engineering

    Expertise in performance analysis, profiling, optimization, memory management, concurrency patterns, and debugging production performance issues in large-scale systems.

  • Technical Leadership and Architecture

    Ability to lead complex engineering initiatives, make sound architectural decisions, mentor engineers, and communicate technical trade-offs clearly to diverse stakeholders.

  • Production Systems Ownership

    Experience owning critical production components including design, implementation, validation, observability, incident investigation, and rollout practices.

Preferred

  • Apache Spark Internals

    Nice to have

    Hands-on experience with Apache Spark codebase, Catalyst optimizer framework, Spark Connect architecture, or DataFrame API implementation and optimization.

  • Database Query Engine Development

    Nice to have

    Experience building or optimizing relational database or data warehouse query engines, including logical optimization, physical execution, or metadata management systems.

  • JVM Performance Optimization

    Nice to have

    Specialized experience optimizing JVM applications, including garbage collection tuning, compilation optimization, metadata caching, or native interop performance.

  • Test Framework Development

    Nice to have

    Experience building compatibility testing frameworks, differential testing strategies, performance benchmarking infrastructure, or production observability systems for complex data systems.

  • C++ Systems Programming

    Nice to have

    Advanced proficiency in C++ for systems-level programming, particularly relevant for understanding Snowflake's underlying query execution engine.

  • Distributed Systems and Cloud Architecture

    Nice to have

    Understanding of distributed systems concepts, cloud-native architecture patterns, and integration between compute and storage layers in modern data platforms.

Tech stack

Languages

ScalaJavaC++SQL

Frameworks

Apache SparkCatalyst Query OptimizerSpark Connect

Databases

SnowflakeRelational Database Engines

Tools

Java/Scala Compilers and Build SystemsPerformance Profiling ToolsDebugging and Testing ToolsGit and Version ControlContinuous Integration and Deployment

Other

Intermediate Representation (IR) DesignType System and Semantic ValidationQuery Plan Transformation and LoweringEnd-to-End Observability and MonitoringProduction Safety and Staged Rollouts

Compensation

Pay and benefits.

Base·USD 236,000 – 339,200

Equity·Stock options

Benefits

  • Equity and Stock Options

    Significant equity package as part of total compensation reflecting Snowflake's commitment to aligning employee and shareholder interests in a high-growth public company.

  • Health and Wellness Coverage

    Comprehensive medical, dental, and vision insurance plans with competitive coverage rates and preventative care emphasis.

  • Retirement Planning

    401(k) retirement plan with employer matching contributions to support long-term financial planning and retirement security.

  • Flexible Time Off

    Generous paid time off policy with flexible vacation and sick leave to support work-life balance and employee wellbeing.

  • Professional Development

    Learning and development opportunities including conference attendance, training programs, and internal educational initiatives to support career growth in compiler and systems engineering.

  • Remote Work Flexibility

    Flexibility to work from office locations in Bellevue, WA or Menlo Park, CA, with potential for hybrid or flexible arrangements subject to business needs.

  • Parental Leave

    Comprehensive parental leave policies supporting both maternal and paternal leave for family planning and life balance.

  • Mental Health and Wellness

    Access to mental health support, wellness programs, and employee assistance programs supporting holistic employee wellbeing.

Full posting

Original listing.

At Snowflake, we are powering the era of the agentic enterprise. To usher in this new era, we seek AI-native thinkers across every function who are energized by the opportunity to reinvent how they work. You don’t just use tools; you possess an innate curiosity, treating AI as a high-trust collaborator that is core to how you solve problems and accelerate your impact. We look for low-ego individuals who thrive in dynamic and fast-moving environments and move with an experimental mindset — who rapidly test emerging capabilities to discover simpler, more powerful ways to deliver results. At Snowflake, your role isn't just to execute a function, but to help redefine the future of how work gets done.

Staff Software Engineer – Compiler & Spark Runtime

Location: Bellevue, WA or Menlo Park, CA

About the Team

The Spark Runtime Engineering Team brings Snowflake’s performance, efficiency, and reliability to Spark workloads. Our mission is to help customers scale data processing while preserving the familiar Spark APIs and semantics they rely on. We build the compiler and runtime integration that translate Spark workloads into efficient execution on Snowflake’s cloud-native query engine. This includes tackling complex challenges in compiler infrastructure, query-plan lowering, Spark compatibility, execution integration, and end-to-end observability to deliver faster, more cost-efficient data processing at scale.

We are seeking an experienced, hands-on engineer who combines deep technical expertise with the ability to lead complex engineering initiatives and influence technical direction. You will design and build critical compiler and runtime capabilities, guide architectural decisions, and mentor engineers through technical collaboration.

Role Overview

You will own complex projects from design through implementation and production rollout, contribute directly to the codebase, review critical designs and code, and investigate challenging correctness, performance, and reliability issues. You will provide technical leadership for engineers building production compiler and runtime capabilities for Spark workloads on Snowflake.

In partnership with compiler, query engine, runtime, and product teams, you will translate customer needs into focused technical priorities and measurable engineering outcomes. You will lead through technical judgment, hands-on execution, mentorship, and influence—not through formal reporting relationships.

Responsibilities

  • Lead the technical design and implementation of Spark query compilation and runtime integration, including intermediate representations, type resolution, semantic compatibility, query-plan transformations, and efficient SQL generation.

  • Own complex engineering initiatives from problem definition and architecture through implementation, validation, and production rollout.

  • Provide technical leadership to project teams by clarifying engineering trade-offs, aligning contributors, reviewing designs and code, and resolving technical blockers.

  • Improve end-to-end workload performance and cost-efficiency through profiling, compilation-latency reduction, metadata-access improvements, and collaboration with query-engine teams.

  • Establish strong correctness and compatibility practices, including differential testing against Apache Spark, regression coverage, and clear handling of unsupported behavior.

  • Drive production quality for owned components, including observability, incident investigation, fault recovery, and safe rollout practices.

  • Mentor engineers through design discussions, code reviews, debugging, and knowledge sharing, strengthening the team’s technical capabilities.

  • Partner with senior technical leaders and Product Management to prioritize investments, identify dependencies, and scope deliverable engineering milestones.

  • Validate changes against representative customer workloads and track measurable improvements in compatibility, latency, reliability, and cost-efficiency.

Minimum Qualifications

  • 10+ years of hands-on software engineering experience, including substantial work on compilers, query-processing systems, or runtime infrastructure.

  • Demonstrated experience serving as a technical lead for complex engineering initiatives, guiding engineers and coordinating delivery without requiring formal management authority.

  • A track record of designing, implementing, and delivering production systems, with ownership of significant architectural and implementation decisions.

  • Deep expertise in at least one relevant area, such as compiler architecture and IR transformations, relational query processing, language semantics and type systems, or runtime integration.

  • Proficiency in Scala, Java, C++, or a comparable implementation language, with the ability to write and review production code.

  • Strong systems fundamentals, including concurrency, memory behavior, and performance analysis, together with experience debugging complex production issues.

  • Ability to translate technical and customer requirements into practical engineering priorities and collaborate effectively across teams and locations.

  • Strong technical communication and mentorship skills, including the ability to build alignment around architectural decisions and explain complex trade-offs.

Preferred Qualifications

  • Experience with Apache Spark internals, Catalyst, Spark Connect, or DataFrame APIs.

  • Experience with relational algebra, SQL generation, query optimization, or database execution systems.

  • Experience with Scala/JVM performance, native runtime functions, metadata caching, or compilation-performance optimization.

  • Experience building compatibility-test frameworks, differential testing, performance benchmarks, or production observability for data systems.

  • Experience technically leading systems through customer adoption, production hardening, and staged releases.

  • A degree in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.

Snowflake is growing fast, and we’re scaling our team to help enable and accelerate our growth. We are looking for people who share our values, challenge ordinary thinking, and push the pace of innovation while building a future for themselves and Snowflake.

How do you want to make your impact?

For jobs located in the United States, please visit the job posting on the Snowflake Careers Site for salary and benefits information: careers.snowflake.com

Redirects to Snowflake's application page.

Other roles

More at Snowflake.

View all 111 roles