Principal Software Engineer (Outer Loop), Engineering Systems
Principal Engineer · Principal · Full Time
Opens Snowflake's application page
Role
What you'll do.
Principal Software Engineer responsible for setting technical direction for Snowflake's testing, validation, and release systems in an AI-first world. This role leads the Engineering Systems organization's Outer Loop team, designing agent-first test frameworks, reimagining release pipelines, and driving organization-wide quality improvements. You'll architect solutions that enable Snowflake's engineering organization to scale velocity by an order of magnitude while maintaining quality and developer experience.
Responsibilities
- Design Agent-First Test Frameworks: Architect and build next-generation test frameworks that enable AI agents to autonomously verify software correctness and performance. Develop layered verification systems allowing agents to hill climb on quality metrics, extend observability infrastructure for autonomous test diagnostics, and implement auto-remediation capabilities for detected issues.
- Reimagine Release Pipelines and Processes: Lead strategic redesign of release workflows and automation to dramatically reduce commit-to-production cycle time and release friction. Architect gating mechanisms, observability dashboards, and automation strategies that enable faster, more frequent releases while maintaining or improving quality metrics across the organization.
- Analyze and Eliminate Quality Escapes: Investigate production incidents, regressions, and customer-impacting defects to identify root causes and systemic themes. Translate incident analysis into concrete improvements in testing strategy, tooling, and engineering practices that measurably reduce escape rates organization-wide.
- Define and Execute Multi-Quarter Technical Strategy: Own the technical vision and roadmap for the Outer Loop infrastructure spanning 12+ quarters. Partner across product management, architecture, release management, and product engineering teams to establish engineering-wide standards and deliver cohesive end-to-end solutions.
- Establish Quality Metrics and Data-Driven Prioritization: Define, implement, and continuously refine metrics for test signal quality, test flakiness, release velocity, change failure rate, and quality escape rates. Use data-driven insights to identify highest-leverage investments and track progress against organizational velocity and quality targets.
- Mentor and Develop Engineering Talent: Lead technical mentorship of senior and junior engineers across the Engineering Systems organization. Set hiring bar and talent standards for the AI era, establish ambitious technical goals, and communicate both achievable objectives and breakthrough possibilities enabled by AI transformation.
- Lead Cross-Functional Technical Alignment: Influence without authority across a large, complex engineering organization. Drive adoption of new testing standards, frameworks, and practices. Present technical decisions, trade-offs, and strategic recommendations to engineering leadership and partner teams.
Qualifications
What we look for.
Technical
Large-Scale Test Infrastructure Design
Proven expertise designing and scaling test infrastructure including distributed test execution, intelligent test selection and sharding algorithms, flaky test detection and remediation systems, and hermetic test environments for complex multi-component systems.
CI/CD Pipeline Architecture
Deep hands-on experience building and optimizing continuous integration and continuous deployment pipelines, progressive delivery strategies, release gating mechanisms, automated rollback systems, and production feedback loops at scale.
Release Engineering and Quality Assurance
Comprehensive understanding of release management processes, qualification procedures, change management, incident analysis methodologies, and systematic approaches to reducing quality escapes and improving mean time to detection/resolution.
AI/ML Infrastructure and AI SRE
Hands-on experience building and scaling AI solutions including AI SRE practices. Experience with AI-powered autonomous systems, self-driving codebase systems, and integrating AI agents into engineering workflows is highly valued.
Distributed Systems Fundamentals
Strong grasp of distributed systems concepts including consistency models, fault tolerance, scalability patterns, monitoring and observability at scale, and architectural trade-offs relevant to large engineering infrastructure.
Data Analysis and Observability
Expertise in collecting, analyzing, and visualizing engineering metrics. Experience with incident analysis, root cause analysis, data-driven decision making, and establishing observability frameworks that provide actionable signals.
Education
Bachelor's Degree in Computer Science or Related Field
Four-year degree in Computer Science, Computer Engineering, Software Engineering, Mathematics, or equivalent field covering foundational algorithms, data structures, and systems principles.
Advanced Technical Knowledge
Demonstrated mastery through publications, patents, open-source contributions, or advanced course work in distributed systems, systems programming, architecture patterns, or related advanced computer science domains.
Experience
12+ Years of Industry Experience
Minimum 12 years building infrastructure, platforms, and systems at scale. Should include significant experience in developer infrastructure, test frameworks, CI/CD systems, or release engineering for large, complex codebases with hundreds or thousands of engineers.
Test Infrastructure Scaling
Proven track record designing, building, and scaling test infrastructure that powers testing for multiple product teams and engineering organizations. Demonstrated success reducing flakiness, improving signal quality, and accelerating test execution velocity.
Release Engineering at Scale
Extensive hands-on experience with modern release engineering practices including CI/CD pipeline design, deployment automation, canary deployments, progressive delivery, production monitoring, and incident response for mission-critical systems.
Cross-Functional Technical Leadership
Demonstrated ability to set technical strategy, influence engineering-wide standards, and partner with diverse stakeholders including product managers, architects, and engineering leaders to drive adoption of new practices and frameworks.
Quality-Focused Problem Solving
Track record of analyzing quality metrics, incident data, and defect patterns to identify systemic issues. Experience implementing practice changes and tooling improvements that demonstrably reduce quality escapes across organizations.
Skills
Required
Test Framework Architecture
Design and implementation of large-scale distributed test frameworks that support thousands of concurrent tests, intelligent test scheduling, and hermetic execution environments.
CI/CD Pipeline Engineering
Building and optimizing continuous integration and deployment pipelines, including orchestration systems, artifact management, deployment automation, and progressive delivery strategies.
Production Observability and Monitoring
Designing comprehensive observability systems including metrics instrumentation, log aggregation, tracing systems, dashboarding, and alerting strategies that provide actionable signals.
Incident Analysis and Root Cause Analysis
Systematic approaches to investigating production incidents, analyzing post-mortems, identifying systemic themes, and translating findings into engineering practice improvements.
Distributed Systems Design
Deep understanding of scalability patterns, fault tolerance, consistency models, and architectural trade-offs in large distributed systems used for infrastructure and tooling.
Software Architecture and Design Patterns
Ability to design scalable, maintainable systems with clean abstractions, extensible APIs, and architectural patterns suitable for large engineering organizations.
Data-Driven Decision Making
Proficiency in analyzing engineering metrics, quality data, and performance statistics to identify opportunities, prioritize investments, and measure impact of changes.
Cross-Functional Leadership and Influence
Ability to work effectively with diverse teams, influence without direct authority, communicate complex technical concepts clearly, and drive adoption of new practices and standards.
Preferred
AI/ML Infrastructure Experience
Nice to haveHands-on experience building infrastructure for AI/ML systems, AI agent orchestration, autonomous systems, or AI-powered developer tools. Experience with AI SRE and self-driving codebase projects is particularly valuable.
Large-Scale Data Warehouse Familiarity
Nice to haveUnderstanding of data warehousing architecture, analytics platforms, distributed query processing, or cloud data infrastructure. Familiarity with Snowflake's architecture is beneficial.
Open Source Contributions
Nice to haveTrack record of open-source contributions to testing frameworks, CI/CD tools, infrastructure projects, or similar systems. Demonstrated leadership in open-source communities.
Container Orchestration and Cloud Infrastructure
Nice to haveExpertise with Kubernetes, Docker, or similar container orchestration platforms. Experience with cloud infrastructure services (AWS, GCP, Azure) and infrastructure-as-code approaches.
Flaky Test Detection and Remediation
Nice to haveSpecialized experience with flaky test management systems, statistical approaches to test reliability, and techniques for making distributed test environments deterministic.
Performance Optimization
Nice to haveDemonstrated ability to profile, analyze, and optimize system performance. Experience improving build times, test execution speed, or deployment velocity in large organizations.
Developer Experience and Platform Engineering
Nice to haveBackground in building developer platforms, internal developer portals, or tools that significantly improve engineering productivity. Understanding of developer experience as a measurable outcome.
Mentorship and Technical Talent Development
Nice to haveDemonstrated success mentoring senior and junior engineers, building high-performing technical teams, and developing the next generation of engineering leaders.
Tech stack
Languages
Frameworks
Databases
Tools
Other
Compensation
Pay and benefits.
Base·USD 236,000 – 339,200
Full posting
Original listing.
At Snowflake, we are powering the era of the agentic enterprise. To usher in this new era, we seek AI-native thinkers across every function who are energized by the opportunity to reinvent how they work. You don’t just use tools; you possess an innate curiosity, treating AI as a high-trust collaborator that is core to how you solve problems and accelerate your impact. We look for low-ego individuals who thrive in dynamic and fast-moving environments and move with an experimental mindset — who rapidly test emerging capabilities to discover simpler, more powerful ways to deliver results. At Snowflake, your role isn't just to execute a function, but to help redefine the future of how work gets done.
About the Team
Engineering Systems is the team redefining how Snowflake builds. We develop AI-powered tools, platforms, and workflows that accelerate delivery, strengthen reliability, and drive deeper developer satisfaction. Now, we’re unlocking the next frontier, bringing AI transformation to boost productivity, elevate the developer experience, and set a new standard for engineering efficiency across Snowflake.
What You Will Do
As a Principal Software Engineer on the Outer Loop team in the Engineering Systems organization, you will set the technical direction for how Snowflake tests, validates, and releases software in an AI-first world. You will build the frameworks, tooling, and practices that let our engineering organization scale its velocity by an order of magnitude without compromising quality.
Build agent-first test frameworks: Extend test frameworks from humans to agents. Build layers of verifiers that agents can autonomously hill climb on correctness and performance. Extend observability for agents to detect, diagnose and auto fix test or product issues.
Improve release tooling and process: Re-imagine our release pipelines and processes to drive faster, more frequent releases with higher quality. Shorten commit-to-production cycle time, reduce release friction and toil, and build the automation, gating, and observability needed to ship confidently at scale.
Identify quality escapes and drive systemic fixes: Analyze production incidents, regressions, and customer-impacting defects to identify quality escapes and recurring themes. Turn those insights into concrete changes in testing strategy, tooling, and engineering practices that measurably reduce escapes across the organization.
Set strategy and lead across organizations: Drive multi-quarter technical strategy for the outer loop, influence engineering-wide standards, and partner with product managers, architects, release management, and product engineering teams to deliver end-to-end solutions.
Measure what matters: Define and own metrics for test signal quality, flakiness, release velocity, change failure rate, and escape rates, and use data to prioritize the highest-leverage investments.
Raise the bar: Mentor senior and junior engineers, set the bar on AI era talent, set ambitious goals and understand the art of what is possible with AI transformation.
What We're Looking For
Our ideal Principal Software Engineer will have:
12+ years of industry experience building developer infrastructure, test frameworks, CI/CD, or release systems for large, complex codebases.
Hands on experience with building and scaling AI solutions include AI SRE. Experience with building fully self-driving codebase is a plus.
A proven track record of designing and scaling test infrastructure and frameworks (e.g., distributed test execution, test selection and sharding, flaky test management, hermetic environments) used by many teams.
Deep experience with release engineering: CI/CD pipelines, progressive delivery, release gating and qualification, rollbacks, and production feedback loops.
A data-driven approach to quality: experience analyzing incidents and defect data to identify quality escapes and systemic themes, and driving practice changes that reduce them.
Strong passion for making developers highly productive, and for building platforms that let many teams move faster while keeping systems maintainable and safe.
A strategic mindset and a strong sense of what to prioritize for maximum impact; comfort leading through ambiguity and influencing without authority across a large engineering organization.
Excellent problem-solving skills and strong CS fundamentals, including data structures, algorithms, and distributed systems.
Ability to work in a team environment, collaborate well, and mentor engineers at all levels.
Excellent interpersonal, written, and oral English communication skills.
Snowflake is growing fast, and we’re scaling our team to help enable and accelerate our growth. We are looking for people who share our values, challenge ordinary thinking, and push the pace of innovation while building a future for themselves and Snowflake.
How do you want to make your impact?
For jobs located in the United States, please visit the job posting on the Snowflake Careers Site for salary and benefits information: careers.snowflake.com
Redirects to Snowflake's application page.
Other roles
More at Snowflake.
Software Engineer Intern (AI / ML) - Summer 2027
Intern
Software Engineer Intern (Core, Infrastructure & Security) — Summer 2027
Intern
Software Engineer Intern (Database Engineering) - Summer 2027
Intern
Forward Deployed Engineer, Frontier Engineering
Mid
Senior Software Engineer - Elastic Global Services
Senior