Intermediate/ Senior Software Engineer - Cortex LLM Training Platform

Senior Software Engineer · Senior · Full Time

US-WA-BellevueUSD 200k – 288k1mo ago
Apply for this role

Opens Snowflake's application page

Role

What you'll do.

Senior/Intermediate Software Engineer role on Snowflake's Cortex LLM Training Platform, focusing on building GPU-accelerated distributed systems for post-training LLM workloads. This position combines deep infrastructure expertise with ML knowledge, requiring 3-6+ years of production ML systems experience and proficiency in Kubernetes, PyTorch, and GPU compute optimization to scale enterprise-grade training and inference services.

Responsibilities

  • Full-Stack API & SDK Development: Design and build comprehensive training APIs and SDKs spanning from public-facing interfaces through control plane architecture to GPU data plane implementation, enabling customers to compose workflows ranging from supervised fine-tuning (SFT) to reinforcement learning (RL) at enterprise scale.
  • Distributed Systems Architecture: Scale multi-tenant GPU compute platforms with fault-tolerant scheduling, intelligent placement algorithms, and capacity-aware routing across regional GPU pools, decoupling GPU computation from training loops through primitive APIs that handle complex orchestration transparently.
  • Performance Optimization & Throughput Engineering: Drive end-to-end performance optimization ensuring training, inference, and RL loops maintain low latency and maximum GPU saturation under heavy concurrent load, implementing data plane responsiveness improvements and monitoring systems to sustain enterprise-grade SLAs.
  • Production ML Infrastructure Partnership: Collaborate with Snowflake Research to productionize cutting-edge training and inference techniques, transforming research breakthroughs into reliable, composable components that withstand production demands while maintaining usability for enterprise customers.
  • Reliability & Cost Efficiency: Harden complex distributed systems for production readiness, implementing comprehensive fault tolerance mechanisms, cost optimization strategies, and operational resilience patterns to ensure sustained high availability in multi-tenant GPU environments.

Qualifications

What we look for.

Technical

  • Distributed Systems & Infrastructure

    Expert-level proficiency in distributed systems design patterns, fault tolerance mechanisms, consensus algorithms, load balancing strategies, and cloud-native infrastructure including Kubernetes orchestration and container technologies.

  • GPU Computing & CUDA Programming

    Strong understanding of GPU architecture, CUDA memory models, kernel optimization, multi-GPU synchronization via NCCL, and ability to profile and optimize GPU utilization across distributed training scenarios.

  • Python & Systems Programming Languages

    Expert Python proficiency for ML infrastructure development, with solid systems programming capability in C++ or Rust for performance-critical components, distributed training frameworks, and low-latency compute scheduling.

  • Large Language Model Fundamentals

    Strong conceptual and practical understanding of LLM architectures, transformer models, post-training techniques including SFT and RLHF, model serving requirements, and inference optimization strategies for production deployments.

  • Containerization & Orchestration

    Production-grade expertise with Docker containerization, Kubernetes cluster management, multi-tenant isolation strategies, resource quotas, scheduling policies, and cloud provider GPU instance management (AWS, GCP, Azure).

  • Monitoring, Observability & Debugging

    Proficiency with distributed systems monitoring stacks (Prometheus, Grafana, ELK), distributed tracing (Jaeger, Zipkin), structured logging frameworks, and sophisticated debugging techniques across multi-layer infrastructure.

Education

  • Bachelor's Degree in Computer Science

    BS in Computer Science or closely related field (Mathematics, Computer Engineering, Physics) providing strong foundational computer science theory essential for distributed systems and infrastructure work.

  • Advanced Degree (Preferred)

    Master's or PhD in Computer Science, Machine Learning, or related field significantly strengthens candidacy, particularly with research focus on distributed systems, GPU computing, or machine learning optimization.

Experience

  • Production ML Systems Engineering

    6+ years building, deploying, and operating production machine learning systems at scale, with proven track record shipping ML infrastructure components that serve enterprise customers in distributed environments.

  • Distributed Systems Design & Implementation

    Deep expertise designing and building scalable, fault-tolerant microservices architectures with experience operating production systems on Kubernetes, including exposure to multi-node orchestration, service mesh patterns, and cloud-native deployment strategies.

  • GPU & LLM Infrastructure Specialization

    Hands-on production experience with GPU compute frameworks and LLM optimization tools including PyTorch, DeepSpeed/FSDP, Ray, CUDA/NCCL, and vLLM, with capability to debug across data layer, infrastructure layer, and GPU compute layer issues.

  • LLM Post-Training & Fine-Tuning (Preferred)

    Direct hands-on experience with large language model post-training workflows including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and model adaptation techniques, combining infrastructure expertise with practical ML modeling intuition.

  • High-Performance Systems Optimization

    Demonstrated ability to architect and implement systems optimized for reliability, throughput, and cost efficiency in production environments, with proven experience hardening complex infrastructure for sustainable performance under production stress.

Skills

Required

  • PyTorch

    Deep production experience with PyTorch framework for building and training neural networks, distributed training integration, and GPU memory optimization in large-scale ML systems.

  • Kubernetes

    Expert-level Kubernetes operational experience including cluster architecture, resource management, scheduling, StatefulSets, job orchestration, and production troubleshooting in multi-tenant environments.

  • DeepSpeed or FSDP

    Hands-on experience with distributed training frameworks like DeepSpeed or PyTorch FSDP for multi-node GPU training, optimization techniques, and inference serving at scale.

  • CUDA & GPU Programming

    Solid CUDA proficiency for GPU kernel optimization, memory management, and performance profiling, with ability to debug GPU-related bottlenecks in distributed systems.

  • Python

    Expert Python development for ML infrastructure, API design, system integration, and performance-critical components in large-scale distributed training systems.

  • Distributed Systems Architecture

    Strong architectural understanding and practical implementation of distributed computing patterns including consensus, replication, fault tolerance, and service coordination at enterprise scale.

Preferred

  • NCCL & Multi-GPU Communication

    Nice to have

    Experience optimizing NCCL collective communication operations across multiple GPUs and nodes, reducing training latency and improving cluster utilization in distributed training scenarios.

  • vLLM or LLM Serving Frameworks

    Nice to have

    Production experience with LLM serving systems like vLLM, enabling efficient inference serving with batching optimization, token generation pipelining, and throughput maximization.

  • Ray Distributed Computing

    Nice to have

    Practical experience with Ray framework for distributed ML workloads, task scheduling, distributed training integration, and resource management in heterogeneous clusters.

  • C++ Systems Programming

    Nice to have

    Strong C++ capability for implementing performance-critical infrastructure components, GPU interaction layers, and low-latency scheduling systems where Python overhead is prohibitive.

  • LLM Fine-Tuning & RLHF

    Nice to have

    Hands-on experience with LLM post-training methodologies including supervised fine-tuning, reinforcement learning from human feedback (RLHF), and model adaptation for domain-specific applications.

  • Cloud GPU Infrastructure

    Nice to have

    Experience managing cloud GPU resources across AWS (NVIDIA GPU instances), GCP (TPU/GPU), or Azure ML, including spot instance management, cost optimization, and multi-region deployment strategies.

  • Monitoring & Observability Tools

    Nice to have

    Proficiency with production monitoring stacks (Prometheus, Grafana), distributed tracing frameworks, and structured logging for debugging complex multi-layer ML infrastructure issues.

  • LLM Research Understanding

    Nice to have

    Familiarity with current LLM research directions, transformer architecture variations, novel training techniques, and ability to evaluate research papers for production feasibility and integration potential.

Tech stack

Languages

PythonC++CUDA

Frameworks

PyTorchDeepSpeedPyTorch FSDPvLLMRay

Databases

Distributed Key-Value StoresTime-Series Databases

Tools

KubernetesDockerPrometheus & GrafanaNCCLGit & CI/CD Pipelines

Other

gRPC & Protocol BuffersDistributed Systems PatternsGPU Profiling ToolsModel Serving Patterns

Compensation

Pay and benefits.

Base·USD 200,000 – 287,500

Equity·Stock options

Benefits

  • Competitive Equity & Stock Options

    Meaningful equity stake in Snowflake as part of comprehensive compensation package, aligned with company performance and long-term value creation for engineering leadership.

  • Premium Health & Wellness Coverage

    Comprehensive medical, dental, and vision insurance plans with flexible spending accounts, mental health support, wellness programs, and family coverage options.

  • Retirement Planning & 401(k)

    Employer-matched 401(k) retirement savings plan with competitive matching contributions and financial planning resources for long-term wealth building.

  • Unlimited PTO & Work-Life Balance

    Flexible unlimited paid time off policy supporting work-life integration, plus paid holidays and sabbatical options for extended breaks and personal development.

  • Remote-Friendly Work Environment

    Flexible work arrangements supporting remote and hybrid options, with modern collaboration tools and distributed team support across multiple time zones.

  • Professional Development & Learning

    Generous learning budgets for conferences, courses, and certifications; internal technical mentorship programs; and access to expert-led technical seminars and workshops.

  • GPU Compute Access & Research

    Direct access to cutting-edge GPU infrastructure for research and development, supporting continuous learning on latest distributed training techniques and LLM advances.

  • Innovation & Research Collaboration

    Opportunity to partner with Snowflake Research team on state-of-the-art LLM training techniques, with potential for research publications and academic conference contributions.

Full posting

Original listing.

At Snowflake, we are powering the era of the agentic enterprise. To usher in this new era, we seek AI-native thinkers across every function who are energized by the opportunity to reinvent how they work. You don’t just use tools; you possess an innate curiosity, treating AI as a high-trust collaborator that is core to how you solve problems and accelerate your impact. We look for low-ego individuals who thrive in dynamic and fast-moving environments and move with an experimental mindset — who rapidly test emerging capabilities to discover simpler, more powerful ways to deliver results. At Snowflake, your role isn't just to execute a function, but to help redefine the future of how work gets done.

Senior Software Engineer — Cortex Training

The Snowflake ML Platform team's mission is to let customers run their most demanding ML/AI workloads inside Snowflake. Cortex Training is our LLM post-training platform: it turns scarce, expensive GPU capacity into a simple, composable service, so customers can adapt open-weight foundation models to their own business problems while we handle the hard distributed-systems parts, including scheduling, orchestration, multi-node training and inference, fault tolerance, and throughput.

The platform already runs post-training at scale. Under the hood, it decouples GPU computation from the training loop and exposes it as primitive APIs that compose into everything from SFT to full RL workflows. You'll work alongside a team that ships fast & sweats reliability and the researchers behind DeepSpeed. We're looking for an engineer who thrives in the ML infrastructure layer and brings a solid understanding of LLMs and post-training to help us scale and grow it.

 

YOU WILL:

  • Design and build across the full stack — from the public training APIs and SDK through the control plane to the GPU data plane.

  • Scale the distributed systems that make GPU compute serverless — multi-tenant scheduling, placement, and capacity-aware routing across regional GPU pools, with fault tolerance built in.

  • Drive end-to-end performance at scale — keep the training, inference, and RL loops fast and the data plane responsive under heavy concurrent load, with GPUs kept saturated.

  • Productionize research building blocks — partner with Snowflake Research to turn state-of-the-art training and inference techniques into reliable, composable components customers can run at enterprise scale.

     

QUALIFICATIONS:

  • 3 + years (Intermediate) | 6+ years (Senior) building and shipping production ML systems

  • Strong distributed systems and infrastructure foundation — designing scalable, fault-tolerant services and operating them on Kubernetes in production.

  • Familiarity with GPU and LLM infrastructure — e.g., PyTorch, DeepSpeed/FSDP, Ray, CUDA/NCCL, vLLM; able to debug across the data, infrastructure, and GPU layers.

  • Demonstrated ability to harden complex systems for reliability, throughput, and cost efficiency.

  • BS in Computer Science or a related field (MS/PhD a plus).

  • (Bonus) Hands-on LLM post-training / modeling experience — the strongest candidates pair deep infra skills with real post-training intuition.

Snowflake is growing fast, and we’re scaling our team to help enable and accelerate our growth. We are looking for people who share our values, challenge ordinary thinking, and push the pace of innovation while building a future for themselves and Snowflake.

How do you want to make your impact?

For jobs located in the United States, please visit the job posting on the Snowflake Careers Site for salary and benefits information: careers.snowflake.com

Redirects to Snowflake's application page.

Other roles

More at Snowflake.

View all 91 roles