# Intermediate/ Senior Software Engineer - Cortex LLM Training Platform
**Company:** [Snowflake](https://scaleengineer.com/companies/snowflake)
Senior/Intermediate Software Engineer role on Snowflake's Cortex LLM Training Platform, focusing on building GPU-accelerated distributed systems for post-training LLM workloads. This position combines deep infrastructure expertise with ML knowledge, requiring 3-6+ years of production ML systems experience and proficiency in Kubernetes, PyTorch, and GPU compute optimization to scale enterprise-grade training and inference services.
**Role:** Senior Software Engineer
**Seniority:** Senior
**Locations:** US-WA-Bellevue
**Salary:** 200000–287500 USD
[Apply](https://jobs.ashbyhq.com/snowflake/55e9aa63-e7b6-43b7-9ca7-8cc398dbbb00)
Canonical: https://scaleengineer.com/jobs/snowflake/intermediate-senior-software-engineer-cortex-llm-training-platform
---
## Responsibilities

- Full-Stack API & SDK Development: Design and build comprehensive training APIs and SDKs spanning from public-facing interfaces through control plane architecture to GPU data plane implementation, enabling customers to compose workflows ranging from supervised fine-tuning (SFT) to reinforcement learning (RL) at enterprise scale.
- Distributed Systems Architecture: Scale multi-tenant GPU compute platforms with fault-tolerant scheduling, intelligent placement algorithms, and capacity-aware routing across regional GPU pools, decoupling GPU computation from training loops through primitive APIs that handle complex orchestration transparently.
- Performance Optimization & Throughput Engineering: Drive end-to-end performance optimization ensuring training, inference, and RL loops maintain low latency and maximum GPU saturation under heavy concurrent load, implementing data plane responsiveness improvements and monitoring systems to sustain enterprise-grade SLAs.
- Production ML Infrastructure Partnership: Collaborate with Snowflake Research to productionize cutting-edge training and inference techniques, transforming research breakthroughs into reliable, composable components that withstand production demands while maintaining usability for enterprise customers.
- Reliability & Cost Efficiency: Harden complex distributed systems for production readiness, implementing comprehensive fault tolerance mechanisms, cost optimization strategies, and operational resilience patterns to ensure sustained high availability in multi-tenant GPU environments.

## Requirements

### education

- {"name":"Bachelor's Degree in Computer Science","description":"BS in Computer Science or closely related field (Mathematics, Computer Engineering, Physics) providing strong foundational computer science theory essential for distributed systems and infrastructure work."}
- {"name":"Advanced Degree (Preferred)","description":"Master's or PhD in Computer Science, Machine Learning, or related field significantly strengthens candidacy, particularly with research focus on distributed systems, GPU computing, or machine learning optimization."}

### technical

- {"name":"Distributed Systems & Infrastructure","description":"Expert-level proficiency in distributed systems design patterns, fault tolerance mechanisms, consensus algorithms, load balancing strategies, and cloud-native infrastructure including Kubernetes orchestration and container technologies."}
- {"name":"GPU Computing & CUDA Programming","description":"Strong understanding of GPU architecture, CUDA memory models, kernel optimization, multi-GPU synchronization via NCCL, and ability to profile and optimize GPU utilization across distributed training scenarios."}
- {"name":"Python & Systems Programming Languages","description":"Expert Python proficiency for ML infrastructure development, with solid systems programming capability in C++ or Rust for performance-critical components, distributed training frameworks, and low-latency compute scheduling."}
- {"name":"Large Language Model Fundamentals","description":"Strong conceptual and practical understanding of LLM architectures, transformer models, post-training techniques including SFT and RLHF, model serving requirements, and inference optimization strategies for production deployments."}
- {"name":"Containerization & Orchestration","description":"Production-grade expertise with Docker containerization, Kubernetes cluster management, multi-tenant isolation strategies, resource quotas, scheduling policies, and cloud provider GPU instance management (AWS, GCP, Azure)."}
- {"name":"Monitoring, Observability & Debugging","description":"Proficiency with distributed systems monitoring stacks (Prometheus, Grafana, ELK), distributed tracing (Jaeger, Zipkin), structured logging frameworks, and sophisticated debugging techniques across multi-layer infrastructure."}

### experience

- {"name":"Production ML Systems Engineering","description":"6+ years building, deploying, and operating production machine learning systems at scale, with proven track record shipping ML infrastructure components that serve enterprise customers in distributed environments."}
- {"name":"Distributed Systems Design & Implementation","description":"Deep expertise designing and building scalable, fault-tolerant microservices architectures with experience operating production systems on Kubernetes, including exposure to multi-node orchestration, service mesh patterns, and cloud-native deployment strategies."}
- {"name":"GPU & LLM Infrastructure Specialization","description":"Hands-on production experience with GPU compute frameworks and LLM optimization tools including PyTorch, DeepSpeed/FSDP, Ray, CUDA/NCCL, and vLLM, with capability to debug across data layer, infrastructure layer, and GPU compute layer issues."}
- {"name":"LLM Post-Training & Fine-Tuning (Preferred)","description":"Direct hands-on experience with large language model post-training workflows including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and model adaptation techniques, combining infrastructure expertise with practical ML modeling intuition."}
- {"name":"High-Performance Systems Optimization","description":"Demonstrated ability to architect and implement systems optimized for reliability, throughput, and cost efficiency in production environments, with proven experience hardening complex infrastructure for sustainable performance under production stress."}

## Skills

### required

- {"name":"PyTorch","description":"Deep production experience with PyTorch framework for building and training neural networks, distributed training integration, and GPU memory optimization in large-scale ML systems."}
- {"name":"Kubernetes","description":"Expert-level Kubernetes operational experience including cluster architecture, resource management, scheduling, StatefulSets, job orchestration, and production troubleshooting in multi-tenant environments."}
- {"name":"DeepSpeed or FSDP","description":"Hands-on experience with distributed training frameworks like DeepSpeed or PyTorch FSDP for multi-node GPU training, optimization techniques, and inference serving at scale."}
- {"name":"CUDA & GPU Programming","description":"Solid CUDA proficiency for GPU kernel optimization, memory management, and performance profiling, with ability to debug GPU-related bottlenecks in distributed systems."}
- {"name":"Python","description":"Expert Python development for ML infrastructure, API design, system integration, and performance-critical components in large-scale distributed training systems."}
- {"name":"Distributed Systems Architecture","description":"Strong architectural understanding and practical implementation of distributed computing patterns including consensus, replication, fault tolerance, and service coordination at enterprise scale."}

### preferred

- {"name":"NCCL & Multi-GPU Communication","description":"Experience optimizing NCCL collective communication operations across multiple GPUs and nodes, reducing training latency and improving cluster utilization in distributed training scenarios."}
- {"name":"vLLM or LLM Serving Frameworks","description":"Production experience with LLM serving systems like vLLM, enabling efficient inference serving with batching optimization, token generation pipelining, and throughput maximization."}
- {"name":"Ray Distributed Computing","description":"Practical experience with Ray framework for distributed ML workloads, task scheduling, distributed training integration, and resource management in heterogeneous clusters."}
- {"name":"C++ Systems Programming","description":"Strong C++ capability for implementing performance-critical infrastructure components, GPU interaction layers, and low-latency scheduling systems where Python overhead is prohibitive."}
- {"name":"LLM Fine-Tuning & RLHF","description":"Hands-on experience with LLM post-training methodologies including supervised fine-tuning, reinforcement learning from human feedback (RLHF), and model adaptation for domain-specific applications."}
- {"name":"Cloud GPU Infrastructure","description":"Experience managing cloud GPU resources across AWS (NVIDIA GPU instances), GCP (TPU/GPU), or Azure ML, including spot instance management, cost optimization, and multi-region deployment strategies."}
- {"name":"Monitoring & Observability Tools","description":"Proficiency with production monitoring stacks (Prometheus, Grafana), distributed tracing frameworks, and structured logging for debugging complex multi-layer ML infrastructure issues."}
- {"name":"LLM Research Understanding","description":"Familiarity with current LLM research directions, transformer architecture variations, novel training techniques, and ability to evaluate research papers for production feasibility and integration potential."}

## Tech stack

### tools

- {"name":"Kubernetes","description":"Container orchestration platform managing multi-tenant GPU workloads, job scheduling, resource allocation, and production service deployment across distributed clusters."}
- {"name":"Docker","description":"Containerization technology for packaging training jobs, inference services, and infrastructure components with consistent dependency management and reproducible environments."}
- {"name":"Prometheus & Grafana","description":"Monitoring and visualization stack for tracking distributed system metrics, GPU utilization, training progress, and infrastructure health across regional deployments."}
- {"name":"NCCL","description":"NVIDIA Collective Communications Library for optimized multi-GPU and multi-node communication, critical for reducing training latency in distributed scenarios."}
- {"name":"Git & CI/CD Pipelines","description":"Version control and continuous integration/deployment automation for infrastructure code, training job configurations, and rapid iteration on platform features."}

### others

- {"name":"gRPC & Protocol Buffers","description":"High-performance RPC framework for inter-service communication across control plane and data plane, with efficient binary serialization for low-latency distributed operations."}
- {"name":"Distributed Systems Patterns","description":"Implementation of consensus mechanisms, replication strategies, fault tolerance patterns, and capacity-aware routing for resilient multi-tenant GPU scheduling."}
- {"name":"GPU Profiling Tools","description":"NVIDIA Nsight Compute, PyTorch Profiler, and similar profiling utilities for identifying performance bottlenecks in GPU kernels and training loop optimization."}
- {"name":"Model Serving Patterns","description":"Understanding of production inference patterns including batch serving, request aggregation, KV cache management, and throughput optimization for concurrent LLM serving."}

### databases

- {"name":"Distributed Key-Value Stores","description":"Experience with Redis or similar systems for high-throughput metadata caching, job state management, and inter-process communication in control plane systems."}
- {"name":"Time-Series Databases","description":"Operational experience with Prometheus or equivalent for metrics collection and retention, supporting observability of distributed training jobs and GPU utilization patterns."}

### languages

- {"name":"Python","description":"Primary language for ML infrastructure, training orchestration, API development, and system integration across Cortex Training platform components."}
- {"name":"C++","description":"Systems programming for performance-critical GPU interaction layers, scheduling algorithms, and low-latency control plane components requiring minimal overhead."}
- {"name":"CUDA","description":"GPU kernel optimization and custom compute operations for specialized training and inference workloads, GPU memory management, and performance profiling."}

### frameworks

- {"name":"PyTorch","description":"Core deep learning framework for building, training, and inference with large language models, distributed training integration via torch.distributed and gradient aggregation."}
- {"name":"DeepSpeed","description":"Large-scale distributed training framework enabling efficient multi-node LLM training through model parallelism, pipeline parallelism, and gradient checkpointing optimization."}
- {"name":"PyTorch FSDP","description":"Fully Sharded Data Parallel framework for distributed training of large models across multiple GPUs and nodes with automatic gradient sharding and memory optimization."}
- {"name":"vLLM","description":"Fast LLM inference serving framework optimizing throughput via PagedAttention, enabling efficient batched token generation and KV cache management for production deployments."}
- {"name":"Ray","description":"Distributed computing framework for task scheduling, distributed ML workload management, and hyperparameter tuning across heterogeneous GPU clusters."}

## Benefits

### benefits

- {"name":"Competitive Equity & Stock Options","description":"Meaningful equity stake in Snowflake as part of comprehensive compensation package, aligned with company performance and long-term value creation for engineering leadership."}
- {"name":"Premium Health & Wellness Coverage","description":"Comprehensive medical, dental, and vision insurance plans with flexible spending accounts, mental health support, wellness programs, and family coverage options."}
- {"name":"Retirement Planning & 401(k)","description":"Employer-matched 401(k) retirement savings plan with competitive matching contributions and financial planning resources for long-term wealth building."}
- {"name":"Unlimited PTO & Work-Life Balance","description":"Flexible unlimited paid time off policy supporting work-life integration, plus paid holidays and sabbatical options for extended breaks and personal development."}
- {"name":"Remote-Friendly Work Environment","description":"Flexible work arrangements supporting remote and hybrid options, with modern collaboration tools and distributed team support across multiple time zones."}
- {"name":"Professional Development & Learning","description":"Generous learning budgets for conferences, courses, and certifications; internal technical mentorship programs; and access to expert-led technical seminars and workshops."}
- {"name":"GPU Compute Access & Research","description":"Direct access to cutting-edge GPU infrastructure for research and development, supporting continuous learning on latest distributed training techniques and LLM advances."}
- {"name":"Innovation & Research Collaboration","description":"Opportunity to partner with Snowflake Research team on state-of-the-art LLM training techniques, with potential for research publications and academic conference contributions."}

## Compensation

- **max:** 240000
- **min:** 180000
- **currency:** USD
- **stockOptions:** true

## Interview process

### steps

## Full description
At Snowflake, we are powering the era of the agentic enterprise. To usher in this new era, we seek AI-native thinkers across every function who are energized by the opportunity to reinvent how they work. You don’t just use tools; you possess an innate curiosity, treating AI as a high-trust collaborator that is core to how you solve problems and accelerate your impact. We look for low-ego individuals who thrive in dynamic and fast-moving environments and move with an experimental mindset — who rapidly test emerging capabilities to discover simpler, more powerful ways to deliver results. At Snowflake, your role isn't just to execute a function, but to help redefine the future of how work gets done.

# Senior Software Engineer — Cortex Training

The Snowflake ML Platform team's mission is to let customers run their most demanding ML/AI workloads inside Snowflake. Cortex Training is our LLM post-training platform: it turns scarce, expensive GPU capacity into a simple, composable service, so customers can adapt open-weight foundation models to their own business problems while we handle the hard distributed-systems parts, including scheduling, orchestration, multi-node training and inference, fault tolerance, and throughput.

The platform already runs post-training at scale. Under the hood, it decouples GPU computation from the training loop and exposes it as primitive APIs that compose into everything from SFT to full RL workflows. You'll work alongside a team that ships fast & sweats reliability and the researchers behind DeepSpeed. We're looking for an engineer who thrives in the ML infrastructure layer and brings a solid understanding of LLMs and post-training to help us scale and grow it.

## YOU WILL:

* Design and build across the full stack — from the public training APIs and SDK through the control plane to the GPU data plane.
* Scale the distributed systems that make GPU compute serverless — multi-tenant scheduling, placement, and capacity-aware routing across regional GPU pools, with fault tolerance built in.
* Drive end-to-end performance at scale — keep the training, inference, and RL loops fast and the data plane responsive under heavy concurrent load, with GPUs kept saturated.
* Productionize research building blocks — partner with Snowflake Research to turn state-of-the-art training and inference techniques into reliable, composable components customers can run at enterprise scale.

## QUALIFICATIONS:

* 3 + years (Intermediate) | 6+ years (Senior) building and shipping production ML systems
* Strong distributed systems and infrastructure foundation — designing scalable, fault-tolerant services and operating them on Kubernetes in production.
* Familiarity with GPU and LLM infrastructure — e.g., PyTorch, DeepSpeed/FSDP, Ray, CUDA/NCCL, vLLM; able to debug across the data, infrastructure, and GPU layers.
* Demonstrated ability to harden complex systems for reliability, throughput, and cost efficiency.
* BS in Computer Science or a related field (MS/PhD a plus).
* (Bonus) Hands-on LLM post-training / modeling experience — the strongest candidates pair deep infra skills with real post-training intuition.

Snowflake is growing fast, and we’re scaling our team to help enable and accelerate our growth. We are looking for people who share our values, challenge ordinary thinking, and push the pace of innovation while building a future for themselves and Snowflake.

How do you want to make your impact?

For jobs located in the United States, please visit the job posting on the Snowflake Careers Site for salary and benefits information: [careers.snowflake.com](http://careers.snowflake.com)