Scale Engineer

Senior Software Engineer, AI Gateway

Senior Backend Engineer · Senior · Full Time · Remote

United States · RemoteUSD 161k – 261k1w ago
Apply for this role

Opens Docker's application page

Role

What you'll do.

Join Docker as a Senior Backend Engineer to architect and build the AI Gateway, a mission-critical control plane that abstracts the complexity of deploying AI agents to production. This role requires 6+ years of backend systems experience with expertise in distributed systems, API gateway design, and secure infrastructure to build the foundational services that will enable tens of thousands of developers to deploy trustworthy AI applications at scale.

Responsibilities

  • Design and Implement Core Gateway Systems: Architect and develop critical components of the AI Gateway including the model router, MCP (Model Context Protocol) gateway, and control plane. Build abstraction layers that unify disparate model providers while maintaining a seamless developer experience. Implement intelligent routing logic that handles dynamic model selection, cost-based routing, and failover mechanisms for production-grade reliability.
  • Engineer Scalable Infrastructure for Model Orchestration: Develop distributed infrastructure systems that support model selection, auto-failover, cost optimization, and policy enforcement at scale. Design patterns and implementations that enable the gateway to handle production traffic loads from early adopters to large-scale deployments. Focus on performance optimization through intelligent caching, session summarization, rate limiting, and latency reduction strategies.
  • Build Security and Credential Management Systems: Own the implementation of secure credential storage, identity handling, and access control mechanisms for tool integrations. Design cryptographic patterns and OAuth2 implementations that are both airtight and performant. Implement policy enforcement engines that prevent unauthorized access while maintaining fast request processing and audit trails.
  • Develop OpenAI-Compatible APIs: Create developer-facing APIs that maintain compatibility with OpenAI standards and the Model Context Protocol. Design intuitive API contracts that abstract away infrastructure complexity. Ensure API stability, versioning strategy, and backward compatibility while supporting rapid feature iteration.
  • Build Observability and Telemetry Infrastructure: Architect systems for evaluation, telemetry collection, replay capabilities, and backtesting functionality for AI agent workflows. Implement comprehensive logging, tracing, and metrics collection using industry-standard observability frameworks. Provide developers with real-time and historical visibility into agent behavior and performance characteristics.
  • Lead Technical Architecture and Mentorship: Make critical architectural decisions that shape the direction of the AI Gateway platform. Mentor junior and mid-level engineers as the team scales, establishing engineering best practices and code quality standards. Contribute to roadmap planning and technical strategy, balancing short-term shipping needs with long-term architectural maintainability.
  • Performance Optimization and Production Support: Drive continuous performance tuning initiatives focused on reducing latency, improving throughput, and maintaining reliability under production loads. Implement caching strategies, database query optimization, and infrastructure-level optimizations. Participate in on-call rotations and incident response to ensure platform stability and reliability.
  • Cross-Functional Collaboration and Product Development: Partner with product, security, and infrastructure teams to align technical implementations with product vision and security requirements. Contribute to roadmap planning by bringing technical perspective to feature prioritization. Work with external partners and customers to understand real-world requirements and use cases for AI agent deployment.

Qualifications

What we look for.

Technical

  • Backend Systems Architecture

    Demonstrable expertise designing and building production-grade backend systems at scale, including experience with microservices architecture, distributed systems patterns, eventual consistency, and system reliability engineering. Understanding of trade-offs between consistency, availability, and partition tolerance.

  • API Gateway and Proxy Systems

    Proven experience building or working extensively with API gateways, reverse proxies, load balancers, or similar request routing infrastructure. Deep understanding of request/response pipelines, connection pooling, circuit breakers, and request timeout strategies.

  • Go and/or Rust Programming

    Fluency in Go or Rust (both strongly preferred) for building high-performance backend services. Demonstrated ability to write idiomatic code, understand memory management, concurrency patterns, and performance characteristics of chosen languages.

  • Cloud-Native Infrastructure

    Expertise with cloud platforms (AWS, GCP, or Azure) and cloud-native development patterns. Experience with containerization, container orchestration, and designing systems for cloud deployment. Understanding of managed services, auto-scaling, and infrastructure-as-code principles.

  • Secure Services Implementation

    Experience building secure, high-throughput services with strong understanding of authentication, authorization, encryption, and secrets management. Knowledge of OAuth 2.0/2.1, JWT, mTLS, and cryptographic principles. Experience with policy enforcement engines and zero-trust architecture patterns.

  • AI/ML Platform Fundamentals

    Familiarity with AI and machine learning platform architecture, model serving infrastructure, and LLM API patterns. Understanding of ML model deployment, versioning, A/B testing, and monitoring. Knowledge of prompt management and model selection strategies.

Education

  • Bachelor's Degree in Computer Science or Related Field

    Formal education in Computer Science, Software Engineering, or a closely related field such as Mathematics or Physics. Equivalent practical experience developing and shipping production systems will be considered in lieu of formal degree.

Experience

  • 6+ Years Backend Engineering

    Minimum six years of professional backend software engineering experience building production-grade systems. Significant hands-on experience with system design, code implementation, and operational ownership of critical infrastructure.

  • Distributed Systems at Scale

    Proven track record designing and debugging distributed systems handling high throughput and low latency requirements. Experience with scalability challenges including caching strategies, database optimization, and handling traffic spikes in production environments.

  • Security-Critical Systems

    Experience building systems where security is paramount, including credential management, access control, audit logging, and policy enforcement. Familiarity with security best practices, threat modeling, and compliance considerations in infrastructure design.

  • Infrastructure and DevOps Mindset

    Strong understanding of infrastructure concerns including observability, monitoring, alerting, incident response, and operational excellence. Experience with infrastructure automation, CI/CD pipelines, and infrastructure-as-code approaches.

Skills

Required

  • Go Programming Language

    Production-level proficiency in Go with understanding of goroutines, channels, concurrency patterns, and building performant networked services.

  • Distributed Systems Design

    Deep understanding of distributed systems concepts including consistency models, failure modes, consensus algorithms, and strategies for building reliable systems across multiple machines.

  • API Design and Development

    Expertise designing and implementing RESTful and modern APIs with focus on versioning, backward compatibility, rate limiting, authentication, and developer experience.

  • System Architecture and Scalability

    Ability to architect systems that scale horizontally, design for high availability, implement caching strategies, and optimize for latency and throughput under production loads.

  • Security Engineering Fundamentals

    Strong understanding of authentication, authorization, encryption, secret management, and secure coding practices for building systems that handle sensitive data and enforce policies.

  • Production Operations Experience

    Hands-on experience monitoring systems in production, implementing observability solutions, debugging complex issues, and participating in incident response.

Preferred

  • Rust Programming Language

    Nice to have

    Production experience with Rust including ownership model, pattern matching, and building systems-level applications with memory safety guarantees.

  • LLM and AI Platform Experience

    Nice to have

    Prior experience integrating with or building systems that work with OpenAI, Anthropic, or similar LLM providers. Understanding of model serving, inference optimization, and LLM-specific challenges.

  • Kubernetes and Container Orchestration

    Nice to have

    Hands-on experience with Kubernetes cluster design, container networking, resource management, and deploying distributed systems on Kubernetes infrastructure.

  • Policy Engines and Access Control

    Nice to have

    Experience with open-source policy engines like OPA (Open Policy Agent), or building custom authorization systems with complex policy evaluation requirements.

  • Vector Databases and RAG Architectures

    Nice to have

    Familiarity with vector databases (Pinecone, Weaviate, Milvus), retrieval-augmented generation patterns, and building systems that integrate semantic search capabilities.

  • Observability and Monitoring Systems

    Nice to have

    Experience with observability frameworks like OpenTelemetry, distributed tracing, metrics collection, and building dashboards for complex system monitoring.

  • Agent Frameworks

    Nice to have

    Familiarity with AI agent frameworks such as LangChain, CrewAI, or AutoGPT, understanding of agent orchestration patterns and multi-step reasoning workflows.

  • Model Context Protocol (MCP)

    Nice to have

    Understanding of MCP server architecture, tool definition, and building extensible systems that support standardized tool integration for AI applications.

Tech stack

Languages

GoRust

Frameworks

gRPCREST/HTTP APIsModel Context Protocol (MCP)

Databases

PostgreSQLRedisVector Databases (Pinecone, Weaviate, Milvus)

Tools

KubernetesDockerOpenTelemetryVault or Similar Secret ManagementOPA (Open Policy Agent)

Other

OAuth 2.0/2.1mTLS (Mutual TLS)Circuit Breaker PatternsLoad Balancing Algorithms

Compensation

Pay and benefits.

Base·USD 160,900 – 260,700

Equity·Stock options

Benefits

  • Remote-First Work Environment

    Work from anywhere globally with flexible location options. Physical offices available in Seattle and Paris for optional in-person collaboration and team connection events, supporting a distributed-first culture that emphasizes trust and autonomy.

  • Flexible Work Schedule

    Manage your own schedule while delivering high-quality work. Docker trusts engineers to balance productivity with personal needs, enabling deep work periods while maintaining team collaboration and communication.

  • Generous Time Off

    Comprehensive paid time off including standard vacation days, quarterly 'Whaleness Days' for personal recharge, and a designated end-of-year 'Whaleness break' to disconnect and return refreshed.

  • Home Office Support

    Monthly technology stipend of $100 USD net to support your home office setup including ergonomic furniture, monitors, keyboards, and other equipment for comfortable and productive remote work.

  • Professional Development Budget

    Annual stipend for conferences, online courses, certifications, and professional development. Docker invests in engineer growth enabling continuous skill development in emerging technologies and engineering practices.

  • Parental Leave

    16 weeks of paid parental leave available after six months of employment, supporting work-life balance during significant life transitions.

  • Equity Compensation

    Stock options for all full-time employees, enabling you to share in Docker's long-term success and financial growth as the company scales and the containerization ecosystem expands.

  • Comprehensive Health and Retirement Benefits

    Medical, dental, vision, and mental health coverage with flexible healthcare options. Retirement savings programs including 401(k) or equivalent retirement accounts varying by country of employment.

  • Paid Holidays

    Standard paid holidays and country-specific holidays observed across global offices, with additional flexibility for personal or cultural observances based on regional practices.

  • Docker Community and Swag

    Access to Docker community events and exclusive merchandise celebrating the iconic Docker brand, fostering community connection and professional networking within the open-source container ecosystem.

Full posting

Original listing.

About Docker

Docker has been one of the most loved brands in developer tooling, trusted by more than 20 million monthly users and over 20 billion container image pulls. From solo founders to the world's largest companies, developers rely on Docker to build, share, and run their applications across our suite of products including Docker Desktop, Docker Hub, and Docker Scout.

We are a globally distributed, remote-first team building the tools that define how software gets built and delivered. As AI agents redefine software development, Docker is at the center of that shift, providing the sandboxed environments, verified images, and secure infrastructure that make autonomous workflows trustworthy by default.

_______________________________________________________________________

Docker AI Gateway is our answer to the complexity of taking AI agents from prototype to production. It’s a powerful, intelligent, and secure control point that eliminates the toil of model orchestration, tool management, observability, and governance—so developers can focus on building incredible AI agents, not gluing together infrastructure.

The Gateway sits at the center of modern AI applications, offering:

  • A model and tool routing layer with built-in security and cost optimization

  • A familiar OpenAI-compatible interface and MCP server

  • Unified observability and policy enforcement

  • Auto-RAG, tool injection, session summarization, and more

We’re just getting started—and we need exceptional engineers to help us build the backbone of the future of agent-based development.

What You’ll Do

  • Design and implement core systems powering the AI Gateway, including the model router, MCP gateway, and control plane

  • Build infrastructure that supports dynamic model selection/routing, auto-failover, cost-based routing, and policy enforcement

  • Own critical capabilities such as secure credential storage, session summarization, caching, and rate limiting

  • Develop APIs for developers building with OpenAI-compatible interfaces and the Model Context Protocol

  • Build the underlying infrastructure to support evaluation, telemetry, replay, and backtesting for agents and LLM workflows

  • Lead architectural decisions and mentor engineers as the team scales

  • Performance tuning - caching, session summarization, and rate limiting that keep the gateway fast under real production load

  • Contribute to roadmap planning, technical strategy, and cross-functional alignment

Key Problems You’ll Help Solve

  • Building a unified abstraction over fundamentally different model and tool providers without leaking their differences into the developer experience

  • Making credential and identity handling for tool access both airtight and fast

  • Giving developers real-time and historical visibility into what their agents are doing, with policy enforcement that can't be bypassed

  • Keeping latency low and reliability high as traffic scales from early adopters to production fleets

Who You Are

Required Qualifications:

  • 6+ years of backend engineering experience with production-grade systems

  • Deep knowledge of distributed and highly scalable systems, cloud-native infrastructure, and API design

  • Experience building secure, high-throughput services (e.g., gateways, proxies, load balancers, policy engines)

  • Fluency in Go, and/or Rust (both preferred)

  • Familiarity with AI/ML platforms or model serving infrastructure

  • A strong product mindset—you're excited about building developer-facing tools

  • Ownership mentality with a bias for shipping, learning, and iterating

  • Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience

Bonus Qualifications:

  • Prior experience with OpenAI, Anthropic, or similar LLM APIs

  • Familiarity with RAG architectures, vector databases, or agent frameworks (e.g., LangChain, AutoGPT, CrewAI)

  • Experience with policy engines (e.g., OPA), observability frameworks (e.g., OpenTelemetry), or API gateways

  • Understanding of OAuth2.1, secret management, and cloud IAM systems

  • Experience with Kubernetes, Docker, and microservices architecture

Why Join Us?

  • Work on what’s next: The agent era is just beginning. Help define the infrastructure stack for the next generation of software.

  • Own mission-critical infrastructure: Be part of a small team building the backbone of secure, intelligent agent applications.

  • Impact at scale: Your work will touch tens of thousands of developers building AI-first applications.

  • Engineering excellence: Join a world-class engineering team that values simplicity, code quality, and deep technical thinking.

  • Remote-first culture: Work from anywhere with a team that trusts and empowers you.

Docker considers visa sponsorship on a case-by-case basis based on business needs.

 

Compensation & Equity

United States: $160,900 – $260,700 + equity

______________________________________________________________________

Posting Information

  • Open vacancy: This posting is for an existing open role.

  • AI in hiring: Docker may use AI-assisted tools during our recruiting process.

  • Interview recordings: Candidates will be invited to opt in to interview recordings to support interviewer calibration and consistent evaluations. Recordings are optional and require explicit consent.

______________________________________________________________________

Perks & Benefits

  • Remote-first by design – Work from your home, with offices in Seattle and Paris for connection and collaboration.

  • Flexibility that fits your life – We trust you to manage your schedule while delivering great work.

  • Time to recharge – Generous PTO, designated quarterly Whaleness Days, and a designated end-of-year Whaleness break.

  • Home office support – Set up your workspace for comfort and success.

  • Technology stipend – Equivalent to US$100 net per month to help support your work.

  • Learning & development – Annual stipend for conferences, courses, certifications, and continued learning.

  • Parental leave – 16 weeks of paid parental leave after six months of employment.

  • Equity for all full-time employees – Share in Docker's long-term success as we continue to grow.

  • Comprehensive benefits – Medical, retirement, and paid holidays vary by country.

  • Docker swag – Because representing the whale never gets old.

Docker is proud to be an equal opportunity employer. We are committed to building a team that reflects a broad range of backgrounds, experiences, and perspectives. We believe diverse teams build better products, make better decisions, and better serve our global community.

#LI-REMOTE

Apply for this position

Redirects to Docker's application page.

Other roles

More at Docker.

View all 22 roles