OpenAI

Software Engineer, Data Infrastructure

OpenAI20 months ago
Location

San Francisco

Type

Full Time

Salary

USD 185,000 – 385,000

Level

Senior

Role

Data Engineer

Posted

Jun 27, 2024

Full TimeSenior

The role

Summary

OpenAI is seeking a Software Engineer for Data Infrastructure to build and operate large-scale data platforms powering their AI products and research. You'll work with massive Spark compute fleets, streaming systems, and exabyte-scale architecture while ensuring reliable, secure data access for machine learning and analytics workflows.

What you'll do

Data Infrastructure Design: Design, build, and maintain distributed compute, data orchestration, and streaming infrastructure systems
Scalability Engineering: Ensure data platform can scale by orders of magnitude while maintaining reliability and efficiency
Developer Tooling: Accelerate company productivity by building excellent data tooling and systems for engineering teams
Cross-functional Collaboration: Work with product, research and analytics teams to build technical foundations for new features
Production Operations: Own reliability of systems including participation in on-call rotation for critical incidents
Full Lifecycle Ownership: Take responsibility for architecture, implementation, production operations, and monitoring

What we look for

Technical

Data Infrastructure Experience4+ years in data infrastructure engineering or infrastructure engineering with strong data interest
Big Data PlatformsExperience supporting Spark, Kafka, Flink, Airflow, Trino, or Iceberg as platforms
Infrastructure ToolingProficiency with infrastructure tools like Terraform for cloud resource management
Distributed SystemsExperience debugging and operating large-scale distributed systems in production
System ReliabilityTrack record of building and operating scalable, reliable, and secure systems

Education

Bachelor's DegreeBachelor's degree in Computer Science, Engineering, or related technical field preferred

Experience

Production Experience4+ years of hands-on experience with production data infrastructure systems
AI/ML ContextExperience with data infrastructure in machine learning or AI environments preferred
On-call ExperienceExperience participating in on-call rotations for critical production systems

Skills

Required skills

Apache SparkHands-on experience with Spark for large-scale data processing and compute fleet management
Streaming SystemsProficiency with Kafka, Flink, or similar high-throughput streaming platforms
Infrastructure as CodeStrong experience with Terraform and cloud infrastructure automation
Distributed SystemsDeep understanding of distributed system design, debugging, and operations
Data Storage SystemsExperience with Iceberg, Delta Lake, or similar data lake technologies

Nice to have

Workflow OrchestrationExperience with Apache Airflow for data pipeline orchestration
Query EnginesFamiliarity with Trino, Presto, or similar distributed SQL query engines
ML Feature EngineeringKnowledge of ML feature stores and tools like Chronon
Cloud PlatformsExperience with AWS, GCP, or Azure for large-scale data infrastructure
Monitoring & ObservabilityExperience with monitoring tools for distributed data systems

Compensation & benefits

Salary

USD 185,000 – 385,000 (annual)

Stock options

Available

Benefits

Equity Package

Comprehensive equity compensation as part of total compensation package

Relocation Assistance

Full relocation assistance provided to new employees moving to San Francisco

Hybrid Work Model

Flexible hybrid work arrangement with 3 days per week in San Francisco office

Health Benefits

Comprehensive health, dental, and vision insurance coverage

Professional Development

Opportunities to work on cutting-edge AI infrastructure and learn from industry leaders

Interview process

  1. 1
    Initial Screening — Phone or video call with recruiting team to discuss background and interest
  2. 2
    Technical Phone Screen — 45-60 minute technical interview focusing on data infrastructure concepts and system design
  3. 3
    Technical Deep Dive — Detailed technical interview covering distributed systems, data processing frameworks, and architecture design
  4. 4
    System Design Interview — Design a large-scale data infrastructure system similar to OpenAI's data platform requirements
  5. 5
    Onsite Interviews — Full day of interviews including technical, behavioral, and team fit assessments in San Francisco office
  6. 6
    Final Interview — Discussion with senior leadership about role expectations, career goals, and cultural alignment
Apply for this position

You'll be redirected to the company's application page

OpenAI

OpenAI

View all jobs

OpenAI is an American artificial intelligence research organization developing advanced AI models like GPT. Focused on ensuring AI benefits humanity, it creates tools for natural language processing and generative AI applications.

San Francisco, California, United StatesFounded 2015openai.com

Tech Stack

Languages
PythonJavaScala
Frameworks
Apache SparkApache FlinkChronon
Databases
Apache IcebergDelta LakeApache Kafka
Tools
Apache AirflowTerraformTrino
Other
KubernetesDockerAWS/GCP

Interview Guides

5 guides available for OpenAI

Apply Now

On this page