Senior Software Engineer, AI Evals

Sentry1 months ago

Location

San Francisco, California

Type

Full Time

Salary

USD 240,000 – 280,000

Level

Senior

Role

Senior Software Engineer

Posted

Jan 28, 2026

Full TimeSenior

The role

Summary

Sentry is seeking a Senior Software Engineer specializing in AI Evaluation to build robust infrastructure for measuring and improving AI system performance. The ideal candidate will design comprehensive evaluation frameworks, create high-quality datasets, and develop automated test harnesses to ensure the reliability and accuracy of AI-powered debugging tools.

What we look for

Technical

Programming LanguagesProficiency in Python and TypeScript for production-quality code development

AI/ML InfrastructureExperience building testing and evaluation infrastructure for complex AI/ML systems

Data HandlingExpertise in working with structured and unstructured datasets, labeling workflows, and data quality pipelines

Education

Degree RequirementBachelor's degree in Computer Science, Machine Learning, or related technical field

Experience

Professional ExperienceMinimum 5+ years of professional software engineering experience

AI EvaluationExperience with ML systems evaluation techniques, including offline metrics, online evaluation, and regression testing

Skills

Compensation & benefits

Salary

USD 240,000 – 280,000 (annual)

Stock options

Available

Benefits

Hybrid Work Model

Flexible work arrangement with in-office collaboration on Mondays, Tuesdays, and Thursdays

Competitive Compensation

Salary range of $240,000 to $280,000 USD with potential equity grants

Interview process

1
Initial Screening — Preliminary review of application and qualifications
2
Technical Assessment — Evaluation of technical skills and AI evaluation expertise
3
Interview Rounds — Multiple interviews with engineering and product leadership teams