# Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency) **Company:** [Perplexity AI](https://scaleengineer.com/companies/perplexity) Join Perplexity AI's Enterprise Agent Evals & Efficiency team as a Member of Technical Staff to build evaluation infrastructure and improve agent reliability across enterprise systems. You'll own complex production systems from design to deployment, developing realistic evaluation environments, automated grading systems, and optimization strategies that measure and enhance agent task success, latency, and cost efficiency. This hands-on role requires 5+ years of software engineering expertise with deep experience in LLM applications, system design, and experimental methodology to drive measurable improvements in enterprise AI automation. **Role:** Software Engineer **Seniority:** Senior **Locations:** San Francisco **Salary:** 220000–405000 USD [Apply](https://jobs.ashbyhq.com/perplexity/f3814af2-55a3-49a5-b0cb-67199a2a45fb) Canonical: https://scaleengineer.com/jobs/perplexity/member-of-technical-staff-software-engineer-enterprise-agent-evals-efficiency --- ## Responsibilities - Build Enterprise Evaluation Environments: Design and implement realistic multi-system evaluation environments that span connected applications, representative datasets, user roles, and permission models. Create complex, multi-step workflows with reproducible system state that accurately reflect production enterprise scenarios and edge cases. This includes setting up stateful test data, managing test isolation, and ensuring evaluation environments remain synchronized with production system changes. - Develop Evaluation Infrastructure and Grading Systems: Own the complete evaluation infrastructure stack including task suites, automated grading systems, and measurement frameworks that assess agent performance against enterprise work. Calibrate graders against human judgment and ground measurements in end-to-end business outcomes. Implement scoring mechanisms that capture task success rates, error recovery effectiveness, and task completion quality. - Analyze Agent Failures and Implement Improvements: Conduct root cause analysis on agent trajectories and production failures by examining decision logs, tool usage patterns, and workflow execution. Identify systemic failure modes and ship targeted improvements across agent harness components, tool integrations, and skill implementations. Prioritize improvements based on frequency and impact to maximize measurable gains in task success and reliability. - Design and Optimize Reusable Skills and Tool Interfaces: Develop reusable skill libraries and improve tool abstractions, schemas, and result handling mechanisms that enable agents to select appropriate tools, execute workflows correctly, and recover from errors efficiently. Reduce unnecessary tool calls through better tool description clarity, improved error recovery logic, and smarter tool composition patterns. - Build Model Selection and Routing Strategies: Design and evaluate model-selection and intelligent routing strategies that match each task to the optimal model variant and inference effort level. Balance quality, latency, reliability, and cost considerations through systematic experimentation and measurement. Implement dynamic routing logic that adapts to task characteristics and system resource availability. - Optimize Agent Efficiency and Cost: Implement context management strategies, skill loading optimizations, caching mechanisms, and execution strategies that reduce token consumption and cost per successful task. Maintain quality and safety standards while systematically eliminating wasted computation. Track efficiency metrics including cost per task, tokens per task, and latency improvements over time. - Establish Regression Testing and Launch Readiness: Build regression gates and launch-readiness signals that validate changes across model versions, connectors, tools, skills, and agent harness components. Create automated testing suites that catch performance regressions before production deployment. Validate that evaluation gains translate directly to measurable production improvements in real-world enterprise scenarios. ## Requirements ### education - {"name":"Bachelor's Degree in Computer Science or Related Field","description":"Formal education in computer science, software engineering, electrical engineering, mathematics, or related technical discipline, or equivalent practical experience demonstrating mastery of core computer science fundamentals."} ### technical - {"name":"Backend Language Proficiency","description":"Advanced proficiency in Python, Go, or Rust with demonstrated ability to write production-quality backend code. Experience building scalable, maintainable systems with these languages. Python expertise is particularly valuable for data processing and evaluation infrastructure."} - {"name":"LLM Application Development","description":"Hands-on experience building, evaluating, or improving LLM-based applications and agent systems, including implementing tool-use patterns and multi-step workflow orchestration. Understanding of prompt engineering, few-shot learning, and agent behavior shaping through context and tool selection."} - {"name":"System Design and Architecture","description":"Strong system design fundamentals with proven ability to architect reliable, scalable infrastructure for experimentation and data-intensive applications. Experience designing systems for high throughput, low latency, and cost efficiency. Familiarity with distributed systems concepts and trade-offs."} - {"name":"Evaluation Infrastructure and Benchmarking","description":"Deep expertise in designing evaluation frameworks, building benchmark suites, and implementing automated grading systems. Experience calibrating evaluations against ground truth and understanding bias in measurement systems. Knowledge of statistical methods for detecting significant improvements versus noise."} - {"name":"Agent Architecture and Tool Integration","description":"Hands-on experience with agent harnesses, tool interfaces, schema design, and skill libraries for complex multi-step workflows. Understanding of agent planning, execution, and error recovery patterns. Experience designing tool abstractions that enable flexible agent behavior."} - {"name":"Test Environment and Simulation Design","description":"Experience building realistic test environments and simulations that span multiple interconnected systems, datasets, permission models, and stateful workflows. Ability to create reproducible test scenarios that accurately reflect production conditions and edge cases."} - {"name":"Inference Optimization and Cost Management","description":"Knowledge of model selection, routing strategies, and inference efficiency techniques including context management, caching, token optimization, and cost-per-task optimization. Understanding of model capabilities, capabilities-cost-latency trade-offs, and dynamic routing patterns."} - {"name":"Experimental Methodology and Analytics","description":"Strong statistical judgment and experimental design skills including task representative selection, success criteria definition, failure diagnosis, and statistical significance testing. Ability to distinguish genuine improvements from noise and establish causal attribution for changes."} ### experience - {"name":"5+ Years Software or ML Engineering","description":"Minimum 5 years of professional software engineering or machine learning engineering experience with demonstrated track record of owning complex production systems through the complete lifecycle from design through deployment and ongoing operation."} - {"name":"Production System Ownership","description":"Proven experience taking full ownership of complex systems in production, including responsibility for system reliability, performance, scalability, and operational excellence. Experience managing trade-offs between feature velocity and system stability."} - {"name":"Cross-Functional Leadership","description":"Track record of leading ambiguous, cross-functional initiatives from initial architecture through production adoption. Experience setting technical direction, resolving competing priorities across teams, and raising overall engineering quality through mentorship and knowledge sharing."} - {"name":"LLM and AI Application Experience","description":"Hands-on experience building or improving LLM-based applications and agent systems in production environments. Understanding of LLM capabilities, limitations, and how to design systems that effectively leverage and compensate for model characteristics."} ## Skills ### required - {"name":"Python","description":"Advanced Python proficiency for building evaluation infrastructure, data processing pipelines, and LLM application backends. Experience with async programming, data structures, and performance optimization."} - {"name":"System Design","description":"Ability to architect scalable, reliable systems with careful attention to distributed system concepts, failure modes, monitoring, and operational concerns. Strong understanding of database design, caching strategies, and infrastructure patterns."} - {"name":"LLM Application Development","description":"Hands-on experience implementing LLM-based agents and applications including prompt design, tool integration, multi-step planning, and workflow orchestration. Deep understanding of model capabilities and limitations."} - {"name":"Evaluation and Testing","description":"Expertise designing comprehensive test suites and evaluation frameworks. Ability to define meaningful success metrics, create representative test cases, and build automated grading systems that scale."} - {"name":"Data Analysis and Statistics","description":"Proficiency with statistical analysis, experimental design, and data interpretation. Ability to identify significant patterns in data, quantify improvements with rigor, and communicate findings clearly."} - {"name":"Production Operations","description":"Experience building, deploying, and operating complex systems in production. Familiarity with monitoring, alerting, debugging tools, and incident response practices."} ### preferred - {"name":"Go or Rust","description":"Production experience with Go for concurrent systems or Rust for systems programming. These languages are valuable for building high-performance evaluation infrastructure and agent harness components."} - {"name":"Enterprise API Integration","description":"Experience integrating with enterprise SaaS platforms and building connectors for business applications. Familiarity with OAuth, permission models, and enterprise data security patterns."} - {"name":"Machine Learning Engineering","description":"Background in machine learning engineering with experience building training infrastructure, evaluating model performance, and optimizing model selection strategies."} - {"name":"Cost Optimization","description":"Track record of optimizing system costs through thoughtful architecture decisions. Experience analyzing cost drivers, implementing cost-aware strategies, and reducing per-unit costs at scale."} - {"name":"Agent Frameworks","description":"Familiarity with popular agent frameworks, reasoning engines, or open-source LLM agent libraries. Understanding of different approaches to agent design and execution patterns."} - {"name":"Distributed Systems","description":"Strong background in distributed systems including consensus algorithms, eventually consistent systems, and distributed debugging. Experience operating complex, multi-component systems."} ## Tech stack ### tools - {"name":"Logging and Monitoring","description":"Observability tools for understanding agent behavior, tracking evaluation metrics, and identifying performance regressions. Examples include logging aggregation, metric collection, and tracing systems."} - {"name":"Version Control","description":"Git-based version control for managing code, evaluation configurations, and test case evolution. Integration with CI/CD systems for automated testing."} - {"name":"Continuous Integration/Deployment","description":"CI/CD pipelines for automated testing, validation, and deployment of agent systems and evaluation infrastructure. Tools for managing evaluation gate testing."} - {"name":"Container Orchestration","description":"Container technologies and orchestration platforms like Docker and Kubernetes for managing agent execution environments and evaluation workloads at scale."} - {"name":"Cloud Platforms","description":"Cloud infrastructure services for running evaluation environments, managing compute resources, and handling data storage. Likely AWS, GCP, or similar platforms."} ### others - {"name":"Prompt Engineering","description":"Systematic approaches to designing prompts that guide agent behavior, including few-shot learning, chain-of-thought patterns, and context formatting for optimal LLM performance."} - {"name":"Enterprise Connector Architecture","description":"Design patterns for building connectors to enterprise SaaS platforms and business applications. Understanding of enterprise data models, permissions, and API patterns."} - {"name":"Workflow Orchestration","description":"Patterns and tools for orchestrating complex, multi-step workflows including error handling, retry logic, and state management across tool calls and agent steps."} - {"name":"Cost Attribution and Analysis","description":"Methods for tracking and analyzing costs in LLM-based systems including token counting, model-specific pricing, and identifying cost optimization opportunities."} ### databases - {"name":"Relational Databases","description":"PostgreSQL or similar relational databases for storing evaluation results, test cases, and task definitions. Critical for maintaining test state and evaluation history."} - {"name":"Document Databases","description":"Document stores for storing complex agent trajectories, execution logs, and flexible evaluation data. Examples include MongoDB or similar NoSQL systems."} - {"name":"Vector Databases","description":"Vector databases for similarity search and semantic evaluation. Useful for storing and retrieving relevant examples or evaluation baselines."} - {"name":"Time-Series Databases","description":"Time-series databases for tracking evaluation metrics, cost trends, and performance metrics over time."} ### languages - {"name":"Python","description":"Primary language for evaluation infrastructure, data processing, and backend systems. Used throughout the stack for LLM application development and evaluation automation."} - {"name":"Go","description":"Used for high-performance backend services and concurrent agent harness components. Valuable for building scalable evaluation infrastructure."} - {"name":"Rust","description":"Used for performance-critical components and systems programming. Useful for building efficient agent harness and tool integration layers."} ### frameworks - {"name":"LLM Agent Frameworks","description":"Agent orchestration frameworks and libraries for building multi-step workflows, handling tool use, and managing agent state. Examples include LangChain, Llamaindex, or custom agent harnesses."} - {"name":"Evaluation Frameworks","description":"Frameworks for building evaluation suites, automated grading systems, and benchmark platforms. May include custom-built infrastructure for enterprise evaluation."} - {"name":"Web Frameworks","description":"Backend web frameworks for building APIs and service-oriented architecture. Examples include FastAPI, Flask, or custom Go/Rust HTTP servers."} ## Benefits ### benefits - {"name":"Equity Compensation","description":"Stock options as part of compensation package, providing upside participation in Perplexity AI's growth as a rapidly scaling AI infrastructure company."} - {"name":"Comprehensive Health Insurance","description":"Medical, dental, and vision coverage for employees and dependents, supporting overall health and wellness."} - {"name":"401(k) Retirement Plan","description":"Company-sponsored retirement savings plan with potential employer matching contributions."} - {"name":"Paid Time Off","description":"Generous PTO policy for vacation, sick leave, and personal days to support work-life balance."} - {"name":"Professional Development","description":"Learning and development opportunities including conference attendance, course reimbursement, and internal knowledge sharing sessions to support continuous skill growth."} - {"name":"Remote Work Flexibility","description":"Flexible work arrangements supporting both remote and in-office collaboration as needed."} - {"name":"Innovation Time","description":"Dedicated time for personal projects and experimentation to foster innovation and continuous learning."} ## Compensation - **max:** 250000 - **min:** 180000 - **currency:** USD - **stockOptions:** true ## Interview process ### steps ## Full description ## **About the Role** The Enterprise Agent Evals & Efficiency team makes Perplexity's agents better at completing real work across enterprise connectors, tools, and data. Our goal is to help agents choose the right tools, use the right context, and complete complex workflows reliably, with less time and compute. This team owns the improvement loop: define what successful work looks like, build evaluations that measure it, understand where agents fail, and ship changes that improve outcomes. This is a hands-on engineering role with end-to-end ownership, from building realistic enterprise evaluation environments to shipping improvements across the agent harness, tools, and skills. You'll create complex scenarios spanning connected systems, data, and permissions, use them to identify where agents struggle, and improve the tools agents call, the skills they follow, the context they receive, and the models they use. Working with AI Research, Connectors, and Platform teams, you'll turn experimental results into measurable gains in task success, latency, and cost per successful task. The goal is not just to measure agents, but to make more enterprise work reliable and practical to automate. ## **Key Responsibilities** * Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, and permissions, with complex, multi-step workflows and reproducible system state. * Own evaluation infrastructure, task suites, and automated graders that measure how well agents complete enterprise work, calibrated against human judgment and grounded in end-to-end outcomes. * Analyze agent trajectories and production failures, identify root causes, and ship improvements across the agent harness, tools, and skills that measurably improve task success and reliability. * Develop reusable skills and improve tool descriptions, schemas, and result handling so agents can select the right tools, execute workflows correctly, and recover from errors with fewer unnecessary calls. * Build and evaluate model-selection and routing strategies that match each task to the right model and effort level, balancing quality, latency, reliability, and cost. * Improve agent efficiency through context management, skill loading, caching, and execution strategies that reduce wasted tokens and cost per successful task without compromising quality or safety. * Build regression gates and launch-readiness signals for model, connector, tool, skill, and harness changes, and validate that evaluation gains translate into better production outcomes. ## **Qualifications** * 5+ years of software engineering or machine learning engineering experience, with a track record of owning complex production systems from design through deployment and operation. * Strong system design skills, including building reliable, scalable infrastructure for experimentation, evaluation, or data-intensive applications. * Strong proficiency in a backend language such as Python, Go, or Rust, and the ability to work effectively across a multi-language production stack. * Hands-on experience building, evaluating, or improving LLM-based applications or agents, including tool use and multi-step workflows. * Deep expertise in at least one of the following: * Evaluation infrastructure, benchmark design, and automated grading. * Agent harnesses, tool interfaces, and reusable skills for complex workflows. * Realistic test environments or simulations spanning multiple systems, datasets, permissions, and stateful workflows. * Model selection, routing, and inference efficiency, including context management and caching. * Strong experimental and statistical judgment, including choosing representative tasks, defining meaningful success criteria, diagnosing failures, and distinguishing real improvements from noise. * Sound technical judgment in balancing task quality, latency, reliability, and cost, with a track record of translating evaluation findings into shipped improvements. * A track record of leading ambiguous, cross-team initiatives from architecture through production adoption, setting technical direction, and raising engineering quality through mentorship. If you’re excited about this role, we encourage you to apply even if you don’t meet every qualification. We value demonstrated impact, deep ownership, and the initiative to turn ambiguity into results.