Scale Engineer

Staff Backend Engineer, AI Systems

Backend Engineer · Staff · Full Time · Remote

United States Remote · RemoteUSD 181k – 226k3mo ago
Apply for this role

Opens Mural's application page

Role

What you'll do.

Staff Backend Engineer role at Mural's AI Innovation Team, pioneering generative AI systems for visual collaboration. Design and build core backend platforms powering agent orchestration, durable execution, contextual memory, and tool integration. Requires 7+ years of software engineering expertise in distributed systems, production reliability, and strong AI/LLM systems knowledge.

Responsibilities

  • Design and Build Core AI Backend Systems: Architect and implement the foundational backend systems powering Mural's agent platform, including agent orchestration, durable execution, tool execution, memory layers, observability infrastructure, and evaluation systems that enable reliable agentic AI at scale.
  • Develop Scalable APIs and Services: Design and deploy scalable backend services and APIs enabling AI agents to retrieve contextual information, coordinate multi-step workflows, interact with Mural platform data, and execute actions reliably on behalf of users with high availability and performance.
  • Build Advanced Memory and Context Systems: Engineer sophisticated agent memory layers including conversation context management, product context storage, semantic retrieval mechanisms, context summarization, memory compaction strategies, and long-term context preservation for persistent agent intelligence.
  • Implement Observability and Debugging Infrastructure: Create comprehensive monitoring, debugging, and improvement infrastructure for agent behavior including distributed tracing, metrics collection, feedback loop systems, and offline evaluation frameworks to ensure production quality and continuous optimization.
  • Translate Requirements into Technical Architectures: Convert complex, ambiguous product requirements into clear backend architectures, well-defined service boundaries, robust data models, and detailed implementation plans that align technical capabilities with user value and business objectives.
  • Define Technical Strategy for Agentic AI: Contribute to and help establish the long-term technical direction, architecture patterns, and strategic vision for implementing agentic AI systems at Mural, ensuring scalability and maintainability across the platform.
  • Champion Engineering Excellence and Mentorship: Lead by example in establishing best practices for building reliable AI systems, mentoring junior and mid-level engineers, conducting code reviews, and fostering a culture of technical excellence and continuous improvement across the team.
  • Drive Team Growth and Culture: Contribute to team expansion through hiring and interview participation, implement process improvements, and foster an inclusive engineering culture emphasizing collaboration, curiosity, and knowledge sharing within the AI Innovation Team.

Qualifications

What we look for.

Technical

  • Distributed Systems Architecture

    Expert-level experience designing and operating distributed systems handling complex workflows including orchestration patterns, asynchronous processing, job execution engines, retry mechanisms, event-driven architectures, long-running process management, and sophisticated failure recovery strategies.

  • Production System Reliability

    Deep expertise in building and operating production-grade systems with strong focus on observability, reliability engineering, comprehensive testing strategies, security practices, maintainability patterns, and incident response protocols.

  • Backend API and Service Design

    Strong capability in designing scalable RESTful APIs, microservices architectures, service boundaries, and data models that support high-performance, reliable backend systems serving diverse client applications.

  • LLM and AI Systems Knowledge

    Practical familiarity with large language models, agentic AI systems, and related concepts including tool use patterns, context retrieval mechanisms, model evaluation techniques, and multi-model orchestration strategies.

  • Node.js Backend Development

    Professional proficiency with Node.js for building scalable backend services, understanding async patterns, event-driven architecture, and integrating with modern cloud platforms and third-party AI services.

  • Database and Data Modeling

    Strong experience with MongoDB and NoSQL databases, designing flexible schemas, managing large-scale data, implementing efficient queries, and understanding trade-offs between schema flexibility and data integrity.

  • Cloud Services Integration

    Experience integrating with Azure cloud services, Azure OpenAI APIs, and managing cloud-based infrastructure for AI applications including API rate limiting, cost optimization, and multi-service orchestration.

Education

  • Bachelor's Degree in Computer Science or Related Field

    Formal education in Computer Science, Software Engineering, or equivalent technical discipline providing foundational knowledge in data structures, algorithms, distributed systems, and software architecture principles.

Experience

  • Senior Software Engineering

    Minimum 7+ years of progressive software engineering experience designing, building, and operating reliable production systems with demonstrated impact on scalable services, complex system architectures, and cross-functional collaboration.

  • Distributed Systems at Scale

    Proven experience architecting and maintaining distributed systems handling significant complexity, including workflow orchestration, asynchronous job processing, event-driven systems, and comprehensive failure handling across multiple services.

  • Production System Operations

    Hands-on experience building and maintaining production systems with emphasis on observability, incident response, security hardening, performance optimization, and long-term system reliability and maintenance.

  • AI/LLM Systems Implementation

    Direct experience working with AI systems, LLMs, or agentic platforms, understanding concepts such as tool use, context retrieval, model evaluation, and working effectively with AI-powered features in production.

  • Cross-Functional Collaboration

    Experience collaborating with frontend engineers, product managers, and designers, preferably with full-stack exposure or deep involvement in product-facing experiences to understand end-to-end system implications.

Skills

Required

  • Distributed Systems Design

    Architecture and implementation of distributed systems including service orchestration, event-driven processing, and failure recovery mechanisms.

  • Node.js and Backend Development

    Production-grade backend development using Node.js with expertise in async patterns, scalable APIs, and integration with third-party services.

  • MongoDB NoSQL Database

    Design and optimization of MongoDB schemas, indexing strategies, and handling large-scale document data with focus on performance and scalability.

  • System Architecture and Design

    Ability to design well-defined service architectures, clear service boundaries, robust data models, and scalable system implementations addressing complex requirements.

  • Production Observability and Monitoring

    Implementation of comprehensive monitoring, logging, tracing, and metrics collection to ensure system reliability and enable rapid incident diagnosis and resolution.

  • LLM and AI Systems Knowledge

    Understanding of large language models, agent-based systems, tool use, context retrieval, evaluation frameworks, and practical experience integrating AI into production systems.

  • Technical Communication and Documentation

    Strong ability to articulate complex technical concepts, produce clear documentation, and communicate architectural decisions across technical and non-technical stakeholders.

  • Software Engineering Best Practices

    Expertise in testing strategies, code quality, performance optimization, security hardening, and establishing engineering excellence standards across teams.

Preferred

  • LLM Evaluation and Experimentation

    Nice to have

    Advanced experience with LLM evaluation methodologies, A/B testing frameworks, experimentation platforms, and data-driven approaches to model optimization and selection.

  • Embeddings and Semantic Search

    Nice to have

    Practical experience implementing semantic search, vector databases, embedding-based retrieval systems, and similarity-based ranking for AI-powered features.

  • Knowledge Graphs and Graph Databases

    Nice to have

    Experience designing and implementing knowledge graphs, relationship-aware data structures, entity resolution systems, and graph-based retrieval mechanisms for complex data relationships.

  • Agentic AI Frameworks

    Nice to have

    Familiarity with agent frameworks, orchestration patterns, tool calling mechanisms, and building autonomous systems that can reason and act in complex environments.

  • Azure Cloud Platform

    Nice to have

    Deep experience with Azure services, Azure OpenAI APIs, cloud architecture patterns, and building cloud-native applications with strong security and cost optimization.

  • React Frontend Integration

    Nice to have

    Understanding of React-based frontend development to effectively collaborate with frontend engineers and design APIs that serve modern web applications efficiently.

  • AI Systems Reliability and Safety

    Nice to have

    Experience implementing reliability patterns, safety checks, bias detection, and guardrails for AI systems operating at scale with focus on user trust and ethical considerations.

  • Performance Optimization and Profiling

    Nice to have

    Expertise in identifying performance bottlenecks, profiling distributed systems, optimizing database queries, and implementing caching strategies for high-performance systems.

Tech stack

Languages

TypeScript/JavaScriptPythonSQL

Frameworks

Node.jsExpress.js or SimilarReact

Databases

MongoDBVector Database or Embeddings Storage

Tools

Azure OpenAI APIDistributed Tracing and ObservabilityAsync Task QueuesGit and Version ControlCI/CD and Deployment PlatformsContainerization and Orchestration

Other

Agent Orchestration PatternsRetrieval-Augmented Generation (RAG)Event-Driven ArchitectureMemory Management for LLMsEvaluation Frameworks for AI Systems

Compensation

Pay and benefits.

Base·USD 181,000 – 226,000

Equity·Stock options

Benefits

  • Remote-First Collaboration

    Work with a globally distributed team using asynchronous communication and collaboration tools, maintaining flexibility in work location while being part of a cohesive engineering culture.

  • Comprehensive Health and Wellness

    Medical, dental, and vision insurance coverage with company contributions supporting employee health and well-being across physical and mental health needs.

  • Retirement and Financial Planning

    Competitive 401(k) retirement plan with company matching contributions and financial planning resources to support long-term wealth building.

  • Professional Development

    Learning and development budget for conferences, courses, certifications, and technical skill development to stay current with emerging technologies and industry practices.

  • Generous Time Off

    Unlimited paid time off (PTO) or competitive vacation allowance, plus paid holidays, enabling work-life balance and personal wellness throughout the year.

  • Stock Options and Equity

    Equity participation in Mural as a growing company, aligning employee interests with long-term company success and providing wealth-building opportunities.

  • Flexible Work Environment

    Supportive remote-first culture with flexibility in work arrangements, recognizing productivity and outcomes rather than rigid schedules or location requirements.

  • Parental Leave

    Paid parental leave supporting employees during important life transitions and family-building decisions.

Full posting

Original listing.

ABOUT THE TEAM

The AI Innovation Team at Mural is pioneering how generative AI transforms visual collaboration and decision-making. We’re a remote-first group of engineers, designers, and product thinkers focused on helping teams work together more effectively. Our goal isn’t to replace human creativity. It’s to amplify it, building AI that enhances how people align, communicate, and make decisions visually.

 

YOUR MISSION

You will design and build the core AI systems and platforms that enable Mural’s next wave of agentic, AI-driven collaboration experiences.

Rather than building isolated AI features, you’ll work on the core backend systems that power Mural’s agent platform, including agent orchestration, durable execution, contextual memory, tool integration, observability, and evaluation. Your work will enable intelligent agents to reason over product context, act on behalf of users, and operate reliably and safely at scale. Our stack at Mural includes Azure OpenAI, React, Node, MongoDB.

WHAT YOU'LL DO

  • Build the core backend systems that power Mural’s agent platform, including orchestration, durable execution, tool execution, memory, observability, and evaluation infrastructure

  • Design scalable services and APIs that allow AI agents to retrieve context, coordinate multi-step workflows, interact with Mural data, and act reliably on behalf of users

  • Develop the agent memory layer, including systems for conversation context, product context, retrieval, summarization, compaction, and long-term context management

  • Create infrastructure to monitor, debug, and improve agent behavior through traces, metrics, feedback loops, and offline evaluation

  • Translate complex, open-ended product needs into clear backend architectures, service boundaries, data models, and implementation plans that align technical capabilities with user value

  • Help define the technical direction for agentic AI at Mural, contributing to long-term architecture and strategy

  • Champion engineering excellence, mentoring others and establishing best practices for building reliable AI systems

  • Contribute to team growth through hiring, process improvements, and fostering a culture of curiosity, collaboration, and inclusivity 

WHAT YOU'LL BRING

  •  7+ years of software engineering experience, with strong expertise designing, building, and operating reliable production systems, scalable services and APIs, and well-defined system architectures

  • Experience designing and operating distributed systems for complex workflows, including orchestration, asynchronous processing, job execution, retries, event-driven architectures, long-running processes, and failure recovery

  • Familiarity with AI, LLM, or agentic systems, including concepts such as tool use, context retrieval, evaluation, or model orchestration

  • Strong focus on production quality, including observability, reliability, testing, security, maintainability, and incident response

  • Strong communication skills, with the ability to produce clear technical documentation and collaborate cross-functionally

  • Experience working full stack or collaborating closely on product-facing frontend experiences

  • Deeper AI/ML experience, especially with LLM evaluation, experimentation, embeddings, retrieval systems

  • Experience with knowledge graphs, graph-based data modeling, entity resolution, or relationship-aware retrieval systems

The base salary + variable pay for this role ranges from $181,000 - $226,000 + benefits. Compensation offered will be determined by factors such as location, level, job-related knowledge, skills, and experience. .

Equal Opportunity 

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

Apply for this position

Redirects to Mural's application page.

Other roles

More at Mural.

View all 7 roles