# Staff Backend Engineer, AI Systems **Company:** [Mural](https://scaleengineer.com/companies/mural) Staff Backend Engineer role at Mural's AI Innovation Team, pioneering generative AI systems for visual collaboration. Design and build core backend platforms powering agent orchestration, durable execution, contextual memory, and tool integration. Requires 7+ years of software engineering expertise in distributed systems, production reliability, and strong AI/LLM systems knowledge. **Role:** Backend Engineer **Seniority:** Staff **Locations:** United States Remote **Remote:** yes **Salary:** 181000–226000 USD [Apply](https://jobs.ashbyhq.com/mural/7a781912-3c72-4843-8810-8a9d0c550569) Canonical: https://scaleengineer.com/jobs/mural/staff-backend-engineer-ai-systems --- ## Responsibilities - Design and Build Core AI Backend Systems: Architect and implement the foundational backend systems powering Mural's agent platform, including agent orchestration, durable execution, tool execution, memory layers, observability infrastructure, and evaluation systems that enable reliable agentic AI at scale. - Develop Scalable APIs and Services: Design and deploy scalable backend services and APIs enabling AI agents to retrieve contextual information, coordinate multi-step workflows, interact with Mural platform data, and execute actions reliably on behalf of users with high availability and performance. - Build Advanced Memory and Context Systems: Engineer sophisticated agent memory layers including conversation context management, product context storage, semantic retrieval mechanisms, context summarization, memory compaction strategies, and long-term context preservation for persistent agent intelligence. - Implement Observability and Debugging Infrastructure: Create comprehensive monitoring, debugging, and improvement infrastructure for agent behavior including distributed tracing, metrics collection, feedback loop systems, and offline evaluation frameworks to ensure production quality and continuous optimization. - Translate Requirements into Technical Architectures: Convert complex, ambiguous product requirements into clear backend architectures, well-defined service boundaries, robust data models, and detailed implementation plans that align technical capabilities with user value and business objectives. - Define Technical Strategy for Agentic AI: Contribute to and help establish the long-term technical direction, architecture patterns, and strategic vision for implementing agentic AI systems at Mural, ensuring scalability and maintainability across the platform. - Champion Engineering Excellence and Mentorship: Lead by example in establishing best practices for building reliable AI systems, mentoring junior and mid-level engineers, conducting code reviews, and fostering a culture of technical excellence and continuous improvement across the team. - Drive Team Growth and Culture: Contribute to team expansion through hiring and interview participation, implement process improvements, and foster an inclusive engineering culture emphasizing collaboration, curiosity, and knowledge sharing within the AI Innovation Team. ## Requirements ### education - {"name":"Bachelor's Degree in Computer Science or Related Field","description":"Formal education in Computer Science, Software Engineering, or equivalent technical discipline providing foundational knowledge in data structures, algorithms, distributed systems, and software architecture principles."} ### technical - {"name":"Distributed Systems Architecture","description":"Expert-level experience designing and operating distributed systems handling complex workflows including orchestration patterns, asynchronous processing, job execution engines, retry mechanisms, event-driven architectures, long-running process management, and sophisticated failure recovery strategies."} - {"name":"Production System Reliability","description":"Deep expertise in building and operating production-grade systems with strong focus on observability, reliability engineering, comprehensive testing strategies, security practices, maintainability patterns, and incident response protocols."} - {"name":"Backend API and Service Design","description":"Strong capability in designing scalable RESTful APIs, microservices architectures, service boundaries, and data models that support high-performance, reliable backend systems serving diverse client applications."} - {"name":"LLM and AI Systems Knowledge","description":"Practical familiarity with large language models, agentic AI systems, and related concepts including tool use patterns, context retrieval mechanisms, model evaluation techniques, and multi-model orchestration strategies."} - {"name":"Node.js Backend Development","description":"Professional proficiency with Node.js for building scalable backend services, understanding async patterns, event-driven architecture, and integrating with modern cloud platforms and third-party AI services."} - {"name":"Database and Data Modeling","description":"Strong experience with MongoDB and NoSQL databases, designing flexible schemas, managing large-scale data, implementing efficient queries, and understanding trade-offs between schema flexibility and data integrity."} - {"name":"Cloud Services Integration","description":"Experience integrating with Azure cloud services, Azure OpenAI APIs, and managing cloud-based infrastructure for AI applications including API rate limiting, cost optimization, and multi-service orchestration."} ### experience - {"name":"Senior Software Engineering","description":"Minimum 7+ years of progressive software engineering experience designing, building, and operating reliable production systems with demonstrated impact on scalable services, complex system architectures, and cross-functional collaboration."} - {"name":"Distributed Systems at Scale","description":"Proven experience architecting and maintaining distributed systems handling significant complexity, including workflow orchestration, asynchronous job processing, event-driven systems, and comprehensive failure handling across multiple services."} - {"name":"Production System Operations","description":"Hands-on experience building and maintaining production systems with emphasis on observability, incident response, security hardening, performance optimization, and long-term system reliability and maintenance."} - {"name":"AI/LLM Systems Implementation","description":"Direct experience working with AI systems, LLMs, or agentic platforms, understanding concepts such as tool use, context retrieval, model evaluation, and working effectively with AI-powered features in production."} - {"name":"Cross-Functional Collaboration","description":"Experience collaborating with frontend engineers, product managers, and designers, preferably with full-stack exposure or deep involvement in product-facing experiences to understand end-to-end system implications."} ## Skills ### required - {"name":"Distributed Systems Design","description":"Architecture and implementation of distributed systems including service orchestration, event-driven processing, and failure recovery mechanisms."} - {"name":"Node.js and Backend Development","description":"Production-grade backend development using Node.js with expertise in async patterns, scalable APIs, and integration with third-party services."} - {"name":"MongoDB NoSQL Database","description":"Design and optimization of MongoDB schemas, indexing strategies, and handling large-scale document data with focus on performance and scalability."} - {"name":"System Architecture and Design","description":"Ability to design well-defined service architectures, clear service boundaries, robust data models, and scalable system implementations addressing complex requirements."} - {"name":"Production Observability and Monitoring","description":"Implementation of comprehensive monitoring, logging, tracing, and metrics collection to ensure system reliability and enable rapid incident diagnosis and resolution."} - {"name":"LLM and AI Systems Knowledge","description":"Understanding of large language models, agent-based systems, tool use, context retrieval, evaluation frameworks, and practical experience integrating AI into production systems."} - {"name":"Technical Communication and Documentation","description":"Strong ability to articulate complex technical concepts, produce clear documentation, and communicate architectural decisions across technical and non-technical stakeholders."} - {"name":"Software Engineering Best Practices","description":"Expertise in testing strategies, code quality, performance optimization, security hardening, and establishing engineering excellence standards across teams."} ### preferred - {"name":"LLM Evaluation and Experimentation","description":"Advanced experience with LLM evaluation methodologies, A/B testing frameworks, experimentation platforms, and data-driven approaches to model optimization and selection."} - {"name":"Embeddings and Semantic Search","description":"Practical experience implementing semantic search, vector databases, embedding-based retrieval systems, and similarity-based ranking for AI-powered features."} - {"name":"Knowledge Graphs and Graph Databases","description":"Experience designing and implementing knowledge graphs, relationship-aware data structures, entity resolution systems, and graph-based retrieval mechanisms for complex data relationships."} - {"name":"Agentic AI Frameworks","description":"Familiarity with agent frameworks, orchestration patterns, tool calling mechanisms, and building autonomous systems that can reason and act in complex environments."} - {"name":"Azure Cloud Platform","description":"Deep experience with Azure services, Azure OpenAI APIs, cloud architecture patterns, and building cloud-native applications with strong security and cost optimization."} - {"name":"React Frontend Integration","description":"Understanding of React-based frontend development to effectively collaborate with frontend engineers and design APIs that serve modern web applications efficiently."} - {"name":"AI Systems Reliability and Safety","description":"Experience implementing reliability patterns, safety checks, bias detection, and guardrails for AI systems operating at scale with focus on user trust and ethical considerations."} - {"name":"Performance Optimization and Profiling","description":"Expertise in identifying performance bottlenecks, profiling distributed systems, optimizing database queries, and implementing caching strategies for high-performance systems."} ## Tech stack ### tools - {"name":"Azure OpenAI API","description":"Cloud service providing access to OpenAI models including GPT-4, embeddings, and other LLM capabilities integrated into Mural's agentic systems."} - {"name":"Distributed Tracing and Observability","description":"Tools for collecting traces, metrics, and logs across distributed systems including solutions like DataDog, Jaeger, or similar observability platforms."} - {"name":"Async Task Queues","description":"Job execution and orchestration tools such as Bull, RabbitMQ, or similar systems for handling durable execution, retries, and long-running processes."} - {"name":"Git and Version Control","description":"Code collaboration and version control system for managing distributed team development, code review, and deployment workflows."} - {"name":"CI/CD and Deployment Platforms","description":"Continuous integration and deployment tools for automated testing, building, and deploying backend services with monitoring and rollback capabilities."} - {"name":"Containerization and Orchestration","description":"Docker for containerizing applications and Kubernetes or similar platforms for managing distributed deployments at scale."} ### others - {"name":"Agent Orchestration Patterns","description":"Architectural patterns for coordinating multiple AI agents, managing agent state, handling tool execution, and implementing reliable agent workflows."} - {"name":"Retrieval-Augmented Generation (RAG)","description":"Techniques for augmenting LLM prompts with retrieved context, implementing efficient retrieval systems, and building knowledge-aware AI agents."} - {"name":"Event-Driven Architecture","description":"Design patterns for building systems based on event emission and consumption, supporting asynchronous communication and decoupled service interactions."} - {"name":"Memory Management for LLMs","description":"Systems for managing conversation context, implementing context windows, memory compression, summarization, and long-term context retrieval for stateful AI agents."} - {"name":"Evaluation Frameworks for AI Systems","description":"Methodologies and tools for evaluating agent behavior, measuring performance, implementing feedback loops, and conducting offline evaluation of AI systems."} ### databases - {"name":"MongoDB","description":"Primary NoSQL database for Mural's stack, supporting flexible document-based schemas for storing product data, agent memory, conversation history, and context information."} - {"name":"Vector Database or Embeddings Storage","description":"Storage layer for semantic embeddings, supporting efficient similarity search and retrieval-augmented generation use cases for AI agents."} ### languages - {"name":"TypeScript/JavaScript","description":"Primary backend language using Node.js runtime for building scalable server-side applications, APIs, and supporting the full JavaScript ecosystem."} - {"name":"Python","description":"Commonly used for AI/ML operations, data processing, evaluation scripts, and integration with machine learning frameworks and LLM tools."} - {"name":"SQL","description":"Essential for complex queries, data analysis, and understanding relational database concepts even in predominantly NoSQL environments."} ### frameworks - {"name":"Node.js","description":"Runtime environment for building scalable server-side applications with asynchronous I/O, supporting event-driven architecture and high-performance backend services."} - {"name":"Express.js or Similar","description":"Web framework for building RESTful APIs and backend services in Node.js with routing, middleware support, and request handling capabilities."} - {"name":"React","description":"Frontend framework used by Mural for building user interfaces; backend engineers should understand React patterns for effective API design and frontend collaboration."} ## Benefits ### benefits - {"name":"Remote-First Collaboration","description":"Work with a globally distributed team using asynchronous communication and collaboration tools, maintaining flexibility in work location while being part of a cohesive engineering culture."} - {"name":"Comprehensive Health and Wellness","description":"Medical, dental, and vision insurance coverage with company contributions supporting employee health and well-being across physical and mental health needs."} - {"name":"Retirement and Financial Planning","description":"Competitive 401(k) retirement plan with company matching contributions and financial planning resources to support long-term wealth building."} - {"name":"Professional Development","description":"Learning and development budget for conferences, courses, certifications, and technical skill development to stay current with emerging technologies and industry practices."} - {"name":"Generous Time Off","description":"Unlimited paid time off (PTO) or competitive vacation allowance, plus paid holidays, enabling work-life balance and personal wellness throughout the year."} - {"name":"Stock Options and Equity","description":"Equity participation in Mural as a growing company, aligning employee interests with long-term company success and providing wealth-building opportunities."} - {"name":"Flexible Work Environment","description":"Supportive remote-first culture with flexibility in work arrangements, recognizing productivity and outcomes rather than rigid schedules or location requirements."} - {"name":"Parental Leave","description":"Paid parental leave supporting employees during important life transitions and family-building decisions."} ## Compensation - **max:** 226000 - **min:** 181000 - **currency:** USD - **stockOptions:** true ## Interview process ### steps ## Full description ### ABOUT THE TEAM The AI Innovation Team at Mural is pioneering how generative AI transforms visual collaboration and decision-making. We’re a remote-first group of engineers, designers, and product thinkers focused on helping teams work together more effectively. Our goal isn’t to replace human creativity. It’s to amplify it, building AI that enhances how people align, communicate, and make decisions visually. ### YOUR MISSION You will design and build the core AI systems and platforms that enable Mural’s next wave of agentic, AI-driven collaboration experiences. Rather than building isolated AI features, you’ll work on the core backend systems that power Mural’s agent platform, including agent orchestration, durable execution, contextual memory, tool integration, observability, and evaluation. Your work will enable intelligent agents to reason over product context, act on behalf of users, and operate reliably and safely at scale. Our stack at Mural includes Azure OpenAI, React, Node, MongoDB. ### WHAT YOU'LL DO * Build the core backend systems that power Mural’s agent platform, including orchestration, durable execution, tool execution, memory, observability, and evaluation infrastructure * Design scalable services and APIs that allow AI agents to retrieve context, coordinate multi-step workflows, interact with Mural data, and act reliably on behalf of users * Develop the agent memory layer, including systems for conversation context, product context, retrieval, summarization, compaction, and long-term context management * Create infrastructure to monitor, debug, and improve agent behavior through traces, metrics, feedback loops, and offline evaluation * Translate complex, open-ended product needs into clear backend architectures, service boundaries, data models, and implementation plans that align technical capabilities with user value * Help define the technical direction for agentic AI at Mural, contributing to long-term architecture and strategy * Champion engineering excellence, mentoring others and establishing best practices for building reliable AI systems * Contribute to team growth through hiring, process improvements, and fostering a culture of curiosity, collaboration, and inclusivity ### WHAT YOU'LL BRING * 7+ years of software engineering experience, with strong expertise designing, building, and operating reliable production systems, scalable services and APIs, and well-defined system architectures * Experience designing and operating distributed systems for complex workflows, including orchestration, asynchronous processing, job execution, retries, event-driven architectures, long-running processes, and failure recovery * Familiarity with AI, LLM, or agentic systems, including concepts such as tool use, context retrieval, evaluation, or model orchestration * Strong focus on production quality, including observability, reliability, testing, security, maintainability, and incident response * Strong communication skills, with the ability to produce clear technical documentation and collaborate cross-functionally * Experience working full stack or collaborating closely on product-facing frontend experiences * Deeper AI/ML experience, especially with LLM evaluation, experimentation, embeddings, retrieval systems * Experience with knowledge graphs, graph-based data modeling, entity resolution, or relationship-aware retrieval systems The base salary + variable pay for this role ranges from $181,000 - $226,000 + benefits. Compensation offered will be determined by factors such as location, level, job-related knowledge, skills, and experience. . ### Equal Opportunity We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.