Senior Technical Program Manager, Machine Learning Infrastructure

Technical Program Manager · Senior · Full Time · Remote

Canada · RemoteUSD 160k – 220k1mo ago
Apply for this role

Opens Cohere's application page

Role

What you'll do.

Senior Technical Program Manager overseeing Machine Learning infrastructure programs at Cohere, a leading enterprise AI company. This strategic role manages inference, serving, efficiency, and endpoint systems while coordinating cross-functional efforts between ML modeling teams and customer-facing operations. Requires 5+ years of technical program management experience in ML infrastructure with strong execution capabilities in fast-paced, complex environments.

Responsibilities

  • ML Infrastructure Program Portfolio Management: Own the complete program portfolio spanning model inference, serving efficiency, and endpoint infrastructure. Ensure systems scale reliably to support rapidly expanding internal and external user bases while maintaining performance standards and architectural integrity across distributed systems.
  • Cross-Functional Coordination and Execution: Lead end-to-end program coordination with particular emphasis on orchestrating collaboration between ML modeling teams, infrastructure engineers, and customer-facing product teams. Drive alignment on technical dependencies, timeline requirements, and resource allocation across organizational boundaries.
  • Process Optimization and Best Practices: Identify infrastructure pain points and establish operational processes that enable engineering teams to maintain high development velocity while consistently meeting stakeholder needs. Implement and refine engineering best practices to balance rapid iteration with system reliability.
  • Incident Management and Continuous Improvement: Partner with incident management leads to foster a culture of continuous improvement. Ensure thorough root cause analysis, timely issue resolution, and effective implementation of preventative measures. Track and communicate learnings across the organization.
  • Strategic Prioritization and Program Management: Ruthlessly prioritize competing demands and overlapping projects to align with company strategic objectives. Manage scope creep, resource constraints, and competing stakeholder needs while maintaining focus on highest-impact deliverables.
  • Stakeholder Communication and Program Tracking: Establish and maintain clear tracking of timelines, deliverables, budgets, and scope across all programs. Communicate consistent, timely updates to engineering teams, leadership, and non-technical stakeholders on program status, risks, and incidents.
  • Strategic and Tactical Partnership: Serve as trusted partner to senior technical leaders, providing both granular tactical support on day-to-day execution and strategic counsel on high-level problems with company-wide impact. Bridge communication between technical depth and strategic vision.

Qualifications

What we look for.

Technical

  • ML Infrastructure Systems Knowledge

    In-depth understanding of machine learning infrastructure architecture including model inference systems, serving frameworks, distributed systems, and performance optimization techniques.

  • Hands-On Technical Experience

    Direct technical experience in software engineering, infrastructure engineering, or related roles. Practical familiarity with building scalable systems, debugging complex technical issues, and understanding engineering trade-offs.

  • Machine Learning Fundamentals

    Working knowledge of how machine learning models are trained, deployed, and served in production. Understanding of model performance characteristics, inference latency requirements, and serving infrastructure patterns.

  • Program Management Tools and Methodologies

    Proficiency with project tracking tools, methodologies for managing technical programs (Agile, Scrum, Waterfall variants), and approaches for stakeholder communication and status reporting.

Education

  • Bachelor's Degree in Computer Science or Related Field

    Formal education in computer science, software engineering, mathematics, or related technical discipline providing foundational understanding of computational systems and technical principles.

Experience

  • Technical Program Management

    5+ years of proven technical program management experience with demonstrated success managing complex, cross-functional programs in fast-paced environments.

  • ML Infrastructure Specialization

    Focused experience in machine learning infrastructure domains including model inference optimization, model serving architecture, computational efficiency, and endpoint design and implementation.

  • Fast-Paced Environment Execution

    Demonstrated ability to operate effectively in chaotic, low-structure, rapidly-scaling environments. Proven pragmatism, resourcefulness, and capability to perform tactical hands-on work while maintaining strategic perspective.

  • Technical Depth and Subject Matter Expertise

    Strong working knowledge of ML infrastructure systems design and implementation. Ability to develop deep technical understanding of internal systems and establish credibility as a subject matter expert in the domain.

Skills

Required

  • Technical Program Management

    Expert-level ability to manage complex multi-team technical programs with clear ownership of scope, timeline, budget, and resource allocation across distributed teams.

  • Cross-Functional Leadership

    Proven ability to lead and influence without direct authority. Drive alignment and collaboration between engineering teams, research teams, and non-technical stakeholders with competing priorities.

  • ML Infrastructure Knowledge

    Deep technical knowledge of machine learning infrastructure domains including inference, serving, efficiency optimization, and scalable endpoint architecture.

  • Execution and Problem Solving

    Exceptional execution capability with pragmatic approach to problem-solving. Comfort with rolling up sleeves to unblock teams and drive tactical outcomes while maintaining strategic perspective.

  • Communication and Stakeholder Management

    Clear, concise communication adapted to audience technical level. Ability to explain complex infrastructure concepts to non-technical stakeholders and translate business needs into technical requirements.

  • Strategic Thinking

    Capability to think strategically about program architecture, organizational processes, and long-term technical direction while executing tactically on immediate priorities.

  • Incident Response and Root Cause Analysis

    Experience leading incident response processes, facilitating root cause analysis, and driving implementation of preventative measures to reduce recurring issues.

Preferred

  • Hands-On Technical Background

    Nice to have

    Direct experience working as a software engineer, infrastructure engineer, or ML engineer providing practical understanding of technical trade-offs and engineering challenges in building scalable systems.

  • Machine Learning Team Collaboration

    Nice to have

    Demonstrated experience working closely with machine learning teams. Deep understanding of research engineering dynamics, model development workflows, and intersection between research velocity and infrastructure stability.

  • Distributed Systems Understanding

    Nice to have

    Working knowledge of distributed systems architecture, scalability challenges, and reliability patterns relevant to managing high-throughput, low-latency inference and serving infrastructure.

  • Enterprise AI or LLM Experience

    Nice to have

    Experience working with large language models, foundation models, or enterprise AI systems. Understanding of unique challenges in deploying and scaling cutting-edge AI models for production use.

  • Cost Optimization and Resource Planning

    Nice to have

    Track record of optimizing infrastructure costs while maintaining performance requirements. Experience with capacity planning and resource allocation decisions balancing business needs with technical constraints.

Tech stack

Frameworks

Machine Learning Serving FrameworksDistributed Computing Frameworks

Databases

Distributed Databases and Caching

Tools

Program Management ToolsMonitoring and Observability ToolsCommunication and Collaboration ToolsInfrastructure Management

Other

Model Inference and Serving ArchitectureScalability and Performance OptimizationEnterprise AI Architecture

Compensation

Pay and benefits.

Base·USD 160,000 – 220,000

Equity·Stock options

Benefits

  • Weekly Lunch Stipend

    Weekly lunch allowance of $75 USD (or equivalent in local currency) to support meal expenses and team building opportunities.

  • Comprehensive Health and Dental Coverage

    Full health and dental benefits with dedicated mental health budget supporting employee wellness across physical and mental health domains.

  • Retirement Planning

    RRSP matching (Canada), 401(k) contributions (US), or Pension Scheme (UK) depending on employment location, supporting long-term financial security.

  • Parental Leave

    100% parental leave top-up for up to 6 months for either parent, supporting family planning and work-life balance.

  • Annual Enrichment Benefits

    Dedicated budgets for arts and culture experiences, fitness and wellness programs, quality time and life experiences, plus annual workspace improvement credit enhancing work environment.

  • Education and Learning Stipend

    Annual budget supporting professional development through conferences, courses, coaching, and continuous learning opportunities aligned with career growth.

  • Paid Time Off

    6 weeks of paid vacation (30 working days annually) providing substantial time for rest, travel, and personal pursuits.

  • Office Travel and Offsite Opportunities

    Budget for traveling to other Cohere offices for remote employees plus attendance at annual company offsite events for team connection and strategic alignment.

  • Remote Work Setup

    One-time $500 home office stipend to establish professional workspace setup supporting productive remote work environment.

  • Co-working Benefits

    Access to co-working spaces in your local city for remote employees not near Cohere offices, enabling flexible professional workspace options.

  • Office Amenities

    For in-office team members: daily lunch program, comprehensive snacks, and regular community and social events fostering team connection.

Process

Interview steps.

  1. 01

    Application Review

    Your application will be reviewed by Cohere recruiters against the job criteria. The company leverages AI-enabled tools to help identify potentially qualified candidates while maintaining human review of all applications.

  2. 02

    Recruiter Screening

    Initial conversation with a Cohere recruiter to discuss your background, experience with ML infrastructure programs, and fit for the Technical Program Manager role. Expect to discuss your approach to managing complex cross-functional initiatives.

  3. 03

    Hiring Manager Discussion

    Conversation with the ML Infrastructure team's leadership to explore your technical depth, program management philosophy, and approach to working in fast-paced, ambiguous environments. Be prepared to discuss specific examples of infrastructure programs you've managed.

  4. 04

    Technical Program Management Assessment

    Detailed discussion or case study focused on your program management capabilities. You may be presented with infrastructure challenges similar to those at Cohere and asked how you would approach prioritization, stakeholder communication, and execution.

  5. 05

    Cross-Functional Stakeholder Conversations

    Potential discussions with engineering team members, ML researchers, or product leaders you would collaborate with. These conversations assess your ability to communicate across technical and non-technical audiences and build collaborative relationships.

  6. 06

    Executive Alignment

    Final conversation with senior leadership to ensure organizational fit, discuss your vision for the ML infrastructure program, and clarify questions about Cohere's technical direction and growth trajectory.

Full posting

Original listing.

Who are we?

Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation AI models and end-to-end products that are designed to solve real-world business problems.

We’re training and deploying frontier models for enterprises who are building AI systems. We believe that our work is instrumental to the widespread adoption of AI and we are looking for folks that want to be part of that.

We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. Cohere is a team of researchers, engineers, designers, and more, who are all passionate about their craft.

We are a global technology company co-headquartered in Toronto and San Francisco, with key offices in London, New York City, Montreal, Seoul, Germany and Paris. Join us!

Why this role?

As Cohere continues to grow, the size and complexity of our programs has continued increasing over time. To help us manage this, we are looking to bring in exceptional Technical Program Managers (TPM) to manage this!

At Cohere, TPMs are pivotal to our success, being seen as operational experts specializing in cross-functional efforts. Great TPMs at Cohere are well-rounded, executing with precision at the tactical level, while partnering with senior leaders and shaping the big picture at a strategic level, and the opportunities are endless in growing the scale and impact of their work. Cohere’s TPMs are often described by their stakeholders as organized, execution-oriented, pragmatic, and adaptable to the needs of the company and the programs they lead.

As a reward for the depth and breadth of expertize that you bring to the table, you will get to work alongside some of the most talented engineers in the world on building cutting edge AI technology. No day will be the same as the one before, and the dynamic high growth environment will be absolutely perfect for anyone excited about solving novel problems, bringing structure to oftentimes chaotic and fast-paced environments, and ambitious about scaling their own impact and growth alongside the company!

Please note: While we are remote-first and accept candidates from all over the world, we are strongly prioritizing hiring for candidates in the Eastern Time (ET) timezone.

As a Technical Program Manager for Machine Learning Infrastructure, you will:

  • Manage the program portfolio covering inference, efficiency, serving, and endpoints to guarantee that Cohere’s infrastructure continues to scale for a rapidly expanding user base of internal and external users

  • Lead the end-to-end coordination and execution across the program, with a particular focus on cross-functional collaboration with Modeling and customer-facing teams

  • Identify pain points and establish processes to ensure that the team can focus on development, while meeting needs of internal and external users of the models, and improving engineering best practices

  • Build a strong culture around continuous improvement, such as liaising with incident management leads, ensuring that problems are accurately root caused, and that required fixes are provided in a timely manner

  • Manage various overlapping projects and programs, ruthlessly prioritizing asks, to ensure that the company’s top priorities are met

  • Collaborate with stakeholders across the company to set, track, and manage timelines, deliverables, budgets, and scope to ensure successful program execution

  • Deliver clear, timely, and consistent updates across engineering, leadership, and non–technical teams on the progress, plans, and incidents

  • Act as a strong tactical and strategic partner to the senior leads across your program, whether that be providing low-level tactical support or help answer high-level strategic problems that drives impact at the company-level.

You may be a good fit if you have:

  • 5+ years of Technical Program Management experience focusing on Machine Learning Infrastructure, specifically covering areas such as model inference, serving, efficiency, and endpoints design & implementation

  • In-depth technical knowledge around ML infrastructure design and implementation, as well as engineering best practices, supporting a balance of velocity and structure in rapidly growing organizations

  • Experience working in a chaotic, fast paced, low structure environment. We need an EPM who is pragmatic, can roll up their sleeves when needed, and can function at the tactical and strategic level to do whatever it takes to make Cohere’s models succeed

  • (optional, but strongly preferred) Have technical experience in a hands-on capacity. We don’t necessarily need someone who has been working hands-on in a technical role, but the programs are inherently extremely technical. You need to dive into the depths of internal systems, and develop a reputation as a subject matter expert who brings in engineering best practices and the right amount of structure to continuously improve the program

  • (optional, but preferred) Experience working with ML teams, and a working understanding of how machine learning models are built in order to advocate for both the needs of the infrastructure teams and modeling teams in building scalable, highly performant ML infrastructure.

If some of the above - other than the ML Infrastructure experience - doesn’t line up perfectly with your experience, we still encourage you to apply! If you consider yourself a thoughtful worker, a lifelong learner, and a kind and playful team member, Cohere is the place for you.

We welcome applicants of all kinds and are committed to providing both an equal opportunity process and work environment. We value and celebrate diversity and strive to create an inclusive work environment for all.

Full-Time Employees at Cohere enjoy these Perks:

  • A weekly lunch stipend of $75/£75 or equivalent in your local currency for lunch.

  • Full health and dental benefits, including a separate budget for mental health.

  • RRSP matching, 401K, Pension Scheme.

  • 100% Parental Leave top-up for up to 6 months, for either parent.

  • Annual enrichment benefits:

    Arts & culture, fitness/wellness, quality time, and a workspace improvement credit.

    Education & learning stipend for conferences, courses, and coaching.

  • 6 weeks of paid vacation (30 working days!)

  • Budget for traveling to other offices if you are remote, plus an annual company offsite.

How and Where We Work:

  • Cohere is remote-friendly. We have offices in Toronto, San Francisco, New York City, London, Paris, Montreal, and more coming soon.

  • For those in the office: a daily lunch program, plenty of snacks, and regular community and social events.

  • For those not near an office: a co-working benefit so you can work alongside others in your city.

  • Everyone receives a $500 home office stipend to set up your workspace properly.

If any of the above doesn’t line up exactly with your experience, we still encourage you to apply.


We strive to create an inclusive work environment for all; we welcome applicants from all backgrounds and are committed to providing equal opportunities. Should you require any accommodations during the recruitment process, please submit an Accommodations Request Form, and we will work together to meet your needs.

We may use AI-enabled tools to screen and assess applicants against the criteria for this position. This helps our recruiters identify potentially qualified candidates, but it doesn't limit the applications our recruiters may review or consider.

Redirects to Cohere's application page.

Other roles

More at Cohere.

View all 26 roles