Staff ML Systems Engineer, Distributed Systems

We are seeking a Staff ML Systems Engineer to architect and build the distributed infrastructure that powers large-scale machine learning workflows across the organization.

This role sits at the intersection of machine learning, distributed systems, and platform engineering. You will be responsible for designing scalable systems that support data processing, model training, evaluation, and post-processing pipelines while enabling ML teams to efficiently develop, operate, and scale production-grade workflows.

You will play a critical role in defining the architectural patterns, tooling, and infrastructure that underpin our machine learning platform.

What You'll Get To Do

• Design and build scalable distributed machine learning pipelines across data processing, model training, evaluation, and post-processing workflows.
• Architect distributed execution systems, including parallelization strategies, workload scheduling, resource allocation, and fault tolerance mechanisms.
• Develop reusable abstractions, frameworks, and libraries that simplify distributed pipeline development.
• Optimize performance across distributed CPU and GPU environments, improving throughput, utilization, and reliability.
• Design systems that effectively manage data partitioning, memory utilization, serialization overhead, and compute efficiency.
• Partner closely with ML engineers, data engineers, and infrastructure teams to productionize research workflows and enable large-scale model development.
• Establish best practices and engineering standards for distributed machine learning infrastructure.
• Evaluate and guide decisions around distributed computing frameworks, infrastructure technologies, and system design trade-offs.
• Improve observability, debugging, monitoring, and operational tooling for distributed systems at scale.

What You Have

• 5+ years of experience building distributed systems, backend infrastructure, machine learning platforms, or large-scale data processing systems.
• Strong Python programming skills, including experience with concurrency, performance optimization, and systems development.
• Experience with distributed computing frameworks such as Ray, Spark, Dask, Flink, or similar technologies.
• Experience designing and scaling data pipelines or machine learning workflows.
• Strong system design skills with demonstrated expertise in scalability, reliability, and performance optimization.
• Experience diagnosing and resolving bottlenecks in distributed environments.
• Ability to work cross-functionally and drive technical decisions across multiple teams.

The Extras That Set You Apart

• Experience building infrastructure for machine learning training and inference systems.
• Familiarity with modern ML frameworks such as PyTorch or TensorFlow.
• Experience with multi-node or multi-GPU training architectures, including DDP, FSDP, DeepSpeed, or similar technologies.
• Experience operating Kubernetes-based infrastructure and large-scale cloud systems.
• Deep understanding of distributed systems concepts including data locality, serialization costs, scheduling, and resource management.
• Experience with distributed debugging, observability, and workflow orchestration platforms.
• Proven ability to establish technical direction and influence architecture across organizations.