Senior / Staff ML Training Optimization Engineer

You will...

• Build standardized distributed training frameworks for research and production, drive our training towards new levels of stability and efficiency.
• Comprehensively profile model runtime and memory to pinpoint performance bottlenecks.
• Identify and evaluate emerging technologies that can be adopted into Waabi’s training and inference frameworks. Examples include designing new CUDA kernels, quantization-aware training and inference, and compilation/deployment techniques.
• Work with researchers and ML engineers on best-practices for optimal resource usage.
• Create and improve tooling and dashboards to ensure broad adoption of your work.

Qualifications:

• MS/PhD or Bachelors degree with a minimum of 4 years of industry experience in Computer Science, Robotics and/or similar technical field(s) of study.
• Solid coding proficiency in a variety of coding languages including Python, C++ or Rust.
• Experience in deep learning frameworks such as PyTorch or Jax.
• Skilled in profiling CPU and GPU code using tools such as PyTorch Profiler and NVIDIA Nsight.
• Open-minded and collaborative team player with willingness to help others.
• Passionate about self-driving technologies, solving hard problems, and creating innovative solutions.

Bonus/nice to have:

• Experience in identifying when custom CUDA kernels are needed, and implementing them.
• Experience in Bazel in a monorepo environment, and integrating third party packages into dev environments.
• Experience with Kubernetes-based training platforms.