Member of Technical Staff - Inference Platform

Job Description:

• Own execution environments for models on the Platform team
• Evolve Sophos, the online and batch inference runtime built on Kubernetes
• Run large batch inference on ephemeral jobs using multi-dimensional admission control through Kueue
• Build and extend the Sophos controller and Kubernetes custom resources
• Optimize model inference engines and feature processing using vectorized, columnar operations
• Serve graphs and data efficiently from Lance-based storage
• Own training, post-training, and fine-tuning jobs in Avra's cloud and customer dataplanes
• Drive autoscaling, GPU serving, performance, and cost optimization with telemetry for every model
• Solve deterministic job sizing, checkpointing and recovery, customer encryption and isolation, input-file resilience, and automatic profiling
• Achieve 99.9% serving availability, low online-inference latency, high batch throughput, optimized costs, GPU utilization, and reliable job completion

Requirements:

• Experience running model serving or large-scale batch compute on Kubernetes
• Experience building Kubernetes controllers or operators
• Skill at profiling and optimizing data-heavy Python pipelines
• Clear sense of cost and compute efficiency
• Production-quality code and code review experience
• Willingness to operate what you build
• Nice-to-have: Ray, Ray Serve, or KubeRay in production
• Nice-to-have: Kueue or other batch scheduling and admission-control systems
• Nice-to-have: GPU serving and performance optimization
• Nice-to-have: Arrow, Parquet, Lance, or other columnar formats
• Nice-to-have: Shipping software to customer-hosted Kubernetes
• Nice-to-have: GCP/AWS, GKE/EKS, and experience in financial services or regulated environments

Benefits: