Staff Product Software Engineer

Requirements:

• 8+ years of backend or systems engineering experience.
• Strong proficiency in Python.
• Experience designing automated testing or evaluation frameworks at scale.
• Experience operating AI-powered or model-dependent systems in production.
• Strong understanding of CI/CD integration and release gating processes.
• Experience defining measurable quality standards across distributed systems.
• Proven ability to influence architectural and engineering standards across multiple teams.

Preferred Qualifications:

• Experience designing benchmarking systems for LLMs or ML models.
• Familiarity with hallucination detection, safety scoring, or response consistency analysis.
• Experience building A/B testing infrastructure for model comparison.
• Exposure to multi-provider model environments (Azure, OpenAI, Bedrock).
• Background in experimentation systems or engineering-led release governance.
• Experience with designing automated regression testing frameworks for AI systems.
• Building structured evaluation datasets and benchmarking pipelines.
• Model comparison systems and A/B testing infrastructure.
• Prompt drift detection and workflow validation. CI/CD integration for automated evaluation gating.
• AI provider APIs and model performance characteristics.
• Observability and analytics systems for quality measurement.
• Cross-functional collaboration with AI Core, Architecture, and Governance leadership.