Sr. Product Software Engineer

The candidate will have responsibilities across the following functions:

Evaluation Pipeline Development:

• Implement automated benchmarking workflows for model versions and providers.
• Build regression detection systems comparing model outputs across updates.
• Develop evaluation harnesses for structured test datasets.
• Integrate evaluation results into CI/CD release workflows.

Model Quality and Testing Systems:

• Support A/B testing systems for model comparison.
• Contribute to response consistency and hallucination analysis tooling.
• Assist in implementing prompt and workflow drift detection automation.
• Ensure reproducibility and traceability of evaluation results.

Collaboration and Execution:

• Partner with AI Core Engineering to validate new model integrations.
• Work with Runtime and Reliability teams to correlate evaluation results with production performance.
• Participate in model readiness reviews and certification processes.

Requirements:

• 5+ years of backend or systems engineering experience.
• Strong proficiency in Python.
• Experience building automated testing or evaluation pipelines.
• Experience integrating external APIs or model services.
• Experience working in cloud-based environments.

Preferred Qualifications:

• Exposure to AI-powered systems or LLM integrations.
• Experience with benchmarking or structured dataset validation systems.
• Familiarity with CI/CD-based release gating workflows.
• Experience contributing to experimentation or validation platforms.

Experience with:

• Backend services and distributed systems.
• Automated testing frameworks and regression detection systems.
• Evaluation dataset processing and validation tooling.
• CI/CD pipelines and automated release gating.
• Observability or analytics systems for performance measurement.
• LLM APIs or model-based services.
• Cloud-native deployment environments.

Required Education:

• Bachelor's degree in engineering or a related technical field (Master's preferred), or equivalent experience.