Founding AI Engineer - AI Evaluations and Systems

We are building a retention product for online merchants, with LLM-drafted content that is validated and human-approved before anything reaches a customer. The question this role answers: How do we know a change to a prompt, a model, or the code made the system better, not worse? You will build the evaluation suites that decide whether a change may ship, and help get that change to production safely. This role suits an engineer early in their career who has already built real evaluations and shipped something, and who wants to grow into owning both quality and reliability.

The candidate will have responsibilities across the following functions:

Make AI Quality Measurable:

• Build golden sets and behavioural assertions for LLM outputs.
• Read real outputs and traces first, and group failures into a taxonomy before choosing metrics.
• Set pass thresholds and track hallucinated or unsupported claims.
• Turn every production failure into a regression case.

Build Evaluation Pipelines That Run in CI:

• Build regression suites for prompts, models, and tools.
• Run them in GitHub Actions or an equivalent platform so a regression blocks the merge.
• Pin model versions and rerun suites on provider upgrades.
• Repeat runs and compare results properly, so noise is not mistaken for progress.
• Review changes against what their descriptions and tests claim, and add checks that would have caught any gaps.
• Make evaluation part of engineering, not a final QA step.

Engineer LLM-as-a-Judge Systems:

• Write structured rubrics and scoring mechanisms.
• Calibrate the judge against human-labelled examples and report agreement using precision, recall, and Cohen's kappa.
• Monitor judge bias, including position, verbosity, and self-preference.

Keep Evaluation Affordable:

• Use deterministic checks for simple cases and reserve LLM judges for more complex cases.
• Batch and parallelise evaluation runs.
• Track cost and latency per run.

Ship and Run It Reliably:

• Containerise services with Docker and automate builds and deployments.
• Keep deployments reversible through rollback steps, health checks, and backups that have been restored at least once.
• Add tracing and cost tracking for model calls, along with actionable alerts.
• Write runbooks and concise technical decision notes.

Requirements:

• 2+ years as a software engineer, with recent hands-on work on an LLM feature that people have used. RAG or agent products count.
• At least one evaluation suite you built and ran, with a clear explanation of what it caught and what you changed because of it.
• Practical experience with LLM-as-a-judge, including rubrics, human labels, and calibration.
• Hands-on experience with at least one of: promptfoo, DeepEval, Ragas, TruLens, LangSmith, Langfuse, Braintrust, Inspect, or an equivalent tool.
• Familiarity with common evaluation measures: faithfulness, answer relevance, task completion, precision and recall, and confidence intervals.

Technical Skills:

• Python 3. x: pytest, Pydantic, httpx or requests, and asyncio basics.
• PostgreSQL / SQL: joins, aggregates, and window functions.
• LLM APIs: Anthropic and/or OpenAI SDKs, structured outputs, and tool calling.
• CI/CD: GitHub Actions or an equivalent platform, including a check that blocks a merge.
• Docker: Building and running containerised services.
• Deployment automation: Basic deploy scripts or pipeline jobs, environment configuration, and secrets handling.
• Cloud: Working-level knowledge of any one of AWS, GCP, or Azure. AWS is a plus. If you have not used a cloud yet, Docker plus solid deployment automation is enough to start.
• Git: Pull-request workflows and code reviews.

Skills

LLMs, Machine Learning, Python, Regression Testing