Senior AI Quality & Evaluation Engineer
Confirmed on the employer's own hiring board on Nov 4, 2026. First seen by Alion on Oct 27, 2026. Avenga scores B on the Alion truth index.
Avenga is an international engineering firm helping businesses operate with AI at the core. With 6,000+ experts worldwide, we combine engineering expertise with AI-native thinking to turn ambitious ideas into real-world impact across industries, technologies, and markets.
At the core of the role
In Bulgaria, are looking for ahighly skilled Senior AQA Engineer to design and build automated testing frameworks for AI-powered applications, with a strong focus on LLM evaluation, AI safety, RAG and agent testing, and integrating quality checks into CI/CD pipelines.
Core skills you’ll bring
5+ years in software quality or test engineering, with strong Python (C#/.NET is a plus). You write test frameworks, not only test cases.
Hands-on experience evaluating LLM applications: evaluation frameworks (for example Promptfoo, DeepEval, Ragas, Langfuse or LangSmith), building datasets, designing judges and metrics.
Working knowledge of RAG and agent architectures and how they fail: retrieval quality, tool-call errors, loops, hallucination, context loss.
Experience with adversarial or security testing, of AI systems or in general.
CI/CD and infrastructure-as-code literacy; comfortable running tests on at least one of Azure, AWS or GCP.
Clear written reporting: you turn findings into decisions.
Full working proficiency in English.
Nice-to-have
Familiarity with the EU AI Act, ISO/IEC 42001 or the NIST AI RMF.
Performance and load testing of LLM-backed services.
Benchmarking open-weight models.
Experience in regulated industries or in client-facing delivery.
The impact you'll create
Design and own the automated test strategy for the platform: unit, integration, contract, end-to-end and load tests, wired into CI/CD.
Build and maintain evaluation suites: datasets, rule-based and LLM-as-judge scorers, and regression tests that run on every prompt, model or tool change.
Evaluate new models and tools against quality, safety, latency and cost criteria before they go into use.
Test AI controls adversarially: prompt injection, jailbreaks, data leakage and tool misuse. Keep a red-team suite current and report findings with severity and reproduction steps.
Define quality gates and release criteria, and make quality visible in dashboards.
Monitor AI quality in production and raise alerts when behavior degrades.
Produce clear quality evidence for security reviews, AI governance requirements and clients.
Coach engineers on testable design and review their test and evaluation coverage.
What supports your journey
Guided by our values, we aim to provide you with the tools, autonomy, trust, and support you need to excel. Our benefits include private health insurance, well-being programs, flexible and hybrid work arrangements, laptops and equipment, training opportunities, social events, great offices, and more.
People are at the core of Avenga. We provide equal opportunities regardless of race, ethnicity, gender identity, sexual orientation, disability, age, religion, or any other characteristic. We are committed to a respectful environment where everyone can be themselves, share their ideas, and feel that they belong.