{"id":3356867,"url":"https://alion.io/job/avenga-senior-ai-quality-evaluation-engineer","title":"Senior AI Quality & Evaluation Engineer","company":{"id":181252,"name":"Avenga","domain":"avenga.com","url":"https://alion.io/company/avenga","size_band":"5000+","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Teamtailor","truth_index":{"grade":"B","score":77,"open_postings":27,"ghost_share":0,"stale_share":0.926,"repost_share":0,"time_to_fill_p50_days":32,"computed_at":"2026-11-05T05:54:30Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Sofia, Bulgaria"],"countries":["BG"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":40000,"max_usd":105000,"period":"year","method":"global_role_cell_scaled_by_country","sample_n":2365},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":".NET","optional":false},{"name":"C#","optional":false},{"name":"CI/CD","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"AWS","optional":true},{"name":"Azure","optional":true},{"name":"DeepEval","optional":true},{"name":"EU AI Act","optional":true},{"name":"GCP","optional":true},{"name":"Hallucination","optional":true},{"name":"Langfuse","optional":true},{"name":"LangSmith","optional":true},{"name":"NIST AI RMF","optional":true},{"name":"Promptfoo","optional":true},{"name":"Ragas","optional":true}],"status":"live","first_seen_at":"2026-10-27T15:55:30Z","employer_posted_date":"2026-10-27","last_verified_at":"2026-11-05T23:43:58Z","board_verified":true,"closed_at":null,"days_open":9,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":9},"description":"Avenga is an international engineering firm helping businesses operate with AI at the core. With 6,000+ experts worldwide, we combine engineering expertise with AI-native thinking to turn ambitious ideas into real-world impact across industries, technologies, and markets. \nAt the core of the role\nIn Bulgaria, are looking for ahighly skilled Senior AQA Engineer to design and build automated testing frameworks for AI-powered applications, with a strong focus on LLM evaluation, AI safety, RAG and agent testing, and integrating quality checks into CI/CD pipelines.\nCore skills you’ll bring\n5+ years in software quality or test engineering, with strong Python (C#/.NET is a plus). You write test frameworks, not only test cases.\n\nHands-on experience evaluating LLM applications: evaluation frameworks (for example Promptfoo, DeepEval, Ragas, Langfuse or LangSmith), building datasets, designing judges and metrics.\n\nWorking knowledge of RAG and agent architectures and how they fail: retrieval quality, tool-call errors, loops, hallucination, context loss.\n\nExperience with adversarial or security testing, of AI systems or in general.\n\nCI/CD and infrastructure-as-code literacy; comfortable running tests on at least one of Azure, AWS or GCP.\n\nClear written reporting: you turn findings into decisions.\n\nFull working proficiency in English.\n\nNice-to-have\nFamiliarity with the EU AI Act, ISO/IEC 42001 or the NIST AI RMF.\n\nPerformance and load testing of LLM-backed services.\n\nBenchmarking open-weight models.\n\nExperience in regulated industries or in client-facing delivery.\n\nThe impact you'll create\nDesign and own the automated test strategy for the platform: unit, integration, contract, end-to-end and load tests, wired into CI/CD.\n\nBuild and maintain evaluation suites: datasets, rule-based and LLM-as-judge scorers, and regression tests that run on every prompt, model or tool change.\n\nEvaluate new models and tools against quality, safety, latency and cost criteria before they go into use.\n\nTest AI controls adversarially: prompt injection, jailbreaks, data leakage and tool misuse. Keep a red-team suite current and report findings with severity and reproduction steps.\n\nDefine quality gates and release criteria, and make quality visible in dashboards.\n\nMonitor AI quality in production and raise alerts when behavior degrades.\n\nProduce clear quality evidence for security reviews, AI governance requirements and clients.\n\nCoach engineers on testable design and review their test and evaluation coverage.\n\nWhat supports your journey\nGuided by our values, we aim to provide you with the tools, autonomy, trust, and support you need to excel. Our benefits include private health insurance, well-being programs, flexible and hybrid work arrangements, laptops and equipment, training opportunities, social events, great offices, and more.\nPeople are at the core of Avenga. We provide equal opportunities regardless of race, ethnicity, gender identity, sexual orientation, disability, age, religion, or any other characteristic. We are committed to a respectful environment where everyone can be themselves, share their ideas, and feel that they belong.","description_format":"text","description_chars":3160,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"Advanced (C1)","optional":false}]},"benefits":["Health insurance","Hybrid work"],"hiring_locations":[{"name":"Bulgaria","iso":"BG","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","IT Outsourcing & Dedicated Teams","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-27T20:13:10Z"}],"visa":[],"liveness":{"score":63,"band":"ok","label":"Likely open","p_open":1,"p_active":0.632,"p_room":1,"age_days":8,"expected_fill_days":32,"reasons":["conf:5","stale_co","velocity","win:early"],"computed_at":"2026-11-05T05:54:30Z"},"pay":null,"html_url":"https://alion.io/job/avenga-senior-ai-quality-evaluation-engineer","json_url":"https://alion.io/job/avenga-senior-ai-quality-evaluation-engineer.json","meta":{"generated_at":"2026-11-06T01:49:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3040,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-07T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":181252},"rest":"https://alion.io/mcp/rest/get_company?id=181252"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Favenga-senior-ai-quality-evaluation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Favenga-senior-ai-quality-evaluation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Favenga-senior-ai-quality-evaluation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/avenga-senior-ai-quality-evaluation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Favenga-senior-ai-quality-evaluation-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3356867}},{"tool":"search_jobs","arguments":{"company":"avenga.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3356867},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}