{"id":43059,"url":"https://alion.io/job/allegis-global-solutions-ai-quality-engineer","title":"AI Quality Engineer","company":{"id":9025,"name":"Allegis Global Solutions","domain":"allegisglobalsolutions.com","url":"https://alion.io/company/allegis-global-solutions","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"SmartRecruiters","truth_index":{"grade":"B","score":75,"open_postings":6,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":27,"computed_at":"2026-10-20T05:45:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":21000,"max_usd":50000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":25},"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Copilot Studio","optional":false},{"name":"DeepEval","optional":false},{"name":"Function Calling","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"LLM","optional":false},{"name":"Power Automate","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Ragas","optional":false},{"name":"TypeScript","optional":false},{"name":"Docker","optional":true},{"name":"Git","optional":true},{"name":"GitHub","optional":true},{"name":"Pytest","optional":true},{"name":"Rest API","optional":true},{"name":"SQL","optional":true}],"status":"closed","first_seen_at":"2026-05-22T11:20:46Z","employer_posted_date":"2026-05-22","last_verified_at":"2026-10-19T13:43:05Z","board_verified":false,"closed_at":"2026-10-19T13:43:05Z","days_open":150,"trust":{"level":"stale","repost_count":0,"flags":["stale"],"days_open":149},"description":"Working at Allegis Global Solutions (AGS) is more than just a job. It’s a career. It’s a community of people who invest in your development and empower you to blaze your own trail. Each of us is here to create real, measurable impact that moves needles. We operate beyond \"roles\" or \"jobs\" to realize the opportunity to make meaningful contributions to a bigger idea. Because we believe that when you build a workforce that’s designed to harness human enterprise, you design a workforce that’s built for impact.\nAt AGS, we help companies all over the world transform their people into a competitive advantage. It’s not about filling seats. It’s about designing workforces to meet missions and unleash the most transformative power in business today: The power of human enterprise.\nWith services around the globe, we have a point of view on the future of work that enables us to be a transformative partner in the way work gets done for our clients’ organizations. Meeting clients where they are, we design a plan and guide them along a transformational journey, applying bold actions and diverse minds to solve the most complex challenges - from permanent and extended workforce management to services procurement, consulting, direct sourcing and our Universal Workforce Model™.\nWe also represent over 100 countries and speak dozens of languages. So as you’re building relationships and doing your job, you’ll be exposed to other cultures and advancement opportunities while expanding your knowledge of global markets and strategies.\nAs a workplace, we focus on relationships - with each other, our clients and our candidates - in fact serving others is one of our core values. We support open communication and recognize that giving constructive criticism can be even harder than receiving it. We appreciate the fearless and the passionate, who force us to be better. Everything we do sits on a pillar of diversity - diverse perspectives, backgrounds and ideas drive innovation and make us successful.\nSee what it’s like to work at AGS by searching #LifeAtAGS on any social network.\n About the Role\nTesting AI systems is a fundamentally different problem than testing traditional software. Outputs are non-deterministic. \"Correct\" is often a spectrum. And the failure modes-hallucinations, drift, prompt injection-don't show up in unit tests. We need an engineer who understands this and can build the testing strategies, evaluation frameworks, and quality infrastructure to keep our agents reliable in production.\nAs an AI Quality Engineer, you'll design how we test intelligent agents, agentic workflows, and Foundation Layer capabilities. This is not a manual QA role-you'll write code, build evaluation pipelines, and create automated testing frameworks that run in CI/CD. You'll define what \"quality\" means for AI systems at AGS and build the systems to measure it.\nYou'll work across every solution the team builds, which means you'll have broad visibility into the architecture and deep understanding of how our agents behave in the real world. If you're an engineer who cares about quality and wants to solve testing problems that most teams haven't figured out yet, this is the role.\nResponsibilities\nTesting Strategy & Design\nDefine testing strategies for AI agents, conversational interfaces, and agentic workflows\nDesign behavioral test suites for non-deterministic outputs-where \"correct\" isn't binary\nBuild evaluation frameworks that measure groundedness, factuality, relevance, and task completion\nIdentify failure modes specific to AI systems: hallucinations, prompt injection, context window limitations, drift\nDevelop testing approaches for each architecture pattern: RAG, function calling, human-in-the-loop, autonomous workflows\nTest Automation & Infrastructure\nBuild automated evaluation pipelines that run as part of CI/CD\nCreate test harnesses for LLM-based systems-mocking, fixtures, and reproducible test scenarios\nDevelop regression suites that detect quality degradation when prompts, models, or data change\nBuild monitoring and alerting for production agent quality (accuracy, latency, error rates)\nMaintain test infrastructure: test data management, environment setup, reporting dashboards\nEvaluation & Metrics\nDefine quality metrics for each solution-what to measure and what thresholds matter\nBuild and maintain evaluation datasets (ground truth, reference outputs, edge case collections)\nConduct systematic prompt evaluation when prompts or models change\nTrack quality trends over time and identify when re-evaluation is needed\nReport quality metrics to the team and stakeholders in clear, actionable terms\nCollaboration & Quality Culture\nPartner with AI Solutions Engineers to define testability requirements during design\nWork with AI Solutions Analysts to translate acceptance criteria into test scenarios\nReview solution designs from a quality and testability perspective\nAdvocate for quality practices across the team-testing isn't an afterthought, it's part of delivery\nContribute to incident response by diagnosing quality failures and building regression tests\n Qualifications\nRequired\n3-7 years of software engineering or quality engineering experience\nStrong programming skills in Python and/or TypeScript-you write test code, not just test cases\nExperience designing and building automated test frameworks\nUnderstanding of AI/ML systems-you know why testing LLM outputs is different from testing deterministic code\nExperience with CI/CD pipelines and integrating automated tests into build processes\nAbility to reason about non-deterministic systems and design meaningful quality metrics\nStrong analytical skills-you can look at agent outputs and determine whether they're good enough\nPreferred\nExperience testing AI/ML applications, conversational interfaces, or chatbots\nBackground in LLM evaluation: prompt testing, groundedness scoring, factuality checking\nFamiliarity with evaluation frameworks (DeepEval, Ragas, custom evaluation pipelines)\nExperience with Microsoft Power Platform (Power Automate, Copilot Studio) testing\nBackground in Azure services and cloud-based test infrastructure\nExperience with load testing and performance testing for API-based systems\nFamiliarity with staffing, HR tech, or workforce management domains\nTechnology Stack\nLanguages: Python, TypeScript\nPlatforms: Azure (Container Apps, Functions, AI Services), Microsoft 365\nTesting: pytest, evaluation frameworks (DeepEval, Ragas, custom), load testing tools\nAI/ML: LLM evaluation, prompt testing, RAG evaluation, behavioral testing\nData: REST APIs, Dataverse, SQL\nTools: Git, GitHub, CI/CD pipelines, Docker, monitoring/alerting (Application Insights)\nWe don't expect expertise in everything. AI quality engineering is a new discipline-we expect strong engineering fundamentals and the ability to figure out new problems.\nWhat We're NOT Looking For\nManual testers who write test cases in spreadsheets\nQA professionals who treat testing as a gate at the end of development rather than a practice woven into it\nPeople who expect deterministic pass/fail for every test-AI quality requires nuance\nEngineers who test to the spec but don't think about how real users will break things\nWhat Makes You Stand Out\nYou've tested a system where \"correct\" was hard to define-and found a way to measure it anyway\nYou write test code that's as clean and maintainable as production code\nYou think about edge cases that nobody else considers\nYou can explain why a particular quality metric matters and what threshold makes sense\nYou've built test automation that actually caught regressions before they hit production\nYou're comfortable saying \"this isn't good enough\" and backing it up with data\nWhat We're Building\nThe AI Engineering team delivers intelligent solutions for AGS's global clients:\nIntelligent Agents - Conversational AI that helps hiring managers, recruiters, and internal teams get work done faster\nAgentic Workflows - Automated processes where AI executes tasks with human oversight\nFoundation Capabilities - Reusable AI services that power multiple solutions\nYou'll make sure these systems work reliably-not just at launch, but as models change, data evolves, and usage scales.\nCareer Growth\nAI quality engineering is an emerging discipline with no ceiling. Growth paths include:\nDepth - Become the team's authority on AI evaluation and testing methodology, influencing quality standards across the organization\nBreadth - Move into a Senior or Lead AI Solutions Engineer role, bringing your quality mindset to architecture and delivery\n Specialization - Build expertise in areas like LLM security testing, AI safety, or evaluation research\n As a workplace, we focus on relationships - with each other, our clients and our candidates - in fact serving others is one of our core values. We support open communication and recognize that giving constructive criticism can be even harder than receiving it. We appreciate the fearless and the passionate, who force us to be better. Everything we do sits on a pillar of diversity - diverse perspectives, backgrounds and ideas drive innovation and make us successful.\nSee what it’s like to work at AGS by searching #LifeAtAGS on any social network.","description_format":"text","description_chars":9221,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"All levels","optional":false}]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Human Resources","Recruiting & Staffing","Business Process Outsourcing (BPO)"],"lifecycle":[{"event":"open","at":"2026-05-22T11:20:46Z"},{"event":"close","at":"2026-10-19T13:43:05Z"}],"visa":[],"liveness":null,"pay":null,"html_url":"https://alion.io/job/allegis-global-solutions-ai-quality-engineer","json_url":"https://alion.io/job/allegis-global-solutions-ai-quality-engineer.json","meta":{"generated_at":"2026-10-20T06:21:27Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":6812,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-21T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":9025},"rest":"https://alion.io/mcp/rest/get_company?id=9025"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fallegis-global-solutions-ai-quality-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fallegis-global-solutions-ai-quality-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fallegis-global-solutions-ai-quality-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/allegis-global-solutions-ai-quality-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fallegis-global-solutions-ai-quality-engineer"}]}