{"id":2398220,"url":"https://alion.io/job/pm-consulting-agentic-ai-application-tester","title":"AGENTIC AI APPLICATION TESTER","company":{"id":4421389,"name":"PM Consulting","domain":"pmconsulting.ph","url":"https://alion.io/company/pmconsulting-2","size_band":"1001-5000","is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":"Manatal","truth_index":null},"role":"QA","role_family":"QA","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Taguig, Philippines"],"countries":["PH"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Anomaly Detection","optional":false},{"name":"Hallucination","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"Playwright","optional":false},{"name":"Python","optional":false},{"name":"CI/CD","optional":true},{"name":"Git","optional":true},{"name":"Grafana","optional":true},{"name":"Prometheus","optional":true},{"name":"Pytest","optional":true},{"name":"SQL","optional":true}],"status":"live","first_seen_at":"2026-10-12T22:49:46Z","employer_posted_date":"2026-10-12","last_verified_at":"2026-10-15T00:22:05Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"ROLE SUMMARY\nWe are looking for an Agentic AI Application Tester to lead quality assurance and testing for AI-powered workflow systems. This role bridges software quality engineering and AI evaluation - you will need both technical testing proficiency and a practical understanding of how agent-based, LLM-driven systems behave across the full lifecycle, from development through live production.\nYou will define evaluation strategies, build and maintain automated test infrastructure, monitor production health, and collaborate with engineering teams to continuously improve the safety, reliability, and performance of AI-enabled applications.\nKEY ACCOUNTABILITIES\nOwn the definition and execution of testing and quality assurance strategies for AI-enabled workflows\nContinuously evaluate and monitor system behavior in live production environments\nSupport auditability, risk management, and ongoing quality improvement initiatives across the organization\nPRINCIPAL RESPONSIBILITIES\nQuality Strategy & Evaluation Design\nSet quality criteria and testing strategies for agentic workflows, spanning accuracy, response latency, safety, compliance, and operational risk\nDesign and build automated evaluation harnesses to measure agent performance across key dimensions, including output hallucination rates, tool misuse, policy adherence, and task completion rates\nApply LLM evaluation frameworks to track output quality, detect regressions, and identify system drift over time\n\nTest Automation & Infrastructure\nBuild and maintain automated test suites covering UI, API, and end-to-end workflow validation\nDevelop custom evaluation scripts and tooling to support continuous quality assessment of agent behaviors\nCreate and maintain runbooks for recurring failure modes and actively contribute to incident response processes\nProduction Monitoring & Reporting\nImplement and manage continuous monitoring systems to identify anomalies, quality degradation, and emerging safety issues in production\nDesign and maintain dashboards and quality reports that surface meaningful metrics and trends for engineering teams and business stakeholders\nCollaboration & Compliance\nPartner with developers to iterate on prompts, tool configurations, and workflow designs based on test findings and quality data\nEnsure all testing, logging, and monitoring activities meet data privacy, audit, and applicable regulatory standards\nQUALIFICATIONS\nEssential Experience\nMinimum 3 years of experience in QA, test automation, or DevOps - or at least 2 years with direct, hands-on experience testing AI or ML-enabled systems\nSolid Python programming skills applied to test automation, evaluation harness development, and data analysis\nStrong attention to detail, with a focus on identifying issues that materially affect system reliability and user trust\nAbility to work effectively in an environment where tools, frameworks, and best practices are actively evolving\nCollaborative approach, with a track record of using evidence-based findings to inform product and engineering decisions\n\nRequired Technical Skills\nProgramming: Python - used for test automation, evaluation harness development, and data analysis tasks\nUI Automation: Playwright - for end-to-end testing of agent-driven workflow interfaces\nAI Evaluation: hands-on experience with LLM evaluation frameworks covering quality assessment, drift detection, and regression analysis\nWorkflow Testing: API and agent workflow validation using custom-built scripts\nMonitoring: production-level quality monitoring and anomaly detection implementation\n\nDesirable Skills\nFamiliarity with Python testing frameworks such as Pytest or equivalent\nSQL skills for querying logs, metrics, or evaluation datasets\nExperience with observability and monitoring tools such as Prometheus, Grafana, or comparable platforms\nUnderstanding of hallucination detection techniques and AI safety design principles\nExposure to CI/CD pipelines and Git-based development workflows","description_format":"text","description_chars":3977,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-12T22:49:46Z"}],"visa":[],"liveness":{"score":52,"band":"ok","label":"Likely open","p_open":1,"p_active":0.516,"p_room":1,"age_days":1,"expected_fill_days":24,"reasons":["conf:1","agency","win:early","comp:brand"],"computed_at":"2026-10-14T06:34:16Z"},"pay":null,"html_url":"https://alion.io/job/pm-consulting-agentic-ai-application-tester","json_url":"https://alion.io/job/pm-consulting-agentic-ai-application-tester.json","meta":{"generated_at":"2026-10-15T02:51:17Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":4219,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4421389},"rest":"https://alion.io/mcp/rest/get_company?id=4421389"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fpm-consulting-agentic-ai-application-tester"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fpm-consulting-agentic-ai-application-tester"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fpm-consulting-agentic-ai-application-tester"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/pm-consulting-agentic-ai-application-tester\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fpm-consulting-agentic-ai-application-tester"}]}