{"id":3385491,"url":"https://alion.io/job/roche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2","title":"Fullstack AI Quality Engineer – Applied & Agentic AI Systems","company":{"id":13531,"name":"Roche","domain":"roche.com","url":"https://alion.io/company/roche","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"A","score":86,"open_postings":224,"ghost_share":0.04,"stale_share":0.612,"repost_share":0.098,"time_to_fill_p50_days":19,"computed_at":"2026-11-01T06:10:25Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":22000,"max_usd":46000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":41},"experience_years_min":7,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"Agile","optional":false},{"name":"AI Agents","optional":false},{"name":"Angular","optional":false},{"name":"Arize Phoenix","optional":false},{"name":"AWS","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude Code","optional":false},{"name":"Cypress","optional":false},{"name":"DeepEval","optional":false},{"name":"GitHub","optional":false},{"name":"GitHub Actions","optional":false},{"name":"Grafana","optional":false},{"name":"Hallucination","optional":false},{"name":"ITSM","optional":false},{"name":"Jest","optional":false},{"name":"Jira","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"Machine Learning","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"NLP","optional":false},{"name":"Playwright","optional":false},{"name":"Promptfoo","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"TypeScript","optional":false},{"name":"JavaScript","optional":true}],"status":"live","first_seen_at":"2026-10-28T07:14:45Z","employer_posted_date":"2026-10-28","last_verified_at":"2026-11-01T21:09:35Z","board_verified":true,"closed_at":null,"days_open":4,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":4},"description":"At Roche you can show up as yourself, embraced for the unique qualities you bring. Our culture encourages personal expression, open dialogue, and genuine connections, where you are valued, accepted and respected for who you are, allowing you to thrive both personally and professionally. This is how we aim to prevent, stop and cure diseases and ensure everyone has access to healthcare today and for generations to come. Join Roche, where every voice matters.\nThe Position\nJob description\nWe are seeking a highly skilled AI Quality Engineer specializing in testing Generative (and non-Generative) AI applications and agentic systems to join our team. The ideal candidate will have expertise in both traditional software testing and specialized testing methodologies for Large Language Models (LLMs)-based and AI-based systems.\nAs a fullstack Quality Engineer within a lean, agentic software team, you will cover the entire spectrum of quality assurance. You will span manual exploratory testing, traditional automated UI/API/MCP testing (across our Python and TypeScript stack), classical machine learning validation, and cutting-edge generative and agentic AI evaluation. This role requires a deep understanding of LLM, agentic and ML systems behaviors, and testing frameworks to ensure the reliability, fairness, and effectiveness of our AI-powered solutions.\nDescription of the area\nAt Roche Digital Technology, we are advancing the boundaries of Applied AI. The Applied AI Engineering Team focuses on architecting, building, and operating high-value AI solutions and services to solve complex business challenges in healthcare.\nIn the 2026 tech landscape, we operate in small, highly autonomous agile teams (e.g., ~9 members) powered by advanced coding agents (like Claude Code) to develop and ship solutions faster than ever before. In this highly regulated environment, quality cannot be an afterthought. You will be the foundational pillar ensuring our rapidly developed agentic workflows and AI, GenAI, and agentic applications are safe, compliant, and robust before they reach the clinical or enterprise user.\nJob Responsibilities\nGenerative & Agentic AI Testing Strategy:Design and implement comprehensive testing strategies specifically tailored for LLM-based applications and agentic systems, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability. You will expand this to evaluate autonomous multi-agent behaviors and tool-use accuracy.\n\nFull-Stack Test Automation:Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests. This spans both our Python backend services and modern TypeScript/React/Angular frontends.\n\nQuality Assurance Framework & Compliance:Utilize Roche's testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required. Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with Roche practices.\n\nEdge Case, Bias, and Fairness Testing:Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns. Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases.\n\nSecurity & Performance Testing:Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns. Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring.\n\nQualifications\nHold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies.\n\nBe passionate about quality assurance and testing, particularly in the context of AI and GenAI applications and agents.\n\nHave strong analytical and problem-solving skills with attention to detail.\n\nUtilize critical thinking and problem-solving when using AI prompts (in e.g. NLP, data distributions and edge cases).\n\nBe proactive in identifying potential issues and suggesting improvements.\n\nEducation / Experience\nExperience:7+ years of experience in software testing, with at least 1 year focused on genAI and/or agentic applications.\n\nRegulated Environments:Be ready to work in a regulated environment with qualified infrastructure and validated applications, understand the difference between GxP and non-GxP products and the importance of CSV and ITSM processes in the delivery.\n\nAgile Integration:Be experienced in working with Agile development teams.\n\nSkills\nMust-have:\nComprehensive Full-Stack Testing Mastery\n\nCore Languages & Testing Frameworks: Advanced command of Python and standard testing frameworks (e.g., pytest, allure), alongside practical familiarity with specialized AI evaluation tools. High competence in TypeScript-driven test frameworks (such as Playwright, Jest, and Cypress) for thoroughly verifying dynamic generative user interfaces.\n\nEnd-to-End Automation & CI/CD: Proven expertise in architecting and maintaining end-to-end automated testing suites that bridge traditional UI/API validation with complex, asynchronous AI/LLM evaluation pipelines.\n\nPipeline Integration: Hands-on experience integrating these comprehensive suites within GitHub Actions or modern CI/CD pipelines to ensure continuous, reliable quality checks throughout the entire development lifecycle.\n\nAdvanced AI Quality & Evaluation Mastery\n\nAI Evaluation Frameworks: Expertise in implementing evaluation harnesses (e.g., RAGAS, DeepEval, Promptfoo) to quantify LLM performance, hallucination rates, and answer relevancy.\n\nAgentic Workflow Validation: Ability to test autonomous agentic loops, including tool-use accuracy, step-by-step reasoning verification, and goal-directed performance metrics.\n\nEvaluation Datasets: Experience in curating, managing, and versioning 'Golden Datasets' for benchmarking model responses against ground truth in both development and production environments.\n\nObservability & Feedback Loops: Leveraging tracing (e.g., LangSmith, Arize Phoenix) to monitor production inputs/outputs, identify drift, and capture user feedback for continuous improvement.\n\nBasic broad AI knowledge: Understanding of LLM architectures, RAG systems, agents, and common failure modes in AI applications.\n\nPython Expertise: Advanced proficiency in writing robust, maintainable Python code for test automation (e.g., pytest). Experience using Python to orchestrate complex AI evaluation workflows, process large datasets for benchmarking, and automate interactions with LLM/AI services. Ability to integrate Python scripts into CI/CD pipelines for automated quality checks and performance observability in agentic systems.\n\nShould-have:\nCloud:Practical experience with testing applications on cloud platforms (AWS) and working with cloud-based AI services.\n\nObservability & ALM:Experience with monitoring and observability tools, log analysis, and performance metrics tracking (Grafana). Experienced in Roche mandatory ALM tools (Jira, GitHub).\n\nAgentic SDLC & Engineering Excellence:Leverage AI coding assistants and autonomous agents (e.g., Claude Code, Ona) daily to accelerate full-stack development and testing cycles. Conduct rigorous code reviews for both human-written and AI-generated code.\n\nSecurity:Understanding of security testing methodologies, particularly in the context of AI applications.\n\nCould-have:\nRegulatory Compliance: Proven understanding of AI quality assurance and testing standards within highly regulated industries.\n\nAdditional Qualifications\nStay current with the latest developments in AI and GenAI testing methodologies and best practices.\n\nHave excellent communication skills and the ability to clearly document findings and recommendations.\n\nBe able to communicate in English at the level of C1+.\n\nLocated in Hyderabad, India, with working hours structured to capture the 'golden hours' of overlap with Central European Time (typically running through the IST evening).\n\nWho we are\nA healthier future drives us to innovate. Together, more than 100’000 employees across the globe are dedicated to advance science, ensuring everyone has access to healthcare today and for generations to come. Our efforts result in more than 26 million people treated with our medicines and over 30 billion tests conducted using our Diagnostics products. We empower each other to explore new possibilities, foster creativity, and keep our ambitions high, so we can deliver life-changing healthcare solutions that make a global impact.\nLet’s build a healthier future, together.\nRoche is an Equal Opportunity Employer.","description_format":"text","description_chars":9089,"description_truncated":false,"requirements":{"experience_years_min":7,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[{"language":"English","level":"Upper-Intermediate (B2)","optional":false}]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Diagnostic Tests & IVD","Biologics & Biosimilars","Oncology Therapeutics"],"lifecycle":[{"event":"open","at":"2026-10-28T07:14:45Z"}],"visa":[],"liveness":{"score":63,"band":"ok","label":"Likely open","p_open":1,"p_active":0.632,"p_room":1,"age_days":3,"expected_fill_days":19,"reasons":["conf:11","stale_co","velocity","win:early","comp:brand"],"computed_at":"2026-11-01T06:10:25Z"},"pay":null,"html_url":"https://alion.io/job/roche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2","json_url":"https://alion.io/job/roche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2.json","meta":{"generated_at":"2026-11-01T23:44:05Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":36184,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-02T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":13531},"rest":"https://alion.io/mcp/rest/get_company?id=13531"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Froche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Froche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Froche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/roche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Froche-fullstack-ai-quality-engineer-applied-agentic-ai-systems-2"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3385491}},{"tool":"search_jobs","arguments":{"company":"roche.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3385491},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}