Lead AI Testing Engineer

We are looking for a Lead AI Testing Engineer to drive the quality strategy for AI-driven and data migration initiatives spanning knowledge graphs, ontologies and RAG-based LLM features. You will define and execute QA practices across data ingestion, graph layers, rule evaluation engines and AI/LLM components while building automation frameworks that scale evaluation across complex, data-heavy systems.

Responsibilities

• Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rate
• Build test harnesses to scale evaluation beyond human-in-the-loop processes
• Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios
• Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation
• Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions
• Establish quality gates, acceptance criteria and test coverage models
• Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers
• Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation
• Integrate test suites into CI/CD pipelines and define release quality gates
• Champion automation-first and shift-left quality practices
• Leverage agentic AI and Gen AI tooling in testing and framework development

Requirements

• 8+ years of experience in software development, testing automation and DevOps
• Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration
• Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL
• Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness
• Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates
• Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems
• Background in backend and API testing (REST) covering data validation, integration and E2E testing
• Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework
• Hands-on experience using coding agents and agentic development daily
• Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks
• Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale
• Experience in Agile delivery using Git and Azure DevOps or JIRA
• English proficiency at B2 level or higher

Nice to have

• Skills in semantic search testing and evaluation
• Familiarity with Vector Database integration and retrieval validation
• Background in data science covering ML concepts, data pipelines or data engineering collaboration
• Proficiency in API tooling such as Postman and Bruno
• Experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or Jenkins

We offer

• We gather like-minded people:
  • Engineering community of industry professionals
  • Friendly team and enjoyable working environment
  • Flexible schedule and opportunity to work remotely within Poland
  • Chance to work abroad for up to 60 days annually
  • Business-driven relocation opportunities
• We provide growth opportunities:
  • Outstanding career roadmap
  • Leadership development, career advising, soft skills, and well-being programs
  • Certification (GCP, Azure, AWS)
  • Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru
  • English classes
• We cover it all:
  • Stable income (Employment Contract or B2B)
  • Participation in the Employee Stock Purchase Plan
  • Benefits package (health insurance, multisport, shopping vouchers)
  • Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more
  • Referral bonuses
  • Corporate, social and well-being events
• Please, note:
  • The set of bonuses might vary based on the role you apply for - specifics will be discussed with our recruiter during the general interview.
  • We will reach out to selected candidates exclusively.

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.