Lead AI Testing Engineer

We are looking for a Lead AI Testing Engineer to drive the quality strategy for AI-driven and data migration initiatives spanning knowledge graphs, ontologies and RAG-based LLM features. You will define and execute QA practices across data ingestion, graph layers, rule evaluation engines and AI/LLM components while building automation frameworks that scale evaluation across complex, data-heavy systems.

Responsibilities

• Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rate
• Build test harnesses to scale evaluation beyond human-in-the-loop processes
• Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios
• Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation
• Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions
• Establish quality gates, acceptance criteria and test coverage models
• Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers
• Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation
• Integrate test suites into CI/CD pipelines and define release quality gates
• Champion automation-first and shift-left quality practices
• Leverage agentic AI and Gen AI tooling in testing and framework development

Requirements

• 8+ years of experience in software development, testing automation and DevOps
• Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration
• Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL
• Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness
• Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates
• Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems
• Background in backend and API testing (REST) covering data validation, integration and E2E testing
• Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework
• Hands-on experience using coding agents and agentic development daily
• Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks
• Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale
• Experience in Agile delivery using Git and Azure DevOps or JIRA
• English proficiency at B2 level or higher

Nice to have

• Skills in semantic search testing and evaluation
• Familiarity with Vector Database integration and retrieval validation
• Background in data science covering ML concepts, data pipelines or data engineering collaboration
• Proficiency in API tooling such as Postman and Bruno
• Experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or Jenkins

We offer

• We gather like-minded people:
• • Top tech minds driving innovation in AI, cloud and digital platform modernization
  • Supportive team and agile, startup-like culture
  • Hybrid by design mode and opportunity to work remotely within Poland
  • Chance to work abroad for up to 60 days annually
  • Business-driven relocation opportunities
• We provide growth opportunities:
• • Career development programs
  • Thought leadership, mentoring, soft skills and well-being programs
  • Certification (Anthropic, Gemini, GCP, Azure, AWS)
  • English classes
• We cover it all:
• • Stable pay
  • Participation in the Employee Stock Purchase Plan with a 15% discount
  • Benefits package (health insurance, multisport, shopping vouchers)
  • Referral bonuses up to $2,000
  • Offices featuring entertainment and relaxation zones, table tennis and football, free snacks, coffee and more
  • Corporate, social and well-being events
• Please, note:
• • Benefits listed above are available to employees only
  • We are open for working with Contractors. Terms of B2B cooperation agreements are agreed individually
  • We will reach out to selected candidates exclusively

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.