Quality Analyst

QA Engineer - Insights (AI/ML)

Location: Remote (US)

Department: Insights (AI/ML)

Reports to: Engineering Manager

About the Role

Irth is building a new AI-driven threat and risk management platform for pipeline asset integrity. The platform brings together three capabilities that have historically been separate at Irth:

• A governed, cross-product data platform built on Databricks and Azure
• An AI-powered ingestion layer that normalizes, repairs, and enriches customer data without services-heavy onboarding
• A reusable analytical layer that runs industry-standard, Irth-developed, and customer-built risk models against the data

We are seeking a mid-level QA Engineer to own quality across the platform end to end, including the user interface, APIs and platform services, and the underlying analytics and model layer.

This role goes beyond traditional application testing. You will validate that data survives ingestion and transformation intact, risk-model outputs are accurate and reproducible, and the evidence trail behind those outputs can withstand regulatory audit. Operators use these outputs to prioritize excavation and repair work, so a silently incorrect result can be far more consequential than a visibly broken interface.

You will own regression coverage and test automation while helping build quality into the delivery pipeline from the beginning rather than inspecting it at the end.

Key Responsibilities

1. End-to-End Test Ownership - Primary Responsibility

• Own the end-to-end test strategy across the user interface, APIs, and analytics layers, defining the appropriate coverage at each level.
• Design, build, and maintain automated test suites covering UI, API contracts and integrations, data validation, and model validation.
• Build and maintain regression coverage that runs on every change and provides a reliable signal for release readiness.
• Create and manage test data, including realistic messy inputs that reflect the quality and variability of actual customer data.
• Perform exploratory testing on new functionality to identify failures that scripted tests may not detect.

2. Data & Model Validation

• Validate data accuracy throughout ingestion and transformation, including row-level and field-level reconciliation, schema conformance, and completeness checks across Bronze, Silver, and Gold layers.
• Validate model outputs against expected results, known baselines, and golden datasets, including verification that identical inputs produce consistent and reproducible outputs.
• Test edge cases throughout the risk-calculation path, including missing attributes, boundary values, unusual segment geometry, and conflicting source records.
• Validate geospatial correctness, including alignment of results to pipeline centerline geometry and consequence-area assignment.
• Verify AI-assisted extraction and gap-filling workflows, ensuring low-confidence outputs are routed for human review rather than silently accepted.

3. CI/CD & Test Automation Infrastructure

• Integrate automated test suites into CI/CD pipelines with appropriate quality gates at each environment promotion.
• Own test-environment configuration, data seeding, and teardown, including infrastructure-as-code contributions where appropriate.
• Build and maintain the test automation framework, keeping suites fast, stable, maintainable, and trustworthy. Treat flaky tests as defects.
• Report quality signals to the team, including coverage, pass rates, defect escape rate, and quality trends over time.

4. Release Readiness & Non-Functional Testing

• Coordinate release testing and sign-off, including assessing what changed and what functionality could plausibly be affected.
• Support performance and load testing of APIs and analytical workloads ahead of release milestones.
• Support security testing activities, including access-control verification and tenant-isolation testing.
• Verify upgrade and migration paths, including data migrations from legacy formats and systems.

5. Audit-Defensible Test Evidence

• Produce and retain test evidence sufficient to support regulatory and audit review of risk-model outputs.
• Maintain traceability from requirements through test cases and execution results.
• Document validation approaches, test results, and known limitations for each model release.

6. Collaboration & Quality Culture

• Work with Product to translate requirements into clear, concrete, and testable acceptance criteria before development begins.
• Partner with engineers on testability, unit and integration coverage, and defect triage.
• Advocate for quality during design reviews and identify risks early rather than waiting until the end of a sprint.
• Maintain clear defect-management practices, including reproducible steps, severity assessment, impact analysis, and verification of fixes.

Requirements

Qualifications

Required Qualifications

• 3-5 years of software quality assurance experience spanning both manual and automated testing.
• Hands-on test automation experience across UI and API layers using tools such as Playwright, Cypress, Selenium, Postman, or REST Assured.
• Strong SQL skills and the ability to independently validate data rather than relying solely on application-level reporting.
• Scripting or programming proficiency, ideally in Python or JavaScript, sufficient to build and maintain an automation framework.
• Demonstrable CI/CD experience, including integrating automated tests into pipelines, implementing quality gates, and supporting environment promotion.
• Experience testing REST APIs, including contract testing, authentication, error handling, and asynchronous workflows.
• Experience with test planning, test-case design, defect management, and release sign-off.
• Working knowledge of Git and branch-based development workflows.
• Strong written communication skills, with the ability to create clear defect reports and test documentation that others can act on.

Preferred Qualifications

• Experience testing data pipelines, ETL processes, or analytics platforms, including data reconciliation and data-quality validation.
• Experience validating machine learning or statistical model outputs, including reproducibility and model regression testing.
• Experience with Databricks, Spark, or modern lakehouse platforms.
• Performance and load testing experience using tools such as JMeter, k6, or Locust.
• Experience with cloud platforms; Azure preferred, including Azure DevOps or GitHub Actions.
• Experience testing multi-tenant SaaS applications, including tenant isolation and access-control verification.
• Familiarity with geospatial data validation and testing.
• Experience using AI-assisted coding tools such as Cursor or GitHub Copilot and/or agentic coding tools such as Claude Code as part of a professional development workflow, including authoring and maintaining automated tests.

Nice to Have

• Experience working in a regulated industry where test evidence is subject to external audit.
• Understanding of pipeline integrity or asset-management concepts.
• Experience with security testing practices, including static and dynamic analysis tools.
• Experience validating data migrations from legacy or spreadsheet-based systems.
• Experience with accessibility testing.

Success Metrics

Success in this role will be measured by:

• Automated regression coverage across UI, API, and analytics layers that runs reliably in CI on every change.
• Low defect escape rates, with critical defects identified and resolved before reaching production.
• Data and model validation processes that identify accuracy issues before they reach customers.
• Audit-defensible test evidence maintained for every model release.
• Stable, fast, and trustworthy test suites that the engineering team relies on rather than routinely bypassing or overriding.
• Quality risks identified and communicated early enough in the development cycle to be addressed proactively.