QA Lead - Automation

Key Roles & Responsibilities:

- Own and drive the end-to-end test automation strategy, defining the roadmap, frameworks, tooling and standards for UI, API, backend and integration testing.

- Lead, mentor and manage the QA team, owning delivery, prioritization, test coverage and quality outcomes.

- Work closely with engineering, product, data science/ML and client service teams to clarify requirements and define acceptance criteria.

- Create and maintain test plans, test scenarios, test data, traceability and release-level quality reports.

- Triage defects, assess business and customer impact, and coordinate defect prioritization with relevant teams.

- Test AI/ML model integrations end to end, including request/response contracts, preprocessing and post-processing, timeout/retry behavior, failure handling and fallback paths.

- Build model-focused test suites using representative and edge-case datasets; assess output correctness against agreed reference labels or acceptance criteria, and track metrics such as precision/recall where applicable.

- Validate LLM-enabled workflows for groundedness, consistency, hallucination risks, prompt/input sensitivity and regression when prompts, models or versions change; use defined evaluation criteria rather than subjective checks alone.

- Check model-service latency, throughput, rate limits, availability, version compatibility and behavior under malformed, incomplete or unexpected inputs.

- Perform end-to-end, system integration and regression testing across UI, backend services, APIs, databases and third party/model services.

- Establish performance baselines and report findings with actionable recommendations for capacity and scalability improvements.

- Plan and execute performance, load, stress, spike and endurance tests; analyze throughput, latency, concurrency, resource utilization and bottlenecks.

- Work with developers to reproduce issues, verify fixes and prevent regressions throughout the release cycle.

- Own test-suite execution in non-production environments and integrate automated checks into CI/CD pipelines where appropriate.

Required Skills:

- Strong experience testing web applications across UI, backend services, APIs, databases and end-to-end workflows.

- Strong QA fundamentals with substantial hands-on automation experience using Python; able to design, build, maintain and scale automation frameworks rather than only execute existing scripts.

- People leadership experience, including leading, mentoring and driving the output of a team of 3 or more QA engineers.

- Strong API testing and debugging skills, including REST APIs, response/schema validation, authentication, error handling and service-to-service integration.

- Experience with performance-testing tools such as JMeter, Locust, k6 or equivalent, and ability to interpret results and identify bottlenecks.

- Working knowledge of AI/ML model evaluation and integration testing; able to define test datasets, expected outcomes, thresholds and regression checks in collaboration with data science/ML engineers.

- Familiarity with testing LLM-based features, including prompt/version regression, groundedness and response-quality evaluation, structured-output validation and safe failure handling.

- Comfortable validating data and service behavior using SQL, logs, metrics and traces; familiarity with CI/CD-based automated test execution is an advantage.

- Good understanding of front-end technologies and browser-based testing, including JavaScript, HTML and CSS.

- Hands-on experience building and maintaining automation frameworks using tools such as Pytest, Robot Framework, Selenium, Cypress or equivalent; experience with API automation is essential.

- Experience working in Agile/Scrum teams and participating in sprint planning, refinement, reviews and release readiness.

- A strong quality mindset and a track record of promoting automation-first, risk-based testing and continuous quality improvement.

- Clear communication skills and the ability to work across engineering, product, QA and AI/ML teams to define and deliver quality outcomes.

Behavioral Expectations:

- Be excited by and have a positive outlook to navigate ambiguity.

- Passion for results and excellence.

- Team player.

- Must be able to get the job done by working collaboratively with others.

- Inquisitive and analytical mind; out-of-the-box thinking.

- Prioritize among competing opportunities, balancing consumer needs with business and product priorities, and clearly articulate the rationale behind product decisions.

- Straightforward and professional.

- Good communicator.

- Maintain high energy and motivate the team.

- A do-it-yourself orientation, consistent with the company's "roll-up-the-sleeves" culture.

- Proactive.

Skills

Automation Testing, Software Quality Assurance, Python, API Testing, Performance Testing, AI/ML Testing, PyTest, Selenium, Cypress, JMeter, CI/CD