Senior Manager - QA Automation

We are looking for an experienced Senior QA Manager (AI, Automation, and Quality Engineering) to lead quality engineering initiatives across critical and complex products. The ideal candidate will have strong hands-on expertise in modern test automation using Playwright, a solid understanding of LLM-based applications and AI testing, and practical knowledge of Evals (AI evaluation). The candidate should be able to define and implement a comprehensive QA strategy, establish scalable automation frameworks, drive quality across the complete SDLC, and independently lead critical, high-impact projects from planning through production.

The core responsibilities for the job include the following:

QA Strategy and Leadership:

• Define and drive the overall QA and quality engineering strategy for complex products and programs.
• Establish quality goals, automation strategy, test coverage targets, release criteria, and quality metrics.
• Drive shift-left testing and ensure quality is embedded throughout the software development lifecycle.
• Own quality for critical projects and ensure timely delivery without compromising product quality.
• Identify quality risks early and establish mitigation plans.
• Drive continuous improvement in testing processes, tools, and engineering practices.

Test Automation; Playwright:

• Lead the adoption and implementation of Playwright-based automation for UI, API, and end-to-end testing.
• Design scalable, maintainable, and reusable automation frameworks.
• Define automation architecture, coding standards, test design patterns, and best practices.
• Drive migration from legacy automation frameworks such as Selenium to Playwright where applicable.
• Establish automated regression suites and integrate them into CI/CD pipelines.
• Improve automation reliability, execution speed, coverage, and maintainability.
• Review automation code and provide technical guidance to QA engineers.
• Drive automation toward meaningful coverage rather than simply increasing test-case counts.

LLM / AI Testing:

• Lead QA strategy for applications and features powered by Large Language Models (LLMs).
• Define testing approaches for prompt quality, response accuracy, relevance, hallucination, consistency, robustness, safety, guardrails, context handling, regression, and latency.
• Develop automated testing approaches for LLM-based applications.
• Work closely with AI/ML and engineering teams to establish measurable quality standards.
• Evaluate AI features across different models, prompts, configurations, and datasets.

Evals / AI Evaluation:

• Demonstrate strong practical knowledge of LLM Evals and AI evaluation methodologies.
• Define evaluation criteria, datasets, test scenarios, and quality thresholds for AI applications.
• Design and implement automated evaluation pipelines for LLM outputs.
• Work with metrics such as accuracy, relevance, faithfulness, toxicity, safety, and consistency as applicable.
• Establish golden datasets and benchmark test suites for continuous AI evaluation.
• Analyze evaluation results and convert findings into actionable engineering improvements.
• Integrate Evals into CI/CD so AI quality is continuously monitored across releases.
• Establish repeatable processes for evaluating prompt, model, RAG, and application changes.

Critical Project Ownership:

• Take end-to-end ownership of quality for business-critical and high-risk projects.
• Lead QA planning from requirements through production release.
• Build risk-based test strategies for complex projects with multiple dependencies.
• Identify critical business flows and ensure comprehensive E2E validation.
• Lead release readiness reviews and provide quality recommendations to leadership.
• Establish war-room processes for critical releases when required.
• Coordinate cross-functional teams during critical production issues.
• Drive RCA, corrective actions, and preventive actions for major defects.
• Ensure critical projects are delivered with predictable quality and minimal production risk.

QA Process and Engineering Excellence:

• Establish standardized QA processes across teams and projects.
• Define and monitor test strategy, automation coverage, defect leakage, regression effectiveness, production incidents, release quality, automation stability, and test execution efficiency.
• Improve defect prevention and early defect detection.
• Establish quality gates for development and release processes.
• Drive continuous improvement using metrics and data.
• Ensure QA processes scale effectively as products and teams grow.

Team Leadership and Mentoring:

• Lead, mentor, and develop QA engineers and automation engineers.
• Build a high-performing quality engineering organization.
• Establish technical and career development plans for team members.
• Conduct technical reviews and provide feedback on automation and testing practices.
• Encourage adoption of AI-assisted testing and modern QA engineering practices.
• Promote engineering ownership, accountability, and quality culture.

Cross-Functional Collaboration:

• Partner closely with engineering, product, DevOps, data science, AI/ML, and business teams.
• Collaborate with architects and developers to ensure testability and observability are built into the product.
• Communicate quality risks and release readiness clearly to senior leadership.
• Represent QA in architecture, design, planning, and release discussions.
• Influence engineering teams to adopt quality-first development practices.

Requirements:

• 15+ years of experience in software quality/quality engineering, with significant experience in automation.
• 5+ years of experience in QA leadership/management or technical leadership.
• Strong experience with modern automation frameworks, preferably Playwright.
• Experience owning quality for complex, enterprise-scale applications.
• Experience working with AI/ML or LLM-based applications is highly preferred.
• Practical knowledge of LLM Evals / AI evaluation is strongly preferred.
• Experience managing multiple teams/projects and driving critical releases.

Automation:

• Strong hands-on experience with Playwright.
• Strong understanding of browser automation and E2E testing.
• Strong programming skills in JavaScript/TypeScript or another suitable language.
• Experience designing automation frameworks from the ground up.
• Strong API automation and integration-testing knowledge.
• Experience with CI/CD tools such as Jenkins, GitHub Actions, or GitLab CI.
• Strong knowledge of Git and software engineering practices.

AI / LLM Testing:

• Strong understanding of LLM applications and AI-based systems.
• Strong practical knowledge of LLM testing and evaluation.
• Understanding of prompts, context windows, embeddings, RAG, agents, and model behavior.
• Experience designing test datasets and evaluation scenarios.
• Knowledge of automated Evals and continuous AI quality monitoring.
• Understanding of AI-specific failure modes such as hallucination, inconsistency, irrelevant responses, and prompt sensitivity.

Testing and Quality:

• Strong understanding of functional integration, API, E2E, regression, performance, risk-based, and production validation testing.
• Strong understanding of SDLC, STLC, Agile, Scrum, and CI/CD practices.

Leadership Competencies:

• Strong ownership and accountability.
• Ability to drive projects independently.
• Excellent analytical and problem-solving skills.
• Ability to work effectively under tight deadlines.
• Strong stakeholder management skills.
• Ability to challenge decisions constructively.
• Strong communication and presentation skills.
• Ability to manage ambiguity and rapidly changing requirements.
• Strong decision-making capability during critical releases.
• Ability to balance speed, risk, quality, and business priorities.

Preferred Qualifications:

• Experience migrating legacy automation frameworks to Playwright.
• Experience building automation frameworks and QA platforms.
• Experience with AI-assisted testing and test-case generation.
• Experience with RAG, AI agents, or conversational AI testing.
• Experience implementing automated Evals in CI/CD.
• Experience with cloud platforms such as AWS, Azure, or GCP.
• Experience with observability, quality metrics, and production monitoring.

Success Metrics:

• Improved automation coverage and reliability.
• Reduction in regression execution time.
• Reduction in escaped production defects.
• Improved release predictability.
• Increased automation stability and maintainability.
• Successful implementation of Playwright-based automation.
• Establishment of reliable LLM evaluations and AI quality metrics.
• Successful delivery of critical projects with minimal production impact.
• Improved QA engineering maturity across teams.