QA Engineer

Staff AI Quality Engineer -

About the Role

As a Staff AI Quality Engineer -you will lead the technical test strategy and architectural validation for Lenovo’s cross-device agentic AI ecosystem. You will serve as the technical authority on evaluating non-deterministic systems, designing automated quality platforms that operate across personal computing form factors (PCs, tablets, and smartphones). You will drive non-functional quality standards-ensuring that privacy, security, accessibility, internationalization, and device thermals/battery efficiency are safeguarded-while acting as a crucial release partner alongside Product Management and PMO leads.

What Will You Do

• Test Automation Architecture: Architect scalable, distributed, cross-platform test automation infrastructure capable of executing synchronized, multi-device end-to-end tests across Windows and Android devices without reliance on UI flakiness.
• Agentic AI Quality Frameworks: Design automated evaluation methodologies to validate multi-step agent reasoning, tool-calling graphs, autonomous error recovery, and transactional rollback mechanisms; establish statistical release gates for non-deterministic model behaviors.
• Hardware & Runtime Resource Profiling: Formulate automated performance and endurance soak suites to benchmark on-device SLM execution against cloud fallback paths, monitoring thermal throttling, memory leaks, and battery draw across hardware device tiers.
• Non-Functional Quality Standards: Lead automated verification across non-functional pillars: red-teaming indirect prompt injection vulnerabilities, securing inter-process data barriers, standardizing automated accessibility (screen-reader verbosity, dynamic token streaming), and validating semantic fidelity across international locales.
• CI/CD Quality Pipelines & Gating: Drive the technical implementation of shift-left gating in continuous integration pipelines, establishing automated failure analytics, regression detection, and flakiness isolation frameworks.
• Cross-Functional Peer Leadership: Serve as a core technical gatekeeper alongside Product Management and PMO; establish release acceptance criteria, lead data-informed risk assessments during schedule compressions, and influence platform design for enhanced testability.

Basic Qualifications

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related technical field.
• 8+ years of progressive software quality engineering, test architecture, and automation experience covering multi-device hardware ecosystems (Windows and Android).
• 3+ years of experience leading quality strategy and automated evaluation frameworks for LLMs, agentic AI workflows, or conversational platforms.
• Advanced programming and architecture skills in Python, C++, C#, or Java, with a track record of building scalable test harnesses and mock environments for OS IPC, APIs, and background daemons.
• Proven experience architecting automated non-functional test frameworks: deep profiling of device performance (NPU/GPU/CPU, battery mA/h, memory), security red-teaming/prompt-injection defense, automated accessibility (WCAG), and i18n semantic evaluation.
• Deep technical expertise in CI/CD pipeline architecture, test orchestration, and statistical quality gating.

Preferred Qualifications

• Hands-on expertise implementing and tuning non-deterministic evaluation platforms (e.g., RAGAS, DeepEval, LangSmith, TruLens) in production pipelines.
• Direct experience testing local NPU runtimes and quantized models (INT4/INT8 vs. FP16) on client silicon (Qualcomm, Intel, AMD).
• Background designing automated testing for multimodal inputs (synchronized voice injection, screen-capture parsing, synthetic system events).
• Demonstrated ability to resolve complex release-gating disputes with Product Management and PMO using objective risk metrics and clear mitigation trade-offs.