Senior Technical Architect (DPO)

Job Summary

Agentic QA Solution Architect

Quality Solution Architect for AI Agents & Agentic Automation

About the Role

AI agents are non-deterministic: the same input can produce different outputs across runs. Testing them requires a fundamentally different discipline than scripted test automation - evaluation frameworks, scoring functions, and statistical judgment instead of exact-match assertions. As an Agentic QA Engineer, you will bring that discipline directly to our customers.

This is a forward-deployed, customer-facing engineering role. You will work side by side with enterprise customers to design, build, test, and harden agentic workflows in their live environments - combining the hands-on coding of a software engineer, the quality mindset of a test automation expert, and the end-to-end design perspective of a solution architect. You will own the question every customer asks before putting an AI agent into production: "How do we know we can trust it?"

Key Responsibilities

What You Will Do

Deploy and validate agentic solutions at customers. Work with enterprise customers to implement, integrate, and operationalize AI agents and agentic workflows in production environments - including APIs, data pipelines, and integration with existing customer systems.

Design testing agents and evaluation strategies for deterministic and non-deterministic systems. Define what "quality" means for each agentic use case and turn it into measurable signals: evaluation suites, scoring functions, regression baselines, and pass/fail criteria that account for variability in agent behavior.

Build and run evaluation frameworks. Implement automated evals (LLM-as-judge, semantic similarity, structured-output checks, human annotation rubrics) that track accuracy, latency, and cost across agent runs, and compare every release against baselines to catch regressions.

Perform adversarial and safety testing. Probe agents for failure modes, hallucinations, prompt-injection vulnerabilities, and out-of-scope actions; validate guardrails, human-in-the-loop checkpoints, and escalation paths before go-live.

Automate the testing lifecycle. Build and maintain automated test suites for both deterministic components (integrations, workflows, UI) and agentic components, wired into CI/CD so quality gates run on every deployment.

Act as a trusted technical advisor. Translate customer requirements into solution and test architecture; explain evaluation results, risks, and trade-offs to audiences ranging from QA engineers to non-technical executives.

Architect for scale. Apply solution-architecture judgment to design quality strategies that span the full solution - agents, RPA/workflow/tools automation, integrations, and human-in-the-loop steps - and that customers can operate and extend themselves.

Close the loop with product and engineering. Feed field findings - failure patterns, missing capabilities, evaluation gaps - back into the product roadmap and internal best practices, and contribute to reusable test assets and accelerators.

Skill Requirements

What You Bring (Required)

Software engineering foundation. 7+ years of hands-on software development experience in Python, C#, VB.NET, or comparable languages, with the ability to write production-quality code and test harnesses.

Testing and QA depth. 3+ years in software testing / test automation, including test strategy design, automated test frameworks, and CI/CD-integrated quality gates.

Hands-on experience with AI agents. Practical experience building, deploying, or testing LLM-based agents and agentic workflows (e.g., tool/function calling, RAG, multi-step orchestration, frameworks such as LangChain/LangGraph or the Model Context Protocol), ideally in customer or production settings.

LLM evaluation literacy. Working understanding of non-determinism, prompting, grounding, and hallucination, and experience setting up evals - including LLM-as-judge approaches, rubric-based scoring, and regression testing of model or prompt changes.

Customer-facing experience. Proven track record in a forward-deployed, professional services, implementation, or consulting role: leading technical workshops, managing stakeholder expectations, and delivering under real-world constraints in customer environments.

Solution architecture background. Experience designing end-to-end solutions across applications, APIs, and automation platforms - able to see the whole system, not just the component under test.

Communication. Excellent written and verbal communication; comfortable presenting technical findings and quality risks to executive stakeholders. Fluent English.

Other Requirements

What Sets You Apart (Preferred)

Experience with agentic testing or AI-augmented QA platforms (e.g., UiPath Test Cloud / Test Manager, or comparable AI-powered testing tools) and with automation platforms more broadly (RPA, workflow orchestration).

Experience with evaluation and observability tooling for LLM applications (e.g., LangSmith, Braintrust, Promptfoo, or custom pytest-based eval harnesses) and with tracing/monitoring agent behavior in production.

Statistical literacy - able to distinguish signal from noise in non-deterministic outputs and reason about sample sizes, baselines, and confidence.

Experience with test management and ALM tooling (e.g., ADO, Jira - Xray, TestRail, Test Manager) and requirements-to-test traceability.

Knowledge of AI governance, safety, and compliance considerations for deploying agents in regulated industries.

Degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Willingness to travel to customer sites as needed.

How Success Is Measured

Customers move agentic use cases from pilot to production with documented, repeatable evaluation and test coverage.

Evaluation suites catch regressions before customers do - measured by escaped-defect and eval-coverage trends.

Customer stakeholders trust and understand the quality evidence behind every go-live decision.

Field learnings are converted into reusable test assets, patterns, and product feedback.