Software Development Engineer in Test - AI

Role Summary:

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and agentic systems.

What you'll do & how you'll make your mark.:

• AI/LLM Quality Engineering
  • Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
  • Develop AI evaluation (Evals) frameworks for: hallucination detection, groundedness, toxicity and bias validation, response quality, context relevance, prompt regression testing, determinism and consistency, safety and guardrail validation.
  • Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
  • Create synthetic datasets and benchmark datasets for AI testing scenarios.
  • Test multi-agent orchestration workflows and agent memory/context retention.
  • Validate MCP (Model Context Protocol) integrations and tool interoperability.
  • Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies.
  • Evaluate latency, token usage, model cost optimization, throughput, and scalability.
• Automation Engineering
  • Develop and maintain scalable UI/API automation frameworks using: Python (mandatory), Robot Framework (preferred), Playwright/Selenium/Cypress.
  • Automate testing for: REST APIs, GraphQL APIs, event-driven systems, streaming systems, webhooks, microservices, MFE (Micro Frontend) architectures.
  • Build reusable automation libraries, utilities, and AI-assisted testing workflows.
  • Integrate automation into CI/CD pipelines and quality gates.
• AI Engineering & Platform Validation
  • Validate RAG architectures including chunking strategies, embedding quality, retrieval accuracy, re-ranking, vector search relevance.
  • Work with vector databases such as: Pinecone, Weaviate, ChromaDB, Qdrant, FAISS.
  • Build or validate AI agents leveraging: LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, OpenAI SDKs.
  • Validate AI workflows involving: tool calling, function calling, agent memory, agent planning/execution, multi-modal AI systems, structured outputs.
• DevOps, Observability & Reliability
  • Build and maintain quality workflows integrated with: GitHub Actions, Jenkins, GitLab CI/CD, Azure DevOps.
  • Work with Docker and Kubernetes environments.
  • Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
  • Validate distributed systems reliability and resiliency.
  • Monitor AI system behaviors using observability platforms/logging/tracing tools.
  • Contribute to shift-left and quality engineering best practices.

Who you are & what you'll need to succeed.:

• Bachelor's/Master's degree in Computer Science, Engineering, AI/ML, or related field preferred.
• Equivalent hands-on experience and demonstrated projects are highly valued.
• Strong programming experience in Python.
• Strong understanding of Software Testing & Quality Engineering principles.
• Hands-on experience with: AI/LLM testing, Prompt Engineering, Context Engineering, AI Evals, RAG systems, AI Agents.
• Experience with: CI/CD pipelines, Docker & Kubernetes, cloud platforms (AWS/Azure/OCI/GCP), API testing & automation, microservices testing, UI automation.
• Experience with SQL and data validation.
• Strong debugging, analytical, and troubleshooting skills.
• Experience working in Agile/Scrum environments.

Good to have skills:

• Experience with Robot Framework.
• Experience with Playwright.
• Knowledge of: LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, MCP (Model Context Protocol).
• Hands-on experience with: Cursor, GitHub Copilot, Claude, Gemini, OpenAI APIs.
• Experience with vector databases and embeddings.
• Exposure to MLOps workflows and ML lifecycle testing.
• Knowledge of performance testing and security testing.
• Experience validating AI governance/compliance requirements.
• Experience with contract testing, chaos testing, or resilience engineering.
• Familiarity with accessibility testing and cross-browser/device testing.
• Exposure to synthetic monitoring and production observability.

Bonus Skills (High Impact):

• AI Red Teaming.
• LLM Security Testing (Prompt Injection, Jailbreaks, Data Leakage).
• Agent Reliability & Safety Testing.
• AI Cost Optimization.
• Distributed Systems Testing.
• Test Data Engineering.
• Feature Flag Testing.
• AI Observability & Telemetry.
• Reinforcement Learning feedback workflows.
• Human-in-the-loop evaluation systems.
• Knowledge Graph integrations.
• Multi-modal AI testing (vision/audio/text).

What We're Looking For:

• Engineers who actively experiment with AI tools and build side projects, prototypes, PoCs, or open-source contributions.
• Strong problem solvers who can combine traditional QE practices with modern AI-native engineering approaches.
• Passion for innovation, developer productivity, automation acceleration, and intelligent testing.
• Ability to rapidly learn emerging AI frameworks, tools, and architectures.

Nice Portfolio Signals:

• Candidates with one or more of the following will stand out:
• GitHub repositories showcasing AI/automation projects.
• AI agents or copilots built personally or professionally.
• Contributions to open-source automation or AI ecosystems.
• Blogs, talks, workshops, or technical content on AI/testing.
• Experience integrating AI into SDLC or QE workflows.
• Experience with autonomous testing systems or self-healing automation.

Skills

API, CrewAI, Cypress, GraphQL, LLM, LangChain, LangGraph, Playwright, Python, RAG, Selenium