Senior Software Engineer in Test - Python, AI Agents

We are seeking a Senior Software Engineer in Test - Python, AI Agents. The person in this role will focus on designing and implementing automated testing solutions for agentic AI systems. This position emphasizes engineering skills and test automation rather than manual QA. The work will include developing Python-based test frameworks, validating probabilistic agent outputs, configuring test environments, and integrating tests within CI/CD pipelines.

Responsibilities

• Develop and maintain Python scripts, test frameworks, and notebooks to validate AI agents and related services.
• Design and implement automated test scenarios and validation logic for agentic systems with non-deterministic outputs.
• Integrate and execute automated tests in CI/CD pipelines to ensure continuous quality checks.
• Perform command-line and Model Context Protocol (MCP) testing of agents and related components.
• Prepare, configure and manage test environments, including containerization and orchestration where applicable.
• Create realistic customer and user scenarios to exercise end-to-end agent behavior and system integrations.
• Build Python notebooks and utilities for metadata extraction, analysis and reporting of test results.
• Collaborate with engineering, product and operations teams to define test strategies and acceptance criteria.
• Contribute to test tooling, observability and documentation to support reproducible and traceable testing practices.

Qualifications

• Proven experience in Python development for test automation, scripting and tooling (senior level).
• Strong expertise in test automation frameworks and practices, including scenario automation and validation of probabilistic outputs.
• Experience with CI/CD systems and integrating automated tests into pipelines.
• Practical knowledge of API and integration testing approaches.
• Comfortable with Git-based workflows and version control.
• Familiarity with Linux/CLI environments and command-line tools.
• Experience with containerization technologies such as Docker.
• Applied experience with AI, large language models or agentic systems and their evaluation approaches.
• Working knowledge of Model Context Protocol (MCP) or equivalent agent interaction protocols is desirable.
• Familiarity with Jupyter or other notebook environments for prototyping and metadata analysis.
• Ability to design realistic customer scenarios and to reason about non-deterministic system outputs.
• Strong communication skills and the ability to collaborate across technical teams.

Benefits

• Opportunities to work on advanced AI agent testing and to influence testing strategy and tooling.
• Access to modern development and testing infrastructure, including containerized environments and CI/CD platforms.
• Remote work. Flexible working arrangements consistent with role requirements.
• Unique TEAL culture, relationship- and respect-driven community, non-corporate atmosphere.
• Agile approach and no bureaucracy.
• Outstanding integration trips to various places in Europe.
• Activities to support your well-being and health.
• Luxmed Gold Extended medical care and Multisport Plus benefit.