AI Evaluation Engineer (Testing oraz Development)
First seen by Alion on Nov 9, 2026. ALTEN scores B on the Alion truth index.
Why us? 🤝
We are part of the ALTEN Group – a leading European provider of engineering and technology consulting servics.
Join ALTEN Polska and let’s grow together!
🌎Location: Kraków- hybird work (6 x month)
👉In this role, you will:
We are seeking an AI Evaluation & Development Engineer who will play a key role in testing, evaluating, and improving AI agents and GenAI applications while contributing to the development of supporting services and integrations.
Tasks associated with this role include:
Design, build, and maintain evaluation frameworks for AI agents and Generative AI solutions.
Define testing methodologies, benchmarks, and quality metrics for agent performance.
Evaluate AI outputs for accuracy, relevance, consistency, safety, and adherence to business requirements.
Identify and analyze hallucinations, failure cases, and context-related issues.
Create automated testing pipelines and regression testing strategies for GenAI applications.
Validate Retrieval Augmented Generation (RAG) systems, memory stores, and context-engineering mechanisms.
Collaborate with AI Engineers, Data Engineers, and Product teams to improve agent behavior and user experience.
Develop Python-based services and tools supporting evaluation and monitoring processes.
Analyze prompts and agent orchestration workflows to optimize performance and quality.
Support production monitoring and continuous improvement of AI solutions.
👉Key Skills and Qualifications:
Strong experience with Python development.
Hands-on experience testing AI, Machine Learning, or Generative AI systems.
Understanding of Large Language Models (LLMs), AI agents, and RAG architectures.
Experience creating test scenarios, evaluation datasets, and automated validation frameworks.
Knowledge of API testing and automation tools.
Experience working with cloud environments such as Azure or Google Cloud Platform.
Strong analytical mindset and ability to investigate complex AI behaviors.
🎯 We offer
We offer a full-time employment contract
Stable and long-term cooperation
Well-defined career path at the European leader in engineering & IT consulting
Opportunity to relocate and work in different ALTEN Polska branches
After completion of the project, opportunity to engage in a subsequent one within the company
Introduction and cooperation with dedicated Competences Manager
🎯Benefits:
Medicover medical care
Medicover dental care
Medicover Benefits platform
Medicover Sport card
Referral Program
Layette for a newborn employee’s child
Group life insurance
Pension scheme
🎯We also offer cooperation based on a B2B or Contract of mandate (UZ)
🎯We also guarantee:
Access to professional certifications with our technology partners (e.g. Google, ISTQB)
Access to relevant industry events
Access to the Knowledge Sharing Program
Access to Referral program
Do not hesitate and join our team!
NOTICE: We kindly inform you that we will contact the selected people.
Do not hesitate and join our team!
Tech stack
- Python
- Machine Learning
- RAG
- AI