{"id":3960648,"url":"https://alion.io/job/alten-ai-evaluation-engineer-testing-oraz-development","title":"AI Evaluation Engineer (Testing oraz Development)","company":{"id":52934,"name":"ALTEN","domain":"alten.com","url":"https://alion.io/company/alten-com","size_band":"5000+","is_staffing_agency":false,"employer_type":"staffing","is_intermediary":false,"listed_via":null,"ats_vendor":"SmartRecruiters","truth_index":{"grade":"B","score":76,"open_postings":91,"ghost_share":0,"stale_share":0.967,"repost_share":0,"time_to_fill_p50_days":27,"computed_at":"2026-11-09T06:03:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Kraków, Poland"],"countries":["PL"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Azure","optional":false},{"name":"GCP","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false}],"status":"live","first_seen_at":"2026-11-09T10:24:21Z","employer_posted_date":null,"last_verified_at":"2026-11-09T10:24:21Z","board_verified":false,"closed_at":null,"days_open":0,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":0},"description":"Why us? 🤝\n\nWe are part of the ALTEN Group – a leading European provider of engineering and technology consulting servics.\nJoin ALTEN Polska and let’s grow together!\n\n🌎Location: Kraków- hybird work (6 x month)\n\n👉In this role, you will:\n\nWe are seeking an AI Evaluation & Development Engineer who will play a key role in testing, evaluating, and improving AI agents and GenAI applications while contributing to the development of supporting services and integrations.\n\nTasks associated with this role include:\n\nDesign, build, and maintain evaluation frameworks for AI agents and Generative AI solutions.\n\nDefine testing methodologies, benchmarks, and quality metrics for agent performance.\n\nEvaluate AI outputs for accuracy, relevance, consistency, safety, and adherence to business requirements.\n\nIdentify and analyze hallucinations, failure cases, and context-related issues.\n\nCreate automated testing pipelines and regression testing strategies for GenAI applications.\n\nValidate Retrieval Augmented Generation (RAG) systems, memory stores, and context-engineering mechanisms.\n\nCollaborate with AI Engineers, Data Engineers, and Product teams to improve agent behavior and user experience.\n\nDevelop Python-based services and tools supporting evaluation and monitoring processes.\n\nAnalyze prompts and agent orchestration workflows to optimize performance and quality.\n\nSupport production monitoring and continuous improvement of AI solutions.\n\n👉Key Skills and Qualifications:\n\nStrong experience with Python development.\n\nHands-on experience testing AI, Machine Learning, or Generative AI systems.\n\nUnderstanding of Large Language Models (LLMs), AI agents, and RAG architectures.\n\nExperience creating test scenarios, evaluation datasets, and automated validation frameworks.\n\nKnowledge of API testing and automation tools.\n\nExperience working with cloud environments such as Azure or Google Cloud Platform.\n\nStrong analytical mindset and ability to investigate complex AI behaviors.\n\n🎯 We offer\n\nWe offer a full-time employment contract\nStable and long-term cooperation\n\nWell-defined career path at the European leader in engineering & IT consulting\n\nOpportunity to relocate and work in different ALTEN Polska branches\n\nAfter completion of the project, opportunity to engage in a subsequent one within the company\n\nIntroduction and cooperation with dedicated Competences Manager\n\n🎯Benefits:\nMedicover medical care\n\nMedicover dental care\n\nMedicover Benefits platform\n\nMedicover Sport card\n\nReferral Program\n\nLayette for a newborn employee’s child\n\nGroup life insurance\n\nPension scheme\n\n🎯We also offer cooperation based on a B2B or Contract of mandate (UZ)\n\n🎯We also guarantee:\nAccess to professional certifications with our technology partners (e.g. Google, ISTQB)\n\nAccess to relevant industry events\n\nAccess to the Knowledge Sharing Program\n\nAccess to Referral program\n\nDo not hesitate and join our team!\n\nNOTICE: We kindly inform you that we will contact the selected people.\n\nDo not hesitate and join our team!\nTech stack\nPython\nMachine Learning\nRAG\nAI","description_format":"text","description_chars":3052,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Life insurance"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","Engineering Services"],"lifecycle":[{"event":"open","at":"2026-11-09T11:27:35Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":27,"reasons":["seen:0","win:early","comp:brand"],"computed_at":"2026-11-09T16:18:32Z"},"pay":null,"html_url":"https://alion.io/job/alten-ai-evaluation-engineer-testing-oraz-development","json_url":"https://alion.io/job/alten-ai-evaluation-engineer-testing-oraz-development.json","meta":{"generated_at":"2026-11-09T16:18:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":23617,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-10T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":52934},"rest":"https://alion.io/mcp/rest/get_company?id=52934"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Falten-ai-evaluation-engineer-testing-oraz-development"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Falten-ai-evaluation-engineer-testing-oraz-development"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Falten-ai-evaluation-engineer-testing-oraz-development"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/alten-ai-evaluation-engineer-testing-oraz-development\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Falten-ai-evaluation-engineer-testing-oraz-development"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3960648}},{"tool":"search_jobs","arguments":{"company":"alten.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3960648},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}