{"id":726169,"url":"https://alion.io/job/cortea-seniorstaff-ai-engineer-quality-evals-mfx","title":"Senior/Staff AI Engineer, Quality & Evals (m/f/x)","company":{"id":673528,"name":"Cortea","domain":"cortea.ai","url":"https://alion.io/company/cortea","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":75,"open_postings":5,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-11-04T06:34:25Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"staff","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Berlin, Germany"],"countries":["DE"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":99000,"max_usd":217000,"period":"year","method":"role_seniority_country_cell","sample_n":8},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Braintrust","optional":false},{"name":"ClickHouse","optional":false},{"name":"GCP","optional":false},{"name":"Google BigQuery","optional":false},{"name":"Langfuse","optional":false},{"name":"LLM","optional":false},{"name":"MLFlow","optional":false},{"name":"Multimodal AI","optional":false},{"name":"Python","optional":false},{"name":"Weights & Biases","optional":false},{"name":"ETL/ELT","optional":true}],"status":"live","first_seen_at":"2026-09-09T08:13:53Z","employer_posted_date":"2026-10-08","last_verified_at":"2026-11-04T18:54:46Z","board_verified":true,"closed_at":null,"days_open":56,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":56},"description":"About us\nWe’re Cortea, a Berlin startup transforming audits with AI. Manual, document-heavy audits waste expert time while demand keeps rising. Our AI-powered software and specialized AI agents remove the repetitive work so auditors can focus on judgment.\nBacked by top-tier VCs with >15m EUR funding, with a working product and paying customers, we’re rapidly scaling.\nWe value first-principles thinking, speed, trust, and kindness. We build side by side in our Berlin office.\nYour Role\nWe are looking for an engineer with strong backend, data, and AI systems experience to build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows.\nThis role sits at the intersection of backend engineering, data infrastructure, and AI quality. You will build the evaluation systems that power our multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines.\nYou’ll work at the edge of applied AI and information retrieval, building multimodal agentic pipelines and solving hard context and agent-harness engineering problems.\nThis is not a traditional analytics, BI, or dashboarding role. You should expect to write production code, design data architecture, work inside backend systems, and directly improve the quality, cost, reliability, and performance of LLM-based agents.\nWhat you’ll do\nYou will help build and operate the technical systems around our AI agents, with a focus on data infrastructure, evaluation, observability, and optimization. You will:\nBuild online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results.\n\nCreate automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.\n\nAnalyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities.\n\nBuild reliable data retention and replay mechanisms for long-term analysis of production agent behavior.\n\nManage observability tools for tracing, monitoring, debugging, and experiment management of our audit agents.\n\nTeam up with backend engineers to improve the speed and reliability of our retrieval and reasoning agents.\n\nYou will fit into the role if you...\nHave strong Python and/or backend engineering experience.\n\nHave a solid understanding of how LLM and agent systems are evaluated-including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics-and can reason about when each approach is appropriate.\n\nHave deployed and operated systems in the cloud, ideally on GCP.\n\nHave hands-on experience building end-to-end retrieval or ML pipeline evaluation systems and using LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.\n\nAre comfortable working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data.\n\nUnderstand system design, reliability, observability, monitoring, logging, debugging, and operational trade-offs.\n\nBring senior-level engineering judgment: you can make architectural decisions, communicate trade-offs, and build systems that other engineers can extend.\n\nAre comfortable with ambiguity, able to reason from first principles, and excited to build infrastructure for AI systems that are actively used in production.\n\nNice-to-haves that are a plus:\nDesigning data pipelines, ETL/ELT workflows, event-processing systems, or feedback loops for production data.\n\nBuilding infrastructure around LLM-based products or agentic systems, including optimizing LLM usage, context windows, reasoning tokens, or model selection.\n\nWorking with production traces from complex distributed systems.\n\nBuilding internal platforms for engineers, domain experts, or operations teams.\n\nUsing workflow orchestration systems such as Temporal or similar.\n\nFamiliarity with audit, finance, compliance, or other high-accuracy domains.\n\nExperience in an early-stage startup or fast-moving engineering environment.\n\nNo-one checks every box. If you’ve shipped retrieval systems and like owning evaluations and pipelines, let’s talk.\nWhat we offer\nHigh impact & growth: Shape strategy at a scaling AI startup from day one.\n\nMission-driven culture: Ambitious team valuing first-principles thinking and bold ideas.\n\nAttractive compensation: competitive salary plus significant equity.\n\nBest tools for the job: Generous coding tools budget so you can use the best tools for the job.\n\nStartup perks: Flexible vacation, team lunches, retreats, central Berlin office.\n\nInterview process\nFirst Call - Intro to Cortea with Liza\n\nSecond Call - Technical interview with Vlad\n\nThird Call - Deep dive into our culture with our Co-Founder Philipp\n\nOn-site Day (Berlin) - Meet the team and work on a real problem together\n\nWe’re an equal-opportunity team and encourage women and underrepresented groups to apply.","description_format":"text","description_chars":5114,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Financial AI","Accounting & Tax Software"],"lifecycle":[{"event":"open","at":"2026-09-11T08:27:04Z"}],"visa":[],"liveness":{"score":29,"band":"fade","label":"Fading","p_open":1,"p_active":0.637,"p_room":0.45,"age_days":55,"expected_fill_days":37,"reasons":["conf:1","win:tail"],"computed_at":"2026-11-04T06:34:25Z"},"pay":null,"html_url":"https://alion.io/job/cortea-seniorstaff-ai-engineer-quality-evals-mfx","json_url":"https://alion.io/job/cortea-seniorstaff-ai-engineer-quality-evals-mfx.json","meta":{"generated_at":"2026-11-04T19:52:57Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":30594,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-05T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":673528},"rest":"https://alion.io/mcp/rest/get_company?id=673528"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcortea-seniorstaff-ai-engineer-quality-evals-mfx"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcortea-seniorstaff-ai-engineer-quality-evals-mfx"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcortea-seniorstaff-ai-engineer-quality-evals-mfx"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cortea-seniorstaff-ai-engineer-quality-evals-mfx\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcortea-seniorstaff-ai-engineer-quality-evals-mfx"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":726169}},{"tool":"search_jobs","arguments":{"company":"cortea.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":726169},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}