{"id":4488012,"url":"https://alion.io/job/black-forest-labs-research-lead-evaluations","title":"Research Lead, Evaluations","company":{"id":13106,"name":"Black Forest Labs","domain":"bfl.ai","url":"https://alion.io/company/bfl","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"A","score":93,"open_postings":11,"ghost_share":0,"stale_share":0.091,"repost_share":0,"time_to_fill_p50_days":166,"computed_at":"2026-11-17T05:47:56Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Freiburg im Breisgau, Germany"],"countries":["DE"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":255000,"max":295000,"currency":"EUR","period":"year","gross":null,"usd_annual":324933},"salary_estimate":null,"experience_years_min":1,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Computer Use","optional":false},{"name":"Multimodal AI","optional":false},{"name":"Post-training","optional":false},{"name":"Pre-training","optional":false},{"name":"Python","optional":false},{"name":"Reinforcement Learning","optional":false},{"name":"Stable Diffusion","optional":false}],"status":"live","first_seen_at":"2026-11-17T17:21:45Z","employer_posted_date":"2026-11-17","last_verified_at":"2026-11-17T21:20:58Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"About Black Forest Labs\nWe’re the team building the foundational layer for visual intelligence: multimodal models trained jointly on images, video, audio, and action to see, generate, and understand the world.\nOur founding team pioneered latent diffusion and Stable Diffusion, the breakthroughs that made generative AI accessible to millions. With FLUX, we're pushing into generative media at scale, machines that act in the physical world, and emerging applications in computer use.\nOur FLUX models top public benchmarks and run in production in some of the world's biggest platforms. We're a small, fast growing research lab competing with the largest companies in the world. From our labs in Freiburg to San Francisco, we're building what comes next.\nWhy This Role\nBFL’s research organization is split into different focus areas, such as pre-training, post-training, reinforcement learning, data, representation learning, evaluations and more. During the day to day work, researchers from these different focus areas work together to solve exciting and hard problems towards solving visual intelligence.\nHaving the right means to measure what matters is the central starting point for every research project. Researchers make decisions based on the outcomes of evaluations. The right decision can only be made if an evaluation is precise and accurate. As the Evaluations Lead you’re the core driver to ensure that. Central responsibilities include building our eval roadmap in collaboration with the research and product teams, managing and extending our evaluation team, and cross-functional work with project leads to make sure each research project has the best evaluation toolbox to help the people working on it to make the right decisions. Finally, you’re responsible for growing the people in your team and maintain an eye on their long term development during the daily project work.\nWhat You’ll Work On\nDefiningwhatwe evaluate: You are the core driver to create, maintain and extend our evaluations roadmap by collaborating with the product and other research leads who provide the required input.\n\nDefininghowwe evaluate: You’ll work with the team to implement our evals roadmap. This includes building new evals. Both benchmarks and the necessary objective functions, involving both automated, AI-based, and human evaluations.\n\nMinimizing noise in our eval pipeline: Developing frontier AI system requires the most accurate means to measure what matters! You will ensure that results are reproducible and trustworthy, that we understand the error bars behind them and that we measure what we want to measure.\n\nUnderstanding what we cannot measure: You’re responsible for highlighting and clearly communicate limitations of each individual evaluation and you will support our project teams to ensure that their chosen set of evaluations has no blind spots in when combined. ****\n\nObserving training runs and visualizing important findings: You will help with monitoring our most critical training runs from an independent standpoint to ensure that regressions and emerging capabilities are caught early and are clearly communicated.\n\nManaging our evals team: Your role is a hands-on management role - you’re responsible for growing the team and the people in it. You will define our hiring roadmap and make sure that each team member can grow within BFL, both in terms of their technical skillset and their career.\n\nWhat We’re Looking For\nProven eval experience: You’ve previously designed, run and maintained large and complex evaluation suites for frontier AI systems, involving various kinds of evaluation methods involving both human and automated (both with or without AI) approaches.\n\nStrong research background: You should have proven experience of designing and implementing evaluations for AI research. Strong understanding of Python is required. A strong understanding of agentic coding is a must.\n\nExperience as a people manager: One of your core responsibilities will be to grow the team and its members. Understanding what it means to lead a team and how it’s different from an IC role is a must. You should have actively managed a team for at least one year.\n\nStrong understanding in statistics and good judgement about numbers: You should be familiar with variance, confidence intervals and significance testing. You should be able to identify and resolve situations when blindly trusting numbers leads to wrong conclusions.\n\nExcellent communication skills: You will have to communicate results with project leads and research leadership. Understanding how to communicate these results and what they don’t tell us is important since they’re the foundation for important decisions.\n\nFirst principles thinking: Crafting evaluations for new capabilities is not straightforward. You need to be able to derive new evals which requires proficiency in thinking from first principles\n\nHow We Work Together\nWe’re a distributed team with real offices that people actually use. This role is based in Freiburg, and you’ll join us in the office at least 2 days a week, or one full week every other week.\nEverything we do is grounded in four values:\nObsessed. We are a frontier research lab. The science has to be right, the understanding deep, the product beautiful.\n\nLow Ego. The work speaks. The best idea wins, no matter who said it. Credit is shared. Nobody is above any task.\n\nBold. We take the ambitious bet. We ship, we do not wait for conditions to be perfect.\n\nKind. People over politics. We treat each other with genuine warmth. Agency without empathy creates chaos.\n\nIf this sounds like work you’d enjoy, we’d love to hear from you.\nCompensation\nEUR255K- 295K + equity, depending on profile\nNote: We may use AI-assisted tools to help organize applications. All hiring decisions are always made by our team.","description_format":"text","description_chars":5832,"description_truncated":false,"requirements":{"experience_years_min":1,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Generative Image","Foundation Models"],"lifecycle":[{"event":"open","at":"2026-11-17T21:20:58Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":166,"reasons":["conf:1","win:early"],"computed_at":"2026-11-17T23:08:47Z"},"pay":{"stated_usd_annual":324933,"is_top_pay":true},"html_url":"https://alion.io/job/black-forest-labs-research-lead-evaluations","json_url":"https://alion.io/job/black-forest-labs-research-lead-evaluations.json","meta":{"generated_at":"2026-11-17T23:08:47Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":7537,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":13106},"rest":"https://alion.io/mcp/rest/get_company?id=13106"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fblack-forest-labs-research-lead-evaluations"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fblack-forest-labs-research-lead-evaluations"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fblack-forest-labs-research-lead-evaluations"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/black-forest-labs-research-lead-evaluations\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fblack-forest-labs-research-lead-evaluations"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4488012}},{"tool":"search_jobs","arguments":{"company":"bfl.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4488012},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}