{"id":2442093,"url":"https://alion.io/job/black-forest-labs-applied-research-engineer-evaluation","title":"Applied Research Engineer, Evaluation","company":{"id":13106,"name":"Black Forest Labs","domain":"bfl.ai","url":"https://alion.io/company/bfl","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"C","score":63,"open_postings":13,"ghost_share":0.615,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-14T06:34:16Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Freiburg im Breisgau, Germany"],"countries":["DE"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":110000,"max_usd":256000,"period":"year","method":"role_country_seniority_unknown","sample_n":60},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"DPO","optional":false},{"name":"Fine-tuning","optional":false},{"name":"LLM","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Multimodal AI","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"Reward Modeling","optional":false},{"name":"RLHF","optional":false},{"name":"Stable Diffusion","optional":false}],"status":"live","first_seen_at":"2026-10-13T11:52:34Z","employer_posted_date":"2026-10-13","last_verified_at":"2026-10-14T09:27:41Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":1},"description":"About Black Forest Labs\nWe're the team behind Latent Diffusion, Stable Diffusion, and FLUX - foundational technologies that changed how the world creates images and videos. Our FLUX models are among the most advanced in the world, and we're just getting started.\nHeadquartered in Freiburg, Germany with a growing presence in San Francisco, we're scaling fast while staying true to what makes us different: research excellence, open science, and building technology that expands human creativity.\nWhy This Role\nFrontier image and video models are now good enough that the hard question is no longer whether they work, but whether a new one is actually better, and how you'd know. Aesthetics, prompt following, editing fidelity, text rendering, character consistency: none of these collapse into a single number, and public benchmarks have fallen well behind the models they're meant to measure.\nYou'll work on the system BFL uses to answer that question. Every FLUX checkpoint, every finetuning and RL run, and every competitor release goes through our evaluation stack, and you'll help build and sharpen it. Product Research relies on what it shows to decide what to fix next, and to confirm the fix worked.\nIt gets harder from here. FLUX 3 is our first multimodal model, spanning image, video, audio and action, and each new modality needs its own way of being measured.\nWhat You'll Work On\nWhat you focus on will depend on where you're strongest and what the team needs most when you join.\nIn scope:\nBuilding and maintaining the evaluation stack that every model checkpoint has to pass through and prove itself on, from automated benchmarking and LLM-as-judge pipelines, to human evaluation through panels (Prolific and otherwise).\n\nDesigning the breadth of evaluation across the taxonomy, from prompt adherence, editing fidelity, aesthetics, text rendering, subject and character consistency, to safety.\n\nEvaluating the evaluation stack itself: checking that human and automated evals agree, that results stay stable over time, and improving them when they don't.\n\nBenchmarking FLUX against the frontier models continuously, while obtaining a meaningful perspective on where we are better or worse than competitors and what such differences really mean.\n\nFinding the ways in which the quality of the model's output falls short that are not captured by aggregate numbers.\n\nClosing the loop, ideally, with the RL and finetuning researchers, by defining target metrics for them to aim for, and verifying that a new checkpoint improved the weakness without introducing new ones.\n\nDesigning how best to exploit the signals we already capture through the API and MCP, and what new signals we should capture, with help from research, data engineering and legal.\n\nMaintaining the usefulness of existing metrics as their natural lifecycle brings up concerns of gaming and overfitting, and anticipating this for new metrics.\n\nWhat We're Looking For\nYou have made meaningful contributions to the evaluation of generative models already. You understand that a benchmark figure without careful consideration of sample size and judge agreement with humans is at best uninformed optimism and at worst dangerously misleading, and you have been the person that found that inconvenient regression that nobody wanted to talk about.\nYou likely have:\nAdvanced degree (MS/PhD) in a relevant field or demonstrated equivalent practical experience\n\nPractical experience with evaluating generative models, in a hands-on way. You may have worked with diffusion or flow-matching based image or video models already, but the capability to evaluate them in earnest transfers to other modalities too\n\nVery strong Python and software engineering skills, with pipelines that run reliably and that other people can read if needed\n\nWorking knowledge of PyTorch and modern ML infrastructure and practices; you are not a pure Python developer, you know what the modern ML code looks like\n\nStrong grasp of statistics and its shortcomings in practice, and how to work with those; sample sizes, inter-annotator agreement, confidence levels, etc., and a healthy skepticism about what those numbers say about the world\n\nPractical experience with building and operating LLM-as-judge systems, and understanding their limitations\n\nWorking knowledge of modern benchmarking practices and of standard datasets and benchmarks in the field already; you have an idea of which to trust and which to disregard\n\nThe inclination to look at the actual outputs your system is judging, and to care about what is there beyond an aggregate score\n\nWe'd be especially excited if you:\nHave practical experience with human evaluation, including study design and execution via panels, and analysis of disagreements and annotator performance\n\nHave worked where evaluation meets training (reward models, preference data, RLHF, DPO or similar) and know that whatever you optimize for is what the model learns\n\nHave evaluated agentic or multi-step systems and know how hard it is to attribute success or failure to a single step\n\nHave built tools that people actually want to use internally\n\nHave made public contributions to the field: papers, open source, or otherwise\n\nHow We Work Together\nWe're a distributed team with real offices that people actually use. This role is based in Freiburg, where you'll join us at least 2 days a week (or one full week every other week). We'll cover reasonable travel costs to make this possible. We think in-person time matters, and we've structured things to make it accessible to all. We'll discuss what this will look like during our interview process.\nEverything we do is grounded in four values:\nObsessed. We are a frontier research lab. The science has to be right, the understanding deep, the product beautiful.\n\nLow Ego. The work speaks. The best idea wins, no matter who said it. Credit is shared. Nobody is above any task.\n\nBold. We take the ambitious bet. We ship, we do not wait for conditions to be perfect.\n\nKind. People over politics. We treat each other with genuine warmth. Agency without empathy creates chaos.\n\nIf this sounds like work you'd enjoy, we'd love to hear from you.\nCompensation : €150K-€220K base + equity depending on experience","description_format":"text","description_chars":6232,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"master","optional":false},"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Generative Image","Foundation Models"],"lifecycle":[{"event":"open","at":"2026-10-13T12:23:56Z"}],"visa":[],"liveness":{"score":60,"band":"ok","label":"Likely open","p_open":1,"p_active":0.602,"p_room":1,"age_days":0,"expected_fill_days":29,"reasons":["conf:2","stale_co","win:early"],"computed_at":"2026-10-14T06:34:16Z"},"pay":null,"html_url":"https://alion.io/job/black-forest-labs-applied-research-engineer-evaluation","json_url":"https://alion.io/job/black-forest-labs-applied-research-engineer-evaluation.json","meta":{"generated_at":"2026-10-14T12:39:59Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15517,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-15T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":13106},"rest":"https://alion.io/mcp/rest/get_company?id=13106"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fblack-forest-labs-applied-research-engineer-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fblack-forest-labs-applied-research-engineer-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fblack-forest-labs-applied-research-engineer-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/black-forest-labs-applied-research-engineer-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fblack-forest-labs-applied-research-engineer-evaluation"}]}