{"id":2182499,"url":"https://alion.io/job/cosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms","title":"ML Systems Engineer - Model Training and Infrastructure (SWE-focused LLMs)","company":{"id":177875,"name":"Cosine","domain":"cosine.sh","url":"https://alion.io/company/cosine-2","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["London, United Kingdom"],"countries":["GB"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"Docker","optional":false},{"name":"GCP","optional":false},{"name":"Kubernetes","optional":false},{"name":"Python","optional":false},{"name":"Reinforcement Learning","optional":false},{"name":"SFT","optional":false},{"name":"Synthetic Data","optional":false},{"name":"Apache Iceberg","optional":true},{"name":"DuckDB","optional":true},{"name":"GDPR","optional":true},{"name":"JAX","optional":true},{"name":"LLM","optional":true},{"name":"PyTorch","optional":true},{"name":"Slack","optional":true},{"name":"SQL","optional":true},{"name":"TensorFlow","optional":true}],"status":"live","first_seen_at":"2026-10-09T12:48:50Z","employer_posted_date":"2026-10-09","last_verified_at":"2026-10-28T10:50:33Z","board_verified":true,"closed_at":null,"days_open":19,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":19},"description":"Job title: ML Systems Engineer - Model Training and Infrastructure (SWE-focused LLMs)\nLocation: London; full in-office working as default\nStart date: ASAP\nCompensation: Salary and Equity options available for discussion\nApplication deadline: 08/11/26 at 23:59 UK time\n___________________________________________________________________________\nWe’re looking for an ML Systems Engineer to collaborate in training our Lumen models - our open-source-based software engineering LLMs.\nThis is a unique, and truly interdisciplinary role that involves developing and deploying our reinforcement learning (RL) training environments, working on large scale synthetic data pipelines and building our ML platform to enable training large models at scale in distributed environments.\nWe want to make sure that the models we train are the best SWEs in the world - this doesn’t just mean training them to get the right answer, it means training them so that they write readable, maintainable code, that fits with the architectural patterns already present in the codebase. We believe we’re now in the anti-slop era of coding agents, where data, RL environments and opinionated reward functions will shape the future standards of SWE models. If this sounds exciting, then this could be the role for you.\n___________________________________________________________________________\nWhat you’ll do\nParticipate in end-to-end training of models:\nSupervised fine-tuning on curated code and conversation datasets.\n\nRL on top of those models to align them with software-engineering objectives.\n\nDevelop and manage synthetic data generation pipelines to curate datasets that will underpin future training runs.\n\nDesign, build and deploy containerized services using Docker and platforms like Kubernetes to enable our RL infrastructure.\n\nBuild and maintain our training platform that enables us to conduct model architecture experiments at scale, train models, and run evaluations on them.\n\nIdeate on novel and opinionated reward functions for the training of SWE agents.\n\nYou’ll collaborate closely with infra, product, and research to decide what to train next, how to train it, and how to measure whether it’s actually better for engineers.\n___________________________________________________________________________\nWhat we’re looking for (must-haves)\nStrong software engineering or computer science background:\nTypically 3-5 years of experience.\n\nYou can read, debug, and write non-trivial production code (you’ll mainly be working across Python and Go).\n\nExperience with tools like Docker and container management/orchestration platforms, like Kubernetes\n\nExperience with at least one major cloud-computing platform like GCP, AWS or Azure\n\nYou care about code quality, correctness, and maintainability as much as model metrics.\n\nData engineering instincts:\nComfortable working with large-scale datasets, object storage, dataset sharding, and filtering.\n\nKnow that data quality and sampling strategies matter as much as architecture.\n\nClear communication and ownership:\nCan take a vague modelling goal (“make Lumen better at X”) and turn it into a concrete plan of experiments.\n\nComfortable documenting decisions and walking others through tradeoffs.\n\n___________________________________________________________________________\nNice to have (bonus)\nYou don’t need all of these, but the more you have, the more you’ll hit the ground running:\nKnowledge of PyTorch/Tensorflow/JAX\n\nExperience with synthetic data generation pipelines\n\nExperience with data tooling like SQL, Apache Iceberg and duckDB\n\nExperience training LLMs in distributed environments\n\nSafety, robustness, and reward shaping:\nExperience with LLM-as-a-judge, reward hacking detection, or robustness evaluation.\n\nOpen-source contributions or research:\nContributions to open-source LLM tooling, RL libraries, etc.\n\n___________________________________________________________________________\nWhy join Cosine\nThis is a rare opportunity to work across the full model-training stack in a high-ownership environment, while still developing deep expertise in the area that interests you most. You’ll be close to the models and the people using them, with the freedom to move from an idea to a production system rather than stopping at an experiment, paper or benchmark. If you care as much about making research work reliably in the real world as you do about developing the underlying techniques, Cosine offers a place to do both.\n___________________________________________________________________________\n Cosine is an equal opportunity employer.\nWe value diverse backgrounds, perspectives, and ways of thinking, and we’re committed to creating an inclusive and respectful workplace.\nWe encourage applications from anyone who meets the role requirements, even if you don’t meet every single qualification. If you need reasonable adjustments at any stage of the hiring process, we’re happy to discuss them.\n___________________________________________________________________________\nCompensation, Benefits & Ways of Working\nWe’re an in-office team, five days a week, by design. We believe the work we’re doing benefits from being together, collaborating closely, and building shared context.\nWhat you can expect:\nCompetitive salary, benchmarked to the market\n\nEquity / share options, so you share in the upside you help create\n\n30 days’ holiday + bank holidays\n\nWork hard in the office, collaborate closely, and switch off properly\n\nDog-friendly office - bring your dog to work\n\nWeekly team breakfast & lunch\n\nMonthly socials\n\nPension\n\nHigh-quality equipment to do your best work\n\nWe care about focus, sustainability, and doing great work - not performative overwork. We value people who show up, contribute thoughtfully, collaborate well with their colleagues, and then go home.\nThis role won’t suit everyone. But if you’re looking for a fast-moving environment, close collaboration, and a team that takes both the work and work-life balance seriously, it’s a great place to be.\n___________________________________________________________________________\nAgency & Data Protection Notice\nTo comply with UK GDPR and our internal data-protection and equal-opportunity obligations, we only accept candidate applications and agency submissions via our Applicant Tracking System (ATS). This ensures appropriate privacy notices, lawful processing, audibility, and consistent retention controls.\nAny CVs or candidate details received outside the ATS (including via email, Slack, or direct message) will be treated as unsolicited, will not be considered as part of the recruitment process, and will not give rise to any fee or payment obligation.","description_format":"text","description_chars":6676,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity","Stock options"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["AI Agents","Code Intelligence"],"lifecycle":[{"event":"open","at":"2026-10-09T15:17:00Z"}],"visa":[],"liveness":{"score":74,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.817,"p_room":0.9,"age_days":18,"expected_fill_days":30,"reasons":["conf:4","velocity","win:mid"],"computed_at":"2026-10-28T05:59:10Z"},"pay":null,"html_url":"https://alion.io/job/cosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms","json_url":"https://alion.io/job/cosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms.json","meta":{"generated_at":"2026-10-28T13:57:38Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":18974,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-29T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":177875},"rest":"https://alion.io/mcp/rest/get_company?id=177875"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcosine-ml-systems-engineer-model-training-and-infrastructure-swe-focused-llms"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":2182499}},{"tool":"search_jobs","arguments":{"company":"cosine.sh","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":2182499},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}