{"id":2845622,"url":"https://alion.io/job/atlassian-senior-machine-learning-engineer-9","title":"Senior Machine Learning Engineer","company":{"id":182,"name":"Atlassian","domain":"atlassian.com","url":"https://alion.io/company/atlassian","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Career site","truth_index":{"grade":"A","score":100,"open_postings":15,"ghost_share":0,"stale_share":0,"repost_share":0.067,"time_to_fill_p50_days":9,"computed_at":"2026-10-20T05:45:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Confluence","optional":false},{"name":"Function Calling","optional":false},{"name":"Jira","optional":false},{"name":"Kotlin","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"SQL","optional":false},{"name":"Tool Use","optional":false}],"status":"live","first_seen_at":"2026-10-19T10:54:22Z","employer_posted_date":"2026-10-19","last_verified_at":"2026-10-20T11:12:13Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"Atlassian is seeking a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio. You'll build the systems that measure how well AI agents perform and use them to make those agents measurably better for thousands of companies.\nResponsibilities\nYour future team\nStudio is where Atlassian customers build, run and manage AI agents under the Rovo brand. Rovo Agents is a model-agnostic cloud agent platform. Teams give an agent instructions, knowledge, tools and skills, then put it to work in Jira and Confluence, inside automation rules, or as an autonomous agent that works on its own.\nBecause customers build these agents themselves, quality can't be checked once before release. Every agent has its own instructions, data and tools, and any of them can be misconfigured or connected to a tool that fails. We ask three questions of every agent: did it understand and follow the task, did it use its tools correctly and report honestly what it did, and did the result actually help?\nThe team is software and machine learning engineers based in Sydney and Melbourne, with remote options. We work closely with product, data science and Atlassian's central AI teams. This is applied ML on real agentic systems in production.\nWhat you'll do\nAs a Senior Machine Learning Engineer, you will design, build, ship and operate the evaluation and optimization systems for Rovo agents.\nBuild offline evaluation suites and benchmarks that reflect realistic agent tasks, using real usage patterns, synthetic scenarios and simulated tools. Run them as regression tests before changes ship.\n\nBuild online quality metrics on production traffic. This covers sampling strategy, cheap behavioral signals such as tool errors, retry loops, failed handoffs and abandoned sessions, and model-based judging of selected agent trajectories.\n\nCalibrate judges against human ratings and independent outcome signals, and evolve evals so they don't saturate as agents improve.\n\nUse evaluation results and production signals to improve agent quality across prompts, context selection, tool use, retrieval and agent routing, including finding the right agent for a request.\n\nImprove quality while reducing cost and latency across production LLM systems.\n\nOwn projects from problem definition through experimentation, deployment and ongoing measurement. Contribute to design and code reviews, coach engineers, and help software engineers on the team write good evals.\n\nQualifications\nYour background\nOn the first day, we'll expect you to have:\nA bachelor's or master's degree, preferably in Computer Science or a related field, or equivalent practical experience.\n\n4+ years of related industry experience in machine learning, including hands-on experience designing, evaluating and operating LLM-based or agentic systems.\n\nStrong coding skills in Python and/or Kotlin, with the ability to write performant, production-quality code using AI\n\nExperience designing experiments, building evaluation datasets and benchmarks, and analyzing model and system behavior to guide improvements. Familiarity with SQL, distributed data processing and cloud data environments.\n\nExperience building and operating ML systems in production, including the practices needed to improve quality, reliability and cost.\n\nRegular use of AI tools in your daily engineering workflow, with the judgment to validate their outputs and take ownership of the results.\n\nA strong ownership mindset and the ability to lead ambiguous technical work from problem definition through delivery.\n\nClear communication, with the ability to explain ML trade-offs to engineers, product managers and leaders.\n\nIt's great, but not required, if you have:\nExperience developing agent evaluations, including trajectory analysis, human evaluation or LLM-as-a-judge.\n\nExperience building agents that use tools, or retrieval-augmented generation applications.\n\nExperience with retrieval, ranking or recommendation.\n\nExperience in enterprise SaaS or similarly large-scale product environments.","description_format":"text","description_chars":4027,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":true},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Service & Asset Management","Project & Work Management Software","Developer Tools","Intranet & Knowledge Management"],"lifecycle":[{"event":"open","at":"2026-10-19T10:54:22Z"}],"visa":[{"country":"GB","licensed_sponsor":true,"evidence":"Licensed UK visa sponsor (Skilled Worker)","filings_12m":null,"filings_prev_12m":null,"green_card_filings_12m":null,"median_offered_wage_usd":null,"route":"Skilled Worker","cap_exempt":false,"checked_at":"2026-10-20T06:21:30+00:00","sources":["UK Home Office: register of licensed sponsors (workers)"],"filings_for_role_12m":0}],"liveness":{"score":99,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.993,"p_room":1,"age_days":0,"expected_fill_days":9,"reasons":["conf:3","wave","velocity","win:early","comp:brand"],"computed_at":"2026-10-20T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/atlassian-senior-machine-learning-engineer-9","json_url":"https://alion.io/job/atlassian-senior-machine-learning-engineer-9.json","meta":{"generated_at":"2026-10-20T12:04:22Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":13357,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-21T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":182},"rest":"https://alion.io/mcp/rest/get_company?id=182"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fatlassian-senior-machine-learning-engineer-9"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fatlassian-senior-machine-learning-engineer-9"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fatlassian-senior-machine-learning-engineer-9"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/atlassian-senior-machine-learning-engineer-9\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fatlassian-senior-machine-learning-engineer-9"}]}