{"id":129930,"url":"https://alion.io/job/canva-principal-research-scientist-evaluations","title":"Principal Research Scientist - Evaluations","company":{"id":181,"name":"Canva","domain":"canva.com","url":"https://alion.io/company/canva","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"SmartRecruiters","truth_index":{"grade":"B","score":75,"open_postings":14,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":34,"computed_at":"2026-10-20T05:45:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"lead","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Sydney, Australia"],"countries":["AU"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":157000,"max_usd":387000,"period":"year","method":"global_role_cell_scaled_by_country","sample_n":1199},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Canva","optional":false},{"name":"Multimodal AI","optional":false},{"name":"Post-training","optional":false},{"name":"Reward Modeling","optional":false},{"name":"RLAIF","optional":false},{"name":"RLHF","optional":false}],"status":"live","first_seen_at":"2026-08-26T18:54:33Z","employer_posted_date":"2026-08-26","last_verified_at":"2026-10-20T11:05:56Z","board_verified":true,"closed_at":null,"days_open":54,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":54},"description":"Join the team redefining how the world experiences design.\nHey, g'day, mabuhay, kia ora,你好, hallo, vítejte!\nThanks for stopping by. We know job hunting can be a little time consuming and you're probably keen to find out what's on offer, so we'll get straight to the point.\nWhere and how you can work\nOur flagship campus is in Sydney. This role is preferred to be based out of our Sydney office, where the majority of this team is located.\n About the role\nCanva's generative models are judged by millions of people who will never read a benchmark. They just know whether the design looks right. Turning that judgement into something measurable is the hardest problem in our research stack, and it gates everything else. If we cannot measure design quality reliably, we cannot train against it, we cannot tell a real improvement from noise, and we cannot decide what ships.\nWe are looking for a Principal Research Scientist who defines what evaluation needs to become as the space gets harder, rather than running the playbook we already have. You will own how Canva evaluates generative quality across the whole of Canva Research, including problems we have not framed yet: new modalities, evaluation that reflects real differences between content types, user segments and markets, and a much tighter link between what our metrics say and what users and the business actually experience.\nThis is a Canva-wide craft leadership role, setting direction across our research groups in Australia, Europe, the US and China. You will be the person others come to when the numbers and the eyes disagree.\nWhat you'll own\nThe evaluation strategy for Canva Research. Define what great evaluation looks like across design, image, video, audio and agentic workflows, and set the long-term direction for how Canva measures generative quality as the space evolves. You will shape the principles teams use to trade off evaluation compute, human data spend and signal fidelity, and you will defend them.\nThe science of measurement itself. Auto-raters and MLLM judges are only as good as their correlation with the thing they proxy for. You will treat that correlation as a research problem: validating metrics against human preference and downstream product outcomes, quantifying judge bias, and catching benchmark saturation and contamination before they quietly stop telling us anything.\nThe link between evaluation and what actually matters. Evaluation should reflect user experience and product impact, not just perform well in isolation. You own closing that gap, including the fact that a good evaluator is not automatically a good reward model for RL. That extends to the full experience rather than the artefact alone, editability included.\nOne standard across every region. Our teams in London, Vienna, San Francisco, Sydney and China all need to know they are measuring the same thing. You will build the shared evaluation layer that makes results comparable, and you will spot the collaboration opportunities nobody has been chartered to own yet. Taking that initiative is a core expectation of this role, not a bonus.\nFocus areas\nHuman preference at scale. Rubric design, rater guidelines, inter-rater reliability, and calibration across markets and cultural contexts. Aesthetic judgement is not universal, and our evaluation systems need to hold that honestly rather than average it away.\nLearned quality models and reward signals. Reward modelling and preference learning that feed post-training (RLHF, RLAIF) and inference-time selection, and being explicit about where a good judge does not translate into a good reward model.\nVision-Language Models for quality understanding. Novel architectures and training approaches for models that understand what makes a design effective, not just well-formed. These become the reward signal and feedback loop for our design generation models, so their failure modes are our failure modes.\nAgentic and automated evaluation. MLLM-as-a-judge systems, model-based grading, and the infrastructure to run hundreds of evaluations against live training checkpoints without the results becoming noise.\nMultimodal and segment-specific evaluation. Extending rigour into photo AI and video as they mature and start moving faster than traditional user research and marketing testing can cover, with evaluation that can be sliced meaningfully by doc type, user group and locale.\nEvaluation in production. Offline evaluation suites and online monitoring, with regression detection that makes a quality drop impossible to miss before it reaches users.\nExternal credibility. Benchmarking Canva's models against the frontier, and building evaluation work others in the industry look to. Publication and public reporting where it serves the mission.\nPrimary responsibilities\nEstablish the evaluation gates that inform launch decisions, and be accountable for the judgement calls when the signal is ambiguous\n\nDiagnose anomalous evaluation results during production training runs, separate model regressions from infrastructure artefacts, and communicate the answer clearly and quickly\n\nPartner with Design Generation, Foundation Models and Agents teams so evaluation shapes their training and inference, rather than reporting on it after the fact\n\nWork directly with designers, creators and product teams to turn subjective creative judgement into measurable criteria\n\nMentor senior research scientists and engineers, and raise the bar for evaluation craft across the group\n\nRepresent Canva's evaluation vision and practice to senior leadership and, where valuable, to the broader industry community\n\nYou're probably a match if you have\nA track record of defining evaluation frameworks and standards from the ground up rather than operating inside someone else's, ideally at an organisation pushing the frontier of GenAI evaluation, and measurement systems that changed how a team made decisions rather than papers about metrics\n\nExperience linking evaluation metrics to downstream business or user outcomes, and diagnosing why they diverge\n\nExperience turning subjective human judgement into reliable, objective evaluation signal through rubric design, human data pipelines and model training\n\nStrong grounding in multimodal generative models (diffusion, transformers, VLMs and MLLMs) and their architectures, deep enough to know where evaluation will break\n\nExperience with reward modelling, preference learning, or alignment methods involving human feedback\n\nExperience setting technical direction across multiple teams or a wide specialty area in a globally distributed organisation, with the instinct to find the gap nobody owns and close it without waiting for a mandate\n\nNice to have\nResearch background in human perception, psychophysics, aesthetics or HCI\n\nExperience evaluating for harm, bias and safety alongside quality\n\nPublication record in evaluation, alignment or generative modelling\n\nBackground or genuine interest in visual arts and graphic design\n\n Don't tick all the boxes? Don't worry about that - nobody does!\nWe’d still love to hear from you! At Canva, we know that great engineers come from a variety of backgrounds, and we value passion, curiosity, and a willingness to learn just as much as specific experience. If you're excited about this role but don’t tick every box, we encourage you to apply, you might a great fit in ways you didn’t expect!\nWhat's in it for you?\nAchieving our crazy big goals motivates us to work hard - and we do - but you'll experience lots of moments of magic, connectivity and fun woven throughout life at Canva, too. We also offer a stack of benefits to set you up for every success in and outside of work.\nHere's a taste of what's on offer:\nEquity packages - we want our success to be yours too\nInclusive parental leave policy that supports all parents & carers\nAn annual Vibe & Thrive allowance to support your wellbeing, social connection, office setup & more\nFlexible leave options that empower you to be a force for good, take time to recharge and supports you personally\nCheck out lifeatcanva.com for more info.\nOther stuff to know\nWe make hiring decisions based on your experience, skills and passion, as well as how you can enhance Canva and our culture. When you apply, please tell us the pronouns you use and any reasonable adjustments you may need during the interview process.\nPlease note that interviews are conducted virtually.","description_format":"text","description_chars":8416,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"All levels","optional":false}]},"benefits":["Equity","Parental leave"],"hiring_locations":[{"name":"Australia","iso":"AU","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Graphic Design","Design & Prototyping Tools"],"lifecycle":[{"event":"open","at":"2026-08-26T18:54:33Z"}],"visa":[],"liveness":{"score":21,"band":"cold","label":"Long shot","p_open":1,"p_active":0.475,"p_room":0.45,"age_days":54,"expected_fill_days":34,"reasons":["conf:3","stale_co","win:tail"],"computed_at":"2026-10-20T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/canva-principal-research-scientist-evaluations","json_url":"https://alion.io/job/canva-principal-research-scientist-evaluations.json","meta":{"generated_at":"2026-10-20T15:05:52Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":17019,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-21T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":181},"rest":"https://alion.io/mcp/rest/get_company?id=181"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcanva-principal-research-scientist-evaluations"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcanva-principal-research-scientist-evaluations"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcanva-principal-research-scientist-evaluations"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/canva-principal-research-scientist-evaluations\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcanva-principal-research-scientist-evaluations"}]}