Agent Reliability & Evaluation Algorithm Engineer

Shopee • Marketplaces • Master's Degree • Python • SQL • 10 hours ago
Smart Apply
Similar jobs
1,513,533open jobs
91,226companies
233,835added this week
Browse all
Salary
≈ $30k – $50k per year (Estimated)
Location
In office (Singapore)
Seniority
Junior · 1+ year exp

Confirmed on the employer's own hiring board on Oct 12, 2026. First seen by Alion on Oct 12, 2026. Shopee scores A on the Alion truth index.

Overview
Company
Impact
Profile match

Shopee

Shopee is a multinational technology company and e-commerce marketplace platform operating as a subsidiary of Sea Limited. The company provides a hybrid B2C and C2C digital marketplace offering retail products, integrated digital payments and financial services via ShopeePay/SeaMoney, and end-to-end logistics and fulfillment solutions through SPX Express. Headquartered in Singapore, the platform serves millions of consumers and merchant partners across Southeast Asia, Taiwan, and Latin America.
shopee.com • HQ: Singapore • Marketplaces • 1001-5000 employees • Est. 2015
Tech stack of Shopee
  • Design and own the evaluation system for LLM Agents, including benchmark construction, LLM-as-Judge with human annotation, and regression/comparison pipelines. Continuously monitor product metrics to guide Agent and model optimization.
  • Build Agent reliability analysis, including error attribution and uncertainty estimation across multi-step trajectories, and turn failure cases into targeted improvements.
  • Drive evaluation-guided Agent optimization, including Harness design (planning, verification, self-refine), tool calling, context engineering, and memory.
  • Lead evaluation-driven post-training of LLMs for vertical domains (SFT, DPO, RL), including data pipelines and reward model design.
  • Build and optimize RAG/retrieval systems and the Agent tool layer (search, code interpreters, sandboxes, MCP) and evaluate their impact on end-to-end Agent quality.
  • Explore frontier research in Agent reliability and evaluation and drive a full closed loop from research to deployment.
  • Master’s degree or above in Artificial Intelligence, Computer Science, Mathematics, or a related field.
  • At least 1 year of full-time industry experience building and deploying production LLM Agent systems (task planning, orchestration, tool calling), including designing evaluation sets and metrics (LLM-as-Judge, human annotation, error attribution) to drive iteration.
  • Hands-on experience fine-tuning LLMs via SFT and DPO (GRPO/RL a plus), and building and optimizing RAG/retrieval systems (recall, ranking, embedding fine-tuning).
  • Good programming skills; proficient in Python (SQL and PyTorch are a plus). Familiar with training and inference frameworks such as verl, vLLM, FSDP, or LLaMA-Factory.
  • Strong analytical skills, with a solid grasp of statistics and experiment design. Sustained curiosity about frontier AI, strong self-drive, and the ability to collaborate closely with teams to drive a full closed loop from research to deployment.
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,513,533 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Industrial Engineering
Similar stack
Same company
Singapore
≈ $25k – $42k per year (Estimated) • In office • Full-Time • Bachelor's Degree • Singapore
Apply
≈ $26k – $43k per year (Estimated) • In office • Full-Time • 1+ year exp • Bachelor's Degree • Singapore
DevOps
AWS
Apply
≈ $25k – $42k per year (Estimated) • In office • Full-Time • Bachelor's Degree • Singapore
Apply
≈ $19k – $39k per year (Estimated) • In office • Full-Time • 1+ year exp • Bachelor's Degree • Pretoria
DevOps
Amazon ECS
Design
AutoCAD
Apply
≈ $36k – $79k per year (Estimated) • In office • Full-Time • 5+ years exp • Pretoria
Design
AutoCAD
Apply
Project Engineer 6 hours ago
≈ $34k – $62k per year (Estimated) • In office • 3+ years exp • Bachelor's Degree • Singapore
Apply
≈ $40k – $80k per year (Estimated) • In office • Full-Time • 5+ years exp • Bachelor's Degree • Singapore
Apply
≈ $38k – $75k per year (Estimated) • In office • 3+ years exp • Singapore
Apply
≈ $28k – $66k per year (Estimated) • In office • Singapore
Management
Microsoft Office
Apply
≈ $40k – $74k per year (Estimated) • Hybrid • Full-Time • 5+ years exp • Singapore
PowerShell
Cybersecurity
DLP
Analytics
Power BI
Management
Power Automate
Power Apps
Outlook
Microsoft Teams
OneDrive
SharePoint
Apply
See all jobs
This is one of many
1,513,533 more open roles from verified company boards, updated every day.
Browse all jobs Get matched roles