{"id":1020012,"url":"https://alion.io/job/furiosa-software-engineer-llm-performance-evaluation","title":"Software Engineer, LLM Performance & Evaluation","company":{"id":31683,"name":"FuriosaAI","domain":"furiosa.ai","url":"https://alion.io/company/furiosa","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"A","score":100,"open_postings":38,"ghost_share":0,"stale_share":0,"repost_share":0.026,"time_to_fill_p50_days":33,"computed_at":"2026-11-12T10:38:37Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Seoul, South Korea"],"countries":["KR"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Hugging Face","optional":false},{"name":"KV Cache","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"Quantization","optional":false},{"name":"Speculative Decoding","optional":false},{"name":"Transformers","optional":false},{"name":"C++","optional":true},{"name":"Linux","optional":true},{"name":"Rust","optional":true},{"name":"SGLang","optional":true},{"name":"TensorRT","optional":true},{"name":"TensorRT-LLM","optional":true},{"name":"TPU","optional":true},{"name":"vLLM","optional":true}],"status":"live","first_seen_at":"2026-09-18T02:04:48Z","employer_posted_date":"2026-10-21","last_verified_at":"2026-11-12T14:36:46Z","board_verified":true,"closed_at":null,"days_open":55,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":55},"description":"About FuriosaAI\nFuriosaAI builds high-performance, high-efficiency AI compute for the Inference Era. Founded in 2017 by veteran semiconductor and AI algorithm engineers, Furiosa operates globally with offices in Korea and Silicon Valley, along with a compiler-focused R&D lab in Lisbon. \nOur vision is to make AI computing sustainable, enabling access to powerful AI for everyone on Earth. We solve the AI hardware energy and operational cost crisis at the architectural level, rather than through brute force, building the world's first truly AI-native compute platform to unlock the full potential of artificial intelligence for every enterprise.\nSoftware Engineer, LLM Performance & Evaluation\nLocation: Seoul, South Korea (Hybrid)\nAbout the Job\nFuriosaAI is seeking a Software Engineer to own the measurement and continuous tracking of LLM inference performance and model accuracy on our NPUs. You will build the benchmarks, evaluation pipelines, and analysis tools that show how individual inference features perform, how accurately each supported model runs, and how both change as Furiosa-LLM (FLM) and our software stack evolve.\nYour work will give engineering teams reliable evidence for optimization priorities, model support, and release decisions. Working with the MLSys, Inference Engine, Compiler, and LLM Serving teams, you will establish reproducible baselines, investigate regressions, and make performance-accuracy trade-offs clear across models, workloads, and configurations.\nKey Responsibilities\nDesign and maintain benchmarks that isolate the impact of inference features such as continuous batching, prefix caching, speculative decoding, and distributed inference. Measure feature interactions and end-to-end behavior across representative input/output lengths, concurrency levels, and deployment configurations.\nMeasure time to first token (TTFT), inter-token and end-to-end latency distributions, throughput, memory use, and accelerator utilization. Compare configurations under explicit workload and latency constraints, and quantify improvements against controlled baselines.\nOwn model-level accuracy evaluation for supported LLMs, selecting representative datasets and task-appropriate metrics. Compare NPU results with trusted reference implementations and prior releases, and characterize the effects of quantization, numerical precision, and inference optimizations on model quality.\nBuild reliable automation for benchmarks and accuracy evaluations, including scheduled runs and change-triggered checks. Version models, datasets, prompts, scoring logic, and execution configurations so results remain reproducible and comparable over time.\nMaintain dashboards and reports that track performance and accuracy by model, inference feature, hardware configuration, and software version. Define regression thresholds and release acceptance criteria with engineering owners, and integrate evaluation checks into CI and release workflows with the Build & Release team.\nInvestigate regressions using profiling, controlled experiments, and targeted reproductions. Distinguish implementation changes from workload variation, evaluation errors, and infrastructure noise; work with component owners to identify causes and verify fixes.\nImprove measurement quality through repeated runs, statistical analysis, and validation of datasets and scoring methods. Maintain a focused regression suite and broader periodic evaluations that balance coverage, execution cost, and feedback speed.\nCommunicate findings, supported operating ranges, and performance-accuracy trade-offs through clear technical reports. Use the evidence to recommend optimization priorities and keep benchmark methodology and model evaluation results documented.\nMinimum Qualifications\nStrong Python programming skills and experience building maintainable automation, data pipelines, or engineering tools.\nHands-on experience measuring and analyzing ML inference or complex systems performance, including profiling, latency/throughput analysis, and regression investigation.\nUnderstanding of transformer-based LLM inference, including prefill/decode, batching, KV-cache behavior, and the effects of workload and numerical precision on performance and model outputs.\nPractical experience evaluating model accuracy or numerical correctness using PyTorch, Hugging Face Transformers, or comparable tools, with the ability to investigate discrepancies against a reference implementation.\nSound experimental design and data analysis skills, including controlled comparisons, variability analysis, and reproducible reporting.\nAbility to communicate quantitative findings clearly and collaborate across engineering teams to drive issues through resolution.\nPreferred Qualifications\nExperience with LLM inference frameworks such as vLLM, SGLang, TensorRT-LLM, including their benchmarking and profiling tools.\nExperience building LLM evaluation suites with tools such as lm-evaluation-harness or comparable frameworks, including dataset preparation, prompt configuration, and scoring validation.\nFamiliarity with quantization, mixed precision, speculative decoding, or distributed inference, and their implications for performance and accuracy.\nExperience benchmarking GPU, NPU, or TPU systems and analyzing memory bandwidth, kernel execution, or communication bottlenecks.\nExperience operating automated evaluation workloads in Linux, container, or cluster environments and integrating results with CI, experiment tracking, or dashboards.\nAbility to read and instrument Rust or C++ code, or contributions to open-source inference, benchmarking, or evaluation projects.\nWhy Join FuriosaAI\nThe defining bottleneck of the AI era is building the right hardware and software stack to run it at global scale. Furiosa is solving this challenge holistically from the ground up.\nWith our flagship chip, RNGD, in mass production today and our next-generation platform in development with Broadcom, we are proving that full-stack, tensor-native compute is the future of AI infrastructure. This is a pivotal moment to join our team, right as we accelerate our global expansion.\nAt Furiosa, you will:\nSolve AI’s Most Urgent Challenge. Help build the high-performance, energy-efficient inference hardware and software required to fulfill the promise of advanced AI.\nPioneer Full-Stack Co-Design. Work with teams that are architecting solutions from silicon up through the compiler (featuring innovations like Tensor Contraction Language and Virtual ISA) and serving frameworks.\nShip Real-World Silicon, Software, and Solutions. Turn breakthrough technology into commercial deployment. RNGD is in mass production with TSMC and running live enterprise workloads for global leaders like LG AI Research and Samsung SDS.\nPartner With the Industry's Best. Collaborate across an elite global ecosystem that includes TSMC, Broadcom, SK Hynix, and GUC.\nDo Your Life’s Best Work. Join a brilliant, low-ego, mission-driven team in a high-trust environment that values autonomy, intellectual curiosity, and shared ambition. \nContact\n ","description_format":"text","description_chars":7104,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"South Korea","iso":"KR","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Processors, MCUs & AI Chips","AI Chips & Accelerators"],"lifecycle":[{"event":"open","at":"2026-09-18T04:06:37Z"}],"visa":[],"liveness":{"score":26,"band":"fade","label":"Fading","p_open":1,"p_active":0.583,"p_room":0.45,"age_days":55,"expected_fill_days":33,"reasons":["conf:1","velocity","win:tail"],"computed_at":"2026-11-12T10:38:37Z"},"pay":null,"html_url":"https://alion.io/job/furiosa-software-engineer-llm-performance-evaluation","json_url":"https://alion.io/job/furiosa-software-engineer-llm-performance-evaluation.json","meta":{"generated_at":"2026-11-12T19:52:42Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":41982,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":31683},"rest":"https://alion.io/mcp/rest/get_company?id=31683"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ffuriosa-software-engineer-llm-performance-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ffuriosa-software-engineer-llm-performance-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ffuriosa-software-engineer-llm-performance-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/furiosa-software-engineer-llm-performance-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ffuriosa-software-engineer-llm-performance-evaluation"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":1020012}},{"tool":"search_jobs","arguments":{"company":"furiosa.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":1020012},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}