{"id":3788147,"url":"https://alion.io/job/supermicro-sr-software-engineer-ai-evaluation-and-knowledge","title":"Sr. Software Engineer - AI Evaluation and Knowledge","company":{"id":58834,"name":"Supermicro","domain":"supermicro.com","url":"https://alion.io/company/supermicro","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"SuccessFactors","truth_index":{"grade":"A","score":98,"open_postings":15,"ghost_share":0,"stale_share":0.067,"repost_share":0,"time_to_fill_p50_days":32,"computed_at":"2026-11-08T05:47:12Z"}},"role":"Backend","role_family":"Backend","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Jose, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":160000,"max":200000,"currency":"USD","period":"year","gross":null,"usd_annual":200000},"salary_estimate":null,"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"AI Agents","optional":false},{"name":"Hadoop","optional":false},{"name":"HPC","optional":false},{"name":"LLM","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Structured Outputs","optional":false},{"name":"A/B Testing","optional":true}],"status":"live","first_seen_at":"2026-10-14T07:00:00Z","employer_posted_date":"2026-10-14","last_verified_at":"2026-11-08T02:25:17Z","board_verified":true,"closed_at":null,"days_open":24,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":24},"description":"Job Req ID: 30248\nAbout Supermicro:\nSupermicro® is a Top Tier provider of advanced server, storage, and networking solutions for Data Center, Cloud Computing, Enterprise IT, Hadoop/ Big Data, Hyperscale, HPC and IoT/Embedded customers worldwide. We are the #5 fastest growing company among the Silicon Valley Top 50 technology firms. Our unprecedented global expansion has provided us with the opportunity to offer a large number of new positions to the technology community. We seek talented, passionate, and committed engineers, technologists, and business leaders to join us.\nJob Summary:\nWe are building an AI Agent-powered platform to handle customer issues related to AI Servers and Racks. This role focuses on the quality of agent outputs, evaluation systems, and continuous improvement of the Knowledge Base, ensuring the agent produces accurate and actionable troubleshooting steps.\nEssential Duties and Responsibilities:\nDesign and maintain the agent evaluation framework (test sets, automated scoring, human sampling review, regression testing)\nOwn Prompt Engineering and structured output optimization to improve step accuracy\nManage Knowledge Base quality processes (content review, version control, retirement of outdated content, writing back new cases)\nAnalyze agent errors and hallucinations, then propose and implement improvements\nWork closely with Domain SMEs to validate the correctness and practicality of troubleshooting steps\nEstablish a feedback loop from on-site execution results back into agent improvements\nHelp define success metrics (e.g., first-time accuracy rate, step usability rate) and track them continuously\nSupport RAG retrieval quality tuning and knowledge presentation improvements\nQualifications:\nExperience in LLM / Generative AI applications, with practical experience in evaluation or quality (2+ years of experience is preferred)\nStrong familiarity with Prompt Engineering and practical methods to reduce hallucinations\nExperience systematically evaluating LLM or Agent outputs\nExperience with RAG or Knowledge Base implementation / management\nStrong analytical and cross-team communication skills\nAbility to turn domain expert feedback into actionable improvements\nFamiliarity with Python and ability to implement basic evaluation or data processing scripts\nExperience in technical support, troubleshooting, or vertical AI Agent applications is a plus\nExperience with LLM-as-judge, automated evaluation pipelines, or A/B testing is a plus\nExperience in knowledge engineering, data annotation, or case library building is a plus\nExperience collaborating with non-AI domain experts is a plus\n\nSalary Range\n$160,000 - $ 200,000\nThe salary offered will depend on several factors, including your location, level, education, training, specific skills, years of experience, and comparison to other employees already in this role. In addition to a comprehensive benefits package, candidates may be eligible for other forms of compensation, such as participation in bonus and equity award programs.\nEEO Statement\nSupermicro is an Equal Opportunity Employer and embraces diversity in our employee population. It is the policy of Supermicro to provide equal opportunity to all qualified applicants and employees without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected veteran status or special disabled veteran, marital status, pregnancy, genetic information, or any other legally protected status.","description_format":"text","description_chars":3505,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Enterprise Storage Systems","Servers & Data Center Hardware"],"lifecycle":[{"event":"open","at":"2026-11-05T04:45:04Z"}],"visa":[],"liveness":{"score":66,"band":"ok","label":"Likely open","p_open":1,"p_active":0.881,"p_room":0.75,"age_days":24,"expected_fill_days":32,"reasons":["conf:3","velocity","win:late","comp:brand"],"computed_at":"2026-11-08T05:47:12Z"},"pay":{"stated_usd_annual":200000,"is_top_pay":true},"html_url":"https://alion.io/job/supermicro-sr-software-engineer-ai-evaluation-and-knowledge","json_url":"https://alion.io/job/supermicro-sr-software-engineer-ai-evaluation-and-knowledge.json","meta":{"generated_at":"2026-11-08T06:58:56Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15123,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-09T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":58834},"rest":"https://alion.io/mcp/rest/get_company?id=58834"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsupermicro-sr-software-engineer-ai-evaluation-and-knowledge"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsupermicro-sr-software-engineer-ai-evaluation-and-knowledge"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsupermicro-sr-software-engineer-ai-evaluation-and-knowledge"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/supermicro-sr-software-engineer-ai-evaluation-and-knowledge\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsupermicro-sr-software-engineer-ai-evaluation-and-knowledge"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3788147}},{"tool":"search_jobs","arguments":{"company":"supermicro.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3788147},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}