{"id":4233783,"url":"https://alion.io/job/solvd-senior-ai-quality-evaluation-engineer","title":"Senior AI Quality & Evaluation Engineer","company":{"id":9288,"name":"Solvd","domain":"solvd.com","url":"https://alion.io/company/solvd","size_band":"501-1000","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"A","score":88,"open_postings":14,"ghost_share":0,"stale_share":0.5,"repost_share":0,"time_to_fill_p50_days":45,"computed_at":"2026-11-12T10:38:37Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon CloudWatch","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"OpenSearch","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"ISO 27001","optional":true},{"name":"LLM","optional":true},{"name":"Machine Learning","optional":true},{"name":"SOC 2","optional":true}],"status":"live","first_seen_at":"2026-11-13T09:46:25Z","employer_posted_date":"2026-11-13","last_verified_at":"2026-11-13T11:15:58Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.\nFollowing the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery -from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.\nWe are looking for a Senior AI Quality & Evaluation Engineer to own the independent proof that retrieval works and that permissions hold. You’ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates - so the engineers who built the system are not grading their own work.\nThis is a 50% allocation role across weeks 3-10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day-to-day with the retrieval engineer.\nWhat you’ll do\nBuild the golden question suite from top contact drivers and needs workshops - question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests.\n\nBuild the negative entitlement test set - queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm.\n\nBuild freshness and retirement tests - source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync.\n\nAutomate the harness to run on schedule and after every content, configuration, or model change - publishing scores to the quality dashboard with trend lines and regression alarms.\n\nDefine the QA gate content analysts apply to remediated content.\n\nRun the suite before every pilot change in weeks 7-9 and report misses with root cause - content, chunking, ranking, or permissions - to the right owner.\n\nWrite the evaluation runbook and train client stewards and platform owners to add questions, re-run the suite, and interpret regressions.\n\nRequired skills & experience\n3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule.\n\nHands-on evaluation of a search or RAG system in production - writing ground truth sets, scoring relevance and grounding, and tracking drift over time.\n\nExperience designing security-focused negative tests - authorization, entitlement, data exposure - and reporting them at the severity a client security team expects.\n\nAbility to work in AWS - invoking Bedrock and OpenSearch APIs, reading CloudWatch, deploying a scheduled job with infrastructure code.\n\nClear written reporting; the suite’s output is read by client stewards, not only engineers.\n\nPreferred qualifications\nAWS Certified AI Practitioner or Machine Learning Engineer Associate.\n\nExperience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.\n\nContact center or knowledge management domain exposure.\n\nFamiliarity with SOC 2 or ISO 27001 evidence practices.\n\nWhen you join Solvd, you'll…\nShape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.\n\nBe part of a global team with equal opportunities for collaboration across continents and cultures.\n\nThrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.\n\nReady to make an impact?\nIf you're excited to build things that matter, champion responsible AI, and grow with some of the industry’s sharpest minds. Apply today and let’s innovate together.\nSolvd is an equal opportunity employer.","description_format":"text","description_chars":4042,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Continuous learning"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Software Testing & QA","Custom Software Development","AI Consulting & Integration","Data Engineering & Migration Services"],"lifecycle":[{"event":"open","at":"2026-11-13T11:15:58Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":48,"reasons":["conf:2","win:early"],"computed_at":"2026-11-13T13:36:06Z"},"pay":null,"html_url":"https://alion.io/job/solvd-senior-ai-quality-evaluation-engineer","json_url":"https://alion.io/job/solvd-senior-ai-quality-evaluation-engineer.json","meta":{"generated_at":"2026-11-13T13:36:06Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":27282,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":9288},"rest":"https://alion.io/mcp/rest/get_company?id=9288"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsolvd-senior-ai-quality-evaluation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsolvd-senior-ai-quality-evaluation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsolvd-senior-ai-quality-evaluation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/solvd-senior-ai-quality-evaluation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsolvd-senior-ai-quality-evaluation-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4233783}},{"tool":"search_jobs","arguments":{"company":"solvd.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4233783},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}