{"id":1742617,"url":"https://alion.io/job/newton-research-qa-engineer-ai-native-quality","title":"QA Engineer-AI Native Quality","company":{"id":679538,"name":"Newton Research","domain":"newtonresearch.ai","url":"https://alion.io/company/newtonresearch","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"QA","role_family":"QA","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Boston, United States","Needham, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":115000,"max":130000,"currency":"USD","period":"year","gross":null,"usd_annual":130000},"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Claude","optional":false},{"name":"Function Calling","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"LLM","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Playwright","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Tool Use","optional":false}],"status":"live","first_seen_at":"2026-10-02T14:36:54Z","employer_posted_date":"2026-10-02","last_verified_at":"2026-10-15T12:16:01Z","board_verified":true,"closed_at":null,"days_open":12,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":12},"description":"QA Engineer, AI-Native Quality\nNewton Research · Research & Development · Boston / Needham, MA\nCompany Description\nNewton Research is a fast-growing software start-up founded by repeat entrepreneurs and well-funded by blue chip venture capital firms. We are building the next generation of the closed loop media lifecycle, developing AI agents that leverage the latest in LLMs and generative AI with specialized knowledge. Our products generate actionable business insights for our customers and partners, assisting in each step of the media planning, buying and measurement lifecycle.\nAbout the Role\nNewton ships on a sprint cadence through a develop, stage and customer-environment pipeline, and the product surface is wide: conversations, blueprints, connectors, scheduled tasks, permissions and sharing, SSO, and AI agents whose behavior is not fully deterministic. A missed regression lands in front of a media planner or a customer's security review.\nWe run everything through an AI-first lens, because it is the only way quality scales. If a quality task is repeatable, an agent does it and you supervise; if it takes judgment, that is where you spend your time.\nThe gap this hire fills: evals and skill-change testing. Code changes already have CI and review, including PRs written by Claude. What has no safety net is behavior change: an edit to a skill, a prompt, a tool definition or a model version can silently change what our agents do, and nothing tests that today. You own that layer: curated eval sets, scoring and regression tracking, so any change to how an agent behaves is measured before it ships.\nMeasure the AI: agent output varies run to run, so “correct” is a range you define with evals, rubrics and scoring, not an exact-match assertion.\nLet agents run the checks: agents run suites, triage failures and draft repro-ready defects; you design their roles and guardrails and review what they produce.\n Make Newton verifiable by agents: you keep the product and pipeline observable and fixture-friendly so agents can verify it without a human in the loop.\nYou are also a release-readiness partner (are we good?), alongside our existing QA lead, and that judgment stays human. But you are not hired to test Claude-driven PRs line by line.\nWhat You Will Do\nOwn the eval suite for Newton's agents: curated datasets of inputs and expected behaviors, rubric and LLM-as-judge scoring, regression tracking across model, prompt, skill and agent releases; validate judges against human-labeled examples\nTest skill and prompt changes before they ship: every edit to a skill, system prompt or tool definition runs against the relevant evals in CI, with a before/after comparison a reviewer can read in one glance; gate releases on the results\nCover the agent flows end to end: tool-call correctness, task completion, multi-turn coherence, blueprint creation from conversations, code generation, scheduled tasks; flag output that is wrong, empty or silently degraded\nHandle non-determinism with rigor: repeated runs, pass-rate thresholds and simple statistics, so a flaky agent is a measured finding, not an anecdote\nTurn production and customer signal into evals: mine logs, error tracking and customer reports so every escaped bad-behavior case becomes a permanent eval, ideally drafted by an agent and reviewed by you\nProbe AI-specific risk: prompt injection, data leakage across users, projects and permissions, hallucinated or ungrounded numbers in analytics output, cost and latency regressions\nAutomate before you repeat: any check you do by hand twice becomes a Playwright test, an eval or an agent workflow; drive manual regression time down every sprint\nDesign agentic QA workflows in CI: agents that run suites, triage failures, draft defects and re-verify fixes, with guardrails, cost limits and escalation rules you define\nKeep human-judgment work sharp: exploratory testing across roles, feature flags and environments, SSO and connector authorization, and the cross-feature bugs only a person thinks to look for\n Write bug reports that are machine- and human-readable, and partner with engineering on testability (observability, seedable data, stable interfaces)\nWhat Makes You a Great Fit\n4+ years in QA or test engineering on a complex web product, ideally B2B SaaS shipping frequently\nHands-on experience building evals for LLM or agent products (datasets, rubrics, LLM-as-judge, regression tracking), or a clear track record of getting there fast\nWorking knowledge of how agents work: prompting, skills, tool calling, context, RAG and orchestration, well enough to tell where a failure originates\nAn automation-first instinct: you can show what you removed from a manual process\nPython (or similar) for eval tooling and data checks; Playwright or similar for end-to-end tests\nStatistical literacy: pass rates, variance and sample size when outputs are not deterministic\nDaily user of AI coding and testing assistants, with the judgment to verify their output rather than trust it\nStrong exploratory instincts and excellent written communication\nNice to have: adtech, martech or marketing analytics domain; SSO / SAML / OAuth flows; data-connector testing; eval or observability tooling\nHow Success Is Measured\nEval suite covering Newton's core agent flows, run on every model, prompt, skill or agent change, with judge accuracy checked against human labels\nShare of skill and prompt changes that ship with a before/after eval result (target: all of them)\nEscaped bad-behavior cases converted into permanent evals\nShare of regression coverage run automatically or by agents, rising every sprint\nRelease calls that hold up: few surprises after release\nSalary range: $115,000-130,000 + Equity","description_format":"text","description_chars":5721,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Marketing Analytics & Attribution"],"lifecycle":[{"event":"open","at":"2026-10-03T03:53:51Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":12,"expected_fill_days":170,"reasons":["conf:1","win:early"],"computed_at":"2026-10-15T05:46:00Z"},"pay":{"stated_usd_annual":130000,"is_top_pay":true},"html_url":"https://alion.io/job/newton-research-qa-engineer-ai-native-quality","json_url":"https://alion.io/job/newton-research-qa-engineer-ai-native-quality.json","meta":{"generated_at":"2026-10-15T12:22:46Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":19043,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":679538},"rest":"https://alion.io/mcp/rest/get_company?id=679538"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fnewton-research-qa-engineer-ai-native-quality"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fnewton-research-qa-engineer-ai-native-quality"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fnewton-research-qa-engineer-ai-native-quality"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/newton-research-qa-engineer-ai-native-quality\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fnewton-research-qa-engineer-ai-native-quality"}]}