{"id":3290088,"url":"https://alion.io/job/delta-exchange-senior-qa-engineer-ai","title":"Senior QA Engineer - AI","company":{"id":175589,"name":"Delta Exchange","domain":"delta.exchange","url":"https://alion.io/company/delta-exchange","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Schema","truth_index":null},"role":"QA","role_family":"QA","seniority":"senior","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":["IN"],"hiring_countries_total":1,"salary":null,"salary_estimate":{"min_usd":69000,"max_usd":168000,"period":"year","method":"global_role_seniority_cell","sample_n":1282},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Aave","optional":false},{"name":"AI Agents","optional":false},{"name":"Copilot","optional":false},{"name":"DeepEval","optional":false},{"name":"Hallucination","optional":false},{"name":"Langfuse","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Promptfoo","optional":false},{"name":"Python","optional":false}],"status":"live","first_seen_at":"2026-10-07T13:32:23Z","employer_posted_date":null,"last_verified_at":"2026-10-07T13:32:23Z","board_verified":false,"closed_at":null,"days_open":21,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":21},"description":"About the CompanyAt Delta, we are reimagining and rebuilding the financial system. Join our team to make a positive impact on the future of finance.\n🎯 Mission Driven: Re-imagine and rebuild the future of finance.\n💡 Most innovative cryptocurrency derivatives exchange. With a daily traded volume of ~$ 10 billion, and increasing. Delta is bigger than all the Indian crypto exchanges combined.\n📈 Offer the widest range of derivative products and have been serving traders all over the globe since 2018 and growing fast.\n💪🏻 The founding team is comprised of IIT and ISB graduates. Business co-founders have previously worked with Citibank, UBS and GIC; and our tech co-founder is a serial entrepreneur who previously co-founded TinyOwl andHousing.com.\n💰 Funded by top crypto funds (Sino Global Capital, CoinFund, Gumi Cryptos) and crypto projects (Aave and Kyber Network).\nSenior QA Engineer - AI🎯 Role OverviewDelta operates three AI products used by live traders: a customer support chatbot, an API Copilot that generates and executes trading scripts, and an MCP server. This role owns the quality of those products and is responsible for measuring it objectively.\nTesting probabilistic systems differs fundamentally from testing deterministic ones. The same input produces different output on every run, an incorrect answer can be entirely fluent, and a prompt change in one flow can degrade another without any visible signal. The core of this role is establishing what correct means for these systems, building the datasets and scoring that measure it, and operating the release gate that prevents a regression from reaching customers.\nThis is an emerging discipline with no established playbook. We are defining the methodology as we build it, and the role carries a high degree of autonomy and ownership.\n🔬 What Sets This Role ApartMost QA roles that mention AI mean using AI to do testing faster: generating test cases from a PRD, healing flaky selectors, exploring an app with an agent. Those are useful and we do them.\nThis role is the other thing. The system under test is itself an AI, and the hard problem is deciding whether its output is correct when the same question produces a different answer every time and a wrong answer reads as convincingly as a right one. That means building evaluation datasets, defining what correct looks like, scoring against it, and defending a number that decides whether a release ships.\nIf you have spent time on the second problem, this role is built for you.\n🛠 Key Responsibilities\nBuild and maintain golden datasets for our AI products, trace mined from production or generated, and verified against a documented source of truth.\n\nDesign layered scoring for AI outputs: deterministic rules where behaviour can be asserted, and model-graded evaluation where it cannot.\n\nManage evaluation runs end to end: schedule and execute them against release candidates, compare results against prior baselines, and triage failures into product defects, incorrect expectations, and platform issues.\n\nOwn and operate the release gate that determines whether a prompt or model change is approved for production.\n\nIdentify failure modes ahead of customers, including hallucination, incorrect tool selection, loss of context across conversation turns, and PII exposure.\n\nInvestigate production traces to distinguish retrieval failures from generation failures from tool failures.\n\nConvert findings into actionable engineering evidence and into permanent regression coverage.\n\nDefine and report quality metrics for AI surfaces, and drive improvement against them.\n\n✅ RequirementsNon-negotiable\n\nAt least 1 year owning quality for AI products in a lead or primary-owner capacity. This includes chatbots and conversational assistants, code or content generation products, and agentic systems. You should have worked directly with prompts, tool and function schemas, and model behaviour, rather than testing around them.\n\nDemonstrated hands-on experience building or operating an evaluation harness for an AI system, whether in-house or using a framework such as DeepEval, RAGAS or Promptfoo. You should be able to describe what the harness measured, the results it produced, and the decisions those results informed.\n\nAlso required\n\n4-6 years of QA / SDET experience.\n\nWorking proficiency in Python. Our evaluation platform, trace analysis and internal tooling are Python-based.\n\nAbility to analyse traces and tool calls, using observability tooling such as Opik, Langfuse or LangSmith or an equivalent, to determine root cause rather than reporting the symptom.\n\nStrong API testing experience. The majority of the surface under test is API-level.\n\nSufficient engineering ability to build your own tooling and automation.\n\nClear written communication. Findings must be documented in a form engineering can act on directly.\n\nBonus\nExperience with trading or exchange platforms, including familiarity with futures and options, crypto derivatives, margin, or settlement flows. Domain knowledge can be picked up on the job, but arriving with it shortens the ramp considerably.\n🚀 Why Join Us?\nPlay a pivotal role in shaping the regulatory landscape for digital assets and Web3 in India.\n\nWork directly with founders and senior leadership on high-impact strategic initiatives.\n\nBe part of a mission-driven, fast-growing organisation at the forefront of financial innovation.\n\nCompetitive compensation, leadership exposure, and significant growth opportunities.\n\nOriginally posted on Himalayas","description_format":"text","description_chars":5515,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Growth opportunities"],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Crypto Exchanges"],"lifecycle":[{"event":"open","at":"2026-10-26T19:11:04Z"}],"visa":[],"liveness":{"score":65,"band":"ok","label":"Likely open","p_open":0.85,"p_active":0.846,"p_room":0.9,"age_days":21,"expected_fill_days":39,"reasons":["seen:21","win:mid"],"computed_at":"2026-10-29T05:49:05Z"},"pay":null,"html_url":"https://alion.io/job/delta-exchange-senior-qa-engineer-ai","json_url":"https://alion.io/job/delta-exchange-senior-qa-engineer-ai.json","meta":{"generated_at":"2026-10-29T11:21:07Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":17221,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-30T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":175589},"rest":"https://alion.io/mcp/rest/get_company?id=175589"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdelta-exchange-senior-qa-engineer-ai"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdelta-exchange-senior-qa-engineer-ai"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdelta-exchange-senior-qa-engineer-ai"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/delta-exchange-senior-qa-engineer-ai\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdelta-exchange-senior-qa-engineer-ai"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3290088}},{"tool":"search_jobs","arguments":{"company":"delta.exchange","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3290088},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}