{"id":82626,"url":"https://alion.io/job/tekion-staff-software-development-test-engineer-ai-evaluation-2","title":"Staff Software Development Test Engineer - AI Evaluation","company":{"id":804,"name":"Tekion","domain":"tekion.com","url":"https://alion.io/company/tekion","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":81,"open_postings":12,"ghost_share":0,"stale_share":0.75,"repost_share":0,"time_to_fill_p50_days":48,"computed_at":"2026-10-17T05:46:00Z"}},"role":"QA","role_family":"QA","seniority":"staff","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":24000,"max_usd":56000,"period":"year","method":"global_role_cell_scaled_by_country","sample_n":191},"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"Embeddings","optional":false},{"name":"Function Calling","optional":false},{"name":"Hallucination","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Synthetic Data","optional":false},{"name":"Tool Use","optional":false},{"name":"A/B Testing","optional":true},{"name":"DeepEval","optional":true},{"name":"LangSmith","optional":true},{"name":"LLM Guardrails","optional":true},{"name":"MLFlow","optional":true},{"name":"Promptfoo","optional":true},{"name":"Ragas","optional":true},{"name":"TruLens","optional":true},{"name":"Weights & Biases","optional":true}],"status":"live","first_seen_at":"2026-07-09T12:43:55Z","employer_posted_date":"2026-07-09","last_verified_at":"2026-10-18T12:49:44Z","board_verified":true,"closed_at":null,"days_open":101,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":101},"description":"About Tekion:\nPositively disrupting an industry that has not seen any innovation in over 50 years, Tekion has challenged the paradigm with the first and fastest cloud-native automotive platform that includes the revolutionary Automotive Retail Cloud (ARC) for retailers, Automotive Enterprise Cloud (AEC) for manufacturers and other large automotive enterprises and Automotive Partner Cloud (APC) for technology and industry partners. Tekion connects the entire spectrum of the automotive retail ecosystem through one seamless platform. The transformative platform uses cutting-edge technology, big data, machine learning, and AI to seamlessly bring together OEMs, retailers/dealers and consumers. With its highly configurable integration and greater customer engagement capabilities, Tekion is enabling the best automotive retail experiences ever. Tekion employs close to 3,000 people across North America, Asia and Europe.\nAbout the Role\nWe are looking for a highly motivated Staff SDET - AI Evaluation to join Tekion’s AI Platform team. Evaluation is the backbone of trustworthy AI: as Tekion scales from a handful of AI agents\nto 100+ across Service, Sales, F&I, and Analytics, this role builds the evaluation platform and frameworks that let every ML team measure, trust, and improve the quality of AI outputs.In this role, you will be responsible for defining and building Tekion’s AI evaluation capabilities as a shared platform service. You will work closely with ML Engineers, Data Scientists, the AI Platform team, and Product Management to design evaluation datasets, automated scoring pipelines, and quality metrics that quantify the accuracy, consistency, and safety of AI\u0002generated outputs across the organization. You will own the systems that answer “is this model or agent good enough to ship, and is it \nstaying good in production?” - from offline benchmarks and LLM-as-judge pipelines to online evaluation and continuous quality monitoring. You will also use AI and LLMs to scale evaluation itself, building automated judges and synthetic datasets that expand coverage faster than manual review ever could.\nWhat You’ll Do\nDevelop a deep understanding of Tekion’s AI agents, ML models, and the quality dimensions that matter for each business domain.\n\nDesign, enhance, and own Tekion’s AI evaluation infrastructure as a shared capability used across ML teams.\n\nCreate, curate, and maintain evaluation datasets (evals) and golden/ground-truth sets across use cases and domains.\n\nDefine quality metrics for AI outputs - accuracy, relevance, faithfulness/groundedness, consistency, safety, and task success.\n\nBuild automated scoring pipelines, including LLM-as-judge, rubric-based, and reference\u0002based evaluation methods.\n\nValidate user intents and measure response accuracy and consistency for AI-powered capabilities such as the Analytics Agent.\n\nIdentify hallucinations, unsafe or biased outputs, and edge cases; design targeted eval suites to catch them.\n\nBuild both offline evaluation (pre-release benchmarking) and online evaluation (production quality monitoring, A/B, drift detection).\n\nEstablish evaluation gates in CI/CD so model, prompt, or data changes are quality-checkedbefore release.\n\nDevelop dashboards and reporting that make AI quality visible and actionable for ML and product teams.\n\nUse AI/LLMs to scale evaluation - automated judges, synthetic data generation, and eval tooling\n\nChampion evaluation and responsible-AI quality best practices across the organization.\n\nWhat We’re Looking For\n8+ years in SDET, quality engineering, ML engineering, or data science, with hands-on experience building evaluation or measurement systems - or a strong SDET background with deep LLM/ML fluency.\n\nStrong programming skills in Python, with the ability to build robust, reusable evaluation pipelines and tooling.\n\nDeep understanding of ML/LLM evaluation, benchmark design, and the pitfalls of evaluating non-deterministic systems.\n\nHands-on experience with LLM-as-judge, rubric-based scoring, or human-in-the-loop evaluation.\n\nSolid grasp of LLM/agent concepts - prompting, RAG, embeddings, tool use - and generative failure modes (hallucination, drift, prompt sensitivity, bias).\n\nExperience designing and curating datasets, including labeling/annotation strategy and data quality.\n\nStrong statistical intuition for interpreting eval results and significance.\n\nExcellent communication skills to translate quality signals into decisions for ML and product teams.\n\nNice to Have\nExperience with eval frameworks/tools such as Ragas, DeepEval, LangSmith, TruLens, Promptfoo, HELM, or provider eval suites.\nExperience building online evaluation, guardrails, or production model monitoring.\nFamiliarity with responsible AI / safety evaluation and red-teaming.\nExperience with experiment tracking (MLflow, Weights & Biases) and A/B testing.\nPrior work standing up evaluation as a platform capability for multiple teams\nCurrent Tekion Employees: Please apply via the Internal Job Board in Ashby\nNote: Tekion recently transitioned to a new recruiting tool and we appreciate your patience and feedback as we adjust to our new system!\nTekion is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, gender (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, victim of violence or having a family member who is a victim of violence, the intersectionality of two or more protected categories, or other applicable legally protected characteristics.\nFor more information on our privacy practices, please refer to our Applicant Privacy Notice here.","description_format":"text","description_chars":5781,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Automotive Software"],"lifecycle":[{"event":"open","at":"2026-07-09T12:43:55Z"}],"visa":[],"liveness":{"score":13,"band":"cold","label":"Long shot","p_open":1,"p_active":0.47,"p_room":0.28,"age_days":99,"expected_fill_days":48,"reasons":["conf:1","stale_co","velocity","win:tail","crowd:"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/tekion-staff-software-development-test-engineer-ai-evaluation-2","json_url":"https://alion.io/job/tekion-staff-software-development-test-engineer-ai-evaluation-2.json","meta":{"generated_at":"2026-10-18T15:58:01Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21967,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":804},"rest":"https://alion.io/mcp/rest/get_company?id=804"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ftekion-staff-software-development-test-engineer-ai-evaluation-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ftekion-staff-software-development-test-engineer-ai-evaluation-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ftekion-staff-software-development-test-engineer-ai-evaluation-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/tekion-staff-software-development-test-engineer-ai-evaluation-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ftekion-staff-software-development-test-engineer-ai-evaluation-2"}]}