{"id":402116,"url":"https://alion.io/job/mastercard-manager-ai-engineering-tester-","title":"Manager, AI Engineering (Tester )","company":{"id":252,"name":"Mastercard","domain":"mastercard.com","url":"https://alion.io/company/mastercard","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"A","score":94,"open_postings":254,"ghost_share":0.016,"stale_share":0.358,"repost_share":0.094,"time_to_fill_p50_days":21,"computed_at":"2026-10-17T05:46:00Z"}},"role":"QA","role_family":"QA","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["O'Fallon, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":140000,"max":231000,"currency":"USD","period":"year","gross":null,"usd_annual":231000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Amazon CloudWatch","optional":false},{"name":"Amazon SageMaker","optional":false},{"name":"Anthropic","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Datadog","optional":false},{"name":"DeepEval","optional":false},{"name":"GCP","optional":false},{"name":"Grafana","optional":false},{"name":"Hallucination","optional":false},{"name":"Hugging Face","optional":false},{"name":"LangChain","optional":false},{"name":"LangGraph","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"LLMOps","optional":false},{"name":"MLFlow","optional":false},{"name":"OpenAI","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"SQL","optional":false},{"name":"TruLens","optional":false},{"name":"Weights & Biases","optional":false}],"status":"live","first_seen_at":"2026-07-24T00:00:00Z","employer_posted_date":"2026-07-24","last_verified_at":"2026-10-17T03:51:18Z","board_verified":true,"closed_at":null,"days_open":86,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":86},"description":"Our Purpose\nMastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.\nTitle and Summary\nManager, AI Engineering (Tester )Mastercard's Business & Market Insights (B&MI) group delivers unparalleled data-driven intelligence and frontier AI solutions that help organizations make smarter, faster, and more impactful decisions. We are currently looking for a AI Tester for the Operational Intelligence Program within B&MI. This is a highly specialized, hands-on AI testing leadership position dedicated to ensuring our Generative AI, LLM, and agentic systems are accurate, safe, reliable, and enterprise-ready. This role will lead AI quality engineering efforts - defining evaluation frameworks, red-teaming strategies, and LLMOps quality gates - while fostering a culture of rigorous, first-class AI testing across the program.Roles and Responsibilities:\nDesign and own end-to-end LLM evaluation frameworks - including automated prompt regression pipelines, output scoring, semantic benchmarking, and hallucination detection across model versions and prompt variations.\nBuild comprehensive test suites for agentic AI systems - validating tool selection, inter-agent coordination, task decomposition, goal completion, and failure handling across multi-step reasoning workflows.\nDevelop RAG pipeline evaluation frameworks assessing retrieval precision, chunk relevance, context faithfulness, answer grounding, and hallucination rates using tools like RAGAS, TruLens, and DeepEval.\nLead structured red-teaming and adversarial testing exercises targeting prompt injection, jailbreaks, data leakage, context poisoning, and model manipulation - building and maintaining an evolving adversarial test library.\nExecute fairness, bias, and Responsible AI audits - testing for demographic bias, sentiment skew, representation gaps, and validating explainability mechanisms, citations, and confidence score accuracy.\nDesign and run inference performance benchmarks - measuring latency, throughput, token efficiency, and degradation under peak load - and enforce LLM quality gates within CI/CD pipelines on Databricks (AWS).\nBuild production monitoring and drift detection pipelines tracking semantic output drift, embedding shifts, retrieval degradation, and anomalous agent behaviors using observability tooling (Grafana, Datadog, CloudWatch).\nDefine the AI testing roadmap and quality standards for the program - establishing evaluation metrics, tooling choices, and documentation practices across all Gen AI workstreams.\nPartner with Gen AI engineers, ML engineers, and product stakeholders to embed quality from day one - reviewing prompt architectures, agent designs, and system workflows for testability and risk.\nContinuously research and adopt frontier evaluation benchmarks (RAGAS, MMLU, TruthfulQA, MT-Bench) and emerging AI testing methodologies to keep quality practices at the cutting edge.\nAll About You:\nMaster's/Bachelor's degree in Computer Science, AI/ML, or Software Engineering, with considerable hands-on experience leading AI/ML quality engineering or LLM testing programs in production environments.\nDemonstrated expertise testing LLM and Gen AI systems - including prompt testing, output evaluation, hallucination detection, RAG pipeline assessment, and agentic workflow validation in real production settings.\nDeep hands-on knowledge of AI evaluation frameworks and tooling: RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, Weights & Biases Evals, or equivalent platforms.\nStrong understanding of Gen AI failure modes - hallucination, prompt injection, retrieval grounding failures, context drift, agent loop failures - and proven methods to surface and document them systematically.\nStrong Python programming skills with the ability to independently build test automation scripts, evaluation pipelines, and API-level integration tests; SQL proficiency required.\nWorking knowledge of LLM ecosystems - OpenAI, Anthropic, Hugging Face, LangChain/LangGraph - sufficient to understand model behavior, prompt structure, and agent architecture deeply enough to test them rigorously.\nFamiliarity with MLOps/LLMOps pipelines (MLflow, Databricks, SageMaker) and experience integrating automated quality gates into CI/CD workflows for AI systems.\nExperience with cloud AI infrastructure (AWS, Azure, or GCP) and observability tooling for monitoring live AI system behavior and output quality in production.\n• Strong analytical, communication, and stakeholder management skills - with the ability to translate complex AI failure patterns into clear risk assessments and remediation recommendations for both technical and business audiences.Mastercard is a merit-based, inclusive, equal opportunity employer that considers applicants without regard to gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. We hire the most qualified candidate for the role. In the US or Canada, if you require accommodations or assistance to complete the online application process or during the recruitment process, please contact  and identify the type of accommodation or assistance you are requesting. Do not include any medical or health information in this email. The Reasonable Accommodations team will respond to your email promptly.\nCorporate Security Responsibility\nAll activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:\nAbide by Mastercard’s security policies and practices;\n\nEnsure the confidentiality and integrity of the information being accessed;\n\nReport any suspected information security violation or breach, and\n\nComplete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.\n\nIn line with Mastercard’s total compensation philosophy and assuming that the job will be performed in the US, the successful candidate will be offered a competitive base salary and may be eligible for an annual bonus or commissions depending on the role. The base salary offered may vary depending on multiple factors, including but not limited to location, job-related knowledge, skills, and experience. Mastercard benefits for full time (and certain part time) employees generally include: insurance (including medical, prescription drug, dental, vision, disability, life insurance); flexible spending account and health savings account; paid leaves (including 16 weeks of new parent leave and up to 20 days of bereavement leave); 80 hours of Paid Sick and Safe Time, 25 days of vacation time and 5 personal days, pro-rated based on date of hire; 10 annual paid U.S. observed holidays; 401k with a best-in-class company match; deferred compensation for eligible roles; fitness reimbursement or on-site fitness facilities; eligibility for tuition reimbursement; and many more. Mastercard benefits for interns generally include: 56 hours of Paid Sick and Safe Time; jury duty leave; and on-site fitness facilities in some locations.Pay Ranges\nO'Fallon, Missouri: $140,000 - $231,000 USD","description_format":"text","description_chars":7667,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["401k plan","Life insurance"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Cards & Card Issuing","Payment Processing & Gateways"],"lifecycle":[{"event":"open","at":"2026-09-06T00:47:14Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 110 · green card filings: 139","filings_12m":110,"filings_prev_12m":161,"green_card_filings_12m":139,"median_offered_wage_usd":137520,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":3}],"liveness":{"score":6,"band":"cold","label":"Long shot","p_open":1,"p_active":0.197,"p_room":0.28,"age_days":85,"expected_fill_days":21,"reasons":["conf:1","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-17T05:46:00Z"},"pay":{"stated_usd_annual":231000,"is_top_pay":true},"html_url":"https://alion.io/job/mastercard-manager-ai-engineering-tester-","json_url":"https://alion.io/job/mastercard-manager-ai-engineering-tester-.json","meta":{"generated_at":"2026-10-18T00:28:52Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":902,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":252},"rest":"https://alion.io/mcp/rest/get_company?id=252"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fmastercard-manager-ai-engineering-tester-"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fmastercard-manager-ai-engineering-tester-"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fmastercard-manager-ai-engineering-tester-"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/mastercard-manager-ai-engineering-tester-\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fmastercard-manager-ai-engineering-tester-"}]}