{"id":3959214,"url":"https://alion.io/job/american-express-senior-technical-product-manager-agent-evaluation-observability","title":"Senior Technical Product Manager – Agent Evaluation & Observability","company":{"id":2209,"name":"American Express","domain":"americanexpress.com","url":"https://alion.io/company/american-express","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Oracle","truth_index":{"grade":"A","score":98,"open_postings":89,"ghost_share":0.034,"stale_share":0.067,"repost_share":0.225,"time_to_fill_p50_days":9,"computed_at":"2026-11-12T10:38:37Z"}},"role":"Product","role_family":"Product","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["New York, United States","Phoenix, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":6,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Function Calling","optional":false},{"name":"LLM","optional":true},{"name":"Model Context Protocol","optional":true}],"status":"live","first_seen_at":"2026-11-04T18:16:12Z","employer_posted_date":"2026-11-04","last_verified_at":"2026-11-12T07:38:35Z","board_verified":true,"closed_at":null,"days_open":7,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":7},"description":"Agentic Partner Membership Experiences\nThe core of our company is not just the products we offer, but the personal connections our customers have with our brand. You can create digital experiences, from payments to rewards to servicing, that keep us connected to our customers and serve them at every touchpoint. From digital servicing and lifestyle features to secure, frictionless payment options, you can make a meaningful difference in our customers’ lives and help set us apart in the industry. Find your place in digital products on #TeamAmex.\nThe Amex Digital Labs team drives innovation at American Express by originating, incubating, and managing products and platforms that are essential to our customers’ digital lives. We are a fast-paced team that leverages emerging technologies to develop digital payment and financial services products and engaging Membership experiences for Card Members. We often collaborate with technology companies and strategic partners to bring products to the market.\nWe are looking for a Senior Technical Product Manager to lead the roadmap for tools and workflows that hlep teams test and improve agent-enabled customer experiences. You will help teams measure how well agent's complete customer tasks, understand failures and use the findings to improve their experiences.\nYou will own the product roadmap, requirements and adoption of shared evaluation tools and workflows. Working with engineering and enterprise platform teams, you will build on existing capabilities to help product teams assess agent performance and decide what to improve. Domain product owners will remain accountable for release decisions.\n Primary responsibilities:\nOwn Evaluation and Quality workflows\nOwn the roadmap for shared agent evaluation and observability capabilities, from user discovery and requirements through launch, adoption and improvement.\nPartner with domain owners to define representative journeys, prompt sets, expected tool behavior and outputs, scoring rubrics and acceptance criteria.\nDefine requirements for automated evaluation, human review, and tools that help teams diagnose failures.\nConnect Evaluations with Operational Evidence\nDefine product requirements for traces, events, diagnostic views and review tools that connect customer intent, tool or API calls and observed outcomes where data access permits.\nWork with engineering, analytics and existing enterprise evaluation and observability teams to extend shared capabilities, reuse instrumentation and standardize evidence.\nIdentify where limited access to partner data prevents a complete assessment.\nPartner with Identity and domain owners to test policy-relevant behavior, including denied actions, revoked access and required approvals.\nMeasure and Improve Agent Experiences\nDefine meaningful measures such as task completion, correct tool use, response quality, failure and recovery rates, latency and cost.\nDefine how testing and monitoring work together across controlled test environments, end-to-end partner integrations and production. Clarify what each can measure and use the findings to prioritize improvements with the responsible teams.\nMeasure adoption and usefulness of evaluation workflows, including review effort, coverage and time to identify problems. Communicate the limits of findings based on sampled data.\nLead a Reusable Product Capability\nTranslate measurement gaps into a prioritized backlog, clear requirements, and actionable acceptance criteria. Recommend when to extend existing tools, integrate a solution or build a missing capability.\nBuild privacy and data governance into evaluation and telemetry requirements, including permitted uses of partner-provided data, appropriate access, minimization and retention. Communicate uncertainty, release risks and tradeoffs clearly.\n Minimum Qualifications\n6-8+ years of experience in product management, product development, software engineering, technical platforms, analytics, quality tooling, or AI products.\nExperience owning evaluation, testing, observability, analytics or related technical products through launch, adoption and iteration, with measurable user or business outcomes.\nPractical understanding of AI or agent-enabled systems, including nondeterministic behavior, tool or API interactions, and the need for both automated tests and human judgment.\nAbility to define metrics, interpret evidence and identify sampling, coverage, or measurement limitations. Familiarity with traces, logs, APIs and technical troubleshooting.\nStrong product discovery, roadmap and prioritization skills, including translating ambiguous quality problems into usable tools and repeatable workflows.\nCross-functional leadership and clear communication, with the ability to balance customer outcomes, engineering constraints and privacy or risk requirements.\nPreferred Qualifications\nExperience with LLM or agent evaluation, scenario design, human review, model-assisted scoring or production failure analysis.\nFamiliarity with MCP, function calling, telemetry standards or evaluation frameworks; ability to use code or queries to investigate evidence.\nExperience building shared capabilities adopted by multiple teams or delivering products in financial services or another regulated industry.\nBachelor’s degree or equivalent experience.\nEmployment eligibility to work with American Express in the United States is required as the company will not pursue visa sponsorship for these positions.","description_format":"text","description_chars":5471,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":8,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Commercial & Retail Banks","Cards & Card Issuing","Payment Processing & Gateways"],"lifecycle":[{"event":"open","at":"2026-11-09T10:32:55Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":24,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 101 · green card filings: 30","filings_12m":101,"filings_prev_12m":132,"green_card_filings_12m":30,"median_offered_wage_usd":139485,"route":null,"cap_exempt":false,"checked_at":"2026-11-12T07:36:15+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)","Alion: sponsorship offered in the employer's own job adverts (24 months)"],"filings_for_role_12m":18}],"liveness":{"score":64,"band":"ok","label":"Likely open","p_open":1,"p_active":0.858,"p_room":0.75,"age_days":7,"expected_fill_days":9,"reasons":["conf:3","velocity","win:late","comp:brand"],"computed_at":"2026-11-12T10:38:37Z"},"pay":null,"html_url":"https://alion.io/job/american-express-senior-technical-product-manager-agent-evaluation-observability","json_url":"https://alion.io/job/american-express-senior-technical-product-manager-agent-evaluation-observability.json","meta":{"generated_at":"2026-11-12T13:29:24Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":28086,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2209},"rest":"https://alion.io/mcp/rest/get_company?id=2209"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Famerican-express-senior-technical-product-manager-agent-evaluation-observability"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Famerican-express-senior-technical-product-manager-agent-evaluation-observability"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Famerican-express-senior-technical-product-manager-agent-evaluation-observability"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/american-express-senior-technical-product-manager-agent-evaluation-observability\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Famerican-express-senior-technical-product-manager-agent-evaluation-observability"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3959214}},{"tool":"search_jobs","arguments":{"company":"americanexpress.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3959214},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}