{"id":2163829,"url":"https://alion.io/job/bmc-software-principal-qa-engineer","title":"Principal QA Engineer","company":{"id":58475,"name":"BMC Software","domain":"bmc.com","url":"https://alion.io/company/bmc-software","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Avature","truth_index":{"grade":"A","score":100,"open_postings":9,"ghost_share":0,"stale_share":0,"repost_share":0.333,"time_to_fill_p50_days":12,"computed_at":"2026-10-12T06:55:09Z"}},"role":"QA","role_family":"QA","seniority":"lead","employment_type":null,"work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"inferred_city","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":["New York, United States"],"countries":["US"],"hiring_countries":["US"],"hiring_countries_total":1,"salary":{"min":152925,"max":254875,"currency":"USD","period":"year","gross":null,"usd_annual":254875},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"Function Calling","optional":false},{"name":"Hallucination","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"Langfuse","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Multi-Agent Systems","optional":false},{"name":"OpenSearch","optional":false},{"name":"Playwright","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Tool Use","optional":false},{"name":"Agile","optional":true},{"name":"AWS","optional":true},{"name":"DeepEval","optional":true},{"name":"OpenShift","optional":true},{"name":"Promptfoo","optional":true},{"name":"Ragas","optional":true}],"status":"live","first_seen_at":"2026-10-09T08:53:20Z","employer_posted_date":"2026-10-09","last_verified_at":"2026-10-12T08:49:59Z","board_verified":true,"closed_at":null,"days_open":3,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":3},"description":"CareerArc Code\nCA-SW\nHybrid: #LI-Hybrid\nBMC empowers nearly 80% of the Forbes Global 100 to accelerate business value, faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.\nThe IZOT product line includes BMC’s Intelligent Z Optimization & Transformation products, which help the world’s largest companies to monitor and manage their mainframe systems. The modernization of mainframe is the beating heart of our product line, and we achieve this goal by developing products that improve the developer experience, the mainframe integration, the speed of application development, the quality of the code and the applications’ security, while reducing operational costs and risks. We acquired several companies along the way, and we continue to grow, innovate, and perfect our solutions on an ongoing basis.\nAbout the Role\nYou build the systems that tell us whether our agentic AI is good enough to ship: evaluation frameworks, golden datasets, and rubrics; automated eval harnesses in CI; and drive down failure modes such as hallucination, drift, and unsafe or non-repeatable output. You are the reason customers can trust what our agents produce.\nAt this level you independently own well-scoped work from definition through delivery.\nScope at this level: Independently owns well-scoped eval work for a feature/team from definition through delivery.\nOrganizational impact expected: Improves outcomes for one team or project.\nKey Responsibilities\nCore responsibilities for this role:\nDefine and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications.\nCreate golden datasets, rubrics, and scoring methods (including LLM-as-a-judge) so quality is measured consistently.\nApply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices.\nStand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship.\nDefine quality metrics and release gates for customer-ready vs prototype.\nDesign testing strategies for multi-agent workflows and AI systems using tools, APIs, MCP, and enterprise apps; Playwright where relevant.\nEvaluate end-to-end agent behaviour (planning, tool use, response generation, task completion).\nIdentify failure modes using observability/tracing; recommend reliability improvements.\nClose the loop from evaluation results into model/prompt improvements with DS and AI Engineering.\nChampion Responsible-AI and human-in-the-loop principles; contribute to governance and model validation.\nMust-Have Skills & Experience\nBachelor's or master's in computer science, AI, or related field (or equivalent experience).\nExperience with Generative AI / LLM applications or AI/ML systems and quality/evaluation frameworks (depth scales with level).\nUnderstanding of LLM/RAG/agentic failure modes; vector DBs and embedding retrieval as level requires.\nPython; building eval sets, rubrics, automated scoring; CI/CD integration.\nLLM observability/tracing (e.g. Langfuse, OTel) and analytics (OpenSearch or similar) as level requires.\nRigorous, data-driven mindset; collaboration across DS, engineering, and product.\nEvidence of owning an eval suite that influenced a release decision.\nBuilds consistent rubrics/datasets - not one-off manual spot checks only.\nWorks independently on scoped quality problems.\nEvidence we will look for in hiring:\nPast experience: Owned a defined evaluation suite or quality workflow for an AI feature and partnered with stakeholders on “good enough.”\nDelivery evidence: Decision-ready quality report; evals integrated into a release or CI path for one product/team.\nShared expectation: Independently owns well-scoped work from definition through delivery.\nNice-to-Have Skills\nRagas, DeepEval, promptfoo, or similar eval frameworks.\nAI safety/governance; enterprise/regulated environments.\nOpenShift/AWS; performance/load/reliability testing of AI apps.\nMLOps; open-weight models/local serving; MCP; Agile/Atlassian\n\n#LI-Remote\nOur commitment to you!\nBMC’s culture is built around its people. We have 6000+ brilliant minds working together across the globe. You won’t be known just by your employee number, but for your true authentic self. BMC lets you be YOU!\nIf after reading the above, You’re unsure if you meet the qualifications of this role but are deeply excited about BMC and this team, we still encourage you to apply! We want to attract talents from diverse backgrounds and experience to ensure we face the world together with the best ideas!\nBMC is committed to equal opportunity employment regardless of race, age, sex, creed, color, religion, citizenship status, sexual orientation, gender, gender expression, gender identity, national origin, disability, marital status, pregnancy, disabled veteran or status as a protected veteran. If you need a reasonable accommodation for any part of the application and hiring process, visit the accommodation request page.\nBMC Software maintains a strict policy of not requesting any form of payment in exchange for employment opportunities, upholding a fair and ethical hiring process.\nThe annual base salary range represents the low and high end of the BMC salary range for this position. Actual salaries depend on a wide range of factors that are considered in making compensation decisions, including but not limited to skill sets; experience and training, licensure, and certifications; and other business and organizational needs.\nThe range listed is just one component of BMC's employee compensation package. Other rewards may include a variable plan and country specific benefits.\nAt BMC, it is not typical for an individual to be hired at /near the top of the range. A reasonable estimate of the current range is $152,925 - $254,875\nMin salary\n152,925\nMid point salary\n203,900\nMax salary\n254,875\nMin Salary - NEW\n152,925\nMax Salary - NEW\n254,875","description_format":"text","description_chars":6144,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":true},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["DevOps & Platform Engineering","IT Service & Asset Management"],"lifecycle":[{"event":"open","at":"2026-10-09T08:53:20Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 12 · green card filings: 3","filings_12m":12,"filings_prev_12m":14,"green_card_filings_12m":3,"median_offered_wage_usd":149500,"route":null,"cap_exempt":false,"checked_at":"2026-10-03T21:08:04+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":0}],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":2,"expected_fill_days":12,"reasons":["conf:3","velocity","win:early"],"computed_at":"2026-10-12T06:55:09Z"},"pay":{"stated_usd_annual":254875,"is_top_pay":true},"html_url":"https://alion.io/job/bmc-software-principal-qa-engineer","json_url":"https://alion.io/job/bmc-software-principal-qa-engineer.json","meta":{"generated_at":"2026-10-12T10:25:05Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15967,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":58475},"rest":"https://alion.io/mcp/rest/get_company?id=58475"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbmc-software-principal-qa-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbmc-software-principal-qa-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbmc-software-principal-qa-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/bmc-software-principal-qa-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbmc-software-principal-qa-engineer"}]}