2,326,972open jobs
146,595companies
277,438added this week
Browse all
Location
Hybrid (Pune, India)
Seniority
Principal
Visa
Sponsors visas in United States, United Kingdom; no register of this country names it

Confirmed on the employer's own hiring board on Nov 17, 2026. First seen by Alion on Nov 17, 2026. BMC Software scores A on the Alion truth index.

Overview
Company
Impact
Profile match
BMC Software provides service management, automation and mainframe tooling for large enterprises. Its Helix suite covers IT service desks, discovery and operations analytics. The company remains a major supplier of mainframe performance and DevOps software.

CareerArc Code

CA-SW

Hybrid: #LI-Hybrid

BMC empowers nearly 80% of the Forbes Global 100 to accelerate business value, faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.

The IZOT product line includes BMC’s Intelligent Z Optimization & Transformation products, which help the world’s largest companies to monitor and manage their mainframe systems. The modernization of mainframe is the beating heart of our product line, and we achieve this goal by developing products that improve the developer experience, the mainframe integration, the speed of application development, the quality of the code and the applications’ security, while reducing operational costs and risks. We acquired several companies along the way, and we continue to grow, innovate, and perfect our solutions on an ongoing basis.

About the Role

You build the systems that tell us whether our agentic AI is good enough to ship: evaluation frameworks, golden datasets, and rubrics; automated eval harnesses in CI; and drive down failure modes such as hallucination, drift, and unsafe or non-repeatable output. You are the reason customers can trust what our agents produce.

At this level you independently own well-scoped work from definition through delivery.

Scope at this level: Independently owns well-scoped eval work for a feature/team from definition through delivery.

Organizational impact expected: Improves outcomes for one team or project.

Key Responsibilities

Core responsibilities for this role:

  • Define and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications.
  • Create golden datasets, rubrics, and scoring methods (including LLM-as-a-judge) so quality is measured consistently.
  • Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices.
  • Stand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship.
  • Define quality metrics and release gates for customer-ready vs prototype.
  • Design testing strategies for multi-agent workflows and AI systems using tools, APIs, MCP, and enterprise apps; Playwright where relevant.
  • Evaluate end-to-end agent behaviour (planning, tool use, response generation, task completion).
  • Identify failure modes using observability/tracing; recommend reliability improvements.
  • Close the loop from evaluation results into model/prompt improvements with DS and AI Engineering.
  • Champion Responsible-AI and human-in-the-loop principles; contribute to governance and model validation.

Must-Have Skills & Experience

  • Bachelor's or master's in computer science, AI, or related field (or equivalent experience).
  • Experience with Generative AI / LLM applications or AI/ML systems and quality/evaluation frameworks (depth scales with level).
  • Understanding of LLM/RAG/agentic failure modes; vector DBs and embedding retrieval as level requires.
  • Python; building eval sets, rubrics, automated scoring; CI/CD integration.
  • LLM observability/tracing (e.g. Langfuse, OTel) and analytics (OpenSearch or similar) as level requires.
  • Rigorous, data-driven mindset; collaboration across DS, engineering, and product.
  • Evidence of owning an eval suite that influenced a release decision.
  • Builds consistent rubrics/datasets - not one-off manual spot checks only.
  • Works independently on scoped quality problems.

Evidence we will look for in hiring:

  • Past experience: Owned a defined evaluation suite or quality workflow for an AI feature and partnered with stakeholders on “good enough.”
  • Delivery evidence: Decision-ready quality report; evals integrated into a release or CI path for one product/team.
  • Shared expectation: Independently owns well-scoped work from definition through delivery.

Nice-to-Have Skills

  • Ragas, DeepEval, promptfoo, or similar eval frameworks.
  • AI safety/governance; enterprise/regulated environments.
  • OpenShift/AWS; performance/load/reliability testing of AI apps.
  • MLOps; open-weight models/local serving; MCP; Agile/Atlassian

Our commitment to you!

BMC’s culture is built around its people. We have 6000+ brilliant minds working together across the globe. You won’t be known just by your employee number, but for your true authentic self. BMC lets you be YOU!

If after reading the above, You’re unsure if you meet the qualifications of this role but are deeply excited about BMC and this team, we still encourage you to apply! We want to attract talents from diverse backgrounds and experience to ensure we face the world together with the best ideas!

BMC is committed to equal opportunity employment regardless of race, age, sex, creed, color, religion, citizenship status, sexual orientation, gender, gender expression, gender identity, national origin, disability, marital status, pregnancy, disabled veteran or status as a protected veteran. If you need a reasonable accommodation for any part of the application and hiring process, visit the accommodation request page.

BMC Software maintains a strict policy of not requesting any form of payment in exchange for employment opportunities, upholding a fair and ethical hiring process.

At BMC we believe in pay transparency and have set the midpoint of the salary band for this role at 4,700,000 INR. Actual salaries depend on a wide range of factors that are considered in making compensation decisions, including but not limited to skill sets; experience and training, licensure, and certifications; and other business and organizational needs.

The salary listed is just one component of BMC's employee compensation package. Other rewards may include a variable plan and country specific benefits.

We are committed to ensuring that our employees are paid fairly and equitably, and that we are transparent about our compensation practices.

(Returnship@BMC)

Had a break in your career? No worries. This role is eligible for candidates who have taken a break in their career and want to re-enter the workforce. If your expertise matches the above job, visit to https://bmcrecruit.avature.net/returnship know more and how to apply.

Min salary

3,525,000

Mid point salary

4,700,000

Max salary

5,875,000

Min Salary - NEW

3,525,000

Max Salary - NEW

5,875,000

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,326,972 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

QA
Similar stack
Same company
Pune
Hybrid · Full-Time · 6+ years exp · Master's Degree · Chennai
Python
JavaScript
TypeScript
C#
Frontend
GraphQL
Angular
React.js
JQuery
Mobile
JUnit
DevOps
Rest API
Azure
CI/CD
Jenkins
AWS
Bamboo
Buildkite
Bitbucket
GitHub
TCP/IP
Management
Jira
Agile
Scrum
Kanban
QA
TestNG
Selenium
JMeter
Cypress
Playwright
Gatling
Postman
Rest-Assured
SoapUI
Apply
In office · 10+ years exp · Bengaluru · Chennai
Python
Java
AI/ML
Anomaly Detection
DevOps
GCP
Azure
CI/CD
AWS
Docker
Kubernetes
Self-Healing
MPLS
QA
Selenium
Pytest
Robot Framework
Apply
Automation Test Lead 11 hours ago
In office · Full-Time · Pune
JavaScript
TypeScript
DevOps
CI/CD
Management
Agile
QA
Selenium
Cucumber
Playwright
Postman
Rest-Assured
Apply
In office · Pune
Java
Apply
Project Lead - QA 1 day ago
In office · 10+ years exp · Pune
TypeScript
SQL
C#
C#
.NET
AI/ML
Copilot
Claude
Prompt Engineering
AI Agents
DevOps
Azure DevOps
Azure
CI/CD
Git
GitHub
Management
Jira
Agile
Scrum
QA
Playwright
Appium
BrowserStack
Rest-Assured
Applitools
Apply
In office · Internship · Bachelor's Degree · Singapore
Python
JavaScript
SQL
AI/ML
ChatGPT
Prompt Engineering
AI Agents
Gradio
RAG
Streamlit
OCR
Frontend
Vue.js
React.js
Apply
$73k – $130k per year · Hybrid · 2+ years exp · Minneapolis
Python
JavaScript
SQL
Databases
Snowflake
Databricks
AI/ML
Copilot
AI Agents
Human-in-the-Loop
Copilot Studio
Analytics
Power BI
Domo
Management
Power Automate
Apply
$92k – $164k per year · Hybrid · 5+ years exp · Minneapolis
Python
JavaScript
SQL
Databases
Snowflake
Databricks
AI/ML
Copilot
Prompt Engineering
AI Agents
Human-in-the-Loop
Copilot Studio
Analytics
Power BI
Domo
Management
Power Automate
Apply
In office · Full-Time · 3+ years exp · Bachelor's Degree · India
Go
JavaScript
Java
Kotlin
Node JS
Java
Spring Boot
Kotlin
Mockito
Databases
MySQL
DynamoDB
AI/ML
Copilot
Cursor
Model Context Protocol
RAG
DevOps
Splunk
Kibana
CI/CD
Jenkins
AWS
Grafana
GitHub
Management
Agile
Scrum
QA
Playwright
Apply
$66k – $99k per year (gross) · In office · Full-Time · Vilnius
Python
SQL
Databases
Google BigQuery
AI/ML
NLP
LLM
Machine Learning
DevOps
GCP
Azure
Apply
In office · Full-Time · 1+ year exp · Pune
C#
Groovy
C#
.NET
Databases
MySQL
PostgreSQL
AI/ML
AI Agents
DevOps
Jenkins
Linux
QA
Cypress
Playwright
Apply
Principal QA Engineer 1 month ago
$153k – $255k per year · Remote (likely United States) · Bachelor's Degree · New York
Python
Databases
OpenSearch
AI/ML
Model Context Protocol
Function Calling
AI Agents
DeepEval
Langfuse
Promptfoo
Ragas
LLM
RAG
Hallucination
Human-in-the-Loop
LLM Evaluation
Multi-Agent Systems
DevOps
OpenShift
CI/CD
AWS
Management
Agile
QA
Playwright
Apply
Hybrid · 8+ years exp · Bachelor's Degree · Pune
Python
C#
C#
.NET
Databases
OpenSearch
AI/ML
AI Agents
LLMOps
DevOps
Terraform
Ansible
GCP
OpenShift
Helm
GitHub Actions
OpenTelemetry
Prometheus
GitLab CI
Azure
CI/CD
Jenkins
AWS
Docker
Kubernetes
Grafana
Management
Agile
Apply
$133k – $222k per year · Remote (United States) · 8+ years exp · United States
Apply
$115k – $191k per year · Remote (United States, Canada) · 6+ years exp · United States
C++
Assembly
Databases
Db2
AI/ML
Cursor
DevOps
GitHub
Apply
≈ $17k – $51k per year (Estimated) · Hybrid · Full-Time · 15+ years exp · Bachelor's Degree · Pune
DevOps
CI/CD
Self-Healing
Cybersecurity
Shift-Left Security
Management
Agile
QA
Selenium
JMeter
Postman
Apply
In office · Full-Time · 9+ years exp · Pune
JavaScript
TypeScript
Node JS
DevOps
Rest API
GCP
Azure
CI/CD
AWS
Docker
Kubernetes
Apply
Staff QA Engineer 1 day ago
In office · Full-Time · 7+ years exp · Pune
Python
Verilog
C++
SystemVerilog
VHDL
DevOps
CI/CD
Git
Chips/EDA
JTAG Debuggers
Apply
Hybrid · Full-Time · Pune
SQL
DevOps
Azure DevOps
Azure
Management
Agile
Scrum
Apply
Project Engineer 1 day ago
In office · Full-Time · 4+ years exp · Bachelor's Degree · Pune
Energy
IEC 61850
Apply
See all jobs
This is one of many
2,326,972 more open roles from verified company boards, updated every day.