Production Engineer - HPC Reliability & Automation

Job Description

• Skill Set :

  Python,Kubernetes,Terraform,Karpenter

• Total Experience :

  5.00 to 15.00 Years

• No of Openings :

  1

• Job Post Date :

  03/11/2026

• Job Expiry Date :

  30/12/2026

• Domain :

  IT

• Location :

  SAN JOSE [United States]

• Job Reference No :

  4116321

Job Summary

Role: Production Engineer - AI Storage & HPC Infrastructure

Location: United States

Duration: Fulltime

Client: This requirement is with Meta through Amazon.

Work Type: Remote-first, with occasional on-site visits to customer offices.

Tech Mahindra is seeking a highly skilled Production Engineer - HPC Reliability & Automation to support and enhance large-scale cloud and High-Performance Computing (HPC) environments powering advanced AI, analytics, and accelerated computing workloads.

This role focuses on reliability engineering, operational excellence, automation, observability, and performance optimization across distributed compute infrastructure. The ideal candidate will have deep expertise in Linux systems, large-scale distributed environments, troubleshooting complex production issues, and building intelligent automation solutions that improve operational efficiency and platform stability.

Success in this role requires strong problem-solving capabilities, operational ownership, and the ability to quickly understand complex infrastructure ecosystems spanning compute, scheduling, storage, networking, cloud platforms, and accelerator technologies.

Key Responsibilities

• Develop deep technical understanding of large-scale HPC and cloud infrastructure environments, including compute clusters, workload schedulers, accelerator platforms, storage systems, networking, and authentication services.

• Proactively identify reliability, scalability, and performance issues by correlating logs, metrics, telemetry, and events across multiple cloud and on-premises environments.

• Troubleshoot complex production issues across Linux systems, containers, distributed applications, schedulers, storage, networking, and compute infrastructure.

• Design and build automation solutions that improve detection, diagnosis, remediation, and operational efficiency while ensuring safety, auditability, and reliability.

• Enhance observability, monitoring, alerting, and operational intelligence capabilities to reduce downtime and improve platform health.

• Collaborate with infrastructure, cloud, security, and engineering teams to implement long-term solutions and improve service resiliency.

• Participate in incident management, root cause analysis, postmortems, and continuous improvement initiatives.

• Develop tooling, scripts, and platform automation to minimize manual operational activities.

Required Skills & Experience

• Strong experience supporting production HPC environments, large-scale distributed compute platforms, GPU-accelerated infrastructure, or reliability-focused engineering environments.

• Deep Linux administration, troubleshooting, and performance engineering experience.

• Strong understanding of networking fundamentals, DNS, authentication, storage I/O, containers, and distributed system operations.

• Hands-on experience with observability and monitoring platforms such as:

  • Prometheus

  • Grafana

  • OpenTelemetry

  • ELK Stack

  • Datadog

  • Splunk

• Strong programming and automation experience using:

  • Python

  • Go (preferred)

  • Shell/Bash scripting

• Experience conducting complex production investigations from issue identification through root cause analysis, resolution, validation, and long-term prevention.

• Experience supporting highly available production platforms with operational ownership responsibilities.

Preferred Qualifications

• Experience with workload orchestration and scheduling platforms such as:

  • Kubernetes

  • Slurm

  • OpenShift

  • Similar cluster management solutions

• Experience supporting AI, machine learning, data science, or GPU-intensive computing environments.

• Understanding of distributed training infrastructure, checkpoint management, job scheduling, workload placement, and compute cluster operations.

• Experience working with cloud platforms such as:

  • AWS

  • Microsoft Azure

  • Google Cloud Platform (GCP)

  • Similar large-scale cloud providers

• Experience with enterprise-scale storage technologies, including:

  • NFS

  • POSIX-compliant storage

  • Lustre

  • Object Storage

  • High-performance storage systems

• Experience with Infrastructure as Code and platform automation tools:

  • Terraform

  • Helm

  • Kubernetes Operators

  • Ansible

• Experience designing and validating resilient infrastructure through testing, fault simulation, workload replay, and controlled recovery procedures.

• Exposure to AI-assisted operations, intelligent automation, or autonomous remediation workflows is beneficial.

The pay range for this role is USD $140,000- $160,000 per annum including any bonuses or variable pay. Tech Mahindra also offers benefits like medical, vision, dental, life, disability insurance and paid time off (including holidays, parental leave, and sick leave, as required by law). Ask our recruiters for more details on our Benefits package. The exact offer terms will depend on the skill level, educational qualifications, experience and location of the candidate.

AI tools may assist in the recruitment process; however, all hiring decisions are made by the recruitment team based on a comprehensive evaluation of candidates.

“Tech Mahindra is an Equal Employment Opportunity employer. We promote and support a diverse workforce at all levels of the company. All qualified applicants will receive consideration for employment without regard to race, religion, color, sex, age, national origin, or disability. All applicants will be evaluated solely on the basis of their ability, competence, and performance of the essential functions of their positions with or without reasonable accommodations. Reasonable accommodations also are available in the hiring process for applicants with disabilities. Candidates can request a reasonable accommodation by contacting the company ADA Coordinator at [email protected].”