GPU Cluster Reliability Engineer

Ishan Technologies is looking for an experienced Cluster Reliability Engineer to manage and maintain the compute infrastructure of Saksham AI Services' 1000+ GPU cluster.

The role involves managing GPU servers, Linux infrastructure, Slurm and Kubernetes environments, cluster reliability, automation and MLOps operations. The selected candidate will play a key role in ensuring high availability, performance, scalability and reliability of AI training and inference workloads.

The position will also serve as an L2/L3 technical escalation point, working closely with the 24x7 NOC team, platform engineers, customers and technology partners.

Key Responsibilities:

• GPU Infrastructure & Server Management
• Receive, rack, configure and maintain GPU servers.
• Manage BIOS, firmware, Linux OS images, GPU drivers and software libraries.
• Perform GPU server burn-in, hardware diagnostics and acceptance testing.
• Maintain consistent server configurations and manage firmware and security updates.
• Coordinate with OEM vendors for hardware faults, replacements and RMAs.
• Cluster Management - Slurm & Kubernetes
• Administer and maintain Slurm and Kubernetes environments for AI training and inference workloads.
• Manage Slurm controllers, partitions, QoS, fairshare and resource allocation.
• Configure Kubernetes GPU Operator, device plugins, Helm and related operators.
• Manage cluster upgrades, patching, tenant onboarding and scheduling.
• Ensure stable and efficient multi-tenant GPU cluster operations.
• Cluster Reliability & Monitoring
• Develop and maintain GPU and node health checks.
• Monitor GPU errors, ECC events, RAS events and hardware failures.
• Implement automated fault detection and remediation mechanisms.
• Develop monitoring dashboards and alerts using Prometheus, Grafana, Loki or ELK.
• Maintain cluster availability in line with the 99.9% uptime commitment.
• Maintain operational logs and monitoring records in accordance with applicable compliance requirements.
• MLOps & AI Workload Support
• Build and maintain optimized framework images for PyTorch, JAX and vLLM.
• Develop ready-to-use job templates for AI training and inference.
• Conduct workload benchmarks and GPU performance testing.
• Support distributed AI workloads, including data, tensor and pipeline parallelism.
• Provide technical assistance to customers and solution architects on workload performance and tooling.
• Incident Management & Automation
• Handle L2/L3 technical escalations from the 24x7 NOC team.
• Lead incident investigations, root cause analysis and post-incident reviews.
• Develop automation scripts using Python and Bash.
• Maintain operational runbooks and auto-remediation procedures.
• Track capacity, GPU utilisation, goodput and service-level performance.
• Identify recurring issues and implement long-term corrective actions.

Required Technical Skills:

• Strong hands-on experience in Linux system administration.
• Experience managing GPU servers, GPU drivers, firmware, BIOS and kernel configurations.
• Hands-on experience with multi-GPU accelerator servers.
• Strong knowledge of Slurm administration and workload scheduling.
• Experience with Kubernetes administration for GPU workloads.
• Knowledge of GPU Operator, Helm, Kubeflow and KubeRay.
• Familiarity with GPU monitoring, ECC/RAS events and vendor diagnostic tools.
• Experience with Prometheus, Grafana, Loki or ELK.
• Working knowledge of PyTorch, JAX, vLLM and GPU collective communication libraries.
• Experience with Docker, Apptainer and container image management.
• Automation experience using Python, Bash and Ansible or Terraform.
• Familiarity with bare-metal provisioning tools such as MAAS, Ironic, Metal3 or Warewulf.

Educational Qualifications:

• Bachelor's degree in Engineering or Technology.
• Preferred specialisations: Computer Science, Information Technology, Electronics or related disciplines.
• Relevant certifications in Linux, Kubernetes or GPU computing will be an added advantage.

Preferred Certifications:

• Certified Kubernetes Administrator (CKA).
• Red Hat RHCSA/RHCE or Linux Foundation Certified System Administrator (LFCS).
• GPU vendor training or certifications.

Experience Requirements:

• 4-8 years of overall IT infrastructure or engineering experience.
• Minimum 2+ years of relevant experience in GPU/HPC cluster operations, Site Reliability Engineering (SRE) or MLOps.
• Hands-on experience with Linux GPU servers and Slurm and/or Kubernetes is essential.
• Experience supporting production environments with on-call and incident management responsibilities.
• Experience with multi-vendor data-centre GPU platforms will be an advantage.
• Exposure to distributed AI training and inference workloads is preferred.

Preferred Additional Exposure:

• NVIDIA GPU ecosystem and related management tools.
• ClusterMAX standards and CMAX audit tool.
• Public cloud platforms such as GCP or AWS.
• HPC infrastructure and large-scale AI compute environments.

Key Competencies:

• Strong troubleshooting and problem-solving abilities.
• Hands-on technical ownership and attention to detail.
• Ability to handle critical production incidents under pressure.
• Strong communication and coordination skills.
• Willingness to participate in a 24x7 on-call support rotation.
• Ability to learn and adapt to evolving GPU, HPC and AI technologies.

Skills

Kubernetes, Linux, Slurm