Network Performance Engineer - GPU/HPC

Ishan Technologies is looking for an experienced Network Performance Engineer to build, operate, optimise and maintain the high-performance network infrastructure supporting Saksham AI Services' 1000+ GPU cluster. The role will focus on backend GPU networking, high-speed Ethernet/RoCE and InfiniBand fabrics, front-end connectivity, storage and management networks. The ideal candidate should have strong hands-on experience in data centre networking, network performance optimisation, RDMA technologies, congestion control and collective communication benchmarking. The selected candidate will also act as an L2/L3 escalation point for network and performance-related issues raised by Ishan Technologies' 24x7 NOC team.

Key Responsibilities:

• Install, configure and operate high-speed backend GPU networks using RoCEv2/Ethernet or InfiniBand.
• Manage front-end, storage and out-of-band management networks.
• Configure and maintain switches, network interface cards (NICs), transceivers and fibre connectivity.
• Work on rail-optimised leaf-spine network architecture.
• Maintain network topology, cable mapping and infrastructure documentation.
• Configure and optimise BGP, EVPN-VXLAN and leaf-spine network architecture.
• Implement congestion control mechanisms, including PFC, ECN/DCQCN, adaptive routing and QoS.
• Monitor network bandwidth, latency, packet loss, link flaps and congestion.
• Troubleshoot GPU-to-GPU communication and collective performance issues.
• Ensure full-bandwidth network performance across different workload and cluster sizes.
• Manage RoCEv2 and InfiniBand fabrics for GPU/HPC environments.
• Work with NVIDIA UFM and InfiniBand management tools.
• Configure and troubleshoot RDMA drivers, SR-IOV and GPUDirect RDMA.
• Work with NVIDIA ConnectX and AMD Pensando Pollara networking technologies.
• Maintain VLAN/VRF segmentation, PKeys and fabric-level tenant isolation.
• Perform collective communication benchmarking using RCCL, NCCL, MPI and related testing tools.
• Conduct RDMA performance testing using perftest, ib_write_bw and ib_read_bw.
• Work with nccl-tests and rccl-tests to validate network performance.
• Understand how data, tensor and expert parallelism impact network utilisation.
• Investigate slow-running AI jobs, communication bottlenecks and straggler GPUs.
• Collaborate with compute and MLOps teams to optimise AI workload performance.
• Develop network monitoring dashboards using Prometheus and Grafana.
• Monitor switch health, link errors, optics performance and network congestion.
• Automate network configurations, health checks and compliance activities using Python and Ansible.
• Maintain network performance baselines and conduct acceptance testing after infrastructure changes.
• Support Kubernetes RDMA networking using Multus, SR-IOV device plugins and network operators.
• Handle L2/L3 network and performance escalations from the 24x7 NOC team.
• Lead network incident investigations and root cause analysis.
• Coordinate with customers, solution architects, vendors and internal engineering teams.
• Plan and execute switch, NIC and optics firmware upgrades under change management.
• Maintain network runbooks, technical documentation and operational standards.
• Support internet edge connectivity, peering and private customer network links.

Required Technical Skills:

• Strong hands-on experience in Data Centre Networking.
• Experience with RoCEv2 or InfiniBand networking.
• Strong understanding of BGP, EVPN-VXLAN and leaf-spine architecture.
• Knowledge of PFC, ECN/DCQCN, QoS and adaptive routing.
• Experience with RDMA, GPUDirect RDMA and SR-IOV.
• Hands-on experience with NVIDIA ConnectX, Spectrum-X or AMD Pensando Pollara.
• Knowledge of RCCL, NCCL, MPI and collective communication benchmarking.
• Experience with NVIDIA UFM and InfiniBand management.
• Understanding of VLAN, VRF, PKeys and network tenant isolation.
• Experience with 400G/800G networking, optics and fibre connectivity.
• Working knowledge of Python, Ansible and network automation.
• Experience with Prometheus, Grafana, SNMP and streaming telemetry.

Educational Qualifications:

• Bachelor's degree in Engineering or Technology.
• Preferred specialisations: Computer Science, Information Technology, Electronics, Telecommunications or related disciplines.
• Relevant data centre networking certifications will be an added advantage.

Preferred Certifications:

• Cisco CCNP/CCIE Data Center.
• Arista ACE.
• Juniper JNCIP-DC.
• NVIDIA Networking (InfiniBand/Spectrum-X) certifications.

Experience Requirements:

• 6-10 years of overall professional experience.
• Minimum 4+ years of relevant experience in Data Centre Networking for GPU, HPC or hyperscale environments.
• Hands-on experience with RoCEv2 or InfiniBand fabrics is essential.
• Experience with large-scale leaf-spine networks, cabling and optics is essential.
• Experience in RCCL/NCCL or MPI performance optimisation is preferred.
• Exposure to AMD Pensando, NVIDIA Spectrum-X or ConnectX networking is preferred.

Key Competencies:

• Strong troubleshooting and analytical skills.
• Hands-on approach to network engineering and performance optimisation.
• Ability to handle critical production incidents under pressure.
• Strong ownership and root cause analysis capabilities.
• Effective communication with customers, NOC teams and technical stakeholders.
• Willingness to participate in a 24x7 on-call support rotation.
• Ability to learn and adapt to evolving AI and GPU networking technologies.

Key Success Outcomes:

• Stable, high-performance GPU networking infrastructure.
• Full-bandwidth collective communication across cluster workloads.
• Verified network isolation between multiple customers and tenants.
• Improved network performance and reduced communication bottlenecks.
• Faster resolution of network incidents and performance escalations.
• Standardised, automated and well-documented network operations.

Skills

Data Center Networking, GPUDirect RDMA, InfiniBand, RDMA, RoCEv2, SR-IOV