Principal Engineer

Job Summary

Job Title: Principal Forward Deployed Engineer, AI Factory (Infrastructure) - NVIDIA Ecosystem Business Unit

Team: Leads the infrastructure workstream of an AI Factory FDE pod; works with the Platform and AI Software Principal and the pod's FDE III engineers

Travel: travel as required to client sites and data centers; indicative 40-60%

Role Summary

HCLTech is seeking a Principal Forward Deployed Engineer to own the physical and fabric layer of client AI Factories on NVIDIA platforms. The Principal takes the solution architect's reference design to detailed design, owns site readiness and acceptance with the OEM and NVIDIA, and leads bring-up and validation of GPU clusters across the pod's engagements, handing over a scheduler-ready factory. The role also coaches FDE III and FDE II engineers and sets infrastructure standards for the bench.

Key Responsibilities

Key Responsibilities

AI Factory Build & Deployment

• Deploy the AI Factory infrastructure stack (Hopper and Blackwell class compute, NVLink and NVSwitch, InfiniBand / Spectrum-X fabrics, BlueField DPUs, storage for AI) to the architect-approved NVIDIA reference design
• Lead bare-metal bring-up and system software: firmware, drivers, OS images, BCM / Mission Control, NVLink partitioning
• Configure and tune InfiniBand / RoCE fabrics and high-performance storage to the approved design
• Run cluster validation (NCCL, HPL, burn-in), telemetry and XID troubleshooting; isolate faults across network, firmware and scheduler layers

Site Readiness & Acceptance

• Check power, cooling and rack-layout readiness against the sizing; flag gaps found on site
• Run acceptance testing with the OEM and NVIDIA and produce validation evidence
• Produce as-built documentation and operable runbooks

Automation

• Build and maintain bring-up automation (Ansible, Python, Bash) and firmware lifecycle tooling for repeatable builds

Client & Team Collaboration

• Hand over a scheduler-ready cluster to the pod's platform FDEs with agreed interfaces
• Brief client technical teams; contribute technical inputs to presales scoping and estimates
• Convert each engagement into reusable bring-up checklists and blueprints

Technical Leadership & Ownership (Principal)

• Own detailed design (L1 to L0) from the solution architect's reference architecture, sized for workload, power and cooling
• Own site readiness and acceptance with the OEM and NVIDIA; sign off cluster readiness
• Lead the infrastructure workstream across the pod's engagements; resolve the hardest cross-layer faults with NVIDIA and OEM engineering
• Coach and review FDE III and FDE II engineers; set bring-up and validation standards
• Engage client infrastructure leadership and NVIDIA counterparts

Skill Requirements

Domain

Target level

What this domain covers

AI infrastructure & systems

4 - Expert

Physical and fabric layer of the AI Factory: GPU servers and rack-scale systems, InfiniBand / RoCE networking, high-performance storage, firmware and bare-metal bring-up, cluster validation and fault isolation.

Platform, orchestration & inference

3 - Proficient

Software layer that makes the cluster usable: Kubernetes and SLURM scheduling, GPU sharing and quotas, multi-tenancy, cluster management (BCM, Run:ai), and model serving (NIM, vLLM, Triton, TensorRT-LLM) tuned for latency, throughput and cost per token.

DevOps & control-plane scripting

3 - Proficient

Automation of the factory lifecycle: infrastructure-as-code, GitOps, CI/CD, provisioning and tenant-onboarding scripts, operators and APIs, observability as code.

Data & applied AI

2 - Working

Data pipelines and use-case delivery on the factory: fine-tuning, RAG and agentic workflows, evaluation and guardrails.

Security, governance & sovereignty

3 - Proficient

Securing and governing the factory: RBAC and tenant isolation, secrets management, hardening, audit, confidential computing, data residency, sovereign and air-gapped builds.

Presales & client engagement

3 - Proficient

Presales and client-embedded delivery: technical scoping inputs, client technical sessions, TCO inputs, as-built documentation, and handover to Delivery Teams.

Other Requirements

Required Skills & Experience

• 12+ years of experience, including 5+ years deploying or operating GPU / HPC infrastructure
• Hands-on InfiniBand or RoCEv2 / RDMA configuration and tuning
• Bare-metal provisioning experience (MAAS, BCM, Redfish / BMC) and Linux at depth
• Automation skills (Ansible, Python, Bash) for repeatable bring-up
• Comfortable debugging across network, firmware and scheduler layers under client pressure
• Client-facing experience; comfortable working on-site in client data centers
• Record of designing and leading GPU cluster builds, including OEM acceptance, and of leading engineers on multi-month deployments

Preferred

• NVIDIA-Certified Associate / Professional credentials (AI infrastructure and operations)
• Experience with rack-scale systems (GB200 / GB300 NVL72) and Mission Control
• Parallel file systems or AI-tier storage (Ceph, WEKA, VAST, DDN)
• Power and liquid-cooling fundamentals for GPU data centers
• SLURM or Run:ai administration; CKA

  Key Performance Indicators

  • Cluster builds delivered on schedule and passing validation (NCCL, burn-in) at first attempt
  • Time from hardware arrival to first production workload
  • Acceptance sign-off with the OEM and NVIDIA without rework
  • Defects and escalations after handover
  • Reusable bring-up assets contributed per engagement
  • NVIDIA certifications completed against the agreed path