Note: The job is a remote job and is open to candidates in USA. Global Data Network is seeking a Director of Infrastructure Operations to lead the operational management of production GPU infrastructure. The role owns live GPU clusters, 24x7 support, incident and change management, SLA performance, vendor coordination, handover readiness, and the scaling of the infrastructure operations organization.
Responsibilities
- Production GPU clusters running customer training and inference workloads
- The 24x7 support model: rota design, escalation paths, on-call structure, sev1 command
- Incident, problem and change discipline across a growing fleet
- SLA performance and the reporting that evidences it - uptime, response, restoration, job success
- Handover standards from deployment engineering, including the authority to refuse a cluster that is not operationally ready
- OEM and vendor coordination on RMA, break-fix and maintenance windows
- Building and scaling the operations organisation itself
Skills
- You need to have operated these, not specified them, not governed them, not attended vendor briefings about them, operated them :
- Fabric - InfiniBand (NDR/HDR) and RoCEv2 in production. Rail-optimised topologies. What a degraded link does to a running collective, and what you do about it
- Collectives - NCCL behaviour at scale, all-reduce performance, when a slow node is poisoning a job
- Scheduling - Slurm and/or Kubernetes for GPU workloads. Gang scheduling, topology-aware placement, queue and preemption behaviour under contention
- Storage - parallel and high-throughput platforms in production: VAST, DDN, WEKA, Lustre
- Hardware - rack-scale high-density AI systems, NVL72-class and equivalent. GPU fault handling, thermals, degraded node management, checkpoint and restart economics
- Telemetry - DCGM and equivalent. Per-job utilisation, MFU, fabric counters, and what you do when the numbers move
- You are running, or have recently run, production GPU infrastructure for external paying customers
- You have carried the pager. You have run the bridge call. You have had a customer on the other end of an SLA breach
- Have you held operational accountability for infrastructure running live workloads for external, paying customers under contractual SLAs?
- Have you personally owned a 24x7 support model for production clusters - rota, escalation, incident command?
Company Overview