Note: The job is a remote job and is open to candidates in USA. Dice is hiring a Sr InfiniBand AI Network Engineer for a remote contract role supporting Artificial Intelligence and High Performance Computing clusters. The role focuses on deploying and validating InfiniBand fabrics, troubleshooting transport and fabric issues, analyzing performance degradation, and documenting remediation workflows.
Responsibilities
- Deploy and validate InfiniBand fabrics supporting distributed Artificial Intelligence training and High Performance Computing workloads
- Troubleshoot issues involving fabric discovery, subnet management, link state, routing, partitioning, congestion, credit starvation, error counters, and host channel adapter behavior
- Diagnose job failures and performance degradation related to collective communication, NCCL transport selection, RDMA pathing, and fabric imbalance
- Validate switch, host channel adapter, firmware, and cable consistency during cluster deployment and expansion activities
- Utilize low level fabric tools to isolate bad links, unstable ports, unhealthy endpoints, topology mismatches, or subnet manager instability
- Partner with Linux, deployment, and Artificial Intelligence validation teams to drive root cause analysis from application symptoms to fabric source
- Define and execute preflight and post change validation workflows for InfiniBand cluster readiness
- Document recurring fault patterns and create remediation playbooks for operational teams
Skills
- 7 or more years of experience in High Performance Computing or advanced networking environments, including direct InfiniBand operations experience
- Strong working knowledge of InfiniBand architecture, subnet management, link training, routing, partitions, congestion behavior, and performance diagnostics
- Experience with RDMA, NCCL related network dependencies, and multi node Artificial Intelligence workload sensitivity to transport issues
- Strong troubleshooting skills using fabric health, counter, topology, and endpoint analysis tools
- Experience with firmware and driver alignment across host channel adapters, switches, and Linux hosts
- Ability to triage complex issues spanning host configuration, fabric state, and application communication behavior
- Strong written and verbal communication skills with excellent documentation capabilities
- Experience supporting DGX, GPU SuperPOD, or equivalent Artificial Intelligence cluster environments
- Familiarity with Unified Fabric Manager (UFM), telemetry pipelines, and automated fabric validation
- Experience correlating InfiniBand anomalies with Artificial Intelligence training performance outcomes
Benefits
- This is a 12 month remote contract opportunity within the United States.
Company Overview