Note: The job is a remote job and is open to candidates in USA. GMI Cloud is an AI-native infrastructure company delivering GPU compute, inference services, and cloud infrastructure for AI workloads. The Network Architect will lead the design, deployment, operation, troubleshooting, and optimization of high-performance InfiniBand and RoCE network fabrics supporting large-scale GPU clusters and global data centers.
Responsibilities
- Architect and design high-performance, highly available network fabrics for GPU clusters (InfiniBand or RoCE), including topology, cabling, switch configuration, subnet/partitioning, and redundancy
- Plan, design, and implement network infrastructure for GMI global data center, including WAN, Core Network, Data Center Network, Firewalls, Load Balancers, DNS, VPN, etc
- Build high-performance network solutions to support AI/ML workloads, encompassing Compute, Storage, Inband, Management, and Out-of-Band (OOB) network fabric using Infiniband and Ethernet RDMA RoCEv2 technologies
- Lead implementation and lifecycle management of network hardware and firmware (HBAs, NICs, RDMA NICs, IB switches, Ethernet switches, etc)
- Configure and fine-tune RDMA, congestion control, QoS, ECN, Priority Flow Control (PFC), and Flow Control to optimize MPI, NCCL, and other GPU-communication patterns
- Hands-on day-to-day operations: provisioning, configuration changes, patching, firmware upgrades, and logging/monitoring of network health
- Rapidly troubleshoot production incidents (fabric errors, performance degradations, link flaps, MTU/flow issues, packet drops), drive root cause analysis, and implement corrective actions
- Develop and maintain automation, scripts, and runbooks for deployment, configuration management, diagnostics, and capacity planning (Ansible, REST APIs)
- Work with compute, storage, platform and SRE engineers to validate end-to-end performance, run benchmarks, and recommend architecture improvements; collaborate with cross-functional teams and stakeholders to understand networking requirements
- Identify suitable network providers, vendors, and solutions to meet organizational needs
- Define and enforce network security, segmentation, and access policies for GPU clusters and management networks
- Maintain documentation: network diagrams, cabling maps, configuration baselines, and operational procedures
- Mentor junior network engineers and participate in on-call rotations for fabric support
- Regional/international travel to GMI data center locations
Skills
- Bachelor's degree in Computer Science or related field
- 10+ years of networking experience, with 3+ years specifically designing and operating InfiniBand/RoCE-based fabrics for large-scale GPU AI clusters (hundreds to thousands of GPUs)
- Deep, hands-on experience with InfiniBand (HDR/XDR) fabrics: subnet manager (UFM, OpenSM), IB routing, partition keys (P\_Key), etc
- Strong expertise with RDMA, RoCE, and Ethernet lossless frameworks: configuring PFC, ECN, DCB, and addressing head-of-line/blocking issues
- Proven troubleshooting experience: packet capture analysis, IB/Ethernet counters, link diagnostics, congestion root-cause, firmware and driver interactions
- Familiarity with GPU communication libraries and patterns: NCCL, MPI, GPUDirect RDMA, and how network settings affect scaling and latency
- Familiarity with storage protocols used in AI environments (NVMe-oF, NFS over RDMA, GPUDirect Storage)
- Hands-on with network hardware from major vendors (Mellanox/NVIDIA Spectrum, Cisco, etc)
- Solid understanding of TCP/IP, VLANs, L3 routing, BGP/OSPF basics, MTU/Jumbo frames, and L2/L3 troubleshooting tools
- Experience with monitoring and observability tools (Prometheus, Grafana, SNMP, ELK, vendor telemetry)
- Experience with network automation and scripting (Ansible, REST API), and configuration management
- Familiar with various routers, switches, firewalls, load balancer, DNS, VPN configuration implementation
- Strong knowledge in network security, DDOS, IDS, etc
- Familiar with optical networking, including fibers, transceivers and optics troubleshooting
- Strong troubleshooting mindset: methodical, data-driven, and calm under production pressure
- Proactive about automation, reliability, and continuous improvement
- Collaborative team player, able to work cross-functionally and to translate technical trade-offs for stakeholders with strong communication skills
- Candidates holding network certifications (e.g. CCNA, CCNP, Nvidia) will be strongly preferred
- Candidates with proven experience in the AI/ML GPU networking environment will be highly considered
- Bilingual English and Chinese will be strongly preferred
Company Overview
Company H1B Sponsorship