Note: The job is a remote job and is open to candidates in USA. Bitdeer is a technology company building Bitcoin mining solutions and AI computational infrastructure, including an AI-operated GPU cloud. The GPU DC East-West Network SRE Expert will operate large-scale InfiniBand and RoCEv2 fabrics, tune GPU communication and network performance, diagnose faults, manage firmware, and provide telemetry to AIOps systems for predictive maintenance and automated remediation.
Responsibilities
- InfiniBand fabrics: fat-tree, rail-optimized, and dragonfly topologies for GPU clusters of 100–10,000 GPUs
- RoCEv2 networks across Nvidia, Arista, and Cisco platforms for RDMA workloads
- UFM (Unified Fabric Manager) for IB fabric monitoring, diagnostics, and subnet management
- IB and RoCE performance monitoring and tuning: adaptive routing, congestion control (DCQCN/ECN), traffic isolation
- NCCL communication tuning: topology detection, ring/tree algorithm selection, GDR configuration
- Firmware lifecycle across IB switches and HCAs
- Fault diagnosis: link flaps, symbol errors, packet drops, routing anomalies, credit stalls
- Coordination with Nvidia/Mellanox support for escalations, bugs, and RMA
- Wire IB/RoCE telemetry (ibdiagnet, perfquery, ibstat, PortRcvErrors, PortXmitDiscards, adaptive-routing state) into the platform's collection pipeline
- Partner with the platform team to define the Link and Straggler predictors: what a "bad optic 30 minutes from failure" looks like in the counters
- Convert every incident into a labeled example the fault-prediction engine can learn from — and every routine mitigation into a workflow the remediation actuator can run
Skills
- 5+ years in data center networking, with at least 3 years focused on InfiniBand or RoCE fabrics
- Hands-on experience deploying and operating Nvidia/Mellanox InfiniBand switches at scale
- Strong understanding of IB subnet management, partitioning, and QoS
- Experience with RoCEv2 deployment including PFC, ECN, DCQCN configuration
- Proficiency with UFM or equivalent IB fabric management tools
- Knowledge of 400G/800G optics, cabling standards, and structured cabling best practices
- Experience diagnosing IB/RoCE network issues using ibdiagnet, perfquery, ibstat, and similar tools
- Understanding of NCCL and how GPU communication maps to network topology
- Instinct for telemetry-driven ops — you've either built dashboards/alerts on RDMA counters at scale, or you can articulate the feature set a fabric-health model would need
- Runbook-as-code mindset — the diagnostics you run today should become automation next quarter
Company Overview