Note: The job is a remote job and is open to candidates in USA. 5C Data Centers is building digital infrastructure that supports hyperscalers, artificial intelligence innovation, and high-performance computing across North America. The Systems Administrator Lead will provide second-shift support for large-scale HPC and AI infrastructure, troubleshooting Linux systems, GPU servers, networking, storage, firmware, and hardware while leading incident escalation and mentoring administrators and engineers.
Responsibilities
- Administer, maintain, and troubleshoot Linux-based HPC and AI compute environments, including large-scale GPU clusters built on NVIDIA HGX, DGX, or equivalent accelerated computing platforms
- Diagnose hardware, OS, driver, firmware, networking, and storage-related failures, and perform root-cause analysis to develop corrective actions for recurring issues
- Serve as a senior escalation point for complex incidents affecting production compute infrastructure, troubleshooting degraded or unstable compute/GPU nodes to restore service
- Support infrastructure across heterogeneous environments - bare-metal, virtualized, containerized, and cloud-hosted - and maintain operational documentation, runbooks, and shift-handoff notes
- Install, configure, validate, and troubleshoot NVIDIA GPU drivers, CUDA components, firmware, and supporting system software using tools such as Nvidia Smi, DCGM, and NVIDIA Fabric Manager
- Investigate GPU Xid errors, NVLink/NVSwitch faults, PCIe errors, ECC events, GPU resets, and thermal or power-related issues
- Diagnose communication and performance issues involving NCCL, GPUDirect RDMA, PCIe topology, NUMA placement, and multi-GPU systems, validating GPU-to-GPU, GPU-to-network, and GPU-to-storage communication after repairs
- Coordinate replacement and RMA activities for GPUs, GPU trays, baseboards, NVSwitch components, system boards, and NICs
- Administer enterprise Linux distributions across the production fleet, troubleshooting boot failures, kernel issues, systemd services, filesystems, and OS performance
- Analyze system and kernel logs (journalctl, dmesg, lspci, dmidecode, ipmitool, ethtool, ss, sar, perf) and manage kernel modules, device drivers, DKMS packages, and OS updates
- Support Linux networking, storage mounts, authentication, and security controls, and investigate OS crashes, kernel panics, and out-of-memory conditions
- Support enterprise server platforms from vendors such as Dell, HPE, ASUS, Supermicro, and NVIDIA, troubleshooting processors, memory, PCIe devices, NICs, storage controllers, and other hardware components
- Manage BIOS, BMC, CPLD, NIC, GPU, switch, and drive firmware, and perform remote console troubleshooting, power cycling, and log bundle generation
- Partner with on-site Data Center technicians on component replacement, cabling validation, break-fix activity, and post-repair testing
- Troubleshoot Ethernet, InfiniBand, and RoCE connectivity within HPC and AI environments, including link state, VLAN/bonding issues, MTU mismatches, packet loss, and RDMA/GPUDirect RDMA connectivity
- Collaborate with Deployment Engineering to troubleshoot fabric instability, link degradation, congestion, and node-level connectivity issues
- Troubleshoot server connectivity to parallel, distributed, and enterprise storage platforms such as NFS, Lustre, BeeGFS, GPFS/Spectrum Scale, and Ceph, diagnosing mount failures, throughput degradation, and capacity issues
- Validate storage network paths and coordinate with storage and networking teams during service-impacting events
- Lead or support incident response for critical infrastructure events, participating in technical bridges and providing clear status updates during outages
- Create detailed timelines, root-cause analyses, and corrective-action plans, maintaining accurate records in Jira Service Management or similar ITSM platforms
- Mentor junior and mid-level Systems Administrators and HPC Engineers, providing technical guidance during complex troubleshooting and infrastructure changes
- Promote consistent troubleshooting methods, documentation standards, and operational discipline across shift handoffs
Skills
- Bachelor's degree in Computer Science, Engineering, Information Technology, or related field
- 7+ years of experience in Linux systems administration, HPC/AI infrastructure or data center operations, including hands-on troubleshooting in production environments
- Hands-on experience supporting enterprise server hardware, bare-metal compute systems, and large-scale NVIDIA GPU infrastructure or accelerated computing platforms
- Strong understanding of server hardware, firmware, BIOS, BMC, PCIe, NUMA, memory, storage, and network interfaces, with experience diagnosing failures using OS logs, BMC logs, and vendor diagnostics
- Experience with NVIDIA drivers, CUDA compatibility, and GPU-management tools such as DCGM, Fabric Manager, or nvidia-smi
- Working knowledge of high-performance networking (InfiniBand, RDMA, RoCE, NVIDIA/Mellanox adapters) and out-of-band management tools (Redfish, IPMI, iDRAC, or iLO)
- Proficiency in one or more automation/scripting languages (Python, Bash, or Ansible) and familiarity with observability platforms such as Prometheus, Grafana, or Datadog
- Proven ability to troubleshoot across hardware, firmware, OS, network, storage, and cluster-management layers, backed by strong documentation and communication skills
- Willingness to work a fixed, recurring shift and participate in scheduled on-call escalation
Benefits
- Medical insurance
- Vision insurance
- Dental insurance
- 401(k)
- Pension plan
- Remote work arrangement
Company Overview