Note: The job is a remote job and is open to candidates in USA. Lightning AI builds an end-to-end platform for developing, training, and deploying AI systems. The company is seeking a Senior Network Engineer to design, deploy, automate, and operate high-performance InfiniBand networking infrastructure for large-scale GPU clusters, while ensuring reliable performance, scalability, monitoring, and troubleshooting for AI workloads.
Responsibilities
- Design, deploy, and maintain large-scale NVIDIA InfiniBand fabrics supporting AI/ML GPU clusters
- Deploy and administer NVIDIA Unified Fabric Manager (UFM) Enterprise for monitoring, provisioning, telemetry, and fabric health
- Configure and optimize NVIDIA Quantum and Quantum-2 InfiniBand switches
- Troubleshoot fabric performance issues impacting NCCL, MPI, GPUDirect RDMA, and AI training jobs
- Implement and validate fat-tree, Dragonfly+, Clos, and spine-leaf network architectures
- Perform firmware lifecycle management for InfiniBand switches, adapters (HCAs), and UFM infrastructure
- Optimize congestion control, adaptive routing, QoS, and traffic engineering for high-performance GPU communication
- Work closely with AI platform, GPU infrastructure, storage, and systems engineering teams to deploy scalable AI Factory environments
- Automate network provisioning using Python, Ansible, Git, REST APIs, and Infrastructure-as-Code methodologies
- Monitor network health using UFM telemetry, Prometheus, Grafana, and other observability platforms
- Support high availability, maintenance windows, incident response, root cause analysis, and capacity planning
- Participate in architecture reviews and define networking standards for AI infrastructure
Skills
- 7+ years of experience in large-scale data center networking
- Experience deploying, operating, and troubleshooting InfiniBand environments
- Strong understanding of Layer 2 and Layer 3 networking, including BGP, EVPN, VXLAN, and spine-leaf architectures
- Strong Linux administration experience
- Experience with network automation using Python, Ansible, Terraform, or similar tools
- Experience with network troubleshooting, observability, and telemetry tooling
- Excellent troubleshooting, documentation, and communication skills
- Experience with Netris, Terraform, or similar infrastructure tooling
- Experience with multi-region backbone design
- Exposure to bare-metal provisioning systems
- Experience operating HPC, GPU-dense, or other large-scale infrastructure environments
- Experience working in high-growth infrastructure startups
- Fully remote work considered for candidates outside of our office hub locations
Benefits
- A discretionary bonus
- Meaningful equity / RSUs that give employees a stake in the company's long-term success.
- Medical, dental, and vision coverage for employees and eligible dependents.
- 401(k) matching (U.S.) and pension contributions (U.K.).
- Unlimited PTO, company holidays, and floating holidays to support work-life balance.
- Two weeks of company closure each winter to disconnect and recharge.
- Paid parental and family leave.
- Annual learning and development allowance to support your professional growth.
- Wellness and work-from-home stipends to support your physical and mental well-being.
- Four weeks of paid sabbatical leave after four years of service.
- Flexible schedules and a hybrid work model for our office-based teams.
- Complimentary meals at our office hubs.
Company Overview
Company H1B Sponsorship