Note: The job is a remote job and is open to candidates in USA. Egen is hiring an ML Platform Engineer to work embedded with a healthcare and life-sciences AI client training foundation models on a large GKE-based GPU cluster. The role focuses on supporting and tuning the training platform, running and debugging distributed training, benchmarking and evaluating performance, and partnering with infrastructure teams and scientists.
Responsibilities
- Work with the client's infrastructure team on the GKE training platform (GPU node pools, job queueing and scheduling, quota and capacity) from the training workload's side
- Diagnose and fix platform failures at the root (for example a controller that keeps running out of memory) rather than restarting them
- Tune multi-node GPU networking (RDMA/GPUDirect, NCCL) for training throughput
- Run, scale and debug multi-node distributed training with PyTorch (FSDP, DeepSpeed, Megatron or NeMo-style) and Ray Train
- Profile jobs and remove bottlenecks across compute, memory, input pipelines and interconnect
- Design and run performance benchmarks for the cluster and training stack: throughput, scaling efficiency, cost per run
- Build and maintain model evaluation harnesses so changes to the platform or training recipes are measured, not assumed
- Instrument training and the platform with telemetry and monitoring, and keep end-to-end pipeline tests running against what users actually run
- Treat the platform as your own: proactively find issues, plug gaps and propose solutions before anyone asks, keep the team informed of status and risks, and look after the scientists who use it
- Explain and defend your design decisions to client engineers and scientists in detail
- Stay effective while the client's priorities and team structure change
Skills
- 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training
- Strong Python software engineering: production-quality code, tests and code review
- GKE and GPU scheduling: Kubernetes on Google Cloud with GPU node pools, job queueing (Kueue or similar), and capacity and quota management
- Distributed PyTorch and Ray Train: has personally run and debugged multi-node training jobs
- RDMA and GPU networking: GPUDirect/RDMA, NCCL tuning, multi-node GB200 (or H200) topology
- Benchmarking and evals: has designed performance benchmarks and model evaluations, and used them to make decisions
- Deep enough understanding to explain how and why a system works, without leaning on AI coding tools to carry the reasoning
- A track record of visible ownership: present, communicative, and accountable on a demanding team
- Observability and telemetry for training platforms: metrics, tracing, alerting, end-to-end pipeline testing
- Healthcare or life-sciences data (genomics, clinical, biomedical), and working under HIPAA
Benefits
- Comprehensive Health Insurance
- Paid Leave (Vacation/PTO)
- Paid Holidays
- Sick Leave
- Parental Leave
- Bereavement Leave
- 401 (k) Employer Match
- Employee Referral Bonuses
Company Overview
Company H1B Sponsorship