Fractional consulting role — remote
DevOpt Labs is looking for a senior practitioner who has personally run, tuned, and diagnosed large-scale AI pretraining jobs on modern multi-GPU systems.
This is not a software-development position. We already have strong algorithm, mathematics, CUDA, optimizer, and systems-programming capability. What we need is someone with deep practical experience in the operational side of large-scale training: designing runs, choosing the right training stack, understanding expected throughput, diagnosing inefficient jobs, establishing fair baselines, and making expensive experiments reproducible and scientifically defensible.
We are preparing a series of optimizer-comparison studies at approximately 1B–14B parameter scale and want someone who can help us avoid wasting time and GPU budget on training-stack inefficiencies or experimental-design mistakes.
What you would help us with
The initial engagement would focus on auditing our current training environment and establishing a reliable methodology for upcoming large-scale experiments.
You would work directly with our technical team to:
• Review our current multi-H100 pretraining and continued-pretraining workflows.
• Determine whether observed training throughput is reasonable for the model size, sequence length, batch configuration, and hardware.
• Diagnose why a run may be 2–3× slower than expected.
• Establish realistic tokens/sec, MFU, GPU utilization, wall-clock time, and cost expectations at 1B, 2B, 4B and larger scales.
• Recommend the most appropriate training stack for rigorous optimizer benchmarking.
• Review PyTorch, Axolotl, Megatron-LM, NeMo, TorchTitan, FSDP/FSDP2, DeepSpeed, Nanotron, or comparable approaches where appropriate.
• Optimize or advise on attention implementation, mixed precision, batching, gradient accumulation, data loading, communication, checkpointing, and distributed execution.
• Help design scientifically fair AdamW-versus-new-optimizer comparisons.
• Review learning-rate schedules, Chinchilla token budgets, checkpoint methodology, stopping criteria, and seed strategy.
• Set up or validate EleutherAI lm-evaluation-harness evaluation.
• Help us distinguish optimizer behavior from training-infrastructure behavior.
• Document the resulting methodology so our internal team can operate it independently.
The person we are looking for
You must have personally operated substantial from-scratch pretraining jobs.
We are not looking for someone whose experience consists mainly of fine-tuning models, writing ML applications, building APIs, or implementing generic PyTorch code.
Strong candidates will have direct experience with several of the following:
• Billion-parameter or larger transformer pretraining
• Multi-GPU H100/A100 systems
• Long-running, expensive training experiments
• PyTorch Distributed, DDP, FSDP/FSDP2
• Megatron-LM, NeMo, TorchTitan, DeepSpeed, Nanotron, or similar training systems
• NCCL and multi-GPU performance
• FlashAttention / SDPA
• BF16 training
• GPU profiling and MFU analysis
• Dataset streaming and high-throughput input pipelines
• Checkpoint/restart strategy
• Hyperparameter and learning-rate sweeps
• AdamW baseline tuning
• Pretraining loss-curve interpretation
• Reproducible optimizer benchmarking
• Cloud GPU cost/performance analysis
The ideal person has experienced the practical realities of multi-day or multi-week training runs and knows how to tell whether a problem is caused by the optimizer, the schedule, the training harness, the hardware, or the data pipeline. Hyperscaler experience preferred.
Initial engagement
We expect the initial engagement to be approximately 40–60 hours over 1-2 months.
The first objective is simple:
Audit our current training setup, determine why our measured throughput differs from highly optimized reference implementations, and establish a credible, reproducible training methodology for upcoming 1B–14B optimizer studies.
If the engagement works well, we would like to retain the consultant on a fractional basis during future large-scale experiments.
Compensation
Expected range:
$175–$300/hour, depending on relevant experience.
We are willing to pay more for someone with unusually strong hands-on experience operating and optimizing large-scale pretraining workloads.
To apply
Please answer these questions briefly in your response:
• What is the largest model you have personally pretrained from scratch?
• Approximately how many tokens were used?
• What GPU type and GPU count were involved?
• What training framework did you use?
• Were you personally responsible for training performance and run execution?
• Have you diagnosed a training job that was materially slower than expected? What was the cause?
• What metrics do you normally use to determine whether an H100 training job is efficient?
• Have you designed or reviewed optimizer-comparison experiments?
• What is your hourly rate?
• What is your availability over the next eight weeks?
Please do not apply with only a generic software-engineering or machine-learning résumé. We specifically need someone who has personally operated large-scale pre and post training experiments in a rigorous scientific environment. USA, CAN or EU based only.