Job Description:
• Build the training system that takes Protocol Learning from an 8B run to frontier scale
• Implement and optimize model-parallel distributed pretraining across heterogeneous GPUs
• Implement data, pipeline, and tensor parallelism over low-bandwidth, high-latency links
• Reduce communication overhead while maintaining model convergence
• Build elasticity and fault tolerance so runs survive node churn
• Implement robust checkpointing, state synchronization, and recovery as participants join and leave
• Build monitoring to show throughput, bottlenecks, and model quality across hundreds of devices
• Train and serve large models on heterogeneous hardware in physically different regions over ordinary internet connections
Requirements:
• Hands-on distributed training required: trained models across many devices in PyTorch using FSDP, DeepSpeed, Megatron, or a custom implementation
• Understanding of data, tensor, and pipeline parallelism
• Production-quality Python
• Concurrency and failure handling experience
• Profiling before optimizing
• Evidence of execution through shipped systems, research code, open-source work, or serious personal projects
• Mission alignment with Protocol Learning
• Professional-level English proficiency, written and spoken
• Nice to have: experience training or serving large language models such as Nemotron, Qwen, or OLMo
• Nice to have: P2P networking and NAT traversal experience
• Nice to have: post-training and reinforcement learning experience
• Nice to have: inference and serving systems experience
• Nice to have: experience at proprietary, open-weight, or open-source AI labs
Benefits:
• Significant ownership/equity for key technical contributors
• High base salary
• Flexible work environment
• Full visa sponsorship and relocation support to either Australia or the US for exceptional candidates
• Opportunity to work on open, unsolved research problems
• Remote-first culture
• Globally distributed team