Note: The job is a remote job and is open to candidates in USA. Cantina Labs is a social AI company developing real-time models for expression, personality, and realism. The Machine Learning Engineer will build state-of-the-art speech and audio generation systems end-to-end, with a focus on joint audio-video modeling, multimodal generation, evaluation, efficiency, and production deployment. The role partners across research, video, data, and infrastructure to develop safe, reliable, and scalable AI systems.
Responsibilities
- Audio Representations: Design, train, and improve the audio VAEs, neural codecs, and vocoders our generative models sit on top of latent design, reconstruction and perceptual objectives, compression-vs-fidelity tradeoffs
- Model Building: Architect, implement, pre-train, fine-tune, and post-train/alignment (e.g., GRPO/DPO) diffusion and flow-matching transformers for large-scale audio and video generation
- Joint Audio-Video Modeling: Design the audio conditioning and cross-modal alignment inside joint AV models, audio latents alongside video latents, reference-audio and multi-speaker conditioning, multi shot generation audio/video modeling
- Experimental Design: Design, run, and analyze scientific experiments to advance our understanding of the models
- Data Ownership: Define data requirements and collaborate on acquisition, curation, AV-sync and quality filtering, annotation quality, and synthetic data strategies for paired audio-video and speech corpora
- Rigorous Evaluation: Design automated objective/subjective evaluations audio fidelity and intelligibility metrics, AV-sync, listening and viewing tests, robustness & bias checks, and red-team studies
- Inference Efficiency: Drive distillation, step-count reduction, quantization, and kernel/memory optimization to meet interactive latency and cost targets
- Pipeline Delivery: Harden the training → evaluation → inference pipeline; profile latency, memory, and cost; and meet production SLAs with robust monitoring and rollback
- GPU Scaling: Partner with infrastructure to run distributed training/inference on cloud fleets and productionize models with reliability and observability
- Project Leadership: Independently lead small research projects while collaborating on larger team initiatives, including cross-team work with video generation
- Tool Development: Develop and improve dev tooling to enhance team productivity
- Safety & Responsibility: Contribute to safety/consent guardrails, watermarking, and misuse/abuse mitigation for responsible voice and likeness technology
Skills
- Exceptional research/development experience with large-scale audio models (>8B parameters, >500k hours of data)
- Deep hands-on experience with diffusion and/or flow-matching transformers, including practical knowledge of samplers, schedules, conditioning mechanisms, and distillation
- Deep hands-on experience training audio VAEs, neural audio codecs, and vocoders latent/tokenizer design, reconstruction and perceptual objectives, adversarial training
- Strong experience with multi-node, multi-GPU distributed training (FSDP/DeepSpeed or equivalent)
- Strong software engineering skills with a proven track record of building complex systems
- Strong with PyTorch and performance work (profiling, CUDA/Triton/C++ as needed) and writing reliable production-quality code
- Shipped large-scale speech/audio or multimodal generative models to production
- Background in working with large-scale ML data, and the ability to iterate on data and triangulate quality using both subjective and objective signals
- Experience with voice cloning, speech control/steerability, or expressive speech generation
- Notable publications and/or open-source contributions in speech/audio/ML
- Experience with multimodal audio-video modeling: joint AV generation of multi-shot, multi-speaker scenes with dialogue, music, and sound design generated jointly with video, and the cross-modal alignment that keeps them in sync
- Experience with video generation: video diffusion/flow-matching transformers, video VAEs, conditioned and multi-shot generation, building data pipelines for video models
- Streaming or real-time generation, causal distillation (e.g., Self Forcing / Self Forcing++)
Benefits
- Competitive salary and generous company equity
- Medical, dental, and vision insurance – 99.99% of premiums covered by Cantina
- 42 days of paid time off, including 15 PTO days, 10 sick days, 15 company holidays, and 2 floating holidays
- Generous parental leave & fertility support
- 401(k) retirement savings plan
- Lifestyle spending account – $500/month to use however you’d like
- Complimentary lunch and snacks for in-office employees
- One Medical membership
Company Overview
Company H1B Sponsorship