Note: The job is a remote job and is open to candidates in USA. Deepgram is the leading platform underpinning the emerging trillion-dollar Voice AI economy, providing real-time APIs for speech-to-text (STT), text-to-speech (TTS), and building production-grade voice agents at scale. They are looking for a Senior Software Engineer - Model Evaluation & AI Systems to validate the quality of their speech, audio, and multilingual models, ensuring they meet performance targets before reaching customers.
Responsibilities
- Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems
- Design, build, and maintain automated evaluation pipelines across batch and streaming (e.g. WER, runaway/hallucination detection, latency and time-to-first-byte), with a focus on correctness, reproducibility, and ease of adoption
- Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines — running against production models and, where needed, large GPU clusters
- Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates
- Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers
- Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure
- Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions
- Integrate evaluation and quality gates into CI/CD so quality is verified continuously, not manually
- Help raise the bar through code reviews, technical design discussions, and strong engineering and QA practices
Skills
- BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience
- 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome)
- Solid backend/scripting experience in a language such as Python, Rust, Go, or similar
- Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems
- Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results — able to distinguish real regressions from noise
- Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams
- Hands-on experience evaluating modern AI systems such as LLMs, RAG pipelines, agents, or multimodal models, including model behavior analysis
- Experience with React Native or other cross-platform mobile frameworks for building tooling that's accessible beyond the desktop
- Experience building or improving evaluation frameworks, benchmarks, or ML infrastructure used by other teams or external users
- A strong appreciation for evaluation quality — correctness, reproducibility, and consistency across environments
- Experience with voice, audio, speech recognition, or real-time systems, and familiarity with metrics like WER, MOS, or latency/TTFB
- Prior involvement in open-source projects, through contributions, reviews, maintenance, or community engagement
- Experience acting as a technical bridge across teams or platforms (evaluation, training, inference, agent frameworks), combining architectural understanding with clear communication and influence
- Familiarity with cloud infrastructure, containerized/ephemeral environments, and monitoring tooling (e.g. Grafana, canaries, anomaly detection)
Company Overview
Company H1B Sponsorship