Note: The job is a remote job and is open to candidates in USA. Toast creates technology to help restaurants and local businesses succeed in a digital world. The Senior Principal Software Engineer will lead the technical direction of the Machine Learning Platform team, shaping the systems that power AI and ML capabilities across Toast.
Responsibilities
- Own technical direction of the ML Platform — feature store, model hosting and serving, experimentation, training infrastructure — driving architectural decisions around scalability, reliability, latency, and cost
- Lead design and delivery of large-scope platform initiatives from conception through production, coordinating across ML, data, and infrastructure teams
- Identify and resolve systemic technical challenges: online/offline feature parity, model deployment friction, experimentation velocity, GPU utilization, cross-team dependencies
- Set and maintain a high engineering quality bar through hands-on code contributions, design reviews, and mentorship of platform and ML-adjacent engineers
- Partner with ML engineering, data science, product, and platform leadership to translate ML strategy into technical roadmaps
- Define the paved paths ML teams use to ship models safely — from feature registration through canary rollout, monitoring, and rollback
- Leverage AI-augmented development tools to increase development velocity and code quality
Skills
- 10+ years delivering complex backend or infrastructure systems at scale
- Direct experience building or operating core ML infrastructure — feature stores, model serving, experimentation platforms, training orchestration, or equivalent
- Mastery of a modern backend language, ideally Java or Kotlin
- Deep proficiency with distributed systems concepts: consistency, latency, throughput, fault tolerance, and observability
- Strong understanding of data modeling, query languages, and the online/offline data patterns that underpin ML systems
- Demonstrated technical leadership, with ability to drive cross-team alignment and influence engineering, product, and business stakeholders
- Bachelor's degree in Computer Science or a related field, or equivalent practical experience
- Hands-on experience with open-source or commercial ML platform components (e.g. Tecton, MLflow, SageMaker, Databricks)
- Experience building or operating experimentation / A-B testing platforms at scale
- Familiarity with real-time streaming systems (Kafka, Flink, Spark Streaming) and their use in feature computation
- Experience serving LLMs or deep-learning models in production, including GPU capacity planning and inference optimization
- Prior work supporting internal-developer-facing platforms with a product mindset
Company Overview
Company H1B Sponsorship