Note: The job is a remote job and is open to candidates in USA. Code Metal is hiring a Staff AI Platform Engineer to build and operate the internal AI platform that supports its engineering teams’ AI-driven products. The role leads platform architecture and delivers production capabilities for model inference, gateways, agent orchestration, context engineering, observability, and experimentation.
Responsibilities
- Set the technical direction and architecture for Code Metal's AI platform and the team building it. Own the design docs and RFCs, help with build-vs-buy decisions, and align with other technical stakeholders
- Deploy, benchmark, and tune elastic, production inference for open-weight models on runtimes such as vLLM, SGLang, and TensorRT-LLM
- Own the model gateway that teams use to reach self-hosted and commercial models, with consistent auth, routing, failover, quotas, and cost attribution
- Design reusable agent harnesses and orchestration primitives that product teams can compose into reliable, verifiable workflows
- Build context-engineering services for memory, retrieval, and data discovery, so agents get the information specific to their tasks
- Instrument the stack end to end, for example, with OpenTelemetry traces and service metrics. Build the experiment-tracking and artifact layer that lets engineers and researchers reproduce and compare results
- Design for productization from day one (versioned APIs, security, deployment in customer environments), and communicate with business and product teams about the roadmap
Skills
- Production-grade Python and strong platform engineering fundamentals: API and service design, distributed systems, containers and Kubernetes, CI/CD, and testing
- Shipped production agentic systems, with a clear sense of where they break and how to make them reliable
- Experience with context engineering: retrieval-augmented generation, embeddings, vector / hybrid search, and memory for agents, ideally over code or large technical corpora
- Experience instrumenting services (for example, with OpenTelemetry tracing and metrics) and operating AI services against SLOs
- Solid data science and AI research fundamentals: how transformers and LLM inference work, experiment design, benchmarking, and model evaluation
- Working familiarity with PyTorch and Hugging Face, and experience fine-tuning, evaluating, or serving language models
- Staff-level technical leadership: owned architecture across multiple systems or teams, written design docs and RFCs, turned ambiguous needs from several internal customers into a roadmap, and mentored engineers
- Experience building an evaluation harnesses for LLMs and agents, and experiment-tracking or artifact management systems such as MLflow or Weights & Biases
- Typically 8+ years of software engineering experience, including 4+ years building and operating ML/AI or LLM systems in production, with demonstrated Staff-level scope and impact
- Highly desirable: Production experience running an LLM gateway or proxy such as SMG or Bifrost, or equivalent experience building an API gateway, including routing, auth, rate limiting, quotas, failover, and cost attribution
- Hands-on experience deploying and tuning LLM inference engines such as vLLM, SGLang, or TensorRT-LLM on GPU infrastructure, with measurable gains in throughput, latency, or cost per token
- Familiarity with inference optimization: speculative decoding, prefix caching, tensor/pipeline/expert parallelism, disaggregated prefill and decode, GPU profiling
- Experience taking an internal platform to an external product: multi-tenancy, SDKs, versioned APIs, and documentation
- Experience deploying AI systems on-prem or in regulated domains such as defense or aerospace
Benefits
- Offers Equity
- Health care plan with 100% premium coverage, including medical, dental, and vision
- 401k with 5% matching
- Paid Time Off (uncapped vacation, plus sick and public holidays)
- Flexible hybrid or remote work arrangement
- Relocation assistance for qualifying employees
Company Overview