Note: The job is a remote job and is open to candidates in USA. TALENT Software Services is recruiting for a confidential client seeking a hands-on engineer to optimize production large language model inference on AWS Trainium. The role focuses on improving decode performance, benchmarking across accelerator platforms, scaling workloads, and enabling customer engineers to operate and tune the stack.
Responsibilities
- Lead decode optimization. Improve decode latency, throughput, and cost on AWS Trainium platforms, validating correctness at every step. Initial priorities include speculative decoding and native multi-token prediction (MTP) for Qwen-family models in vLLM on AWS Neuron
- Benchmark and assess performance rigorously. Design and run matched, reproducible tests against leading GPU platforms, controlling prompts, concurrency, cache state, and warm-up. Deliver latency-throughput Pareto frontiers and price-performance comparisons aligned with customer service-level objectives
- Lead cross-generation optimization. Benchmark and migrate workloads across Trainium generations, including Trainium2 and Trainium3. Scale serving on multi-chip Trainium systems for long-context workloads, including contexts of 100K tokens or more, and apply architecture-specific optimizations
- Extend model coverage. Onboard additional architectures, including hybrid state-space and mixture-of-experts models, reusing and generalizing existing implementations where practical
- Enable customer ownership. Create runbooks, implementation guides, and benchmark documentation, and support hands-on workshops so customer engineers can operate, reproduce, and tune the stack independently
Skills
- 5+ years of experience in machine learning or systems engineering, including 2+ years focused on production model inference or serving
- Strong proficiency in Python and PyTorch, with the ability to read and modify model implementations involving attention, mixture-of-experts routing, and KV-cache management
- Hands-on experience with an LLM serving framework such as vLLM, TensorRT-LLM, SGLang, or Triton Inference Server
- Deep understanding of LLM inference performance, including prefill and decode, continuous batching, and tensor and expert parallelism, and their impact on latency, throughput, and cost
- Experience profiling and optimizing accelerator workloads at the framework, compiler, or kernel level, and designing rigorous performance experiments
- Strong written and verbal communication skills, with the ability to explain engineering trade-offs to technical audiences and business impact to senior stakeholders
- Experience with AWS Trainium or Inferentia and the AWS Neuron SDK, including NxD Inference, vLLM on Neuron, or Neuron Kernel Interface (NKI), or with migrating workloads across accelerator generations
- Experience implementing or evaluating speculative decoding, such as draft-model approaches, Medusa, EAGLE, or MTP
- Kernel development experience in NKI, Triton, or CUDA
- Experience with hybrid or state-space architectures such as Mamba, large mixture-of-experts models, or FP8 / INT8 quantization with model quality and accuracy validation
- Experience in a customer-facing engineering, consulting, or technical enablement role
Company Overview
Company H1B Sponsorship