Note: The job is a remote job and is open to candidates in USA. The client is a consumer AI company focused on privacy, free speech, and user sovereignty. The Engineering Lead, Inference Optimization will own inference performance strategy, lead a specialized engineering team, optimize GPU infrastructure and LLM workloads, and evaluate emerging optimization techniques and hardware.
Responsibilities
- Own the company’s technical strategy for inference performance
- Recruit and lead the Inference Optimization Team
- Optimize the company’s GPU infrastructure across a range of architectures (e.g. H200s, B300s)
- Improve latency, throughput, and cost per token for LLM inference workloads
- Build reproducible benchmarking harnesses across inference engines (e.g. vLLM, SGLang) to identify the optimal engine, quantization scheme, and parallelism strategy per workload and GPU SKU
- Work with the company’s inference routing system to optimize multivariate inference load-balancing algorithms
- Evaluate emerging inference optimization techniques (custom CUDA/Triton kernels), novel attention variants, new quantization schemes, and compilation stack improvements. Hands-on kernel development experience is a strong plus
- Evaluate emerging inference hardware (FPGAs, ASICs, custom silicon) for viability in the company’s stack
Skills
- 8+ years in performance optimization or HPC, with deep GPU architecture and parallel programming knowledge
- 5+ years of experience leading engineering teams
- Proficiency in Python, Rust, or Go
- Hands-on experience with at least one production LLM inference engine (e.g. vLLM, SGLang) running at high volume in production
- Demonstrated experience with LLM inference optimization techniques: continuous batching, PagedAttention/KV cache management, speculative decoding, quantization, CUDA graphs, and torch.compile
- Fluency with quantization tradeoffs, both qualitative and quantitative
- Experience with distributed inference strategies (tensor parallelism, pipeline parallelism, MoE parallelism) in multi-GPU and multi-node environments
- Fluency with GPU profiling (Nsight Systems, Nsight Compute, PyTorch Profiler) and a bias toward measuring before optimizing
- Bonus: C++/CUDA
- Bonus: diffusion/image model inference optimization, custom Triton kernels, contributions to open-source inference frameworks
Benefits
- Equity
- Crypto token compensation
- Remote — US Only
Company Overview