Note: The job is a remote job and is open to candidates in USA. d-Matrix is hiring a Staff Test & Release Engineer for its Inference Cloud Engineering team. The role owns end-to-end testing and release systems for inference services, including automated tests on GPU and Corsair hardware, release strategy, and coordination across engineering teams.
Responsibilities
- Design and build end-to-end test infrastructure for Inference Cloud that stands up homogeneous and heterogeneous inference deployments on demand, runs tests against them, and tears them down
- Turn user stories and acceptance criteria into automated end-to-end tests, including failure cases such as invalid API keys, unavailable endpoints, and failed draft nodes
- Bring real hardware into CI by running tests on GPU and Corsair nodes, scheduling fairly on a limited shared pool, and owning test-signal quality, including flaky tests and fast routing of failures to the right owner
- Own the release strategy for the orchestration platform, Kubernetes tooling, and benchmarking tools, including versioning, cadence, branching, promotion, rollback, release notes, and go/no-go criteria
- Lead the move from a single platform-wide release to independently released components, defining the component breakdown, cadence, versioning, and compatibility and upgrade testing with platform architects
- Define release gates covering functional tests, upgrade and compatibility tests, model accuracy evals, and performance baselines (time to first token, time per output token, throughput)
- Partner with QE, product, platform, and infrastructure teams on a shared test strategy, and push for acceptance criteria specific enough to test
- Enable feature engineers to write and own tests for their own work
Skills
- 6+ years in test infrastructure, software engineering in test, release engineering, or a closely related role, including owning end-to-end testing or releases for a distributed system
- Track record of building test or release automation that measurably improved release reliability or speed
- Experience running test and release infrastructure across on-premises clusters and a public cloud, preferably Azure (AKS, Azure DevOps or GitHub Actions on Azure, Azure networking and identity)
- Strong Kubernetes experience, including deploying and debugging multi-service workloads, Helm, operators and custom resources, and automated test cluster setup and teardown
- Experience designing CI/CD pipelines (such as GitLab CI, GitHub Actions, Buildkite, or Argo), including artifact versioning, environment promotion, and rollback
- Proficiency in Python and pytest or a comparable framework, plus fluency in a second language sufficient to read and test the code under qualification (much of our orchestration layer is Rust)
- Experience testing on shared, limited hardware such as GPU or accelerator fleets, accounting for scheduling, contention, and hardware faults
- Sound judgment about what to test end to end versus lower in the stack, and the ability to explain tradeoffs to code owners
- Record of building consensus across teams and turning input into a strategy people follow
- Fluency with AI and agentic tools for writing tests, triaging failures, and searching logs and code, with the discipline to verify their output
- Experience testing or releasing LLM serving systems such as vLLM, SGLang, or TensorRT-LLM, including accuracy evals (such as GSM8K, HumanEval, MMLU) and performance benchmarks as release gates
- Familiarity with speculative decoding, disaggregated serving, or heterogeneous inference
- Hardware-in-the-loop testing experience, including qualifying software across driver, firmware, and hardware versions
- Experience with operator and controller testing tools such as envtest, kuttl, or chainsaw, and with upgrade and CRD version-migration testing
- Experience shipping software into customer-managed, on-premises Kubernetes environments, including compatibility matrices and customer install packaging
- Experience with canary or progressive rollouts for customer-facing services
- Rust experience
- Startup experience or comfort in environments where priorities shift quickly
Benefits
- Equity
- Bonus
- Remote work in the US
Company Overview
Company H1B Sponsorship