Note: The job is a remote job and is open to candidates in USA. Groq is an AI technology company building a multi-tenant AI inference platform. The Principal Platform Engineer will design, build, and operate distributed control-plane, orchestration, scheduling, and fleet-management systems across bare-metal environments and multiple data center sites. The role also involves improving reliability, observability, automation, tenant isolation, and operational readiness while partnering with cross-functional engineering teams.
Responsibilities
- Own distributed control-plane systems that manage the lifecycle of tenant clusters, bare-metal hosts, accelerator racks, and inference workloads across multiple data center sites
- Build and operate multi-tenant Kubernetes at bare-metal scale, including virtual clusters, host-cluster lifecycle management, node scaling, and automated server provisioning
- Design and develop controllers, operators, custom resources, and reconciliation loops that continuously converge infrastructure and workload state toward the desired configuration
- Own distributed scheduling and workload placement across accelerator topology, rack availability, network fabrics, capacity, tenant constraints, and workload priorities
- Build fleet-management services that track infrastructure state, health, capacity, maintenance status, and software versions across large-scale environments
- Develop platform APIs and state-management services that connect bare-metal provisioning, Kubernetes orchestration, capacity management, scheduling, and managed model deployment
- Design systems that remain reliable under partial failure, network partitions, delayed communication, duplicate events, resource contention, and rapidly changing fleet conditions
- Define clear consistency, availability, durability, and recovery requirements for critical platform services
- Build idempotent workflows and safe retry mechanisms for long-running operations such as provisioning, cluster upgrades, workload placement, rack maintenance, and infrastructure recovery
- Design for horizontal scalability, high availability, fault isolation, backpressure, rate limiting, and graceful degradation
- Develop distributed coordination mechanisms such as leader election, leases, work queues, event streams, caching, and distributed locking
- Harden tenant isolation through network policies, admission controls, per-tenant storage, API-server audit logging, and secrets-management integrations
- Drive automation and GitOps for provisioning, configuration management, drift remediation, upgrades, and day-two operations. Infrastructure as Code is the only path to production
- Establish end-to-end observability for distributed workflows using metrics, logs, traces, events, and correlation identifiers
- Define and monitor service-level indicators for control-plane availability, scheduling latency, provisioning success, reconciliation health, fleet capacity, and recovery time
- Build tooling for safely rolling out changes across large fleets, including staged deployments, compatibility validation, automated rollback, and failure-domain-aware upgrades
- Investigate complex production issues spanning services, clusters, networks, hardware, and data center boundaries, then implement durable systemic fixes
- Partner with inference, model-serving, networking, security, and hardware teams to define platform interfaces and end-to-end system behavior
- Review designs, mentor engineers, and raise the bar for distributed systems correctness, operational readiness, testing, and documentation
- Participate in an on-call rotation for production control-plane services, clusters, and tenant infrastructure
Skills
- A BS in Computer Science or another STEM field and 5+ years of systems, infrastructure, or distributed software engineering experience, or 7+ years of equivalent professional experience
- Experience designing, building, and operating distributed systems in production
- Strong understanding of distributed systems fundamentals, including consistency models, consensus, leader election, replication, partition tolerance, failure detection, idempotency, and eventual consistency
- Experience building control planes, schedulers, orchestration systems, fleet-management platforms, or other systems that coordinate large numbers of resources
- Deep experience operating Kubernetes as a platform, including controllers, operators, custom resources, admission webhooks, schedulers, and API machinery
- Hands-on experience deploying and operating Linux servers across bare-metal and virtualized environments, including automation of the full provisioning lifecycle
- Deep knowledge of Linux internals, including kernel modules, cgroups, namespaces, process management, PKI, networking, and iptables or nftables
- Proficiency in Go and/or Python, with experience writing controllers, services, command-line tools, automation, and API integrations
- Experience designing asynchronous systems using queues, event streams, reconciliation patterns, or workflow engines
- Ability to reason about race conditions, concurrency, state transitions, ordering, retries, timeouts, and failure recovery
- Experience with data stores used by distributed control planes, including relational databases, key-value stores, caches, or strongly consistent coordination systems
- Experience with Infrastructure as Code, configuration management, GitOps, and CI/CD for infrastructure and platform services
- Experience with Kubernetes networking, storage, and container runtimes
- Experience building observability for distributed systems using metrics, logs, traces, dashboards, and service-level objectives
- Able to translate ambiguous platform requirements into clear system boundaries, APIs, state models, invariants, and operational expectations
- Strong operational judgment and a track record of building systems that are testable, repeatable, observable, auditable, and resilient
- Effective collaborators who communicate clearly across software, infrastructure, networking, hardware, security, and operations teams
- This position may require access to technology and/or information subject to U.S. export control laws and regulations, including the Export Administration Regulations (EAR). To comply with these requirements, candidates for this role must meet certain citizenship or residency criteria. Specifically, they must qualify as U.S. Persons for export control purposes (i.e., U.S. citizen, U.S. lawful permanent resident (Green Card holder), or a protected individual under 8 U.S.C. § 1324b(a)(3) such as a refugee or asylee), or otherwise be eligible for an applicable export license
- Experience with multi-region or multi-data-center control planes, topology-aware scheduling, capacity management, service discovery, or distributed configuration is especially valuable
- Bay Area or NYC preferred; open to hybrid or remote for the right roles and candidates
Benefits
- Potential bonus value incorporated directly into base pay through Groq’s Total Cash philosophy
- Long-Term Incentive (LTI) Program
- Open to hybrid or remote for the right roles and candidates
Company Overview