Note: The job is a remote job and is open to candidates in USA. Confluent is pioneering a foundational platform for real-time data streaming and develops highly available, secure, distributed systems at planetary scale. The Senior Principal Engineer II will define the multi-year technical strategy for the Kafka Background Plane, architect high-scale distributed processing capabilities, ensure end-to-end correctness, drive cross-organizational architectural alignment, and mentor senior technical talent.
Responsibilities
- Architect Multi-Year Strategy: Establish the long-term technical vision for the Background Plane, translating strategic business and product objectives into world-class distributed architectures, multi-year engineering roadmaps, and high-impact investment priorities
- Engineer High-Scale Processing Platforms: Design, build, and optimize core platform capabilities including distributed scheduling, task coordination, isolation guarantees, elastic autoscaling, sophisticated backpressure, and cross-dependency execution workflows
- Guarantee End-to-End Correctness: Own the mathematical and operational correctness model for all data and metadata workflows (compaction, rewriting, retention, segment lifecycle, object GC), building bulletproof safety guarantees under extreme concurrency, partial network failures, and degraded dependencies
- Scale Without Bounds: Ensure the Background Plane seamlessly accommodates hyper-growth in partition counts, throughput, storage footprint, and background lag without compromising sub-millisecond foreground latencies
- Drive Cross-Organizational Consensus: Spearhead architectural contracts across organizations—spanning data references, metadata interfaces, object life cycles, encryption models, and task schedulers—leading initiatives from initial framing to global production rollout
- Raise Operational Excellence: Define and enforce global engineering standards for fault tolerance, chaos testing, deep observability, canary safety, and incident mitigation, continually driving down systemic risk across reliability, cost, and velocity
- Serve as Senior Authority: Act as the definitive technical authority during critical architectural reviews, high-severity incident responses, and complex technical trade-off discussions spanning product, infrastructure, and security
- Elevate Engineering Culture: Sponsor and mentor senior and staff engineers across the organization while representing Confluent's technical vision externally through keynotes, research publications, and open-source contributions
Skills
- Bachelor's Degree
- Proven track record of architecting, building, and operating mission-critical distributed systems at planetary scale, with deep expertise in storage engines, distributed log stores, stream processing, replication, or consensus protocols
- Deep insight into Apache Kafka internals (storage semantics, log compaction, tiered storage, segment indexing) or equivalent high-throughput streaming systems
- Demonstrated capability to articulate, execute, and deliver complex multi-year architectural roadmaps across multi-disciplinary engineering organizations
- Exceptional fluency in reasoning about race conditions, distributed concurrency, state machine transitions, edge-case failures, and backpressure mechanisms
- Proven ability to establish measurable SLA/SLO metrics, durability targets, and automated recovery practices across global multi-tenant infrastructure fleets
- Outstanding written and oral communication skills, with an ability to distill highly technical concepts for executive leadership and forge cross-team alignment through trust and expertise
- A passion for mentorship, talent development, and empowering staff/principal engineers to reach their full leadership potential
- Shift: General (daytime)
- Some travel may be required based on business demand
- Master's Degree
- Log-structured storage formats, custom storage engines, cloud object stores, or large-scale data re-writing pipelines
- Fault-tolerant object lifecycle strategies designed to prevent data leakage, orphaned resources, or premature deletion
- Chaos engineering, production traffic replay platforms, deterministic simulation testing, and large-scale load generation
- Multi-cloud, multi-region deployment topologies across AWS, GCP, and Azure
Benefits
- Remote work arrangement
- Healthcare benefits including medical & prescription drug coverage, dental, vision, and mental health & well being
- 401(k)
- Cash balance pension plan
- IBM Employee Stock Purchase Plan
- Financial counseling
- Life insurance
- Short & long-term disability coverage
- Opportunities for performance based salary incentive programs
- 12 paid holidays
- Minimum 56 hours sick time
- 120 hours vacation
- 12 weeks parental bonding leave in accordance with IBM Policy
- Other Paid Care Leave programs
- Paid family leave benefits to eligible employees where required by applicable law
- Training and educational resources on our personalized, AI-driven learning platform
- Industry-recognized certifications to achieve career goals
- Diverse and inclusive employee resource groups
- Giving & volunteer opportunities
- Discounts on retail products, services & experiences
Company Overview
Company H1B Sponsorship