Note: The job is a remote job and is open to candidates in USA. Circana provides technology, artificial intelligence, and data solutions for consumer packaged goods companies, manufacturers, and retailers. The Principal Engineer will design, develop, and optimize a next-generation massively parallel distributed database engine, focusing on database kernels, high-performance computing, distributed systems, hardware optimization, and in-database machine learning infrastructure.
Responsibilities
- **Architecture & Design:** Play a key role in core architectural choices and roadmap for our distributed database kernel, including query execution engines, custom/hybrid columnar storage layers, distributed transaction managers, and cluster coordination protocols
- **Low-Level Development:** Write, optimize, and maintain highly performant, production-ready backend code utilizing modern C++ (C++20/23)
- **Parallel Execution Planning:** Architect MPP execution engines that distribute, schedule, and execute queries across several dozens to hundreds of nodes using MPI and hybrid OpenMP models
- **Hardware Optimization:** Profile and eliminate execution bottlenecks by implementing explicit SIMD vectorization (AVX-512, ARM Neon, SVE) and ensuring absolute cache locality (L1/L2/L3)
- **Distributed Memory & Networks:** Design ultra-low latency cluster communication subsystems utilizing RDMA, RoCE, or kernel-bypass tech (DPDK)
- **In-Database ML Framework Design:** Architect and implement a native, highly performant, zero-copy ML execution framework directly inside the MPP engine to eliminate data serialization and transport bottlenecks between the database and external AI pipelines
- **Functional & High-Performance Algorithms:** Develop mathematically rigorous, cache-oblivious, and highly vectorized algorithms for real-time statistical computations, matrix operations/manipulations on clustered data
- **Vector & Embedding Infrastructure:** Design and optimize or embed/reference distributed high-dimensional vector storage subsystems, custom SIMD-accelerated indexing structures (e.g., HNSW, IVF-PQ), and parallelized approximate nearest neighbor (ANN) search algorithms
- Be comfortable working in a cross country/multi time zone team of elite systems engineers, implementing as well as conducting rigorous peer design and development reviews
- Collaborate with cross-functional teams in defining and developing new platform capabilities
- Actively participate in architecture reviews, code reviews, and technical mentoring
- Investigate and resolve complex performance and scalability issues
Skills
- Minimum - 4 years Computer Science engineering degree from a reputed institute
- 12+ years of production experience writing hands-on, low-level systems in C++ (C++17/20/23 preferred)
- Mastery of template metaprogramming, concepts, coroutines, and custom memory allocators is key
- Very good to expert-level knowledge of cluster-scale distributed programming using MPI (Message Passing Interface) paired with shared-memory thread-level programming via OpenMP and MapReduce frameworks
- Documented experience with microarchitectural optimization, loop vectorization, pointer-aliasing constraints, and manual SIMD intrinsics programming
- Deep understanding of OS internals, memory barriers, lock-free data structures, and atomics (std::atomic)
- Practical experience designing core database subsystems (e.g., vectorized query operators, LSM-trees, B-trees, custom buffer pools, columnar serialization frameworks and query plan optimizers)
- Strong debugging, performance tuning, and problem-solving skills
- Proficiency in one or more leading memory and compute monitoring/performance tools (such as Valgrind, Purify, Intel/AMD/ARM diagnostics etc.)
- Masters in HPC (high performance computing) and Bigdata engineering areas preferred for Senior staff engineer and above roles
- Track record contributing to open-source or work experience on proprietary distributed engines (e.g., ClickHouse, DuckDB, Doris, CockroachDB, RocksDB, ScyllaDB, or Velox)
- Experience with cloud-native distributed, elastic environments, object storage tiering (S3/GCS), or zero-copy networking
- Experience integrating heterogeneous acceleration (NVIDIA CUDA / ROCm) into big data workloads and hybrid CPU/GPU setups
- Experience building or modifying high-performance vector index/search engines, graph-based indices, or localized quantization frameworks
- Functional domain awareness – CPG, Retail, Pharma verticals
Benefits
- Bonus pay
- Paid time off
- Medical insurance for eligible employees
- Dental insurance for eligible employees
- Vision insurance for eligible employees
- 401(k) for eligible employees
- Hybrid work arrangement in Chicago, IL
Company Overview
Company H1B Sponsorship