Note: The job is a remote job and is open to candidates in USA. CCS Global Tech is seeking an Iceberg DBA / Lakehouse Operations Engineer to support an enterprise-scale Iceberg data layer. The role owns reliability, performance, metadata management, data lifecycle operations, production support, and query optimization across Spark, Hive, and Impala in a cloud-based Lakehouse environment.
Responsibilities
- Own day-to-day operations of Apache Iceberg tables supporting multiple enterprise applications
- Ensure data reliability, consistency, and availability across all Lakehouse workloads
- Maintain operational integrity for datasets at multi-terabyte to petabyte scale
- Execute advanced Iceberg table maintenance and optimization strategies:
- Compaction (minor/major) and small file mitigation
- Snapshot expiration and metadata compaction to control metadata growth
- Orphan file cleanup (vacuum) to maintain storage efficiency
- Optimize data layout and performance through:
- File size tuning and distribution strategies
- Partition evolution and pruning optimization
- Clustering and ordering techniques (e.g., Z-ordering or similar patterns)
- Support and enforce data modeling best practices aligned with:
- Normalized data structures (3NF) for source-aligned datasets
- Medallion architecture (Bronze / Silver / Gold layers) for curated data flows
- Ensure Iceberg table design aligns with:
- Data ingestion patterns (raw vs curated layers)
- Downstream consumption and performance requirements
- Assist in structuring datasets to balance:
- Data integrity and normalization
- Query performance and analytical efficiency
- Work with data engineering teams to ensure consistent implementation of layered data architecture across multiple applications
- Ensure consistent and performant query behavior across:
- Spark (CDE)
- Hive / Impala (CDW)
- Troubleshoot and resolve:
- Query performance bottlenecks
- Metadata inconsistencies across engines
- Inefficient execution plans and scan patterns
- Manage Iceberg metadata to ensure:
- Efficient scaling and performance
- Consistent table state across engines
- Execute lifecycle operations:
- Data retention and archival policies
- Snapshot lifecycle management and cleanup
- Time-travel optimization and maintenance
- Provide L2/L3 support for data-related production issues across Iceberg-based Lakehouse workloads
- Participate in on-call rotation to support critical data platforms and ensure timely response to incidents
- Respond to and resolve P1/P2 production incidents within defined SLAs, minimizing impact to downstream applications and reporting
- Troubleshoot:
- Data inconsistencies and reporting discrepancies
- Query failures and performance degradation
- Perform root cause analysis (RCA) and implement preventive measures to avoid recurring issues
- Collaborate with platform and application teams during incident triage and resolution
- Support fine-grained access control using:
- Ranger policies and RBAC
- Own and ensure data validation, reconciliation, and accuracy between source and Iceberg datasets
- Ensure secure and compliant access to data across applications
Skills
- Strong hands-on experience with Apache Iceberg and/or Hive-based data lakes
- Understanding of data modeling concepts (normal forms) and modern Lakehouse patterns (Medallion architecture)
- Expertise in:
- O Table-level optimization and performance tuning
- Large-scale data management (TB/PB scale)
- Experience with:
- O Spark SQL, Hive, Impala, NiFI, Trino
- Strong understanding of:
- O Partitioning strategies
- File formats (Parquet/ORC)
- Distributed query processing
- Experience with:
- O Hive-to-Iceberg or Teradata-to-Iceberg migration
- Cloudera CDP (CDE/CDW)
- Familiarity with:
- O Cloud platforms (AWS, Azure)
- Scripting/automation (Python, Shell)
Company Overview
Company H1B Sponsorship