Note: The job is a remote job and is open to candidates in USA. VySystems is seeking a senior Iceberg DBA / Lakehouse Operations Engineer to own the reliability, performance, and operational integrity of the Apache Iceberg data layer supporting enterprise analytics and business-critical applications. The role operates in a large-scale, multi-engine Lakehouse environment and supports data modernization initiatives from Hive and Teradata to Iceberg.
Responsibilities
- Day-to-day ownership of Iceberg tables: ensure availability, consistency and integrity for TB–PB datasets
- Advanced table maintenance: compaction (minor/major), small-file mitigation, snapshot expiration, metadata compaction, orphan file cleanup (vacuum)
- Data layout optimization: file-size tuning, partition evolution/pruning, clustering/ordering strategies (e.g., Z-ordering patterns)
- Enforce data modeling and Lakehouse design: 3NF where appropriate, medallion (Bronze/Silver/Gold) layers, alignment with ingestion patterns and downstream requirements
- Multi-engine query consistency and tuning across Spark (CDE), Hive/Impala (CDW); resolve metadata inconsistencies and inefficient plans
- Metadata and lifecycle management: retention/archival, snapshot lifecycle, time-travel optimization
- Production support: L2/L3 troubleshooting, on-call rotation, P1/P2 incident response, RCA and preventive action
- Security/governance: support Ranger/RBAC policies, data validation and reconciliation between source and Iceberg
Skills
- Apache Iceberg/Hive experience, Spark SQL/Hive/Impala/NiFi/Trino, partitioning, Parquet/ORC, distributed query processing
- Migration experience (Hive→Iceberg, Teradata→Iceberg), Cloudera CDP, cloud (AWS/Azure), Python/Shell automation
Company Overview