Note: The job is a remote job and is open to candidates in USA. Dice is hiring a Senior Data Software Engineer for a remote W2 position. The role focuses on designing and building large-scale Delta Lake to Apache Iceberg migration frameworks, preserving fine-grained access controls, and supporting AWS data lake governance, validation, and production reliability.
Responsibilities
- Design and build a migration feature/framework to convert tables from Delta Lake format to Apache Iceberg at scale
- Leverage deep understanding of Iceberg internals (metadata layers, manifests, snapshots, partitioning, schema evolution) to ensure correct and efficient migration
- Validate data integrity, schema fidelity, and partitioning strategy parity between source Delta tables and migrated Iceberg tables
- Optimize migration processes for performance and reliability across large volumes of tables and data
- Identify and resolve edge cases, compatibility issues, or performance bottlenecks during migration at scale
- Ensure Row-Level Security (RLS) and Column-Level Security (CLS) policies are correctly preserved or re-implemented on migrated Iceberg tables
- Integrate migration workflows with AWS Lake Formation to maintain governance and access control continuity
- Collaborate with security/governance teams to validate that FGAC policies enforce correctly post-migration
- Support GDC (Global Data Catalog / Governed Data Catalog) integration to ensure migrated tables remain properly cataloged and governed
- Build validation/reconciliation tooling to confirm access control parity between pre- and post-migration states
- Document migration architecture, access control mapping logic, and operational runbooks
- Collaborate with data platform, governance, and security teams throughout the migration lifecycle
- Monitor and troubleshoot production issues related to migrated tables and access enforcement
- Contribute to best practices and reusable tooling for future Delta-to-Iceberg migration phases
Skills
- Strong, hands-on knowledge of **Apache Iceberg internals** (table format, metadata management, snapshots, partitioning, schema evolution)
- Proven experience performing **Delta-to-Iceberg migrations at scale**
- Experience with **AWS Lake Formation** for access control and data governance
- Practical experience implementing or preserving **Row-Level Security (RLS)** and **Column-Level Security (CLS)**
- Experience with **GDC** (Global/Governed Data Catalog) or equivalent data cataloging/governance systems
- Solid understanding of distributed data processing and large-scale data migration challenges
- Strong SQL skills and experience with table format internals (Delta Lake, Iceberg, or similar)
- Experience with **Spark for large-scale data processing and migration tooling**
- **Familiarity with AWS data platform services (S3, Glue, Athena)**
- Experience with data governance frameworks and compliance-driven access control requirements
- Prior experience building reusable migration frameworks or tooling for table format conversions
- S**cripting/automation experience (Python, Scala) for migration validation and tooling**
Benefits
- Remote work in the USA, aligned to Eastern Time (EST)
Company Overview
Company H1B Sponsorship