Note: The job is a remote job and is open to candidates in USA. iTech US Inc is seeking a Senior Data Engineer to support large-scale data platform migrations and governance initiatives. The role focuses on building Delta Lake-to-Apache Iceberg migration frameworks, preserving data integrity and access controls, integrating AWS governance services, and developing validation and operational tooling.
Responsibilities
- Design and build a migration feature/framework to convert tables from Delta Lake format to Apache Iceberg at scale
- Leverage deep understanding of Iceberg internals (metadata layers, manifests, snapshots, partitioning, schema evolution) to ensure correct and efficient migration
- Validate data integrity, schema fidelity, and partitioning strategy parity between source Delta tables and migrated Iceberg tables
- Optimize migration processes for performance and reliability across large volumes of tables and data
- Identify and resolve edge cases, compatibility issues, or performance bottlenecks during migration at scale
- Ensure Row-Level Security (RLS) and Column-Level Security (CLS) policies are correctly preserved or re-implemented on migrated Iceberg tables
- Integrate migration workflows with AWS Lake Formation to maintain governance and access control continuity
- Collaborate with security/governance teams to validate that FGAC policies enforce correctly post-migration
- Support GDC (Global Data Catalog / Governed Data Catalog) integration to ensure migrated tables remain properly cataloged and governed
- Build validation/reconciliation tooling to confirm access control parity between pre- and post-migration states
- Document migration architecture, access control mapping logic, and operational runbooks
- Collaborate with data platform, governance, and security teams throughout the migration lifecycle
- Monitor and troubleshoot production issues related to migrated tables and access enforcement
- Contribute to best practices and reusable tooling for future Delta-to-Iceberg migration phases
Skills
- Strong, hands-on knowledge of Apache Iceberg internals (table format, metadata management, snapshots, partitioning, schema evolution)
- Proven experience performing Delta-to-Iceberg migrations at scale
- Experience with AWS Lake Formation for access control and data governance
- Practical experience implementing or preserving Row-Level Security (RLS) and Column-Level Security (CLS)
- Experience with GDC (Global/Governed Data Catalog) or equivalent data cataloging/governance systems
- Solid understanding of distributed data processing and large-scale data migration challenges
- Strong SQL skills and experience with table format internals (Delta Lake, Iceberg, or similar)
- Experience with Spark for large-scale data processing and migration tooling
- Familiarity with AWS data platform services (S3, Glue, Athena)
- Experience with data governance frameworks and compliance-driven access control requirements
- Prior experience building reusable migration frameworks or tooling for table format conversions
- Scripting/automation experience (Python, Scala) for migration validation and tooling
Benefits
- Remote work arrangement
Company Overview
Company H1B Sponsorship