Note: The job is a remote job and is open to candidates in USA. Hydrogen Group is recruiting for a Senior Data Engineer to support its client's Core Data team, which builds data platforms and analytics capabilities for organizational decision-making. The role focuses on migrating the Vehicle Catalog's ETL/ELT pipelines, datasets, dashboards, and reporting assets to a new Databricks environment while validating data parity, updating operational processes, and troubleshooting migration issues.
Responsibilities
- Working with a Staff Data Engineer to understand the migration plan, sequence work against it, and provide regular progress updates
- Refactoring existing ETL/ELT code written in PySpark, DABs, and dbt from the current Databricks environment to meet the patterns and standards of the new account
- Migrating data pipelines and their underlying datasets, including backfills, and validating parity between source and target
- Building and running reconciliation checks - row counts, schema conformance, freshness, completeness - to confirm migrated data matches the legacy environment
- Migrating dashboards and downstream reporting assets to point at the new datasets, coordinating with the analysts who own them
- Updating orchestration, scheduling, alerting, and runbooks so migrated pipelines are supportable after cutover
- Troubleshooting pipeline failures and data discrepancies during and after cutover
- Documenting what was migrated, what changed, and anything the team needs to know to own it going forward
- Taking on adjacent migration and data engineering requests as the program evolves
Skills
- 6+ years of experience in Data Engineering, Analytics Engineering, or Software Engineering working with production data systems
- Strong expertise in SQL and Python for building scalable data pipelines and transformations
- Hands-on experience building ELT pipelines using modern cloud data platforms (Databricks experience required)
- Experience building robust testing frameworks for data migration (completeness, quality, etc.)
- Deep experience with dbt, including model development, testing, documentation, and CI/CD integration
- Experience designing and operating production-grade data pipelines with monitoring and observability
- Prior data or platform migration experience
- Strong familiarity with AI development tools such as Claude Code or Devin for the software development process
- Strong collaboration skills and ability to partner with engineering teams, analysts, and operational stakeholders
- Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or a related field, or equivalent practical experience
- Candidates in PST, MT, or CST preferred
- Familiarity with Fivetran connector management and ingestion architecture (nice-to-have)
- Experience with data contracts and schema governance (nice-to-have)
- Knowledge of streaming or near-real-time data pipelines (nice-to-have)
Company Overview
Company H1B Sponsorship