Note: The job is a remote job and is open to candidates in USA. Pivotal Solutions is seeking a Senior Data Engineer to join a client’s team on a remote contractor engagement. The role will lead the migration and refactoring of data pipelines and code, using Python, SQL, dbt, Snowflake, and orchestration tools while ensuring validated outputs, coordinated cutovers, and maintainable documentation.
Responsibilities
- Audit pipelines, jobs, and code in the legacy project: what runs, what it produces, who consumes it, and what is actually dead
- Decide, with the team, what gets ported as-is, what gets rewritten to current standards, and what gets retired
- Rewrite pipelines to the new project's patterns, conventions, and directory structure, including modular models, tests, and documented dependencies
- Refactor code that carries accumulated shortcuts: hardcoded values, duplicated logic, undocumented business rules, missing error handling
- Migrate orchestration, scheduling, alerting, and access controls, not just the transformation code
- Build validation to prove migrated pipelines produce equivalent output, and document any intentional differences
- Manage cutover for each pipeline with downstream consumers, including communication and rollback
- Leave behind documentation and diagrams for what you moved so the team can maintain it after the engagement
Skills
- 5+ years in data engineering, including at least one migration or major refactor of an existing production codebase
- Strong SQL and strong Python experience
- Experience with dbt or a comparable transformation framework, including project structure, macros, testing, and documentation
- Hands-on Snowflake or comparable cloud warehouse experience
- Orchestration experience: Airflow or similar, including migrating DAGs between environments or projects
- Solid Git and CI/CD practice. You should be comfortable in a repo with existing standards and reviewers
- Ability to read code without a spec and work out what the business rule was meant to be, then ask the right questions before changing it
Benefits
- Remote work arrangement
Company Overview
Company H1B Sponsorship