Note: The job is a remote job and is open to candidates in USA. Jobot Consulting is seeking a Senior Data Engineer to lead the end-to-end migration of complex structured finance datasets from a legacy proprietary platform to Databricks. The role owns data architecture and modeling, reverse-engineers undocumented business logic, builds migration pipelines, validates financial parity, and optimizes cloud infrastructure for performance and cost.
Responsibilities
- Lead migration of structured finance datasets from a legacy platform to Databricks
- Reverse-engineer undocumented business logic and establish data/financial parity
- Design data models supporting time-series data, point-in-time accuracy, bitemporal history, and late-arriving data
- Build scalable ingestion, transformation, and publishing pipelines
- Optimize Spark/Databricks performance, storage, orchestration, and cloud costs
- Partner with quantitative, risk, and product teams and document architectural decisions
Skills
- 8+ years of Data Engineering experience with architectural ownership
- Strong production experience with Databricks, Delta Lake, Spark/PySpark, Spark SQL, and Unity Catalog
- Strong knowledge of financial/structured finance data, including ABS/MBS, loan-level data, cash flows, and financial time series
- Experience reverse-engineering large, undocumented legacy codebases using SQL, C++, Java, or similar
- Advanced Python and SQL skills
- Experience with CI/CD, Git, Terraform, and cloud data platforms
- Must be available during U.S. Eastern Time business hours
- Must be authorized to work in the U.S. for the duration of the engagement
- Experience fully migrating and decommissioning legacy platforms
- Experience with Delta Live Tables, data governance, and FinOps
- Conference presentations or published technical work in data engineering or financial technology
- NYC-area candidates strongly preferred for periodic onsite collaboration
Benefits
- Renewable 6 Month Contract!
- 6-month contract, with potential extension.
- Remote within the U.S.
Company Overview