Note: The job is a remote job and is open to candidates in USA. Robert Half is recruiting for a leading infrastructure and transportation group seeking a hands-on Senior Data Engineer to architect and build its next-generation Enterprise Reporting & Data Warehouse on Databricks. The role owns target-state design, lakehouse architecture, production data pipelines, legacy warehouse migrations, platform operations, governance, and collaboration with technical and business stakeholders.
Responsibilities
- Architecture & Strategy: Design the ERDW.Next lakehouse (medallion/gold layer star schema) and govern data modeling, batch/streaming, and Delta Live Tables standards
- Build & Migration: Write production-grade PySpark, Spark SQL, and Delta Live Tables pipelines orchestrated via Databricks Workflows. Re-engineer legacy SSIS/T-SQL/Synapse logic and ingest ERP data (JD Edwards, Anaplan) using CDC and Auto Loader
- Platform Operations: Optimize cost/performance using Photon, Liquid Clustering, and OPTIMIZE. Manage CI/CD pipelines via Git and Databricks Asset Bundles while maintaining strict SLAs
- Governance & AI: Implement Unity Catalog governance (entitlements, lineage, audit) and curate Databricks Genie Agents for natural-language analytics
- Partner & Legacy Management: Support legacy platforms during transition and maintain code quality standards across offshore partner teams. Leverage agentic AI coding tools safely to accelerate delivery
Skills
- • Serving as a Databricks SME
- • ERP integrations (strong plus)
- • Python / PySpark
- • Performance tuning and optimization
- • Databricks maintenance, including VACUUM
- • Data governance and best practices
- • Semantic modeling
- • Education: Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
- • Track Record: 5+ years dedicated to Databricks/Apache Spark platforms and 8+ years overall in Data Engineering/BI
- • Track Record: Proven experience taking an enterprise lakehouse end-to-end into production
- • System Experience: Deep hands-on experience migrating legacy warehouses (SQL Server, Synapse, ADF) and sourcing data from ERPs (JD Edwards strongly preferred)
- • Databricks & Spark: Advanced PySpark, Spark SQL, Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, and Photon
- • Data Modeling & Storage: Star Schema, SCDs, Conformed Dimensions, Delta Lake (MERGE, Liquid Clustering, Z-ORDER, Time Travel)
- • Cloud & DevOps: Azure Gen2, Key Vault, Entra ID, Git, Azure DevOps/GitHub Actions, Databricks Asset Bundles, and Infrastructure as Code (Terraform)
- • Legacy & Integration: SQL/T-SQL, SSIS, SSRS, Synapse, ADF, CDC/replication tools, Direct Lake/Power BI integration, and Microsoft Fabric/OneLake
- • AI & Modern Workflow: Agentic AI coding tools, natural language query curation (Databricks Genie), and metadata-driven semantics
- • Strong technical ownership, architectural influence, and clear communication with both executives and engineers
- • Excellent cross-functional collaboration with offshore partners, citizen developers, and business stakeholders
- • Pragmatic problem-solving, critical thinking, and time management across build and legacy support
- • Certifications (Preferred): Databricks Certified Data Engineer Professional/Data Architect; Azure DP-203 or AZ-305
Company Overview
Company H1B Sponsorship