Note: The job is a remote job and is open to candidates in USA. Robert Half is a specialized talent solutions firm connecting qualified job seekers with opportunities at leading companies. The Senior Data Engineer will architect and build an enterprise reporting and data warehouse lakehouse on Databricks, lead legacy platform migrations, and remain hands-on in development. The role also partners with architecture, analytics, business, and offshore teams to implement data engineering, governance, optimization, and AI-enabled analytics capabilities.
Responsibilities
- Architecture & Strategy: Design the ERDW.Next lakehouse (medallion/gold layer star schema) and govern data modeling, batch/streaming, and Delta Live Tables standards
- Build & Migration: Write production-grade PySpark, Spark SQL, and Delta Live Tables pipelines orchestrated via Databricks Workflows. Re-engineer legacy SSIS/T-SQL/Synapse logic and ingest ERP data (JD Edwards, Anaplan) using CDC and Auto Loader
- Platform Operations: Optimize cost/performance using Photon, Liquid Clustering, and OPTIMIZE. Manage CI/CD pipelines via Git and Databricks Asset Bundles while maintaining strict SLAs
- Governance & AI: Implement Unity Catalog governance (entitlements, lineage, audit) and curate Databricks Genie Agents for natural-language analytics
- Partner & Legacy Management: Support legacy platforms during transition and maintain code quality standards across offshore partner teams. Leverage agentic AI coding tools safely to accelerate delivery
Skills
- • Serving as a Databricks SME
- • ERP integrations (strong plus)
- • Python / PySpark
- • Performance tuning and optimization
- • Databricks maintenance, including VACUUM
- • Data governance and best practices
- • Semantic modeling
- • Education: Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
- • Track Record: 5+ years dedicated to Databricks/Apache Spark platforms and 8+ years overall in Data Engineering/BI
- • Track Record: Proven experience taking an enterprise lakehouse end-to-end into production
- • System Experience: Deep hands-on experience migrating legacy warehouses (SQL Server, Synapse, ADF) and sourcing data from ERPs (JD Edwards strongly preferred)
- • Databricks & Spark: Advanced PySpark, Spark SQL, Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, and Photon
- • Data Modeling & Storage: Star Schema, SCDs, Conformed Dimensions, Delta Lake (MERGE, Liquid Clustering, Z-ORDER, Time Travel)
- • Cloud & DevOps: Azure Gen2, Key Vault, Entra ID, Git, Azure DevOps/GitHub Actions, Databricks Asset Bundles, and Infrastructure as Code (Terraform)
- • Legacy & Integration: SQL/T-SQL, SSIS, SSRS, Synapse, ADF, CDC/replication tools, Direct Lake/Power BI integration, and Microsoft Fabric/OneLake
- • AI & Modern Workflow: Agentic AI coding tools, natural language query curation (Databricks Genie), and metadata-driven semantics
- • Strong technical ownership, architectural influence, and clear communication with both executives and engineers
- • Excellent cross-functional collaboration with offshore partners, citizen developers, and business stakeholders
- • Pragmatic problem-solving, critical thinking, and time management across build and legacy support
- • Certifications (Preferred): Databricks Certified Data Engineer Professional/Data Architect; Azure DP-203 or AZ-305
Benefits
- Free online training
- Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance.
- Hired contract/temporary professionals are eligible to enroll in the company 401(k) plan.
Company Overview