Note: The job is a remote job and is open to candidates in USA. Dice is recruiting for a data engineering role supporting the modernization of enterprise data capabilities. The Senior Databricks Data Engineer will build scalable Databricks solutions using medallion architecture, reusable frameworks, dimensional models, batch and streaming pipelines, observability, and data-quality capabilities. The role also contributes to testing, deployment, documentation, production support, and emerging AI-enabled data solutions.
Responsibilities
- Design, develop, and enhance enterprise-scale data solutions using Databricks and medallion architecture
- Build reusable frameworks and common functions that standardize data ingestion, transformation, validation, repair, and pipeline monitoring
- Develop Bronze, Silver, and Gold data layers that improve the usability, consistency, and accessibility of enterprise data
- Create dimensional data structures, including facts, dimensions, derived attributes, and SCD Type 2 implementations
- Develop and orchestrate both batch and streaming data pipelines
- Partner with business stakeholders to translate high-level data requirements into technical solutions
- Collaborate with architects and technical leadership on solution design and engineering standards
- Perform hands-on development, unit testing, system integration testing, debugging, and production support
- Build solutions that improve pipeline observability, SLA monitoring, data quality, and operational transparency
- Optimize Databricks compute and storage performance
- Develop documentation, training materials, and reusable guidance so other engineering teams can adopt frameworks and capabilities
- Explore newer Databricks functionality such as Intelligent Document Processing, Genie, and self-service workspace capabilities and establish practical implementation patterns
Skills
- Minimum **3 years of full-time, hands-on Databricks experience** within Azure or AWS environments
- Strong experience implementing the **medallion architecture**, including Bronze, Silver, and Gold data layers
- **Expert-level SQL** skills
- **Intermediate or stronger PySpark** programming experience
- Demonstrated experience creating reusable data engineering frameworks, common functions, or standardized pipeline components
- Hands-on experience creating dimensional models, fact tables, dimensions, derived business values, and **SCD Type 2** structures
- Strong Databricks orchestration experience supporting batch and streaming pipelines
- Experience with Databricks capabilities such as **Unity Catalog, Spark UI, job scheduling, Lakeflow, and data quality expectations**
- Proven experience with unit testing, SIT, UAT support, debugging, and troubleshooting data defects
- Experience optimizing Databricks compute and storage
- Strong understanding of **CI/CD, Git-based development, and automated deployment processes**
- Experience working within Agile environments, including user stories, technical task definition, and effort estimation
- Ability to communicate effectively with business users, architects, engineering teams, and onshore/offshore partners
- Strong problem-solving, design-thinking, teamwork, and documentation skills
- Candidates with **5+ years of Databricks experience** and experience building major platform capabilities from scratch are strongly preferred
- Insurance data, data governance, or data observability experience is beneficial
Benefits
- W2 employment only (no C2C/1099)
- Fully remote within the U.S., working Eastern Time hours
- Contract-to-hire opportunity
- Opportunity to help define reusable Databricks engineering standards and frameworks used across multiple teams
- Exposure to newer Databricks capabilities, including AI-assisted development and intelligent document processing
Company Overview
Company H1B Sponsorship