Core requirement: Strong data engineering fundamentals with proven, hands-on Databricks experience. Deep experience of Databricks concepts the team uses day to day.
Technical must-haves: Performance tuning and optimization, including Spark internals, join strategies, and data skew handling; strong use of LDP APIs, CDF, and CDC; and experience building data-quality frameworks using expectations, quarantine logic, and alerting.
Governance & orchestration: Hands-on experience with PII/PHI masking, access controls, compliance/governance, plus the ability to orchestrate and stitch Databricks pipelines together end to end.
Location: Remote- US
Required Skills & Experience
5-7+ years of Data Engineering experience building large-scale data platforms • Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS) • Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning • Experience developing and maintaining Delta Live Tables (DLT) pipelines • Strong SQL background with experience supporting legacy data warehouse environments • Experience migrating SQL-based pipelines to modern Python/PySpark architectures • Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies • Proficiency with YAML, JSON, and configuration-driven development patterns • Experience automating CI/CD deployments using DevOps pipelines • Strong understanding of Medallion Architecture (Bronze, Silver, Gold) • Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers • Experience troubleshooting, monitoring, and supporting production data pipelines • Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred • Healthcare industry experience preferred • Medicare, payer, member enrollment, or claims data experience strongly preferred Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred
Job Description
Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables • Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions • Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns • Develop and maintain CI/CD deployment processes for data engineering workflows • Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers • Partner with upstream and downstream teams to ensure data quality, reliability, and scalability • Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments • Enhance monitoring, observability, and operational support processes • Support Medicare and payer-focused initiatives involving member, claims, and operational datasets • Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability