Note: The job is a remote job and is open to candidates in USA. Axiom Path is a global solutions provider leveraging AI and pre-built analytics to deliver services across various domains. The Senior Databricks Data Engineer will design, develop, and support enterprise-scale data solutions using Databricks and medallion architecture, including reusable frameworks, dimensional models, batch and streaming pipelines, and data-quality capabilities. The role will also collaborate with stakeholders and technical leaders, optimize platform performance, and explore emerging Databricks capabilities such as intelligent document processing.
Responsibilities
- Design, develop, and enhance enterprise-scale data solutions using Databricks and medallion architecture
- Build reusable frameworks and common functions that standardize data ingestion, transformation, validation, repair, and pipeline monitoring
- Develop Bronze, Silver, and Gold data layers that improve the usability, consistency, and accessibility of enterprise data
- Create dimensional data structures, including facts, dimensions, derived attributes, and SCD Type 2 implementations
- Develop and orchestrate both batch and streaming data pipelines
- Partner with business stakeholders to translate high-level data requirements into technical solutions
- Collaborate with architects and technical leadership on solution design and engineering standards
- Perform hands-on development, unit testing, system integration testing, debugging, and production support
- Build solutions that improve pipeline observability, SLA monitoring, data quality, and operational transparency
- Optimize Databricks compute and storage performance
- Develop documentation, training materials, and reusable guidance so other engineering teams can adopt frameworks and capabilities
- Explore newer Databricks functionality such as Intelligent Document Processing, Genie, and self-service workspace capabilities and establish practical implementation patterns
Skills
- Minimum 3 years of full-time, hands-on Databricks experience within Azure or AWS environments
- Strong experience implementing the medallion architecture , including Bronze, Silver, and Gold data layers
- Expert-level SQL skills
- Intermediate or stronger PySpark programming experience
- Demonstrated experience creating reusable data engineering frameworks, common functions, or standardized pipeline components
- Hands-on experience creating dimensional models, fact tables, dimensions, derived business values, and SCD Type 2 structures
- Strong Databricks orchestration experience supporting batch and streaming pipelines
- Experience with Databricks capabilities such as Unity Catalog, Spark UI, job scheduling, Lakeflow, and data quality expectations
- Proven experience with unit testing, SIT, UAT support, debugging, and troubleshooting data defects
- Experience optimizing Databricks compute and storage
- Strong understanding of CI/CD, Git-based development, and automated deployment processes
- Experience working within Agile environments, including user stories, technical task definition, and effort estimation
- Ability to communicate effectively with business users, architects, engineering teams, and onshore/offshore partners
- Strong problem-solving, design-thinking, teamwork, and documentation skills
- Insurance data, data governance, or data observability experience is beneficial
- Candidates with 5+ years of Databricks experience and experience building major platform capabilities from scratch are strongly preferred
Benefits
- W2 employment only (no C2C/1099)
- Fully remote within the U.S., working Eastern Time hours
- Contract-to-hire opportunity
- Opportunity to help define reusable Databricks engineering standards and frameworks used across multiple teams
- Exposure to newer Databricks capabilities, including AI-assisted development and intelligent document processing
- Broad ownership across requirements, development, testing, deployment, documentation, and production support
Company Overview