Note: The job is a remote job and is open to candidates in USA. Apolis is recruiting for a confidential client seeking a Databricks Data Engineer. The role will lead the architecture and implementation of a scalable Sales Data Platform on Databricks and AWS, covering ingestion, transformation, modeling, optimization, governance, and production operations. The engineer will also provide technical leadership and mentorship while collaborating with business, source-system, and cloud teams.
Responsibilities
- Define end-to-end lakehouse architecture on Databricks (AWS) for Sales data domains
- Design medallion architecture (Bronze / Silver / Gold) aligned with OneData standards
- Establish data modeling standards for Sales facts, dimensions, hierarchies, and aggregations
- Define scalable ingestion patterns for batch and incremental loads
- Drive performance, scalability, and cost optimization best practices
- Build and guide development of PySpark-based data pipelines in Databricks
- Implement Delta Lake features including ACID transactions, schema evolution, and versioning
- Design and optimize large-scale joins, aggregations, and window functions
- Implement CDC and incremental processing using watermarking and change detection
- Ensure idempotent, restartable, and fault-tolerant pipelines
- Architect solutions using AWS services such as Amazon S3, IAM, AWS Glue, and CloudWatch
- Optimize Databricks cluster configurations
- Implement secrets management and secure connectivity patterns
- Define and implement data quality checks and validations
- Ensure data lineage and metadata capture
- Implement error handling, auditing, and reconciliation frameworks
- Support data governance and access control requirements
- Implement CI/CD pipelines for Databricks notebooks and jobs
- Enforce code versioning, reviews, and deployment standards
- Design monitoring, alerting, and SLA tracking for pipelines
- Support production stabilization and performance tuning
- Act as technical lead / mentor for Databricks data engineers
- Collaborate with source system teams, data consumers, and cloud/platform teams
- Translate business requirements into robust technical designs
Skills
- 8 years in Data Engineering / Data Architecture roles
- 4 years hands-on experience with Databricks
- Strong expertise in PySpark & Spark SQL
- Expertise in dbt and Delta Lake
- Strong knowledge of AWS cloud services (S3, IAM, Glue, CloudWatch)
- Experience in Sales data domain (orders, revenue, pricing, customers, products)
- Understanding of Fact & dimension modeling and Slowly Changing Dimensions (SCD Type 1 / 2)
- Experience handling high-volume, high-velocity datasets
- Experience with enterprise OneData / Data Mesh programs
- Exposure to real-time or near-real-time ingestion patterns
- Experience integrating CRM / Sales systems (e.g., Salesforce, SAP Sales data)
- AWS certifications or Databricks certifications
Benefits
- Remote (Harrisburg, PA)
Company Overview
Company H1B Sponsorship