Note: The job is a remote job and is open to candidates in USA. Artmac is seeking a Senior Databricks Data Engineer to design, develop, and optimize scalable data engineering solutions. The role focuses on data pipelines, Lakehouse architecture, ETL/ELT workflows, cloud data platforms, and reliable, secure data processing.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Apache Spark, and SQL
- Build and optimize ETL/ELT workflows to process large volumes of structured and semi-structured data
- Implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers)
- Develop batch and streaming data processing solutions to support business and analytics requirements
- Optimize Spark jobs, cluster configurations, partitioning, and data storage to improve performance and control processing costs
- Integrate data from multiple sources, including databases, APIs, cloud storage, and enterprise applications
- Implement data validation, reconciliation, error handling, and monitoring to ensure data accuracy and reliability
- Work with cloud platforms such as Microsoft Azure, AWS, or Google Cloud, depending on project requirements
- Implement data governance, access controls, and secure data management using Unity Catalog and related platform capabilities
- Develop automated workflows, job scheduling, and dependency management for production data pipelines
- Troubleshoot production issues, identify root causes, and implement long-term solutions
Skills
- 8 15 years of experience in data engineering, big data development, or related roles
- Strong hands-on experience with Databricks and Apache Spark
- Advanced programming skills in Python and PySpark
- Strong SQL skills, including complex queries, joins, aggregations, and query optimization
- Experience building production-grade ETL/ELT pipelines and data integration workflows
- Hands-on experience with Delta Lake, Delta tables, and Lakehouse architecture
- Understanding of Medallion Architecture and scalable data modeling practices
- Experience with at least one major cloud platform: Azure, AWS, or GCP
- Experience with data pipeline orchestration, scheduling, monitoring, and troubleshooting
- Knowledge of Git, CI/CD pipelines, automated testing, and deployment practices
- Strong understanding of data quality, data governance, security, and performance optimization
- Familiarity with Terraform, infrastructure as code, or cloud resource management
- Experience working with enterprise-scale data platforms and high-volume datasets
- Databricks certifications or relevant cloud data engineering certifications
- Bachelor's degree or equivalent combination of education and experience
Benefits
- Remote work
Company Overview
Company H1B Sponsorship