Note: The job is a remote job and is open to candidates in USA. Quantum World Technologies Inc. is seeking a Databricks Data Engineer to develop and manage scalable data engineering solutions. The role focuses on Databricks architecture and administration, Lakehouse and Delta Lake implementation, ETL/ELT pipelines, data governance, quality frameworks, and cloud data platforms.
Responsibilities
- Strong experience with Databricks Architecture and platform administration
- Hands-on expertise in Databricks Lakehouse Architecture
- Deep understanding of Delta Lake concepts and implementation
- Experience with Unity Catalog / Data Catalog and metadata management
- Knowledge of Databricks Workflows, Jobs, Clusters, and Performance Tuning
- Strong proficiency in PySpark for large-scale data processing
- Advanced Python programming skills
- Expert-level SQL development and query optimization
- Experience in building robust ETL/ELT pipelines
- Strong understanding of data modeling techniques including:
- Star Schema
- Snowflake Schema
- Dimensional Modeling
- Data Vault (preferred)
- Experience implementing data quality frameworks and validation checks
- Knowledge of data lineage, metadata management, and governance processes
- Experience with data reconciliation, profiling, and monitoring tools
- Azure Databricks
- Azure Data Lake Storage (ADLS)
- Azure Data Factory
- Azure Synapse Analytics
- CI/CD pipelines (Azure DevOps, GitHub Actions)
Skills
- Strong experience with Databricks Architecture and platform administration
- Hands-on expertise in Databricks Lakehouse Architecture
- Deep understanding of Delta Lake concepts and implementation
- Experience with Unity Catalog / Data Catalog and metadata management
- Knowledge of Databricks Workflows, Jobs, Clusters, and Performance Tuning
- Strong proficiency in PySpark for large-scale data processing
- Advanced Python programming skills
- Expert-level SQL development and query optimization
- Experience in building robust ETL/ELT pipelines
- Strong understanding of data modeling techniques including:
- Star Schema
- Snowflake Schema
- Dimensional Modeling
- Experience implementing data quality frameworks and validation checks
- Knowledge of data lineage, metadata management, and governance processes
- Experience with data reconciliation, profiling, and monitoring tools
- Azure Databricks
- Azure Data Lake Storage (ADLS)
- Azure Data Factory
- Azure Synapse Analytics
- CI/CD pipelines (Azure DevOps, GitHub Actions)
- Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field
- 4-8 years of experience in Data Engineering and Analytics
- Minimum 3+ years of hands-on experience with Databricks and PySpark
- Experience working in Agile development environments
- Data Vault (preferred)
Company Overview