Note: The job is a remote job and is open to candidates in USA. Shakti Solutions is seeking an AWS Databricks Engineer to design and implement the Databricks platform lifecycle and environments on AWS. The role focuses on data governance, CI/CD, cloud architecture, Spark and Delta Lake technologies, and enabling AI/ML capabilities within Databricks.
Responsibilities
- Design and implement end-to-end lifecycle of Databricks platform on AWS
- Design and implement different environments like Dev/SIT/UAT/PROD for Databricks
- Expertise in AWS
- Implement data access controls, Lineage and auditing using Unity catalog
- Implement CI/CD pipelines between different environments
- Hands-on architecture experience within AWS, Azure, or GCP environments
- Deep expertise in Apache Spark, Delta Lake, SQL, and cluster resource management
- Translate business requirements into technical blueprints, run proof-of-concepts (POCs), and guide engineering teams on best practices
- AI/ML Enablement, experience in setting up AI and Genie for Databricks
- Experience with Delta lake house, Unity catalog, DBX notebooks, Python, Pyspark, Genie, Databricks Apps
Skills
- Expertise in AWS
- Hands-on architecture experience within AWS, Azure, or GCP environments
- Deep expertise in Apache Spark, Delta Lake, SQL, and cluster resource management
- AI/ML Enablement, experience in setting up AI and Genie for Databricks
- Experience with Delta lake house, Unity catalog, DBX notebooks, Python, Pyspark, Genie, Databricks Apps
Company Overview
Company H1B Sponsorship