Note: The job is a remote job and is open to candidates in USA. HMG AMERICA LLC is seeking a Lead Data Engineer (Databricks) to lead end-to-end cloud data initiatives for clients. The role involves architecting AWS Databricks Lakehouse platforms, advising client stakeholders, modernizing legacy data warehouses, and guiding engineering teams.
Responsibilities
- Lakehouse Architecture: Architect, design, and implement scalable, enterprise-grade data platforms on AWS using the Databricks Lakehouse architecture
- Medallion Architecture: Establish robust data architecture patterns across Bronze, Silver, and Gold layers for both real-time streaming and batch processing pipelines
- Data Governance & Security: Implement fine-grained access control, governance, data lineage, and audit trails using Databricks Unity Catalog, AWS IAM, and KMS
- Performance & Cost Optimization: Optimize cluster configurations, auto-scaling parameters, and Delta Lake performance techniques (Z-Ordering, Liquid Clustering, compaction)
- Ecosystem Integration: Seamlessly integrate AWS-native services (S3, AWS Glue, Lambda, EventBridge, Redshift, Athena, Secrets Manager) with Databricks
- Stakeholder Management: Lead discovery workshops, technical assessments, and architecture reviews with C-level executives, VPs of Engineering/Data, and Enterprise Architects
- Technical Roadmaps: Translate complex business requirements into solution blueprints, high-level architecture diagrams, technical roadmaps, and Statements of Work (SOWs)
- Legacy Modernization: Drive client migration strategies from legacy data warehouses (e.g., Teradata, Netezza, Snowflake, On-Prem Hadoop) to AWS Databricks
- Best Practices Guidance: Educate clients on cloud data best practices, FinOps/cost optimization strategies, and modern Lakehouse paradigms
- Technical Mentorship: Guide, direct, and mentor cross-functional teams of Data Engineers, Analytics Engineers, and DevOps Engineers
- Infrastructure as Code (IaC): Enforce IaC best practices using Terraform or CloudFormation for automated enterprise workspace and cluster deployments
- DevOps & Automation: Build and maintain CI/CD pipelines for Databricks notebooks, Delta Live Tables (DLT), and Spark jobs across multi-environment setups (Dev, Test, Prod)
Skills
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related field
- 8+ years in Data Engineering with at least 3+ years in a Lead/Architect capacity
- Deep experience with Databricks , PySpark/Scala, Delta Lake, Unity Catalog, and Delta Live Tables (DLT)
- Hands-on experience with AWS Cloud Data Ecosystem (S3, Glue, Lambda, EMR, Redshift, IAM)
- Proficiency in Python , SQL , and Spark SQL performance tuning
- Strong experience with Infrastructure-as-Code ( Terraform ) and CI/CD tools (GitHub Actions, Azure DevOps, Jenkins)
- Exceptional verbal and written communication skills with the ability to articulate complex technical ideas to executive audiences
- Strong problem-solving skills and client-facing advisory experience
Benefits
- Remote
Company Overview