Note: The job is a remote job and is open to candidates in USA. Diverse Lynx is seeking a highly skilled Databricks Data Engineer with a Solution Architect background to lead the design of enterprise-scale data solutions. The role involves establishing engineering standards, optimizing platform performance, and providing technical leadership for cloud modernization and data governance initiatives.
Responsibilities
- Applies advanced expertise in Databricks, Spark, Delta Lake, and cloud-native architectures to design and govern large-scale enterprise data platforms
- Leads architecture decisions for cloud-based data and analytics solutions, ensuring scalability, security, resiliency, and operational excellence across Azure, AWS, or GCP environments
- Establishes and evolves platform standards, best practices, and governance frameworks for data engineering, streaming, AI/ML, and analytics workloads
- Recommends innovative technologies, architectural patterns, and optimization strategies to improve platform performance, reliability, and cost efficiency
- Works on and may lead highly complex, cross-functional initiatives involving cloud modernization, data platform transformation, and enterprise architecture adoption
- Designs and optimizes high-volume batch and real-time data pipelines, leveraging deep knowledge of Spark internals, Structured Streaming, and distributed computing principles
- Provides architectural leadership for Delta Lake, Unity Catalog, security, data governance, and data-sharing capabilities across multiple business domains
- Drives Infrastructure as Code (IaC), DevOps, and CI/CD adoption using Terraform, Databricks Asset Bundles (DABs), and modern software engineering practices
- Independently determines architectural approaches, technology choices, and implementation strategies for new and evolving platform capabilities
- Receives guidance on strategic business objectives while exercising significant discretion in technical and architectural decision-making
- Provides expert consultation to leadership, engineering teams, and business stakeholders, delivering solutions that may establish enterprise-wide standards and precedents
- Acts as a technical mentor and thought leader, elevating engineering capabilities and promoting architecture best practices across the organization
- Influences long-term cloud, data, and AI platform strategy, ensuring solutions align with business goals, regulatory requirements, and future growth needs
Skills
- Strong expertise in Spark, Delta Lake, Unity Catalog, cloud architecture, Infrastructure as Code (IaC), and modern data platform engineering principles
- Advanced expertise in Databricks, Spark, Delta Lake, and cloud-native architectures
- Experience in designing and governing large-scale enterprise data platforms
- Ability to lead architecture decisions for cloud-based data and analytics solutions
- Knowledge of scalability, security, resiliency, and operational excellence across Azure, AWS, or GCP environments
- Experience in establishing and evolving platform standards, best practices, and governance frameworks for data engineering, streaming, AI/ML, and analytics workloads
- Ability to recommend innovative technologies, architectural patterns, and optimization strategies
- Experience in designing and optimizing high-volume batch and real-time data pipelines
- Deep knowledge of Spark internals, Structured Streaming, and distributed computing principles
- Architectural leadership for Delta Lake, Unity Catalog, security, data governance, and data-sharing capabilities
- Experience in driving Infrastructure as Code (IaC), DevOps, and CI/CD adoption using Terraform and Databricks Asset Bundles (DABs)
- Ability to independently determine architectural approaches, technology choices, and implementation strategies
- Ability to provide expert consultation to leadership, engineering teams, and business stakeholders
- Experience in acting as a technical mentor and thought leader
- Ability to influence long-term cloud, data, and AI platform strategy
Company Overview
Company H1B Sponsorship