Note: The job is a remote job and is open to candidates in USA. Management Science and Innovation (MSI) Consulting, LLC is a mid-size management consultancy serving industry, government, non-profit, and higher-education clients. The Senior Databricks Engineer will architect and implement scalable data pipelines and platforms using Databricks, PySpark, and Delta Lake while establishing engineering standards, governance, CI/CD, and data quality practices. The role also partners with cross-functional teams and mentors junior engineers to deliver production-ready data products.
Responsibilities
- Architect and implement high-performance ETL/ELT pipelines using Databricks, PySpark, and Delta Lake
- Optimize data workflows for scalability, reliability, and cost efficiency
- Partner with data scientists, analytics teams, business stakeholders, and product teams to deliver trusted, production-ready data products
- Implement CI/CD, testing, and data quality processes
- Define data platform architecture standards, engineering best practices, and governance frameworks across Databricks environments
- Mentor junior engineers and conduct code reviews
Skills
- 7+ years of data engineering experience, including 3+ years hands-on Databricks experience
- Deep expertise with Databricks Lakehouse Platform, Delta Lake, and Apache Spark
- Strong Spark performance tuning and optimization skills
- Advanced proficiency in SQL and Python; Scala preferred
- Experience designing batch and streaming data pipelines
- Experience building and deploying production Databricks AI/Agent solutions, ML engineering workflows, GenAI applications, and enterprise-grade dashboards and data products on the Databricks Lakehouse Platform
- Experience implementing CI/CD and automated testing for data engineering solutions
- Strong understanding of Unity Catalog, RBAC, lineage, auditing, and governance
- Experience designing and operating data platforms on AWS, Azure, or GCP, including security, networking, storage, and cost optimization considerations
- Familiarity with Terraform, CloudFormation, ARM, or Bicep
- Experience designing and implementing Medallion (Bronze/Silver/Gold) Lakehouse architectures and scalable data domain models
- Experience supporting Federal government clients
- Bachelor's (Required)
- Databricks Certified Data Engineer Professional
- Experience supporting RFP/RFQ/RFI responses and solution estimation
- Experience leading platform migration or modernization initiatives
- Experience in regulated environments and cloud cost optimization
- Certifications in other skills listed in the Qualifications section
Benefits
- 401(k)
- Dental insurance
- Health insurance
- Life insurance
- Vision insurance
- Paid time off
- Parental leave
- Remote work
Company Overview
Company H1B Sponsorship