Note: The job is a remote job and is open to candidates in USA. MetaPhase is a mission-focused consultancy that values authenticity, collaboration, and practical innovation for its clients. The Data Engineer – Databricks will design, develop, test, deploy, operate, and continuously improve data pipelines and data products in Databricks, while supporting governance, troubleshooting, platform operations, and enterprise data delivery.
Responsibilities
- Develop, test, deploy, and maintain batch and streaming data pipelines using Databricks, Python, SQL, Apache Spark, and Delta Lake
- Build and enhance ingestion, transformation, validation, and publishing processes that move data from source systems into governed data products and analytics-ready datasets
- Implement approved data models, data-quality rules, metadata, and documentation in accordance with established architecture and governance standards
- Configure and maintain Databricks notebooks, workflows, jobs, compute resources, and related deployment artifacts
- Participate in code reviews, peer testing, release preparation, defect remediation, and CI/CD activities
- Monitor pipeline performance, job execution, data-quality results, and platform alerts; troubleshoot issues and support resolution of production incidents
- Collaborate with architects, analysts, data owners, and other engineers to clarify requirements, identify dependencies, and deliver iterative improvements
- Maintain technical documentation for pipelines, data sources, transformations, interfaces, test results, and operating procedures
Skills
- Bachelor's degree in a technical discipline and three or more years of relevant experience in data engineering, software engineering, analytics engineering, or a related field
- Demonstrated proficiency in Python and SQL, with experience developing, debugging, and maintaining ETL/ELT processes and data-processing code
- One or more years of hands-on experience with Databricks, Apache Spark, or a comparable cloud data-engineering platform
- Experience working with structured and/or unstructured data sources, data validation, source-to-target mapping, and production-support activities
- Ability to obtain a U.S. Public Trust suitability determination
- U.S. Citizenship Required
- Databricks Certified Data Engineer Associate certification
- Experience with Databricks capabilities such as Delta Lake, Auto Loader, Databricks SQL, Lakeflow Jobs, Unity Catalog, or streaming data pipelines
- Familiarity with Git-based version control, code reviews, automated testing, CI/CD, and Agile delivery practices
- Experience supporting data governance activities, including metadata documentation, data-quality checks, lineage, and access-control implementation
- Experience with AWS, Azure, or Google Cloud data services and cloud-based integrations
- Experience supporting regulated, public-sector, or security-sensitive data environments
- Additional Databricks certifications (e.g., Databricks Machine Learning Engineer Associate or Professional, Databricks Generative AI Engineer Associate)
Benefits
- Generous PTO
- Federal holidays
- Parental leave
- Comprehensive health coverage (medical, dental, vision, life, and disability)
- 401(k) with company match
- FSA/HSA options
- Commuter benefits
- Remote work
Company Overview