Note: The job is a remote job and is open to candidates in USA. Guidehouse is a consulting and professional services company supporting clients through its AI & Data team. The Databricks Data Engineer will design, build, and maintain scalable batch and streaming data pipelines, transform large-scale datasets, and deliver reliable data products using Databricks, Spark, SQL, Python, and cloud-native technologies. The role also involves troubleshooting, documentation, governance, and collaboration with technical teams and clients.
Responsibilities
- Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, Delta Lake, and related cloud-native data engineering tools
- Develop and support batch and streaming data workflows, including ingestion, transformation, validation, and publishing of curated data products
- Write, optimize, and maintain Python, PySpark, and SQL code for data processing, orchestration, and performance tuning
- Work with large-scale datasets using Databricks, Spark, Delta Lake, Unity Catalog, and cloud storage services
- Troubleshoot and resolve data pipeline failures, performance issues, data quality issues, and workflow bottlenecks
- Translate business and technical requirements into data engineering designs, processing scripts, and job orchestration workflows
- Perform data validation, quality checks, code reviews, and issue resolution to support accurate and reliable data products
- Collaborate with cross-functional teams including solution architects, data scientists, analysts, DevOps engineers, and client stakeholders
- Communicate technical concepts, delivery impacts, and data engineering considerations to technical and non-technical audiences
- Document pipelines, datasets, data models, workflows, and engineering decisions to support maintainability, transparency, and reuse
- Follow data governance, security, lineage, and compliance standards within the Databricks platform
Skills
- * Bachelor's degree in computer science, engineering, mathematics, statistics, or another relevant field
- * 3-8 years of relevant experience in data engineering, data architecture, or cloud data platform implementation
- * Strong experience with Python, PySpark, and SQL for data transformation, pipeline development, and data processing
- * Experience with Databricks, Spark, Delta Lake, Unity Catalog, or similar cloud-native data platforms
- * Experience developing batch or streaming data pipelines, ETL/ELT workflows, and reusable data assets
- * Experience with data modeling, data warehousing, data quality validation, and large-scale data processing concepts
- * Ability to troubleshoot technical issues, communicate engineering recommendations clearly, and work effectively in team-based delivery environments
- * Experience applying data governance, access control, lineage, and security practices within cloud data platforms
- * 2+ years of hands-on experience with the Databricks platform
- * Experience using Unity Catalog for data governance, schema design, data modeling standards, access controls, lineage, and secure management of enterprise data assets
- * Experience designing and building standard medallion data architectures for scalable ingestion, transformation, quality validation, and curated data delivery
- * Active Databricks Data Engineer Associate, Databricks Data Engineer Professional, or related certification
- * Experience with CI/CD tools and practices for notebooks, workflows, orchestration jobs, and infrastructure deployment
- * Experience with cloud platforms such as Azure, AWS, or GCP
- * Experience working in project-based or consulting delivery environments
- * Familiarity with streaming frameworks, orchestration tools, Terraform, or related data platform administration capabilities
- This position offers virtual work flexibility within the United States. While remote candidates will be considered, preference will be given to candidates located near a Guidehouse office in one of the following markets: **Arlington, VA; Washington, DC; New York, NY; Chicago, IL; Austin, TX; Atlanta, GA; Boston, MA; and Boulder, CO.**
Benefits
- Virtual work flexibility within the United States
- A flexible benefits package
- A comprehensive total rewards package
Company Overview
Company H1B Sponsorship