### Senior Databricks Data Engineer / Technical Lead
**Job Summary**
We are seeking a hands-on **Senior Databricks Data Engineer / Technical Lead** to support a U.S. Department of Transportation (DOT) data modernization initiative. The role will focus on building and migrating scalable data pipelines to a **Databricks Lakehouse**, modernizing legacy data sources, and supporting production data operations.
**Key Responsibilities**
* Design and develop scalable **Databricks data pipelines** using Medallion Architecture (Bronze/Silver/Gold).
* Build metadata-driven ingestion and transformation frameworks for legacy and external data sources.
* Develop and optimize **Python, PySpark, and SQL** workloads.
* Work with **Databricks, Delta Lake, Unity Catalog, Auto Loader/Lakeflow, and Databricks Jobs/Workflows**.
* Integrate data from **AWS/S3, CSV/JSON files, REST APIs**, and other sources.
* Implement data quality, validation, reconciliation, schema evolution, lineage, and governance.
* Manage orchestration, dependencies, monitoring, alerting, retries, and production support.
* Collaborate with architects, PMs, DevSecOps, security, governance, and business teams.
* Support migration to the **AWS/OneDOT** environment while ensuring security and performance.
**Required Qualifications**
* 7+ years of professional **Data Engineering** experience.
* Strong hands-on experience with **Databricks/Spark** and production data pipelines.
* Advanced **Python, PySpark, and SQL** skills.
* Experience with **Delta Lake, AWS/S3, Git, CI/CD, and data orchestration**.
* Experience handling sensitive data, access controls, and secrets management.
* Bachelor's degree in Computer Science, IT, Engineering, Data Engineering, or related field, or equivalent experience.
**Preferred**
* Experience migrating **Sybase/SAP IQ or similar legacy platforms to Databricks**.
* Experience with **Unity Catalog, metadata-driven ingestion, Auto Loader/Lakeflow, and Databricks Workflows**.
* Federal/DOT project experience.
* Databricks and/or AWS certifications.
* Experience with streaming, AI/ML data pipelines, data governance, and cloud cost optimization.