Note: The job is a remote job and is open to candidates in USA. Data Capital Incorporation is seeking a Senior Databricks Data Engineer / Technical Lead to support a U.S. Department of Transportation data modernization initiative. The role focuses on designing, developing, migrating, and supporting secure, scalable data pipelines in a Databricks Lakehouse environment, including legacy data migration, metadata-driven ingestion, data quality, production operations, and AWS/OneDOT cloud modernization.
Responsibilities
- Designing, developing, migrating, and supporting secure and scalable data pipelines within a Databricks Lakehouse environment
- Supporting a U.S. Department of Transportation (DOT) data modernization initiative
- Legacy data migration
- Metadata-driven ingestion
- Medallion Architecture
- Data quality
- Reconciliation
- Production operations
- AWS/OneDOT cloud modernization
- Developing and supporting production-grade data pipelines
- Data quality, validation, reconciliation, monitoring, and troubleshooting
- Working effectively with technical and non-technical stakeholders
Skills
- 7+ years of professional data engineering experience
- Strong hands-on experience with Databricks and Apache Spark in production environments
- Advanced proficiency in: • Python • PySpark • SQL
- Strong experience with: • Databricks Lakehouse • Delta Lake • AWS / Amazon S3 • Batch and file-based ingestion • CSV/JSON processing • REST APIs • Git • CI/CD • Data pipeline orchestration
- Experience developing and supporting production-grade data pipelines
- Strong understanding of Medallion Architecture (Bronze/Silver/Gold)
- Experience with data quality, validation, reconciliation, monitoring, and troubleshooting
- Understanding of security practices for PII and sensitive data , including access controls and secrets management
- Ability to work effectively with technical and non-technical stakeholders
Company Overview
Company H1B Sponsorship