Responsibilities
- Design, build, and maintain scalable ETL/ELT pipelines and datasets in Databricks using Python, Spark, and SQL.
- Develop and support batch and streaming data architectures for reporting, analytics, and AI/ML workloads.
- Troubleshoot and optimize Databricks and existing data integration processes for performance, reliability, and data latency.
- Support and enhance existing SQL Server, SSIS, and SSRS solutions while contributing to ongoing platform modernization.
- Implement data quality checks, validation, cleansing, and data warehousing best practices across data pipelines and reporting environments.
- Integrate Databricks with enterprise applications, databases, and other data sources.
- Partner with business users, analysts, and IT teams to translate reporting and data needs into scalable technical solutions.
- Contribute to continuous improvement through emerging data engineering practices, AI-assisted development tools, documentation, and ongoing learning
Requirements
- Bachelor’s degree in Computer Science, Information Systems, or a related field.
- Strong hands-on experience with Databricks, including Python, Spark, and Delta Lake.
- Strong SQL Server and data warehousing experience, including T-SQL.
- Experience with SSIS and SSRS.
- Experience with Microsoft Azure; Microsoft Fabric is a plus.
- Experience using AI-assisted development tools as part of the engineering workflow.
- Strong analytical, problem-solving, communication, and collaboration skills.
Originally posted on