Note: The job is a remote job and is open to candidates in USA. Catalyte is seeking a Mid-Level Data Engineer to support data modernization and migration initiatives. The role focuses on developing and maintaining data pipelines and workflows using Python, PySpark, Azure, and Azure Databricks, while collaborating with data scientists and technical stakeholders to deliver reliable engineering solutions.
Responsibilities
- Develop and maintain data processing and transformation workflows using Python and PySpark
- Build, test, and maintain data pipelines and related engineering processes
- Support the migration of data from legacy sources to modern data platforms
- Work within Azure and Azure Databricks environments
- Develop and maintain automation and CI/CD workflows using GitHub Actions, Bash, and Python
- Troubleshoot data and application issues and identify opportunities to improve reliability and efficiency
- Collaborate with data scientists and other technical stakeholders to understand requirements and translate them into effective engineering solutions
- Participate in code reviews, testing, documentation, and other software development practices
- Take ownership of assigned work, identify issues proactively, and contribute to solutions with limited day-to-day direction
- Learn and apply data engineering concepts, tools, and practices as needed
Skills
- Experience with Python in a professional software development environment
- Experience with PySpark or Spark
- Experience with Git and modern software development practices
- Experience with scripting and automation using Python and/or Bash
- Ability to work independently, take initiative, and manage assigned work
- Strong problem-solving and analytical skills
- Ability to communicate technical concepts clearly to both highly technical and less technical stakeholders
- Willingness and ability to learn new technologies and data engineering practices
- Experience with Azure
- Experience with Azure Databricks
- Experience with GitHub Actions or similar CI/CD platforms
- Experience developing or supporting data pipelines
- Familiarity with data engineering concepts, data migration, ETL/ELT, and data transformation
- Experience working with data scientists, analytics teams, or other technical stakeholders
Company Overview