Job Title: Data Engineer
Location: [Your Location or "Remote"]
About the Role:
We are seeking a highly skilled Data Engineer with strong expertise in Databricks and PySpark. The ideal candidate will also have hands-on experience with Airflow for orchestration and Kafka for streaming data pipelines. You will play a key role in building, optimizing, and maintaining our data infrastructure to support data-driven decision-making across the organization.
Responsibilities:
- Design, build, and maintain scalable data pipelines using Databricks and PySpark.
- Develop and manage ETL workflows and data integration processes.
- Implement data orchestration and workflow automation using Apache Airflow.
- Build real-time and batch data streaming solutions leveraging Apache Kafka.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality solutions.
- Optimize data workflows for performance, scalability, and reliability.
- Ensure data quality, integrity, and security across all pipelines.
- Monitor and troubleshoot data pipelines, ensuring minimal downtime and data loss.
Requirements:
- Bachelor's or Master's degree in Computer Science, Engineering, or related field.
- 3+ years of experience in data engineering or a similar role.
- Strong hands-on experience with Databricks and PySpark.
- Proven experience in building and maintaining data pipelines with Apache Airflow.
- Solid understanding of Apache Kafka, including producing and consuming data streams.
- Proficiency in SQL and working with large-scale data processing.
- Experience with cloud platforms (AWS, Azure, or GCP) is a plus.
- Familiarity with data modeling, warehousing, and governance best practices.
- Excellent problem-solving and communication skills.
Nice to Have:
- Experience with Delta Lake, Spark Streaming, or other big data technologies.
- Knowledge of CI/CD tools and DevOps practices for data workflows.
- Exposure to monitoring tools like Datadog, Grafana, or similar.