Role Overview
We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role covers data ingestion, processing, storage, orchestration, and delivery to Analytics and BI teams within an AWS-based data ecosystem.
Responsibilities
- Design and maintain batch and real-time data pipelines.
- Integrate data from APIs, databases, files, and external sources using Apache Kafka and AWS services.
- Develop and orchestrate workflows with Apache Airflow, including monitoring, retry mechanisms, and alerting.
- Build and optimize distributed data processing solutions using Apache Spark, PySpark, Spark SQL, and AWS EMR.
- Manage Data Lake and Data Warehouse environments using Amazon S3 and Amazon Redshift.
- Design analytical data models, including Star and Snowflake schemas.
- Prepare and optimize datasets for Analytics and Business Intelligence use cases, particularly for Qlik.
- Develop high-quality solutions using Python and SQL, following best practices for Git, code reviews, testing, and documentation.
- Ensure data quality, security, governance, and compliance, including AWS IAM, encryption, and access control.
- Monitor production data pipelines and proactively address operational and performance issues.
Mandatory Requirements
- Proven experience as a Senior Data Engineer or in a similar role.
- Advanced proficiency in Python and SQL.
- Hands-on experience with:
- Apache Spark / PySpark
- Apache Kafka
- Apache Airflow
- Strong AWS experience, particularly with:
- Amazon S3
- Amazon Redshift
- AWS EMR
- Experience with Data Lakes, Data Warehouses, ETL/ELT processes, and data modeling.
- Strong understanding of batch and streaming architectures.
- Experience with performance optimization, Git, testing practices, and production support.
Technical Stack
- Python
- SQL
- Apache Spark / PySpark
- Apache Kafka
- Apache Airflow
- AWS EMR
- Amazon S3
- Amazon Redshift
- Git
- Qlik
Originally posted on