Note: The job is a remote job and is open to candidates in USA. Diverse Lynx is hiring a Senior Data Engineer to support scalable data platforms and pipelines. The role focuses on designing, building, governing, and optimizing ETL/ELT pipelines, data integrations, Airflow workflows, and PySpark-based processing for analytics, reporting, AI/ML, and business operations.
Responsibilities
- Design, build, and maintain scalable ETL/ELT pipelines
- Automate data ingestion from multiple data sources
- Ensure reliable, high-performance, and scalable data movement
- Develop and Eligible to workimize PySpark-based data processing frameworks
- Build and manage Airflow workflows and orchestration pipelines
- Work with Hive and Kafka-based batch and streaming data solutions
- Implement data quality, monitoring, governance, and operational controls
- Support production incidents, performance tuning, and continuous improvement initiatives
- The Data Engineer will be responsible for designing, building, governing, and Eligible to workimizing data platforms, pipelines, and integrations to ensure secure, reliable, and scalable data availability for analytics, reporting, AI/ML, and business operations
Skills
- Google Cloud Platform (Eligible to workP)
- PySpark
- Python
- Airflow
- Hive
- Kafka
- Strong Functional Knowledge in Retail (Search & Personalization) Added advantage
- Strong experience with large-scale Eligible to workP Data Engineering solutions
- Hands-on experience with Airflow DAG development and support
- Experience building and Eligible to workimizing Spark/PySpark data pipelines
- Exposure to Hive-based data processing and Kafka-driven ingestion frameworks
- Experience supporting analytics, personalization, recommendation, or search platforms is highly desirable
Company Overview