Note: The job is a remote job and is open to candidates in USA. PeerIslands is seeking a Data Pipeline Engineer with strong expertise in Databricks, AWS Glue, Python, SQL, and MongoDB. The role involves designing, developing, and optimizing scalable data pipelines to deliver high-quality data for analytics and business applications.
Responsibilities
- Develop and maintain scalable ETL/data pipelines using Databricks and AWS Glue
- Build data ingestion and transformation workflows for structured and semi-structured data
- Write efficient PySpark, Python, and SQL code for large-scale data processing
- Integrate data from multiple sources, including MongoDB and relational databases
- Implement data quality, validation, and reconciliation checks
- Optimize pipeline performance, reliability, and scalability
- Collaborate with architects, analysts, and application teams to deliver data solutions
- Support production deployments, monitoring, and troubleshooting
Skills
- 5+ years of experience in Data Engineering or ETL development
- Strong hands-on experience with Databricks and PySpark
- Experience with AWS Glue and AWS data services
- Proficiency in Python and SQL
- Experience working with MongoDB
- Knowledge of ETL, data modeling, and data warehousing concepts
- Familiarity with Delta Lake/Lakehouse architecture is preferred
- Experience with Git and Agile development practices
- Delta Lake
- Apache Spark optimization
- AWS S3
- CI/CD pipelines
- Healthcare or Benefits domain experience
Company Overview
Company H1B Sponsorship