Note: The job is a remote job and is open to candidates in USA. Dice is seeking a Senior Data Engineer to design and implement data pipelines for API, streaming, and batch processing. The role involves collaborating with engineering and DevOps teams to optimize data workflows and ensure data quality in a remote setting.
Responsibilities
- Design data pipelines for API, streaming, and batch processing to facilitate data loads into the Snowflake data warehouse
- Collaborate with other engineering and DevOps team members to implement, test, deploy, and operate data pipelines and ETL solutions
- Develop scripts to Extract, Load and Transform data and other utility functions
- Optimize data pipelines, ETL processes, and data integrations for large-scale data analytics use cases
- Build necessary components to ensure data quality, monitoring, alerting, integrity, and governance standards are maintained in data processing workflows
- Able to navigate ambiguity and thrives in a fast-paced environment. Takes initiative and consistently delivers results with minimal supervision
- 7+ years of experience in building and maintaining data pipelines and ETL/ELT processes in data-centric organizations
- Strong coding skills using Python. Familiar with Python libraries related to data engineering and cloud services including pandas, boto, etc
- At least 3 years of experience with AWS S3, SQS, Kinesis, Lambda, AWS DMS, Glue/EMR, AWS Batch or similar services
- Hands-on experience building streaming and batch big data data pipelines
- Must have knowledge of building Infrastructure in AWS cloud using Cloud formation or Terraform
- Experience on Anakonda and Jupyter Notebook
- 3+ years of working experience with Snowflake cloud data warehouse including Snowflake data shares, Snowpipes, Snow SQL, Tasks etc
- Must have working knowledge of various databases, SQL and NoSQL
- Must have working knowledge of various file formats like CSV, Json Avro, and Parquet
- Hands-on experience with cloud platforms such as AWS and Google Cloud
- Experience working with agile development methodology
- Experienced in CI/CD and release processes, proficient in Git or other source control management systems, to streamline development and deployment workflows
- Minimum 5 years of designing and implementing operational production grade large-scale data pipelines, ETL/ELT and data integration solutions
- Exposure to multi-tenant/multi-customer environments is a big plus
- Hands on experience with productionized data ingestion and processing pipelines
- Strong understanding of Snowflake Internals and integration of Snowflake with other data processing and reporting technologies
- Experience working with structured, semi-structured, and unstructured data
- Familiarity with MongoDB or similar NoSQL database systems
Skills
- 12+ Years of experience
- 4-6 Years of USA experience
- Bachelor's or master's degree in Computer Science, Information Systems, or related field
- Design data pipelines for API, streaming, and batch processing to facilitate data loads into the Snowflake data warehouse
- Collaborate with other engineering and DevOps team members to implement, test, deploy, and operate data pipelines and ETL solutions
- Develop scripts to Extract, Load and Transform data and other utility functions
- Optimize data pipelines, ETL processes, and data integrations for large-scale data analytics use cases
- Build necessary components to ensure data quality, monitoring, alerting, integrity, and governance standards are maintained in data processing workflows
- Able to navigate ambiguity and thrives in a fast-paced environment. Takes initiative and consistently delivers results with minimal supervision
- 7+ years of experience in building and maintaining data pipelines and ETL/ELT processes in data-centric organizations
- Strong coding skills using Python. Familiar with Python libraries related to data engineering and cloud services including pandas, boto, etc
- At least 3 years of experience with AWS S3, SQS, Kinesis, Lambda, AWS DMS, Glue/EMR, AWS Batch or similar services
- Hands-on experience building streaming and batch big data data pipelines
- Must have knowledge of building Infrastructure in AWS cloud using Cloud formation or Terraform
- Experience on Anaconda and Jupyter Notebook
- 3+ years of working experience with Snowflake cloud data warehouse including Snowflake data shares, Snowpipes, Snow SQL, Tasks etc
- Must have working knowledge of various databases, SQL and NoSQL
- Must have working knowledge of various file formats like CSV, Json Avro, and Parquet
- Hands-on experience with cloud platforms such as AWS and Google Cloud
- Experience working with agile development methodology
- Experienced in CI/CD and release processes, proficient in Git or other source control management systems, to streamline development and deployment workflows
- Minimum 5 years of designing and implementing operational production grade large-scale data pipelines, ETL/ELT and data integration solutions
- Exposure to multi-tenant/multi-customer environments is a big plus
- Hands on experience with productionized data ingestion and processing pipelines
- Strong understanding of Snowflake Internals and integration of Snowflake with other data processing and reporting technologies
- Experience working with structured, semi-structured, and unstructured data
- Familiarity with MongoDB or similar NoSQL database systems
Company Overview