Note: The job is a remote job and is open to candidates in USA. Infinite Computer Solutions is seeking a Lead Data Engineer to build, maintain, and optimize scalable data pipelines on AWS using Snowflake. The role supports business intelligence, analytics, and reporting through data integration, transformation, and warehousing solutions.
Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using AWS services and Snowflake
- Build robust data ingestion frameworks to load data from multiple sources, including relational databases, APIs, flat files, and cloud storage
- Develop and optimize data models, schemas, and database objects in Snowflake
- Implement data transformation logic using SQL, Snowflake Stored Procedures, and Snowpark (preferred)
- Load structured and semi-structured data (CSV, JSON, XML, Parquet, Avro) into Snowflake
- Develop batch and incremental data processing pipelines using AWS services
- Optimize Snowflake performance through clustering, query tuning, partitioning, and warehouse optimization
- Integrate data from AWS S3 into Snowflake using Snowpipe, COPY INTO, or external stages
- Monitor, troubleshoot, and resolve ETL failures and performance bottlenecks
- Implement data quality checks, validation, and reconciliation processes
- Collaborate with business analysts, data architects, and application teams to deliver high-quality data solutions
- Follow coding standards, documentation practices, and data governance policies
- Participate in Agile ceremonies, code reviews, and production support activities
Skills
- 10+ years of experience in building and maintaining scalable data pipelines on the AWS Cloud using Snowflake
- Hands-on experience with AWS data services, ETL/ELT development, cloud data warehousing, and data integration
- Strong hands-on experience with Snowflake Cloud Data Platform
- Experience with Snowflake architecture, virtual warehouses, databases, schemas, stages, and file formats
- Strong SQL development skills with query optimization techniques
- Experience with Snowflake Tasks, Streams, Stored Procedures, Secure Views, and Time Travel
- Experience using Snowpipe and COPY INTO commands for automated data ingestion
- Amazon S3
- AWS Glue
- AWS Lambda
- AWS IAM
- Amazon CloudWatch
- AWS Secrets Manager
- Design and development of ETL/ELT workflows
- Data migration from on-premises systems to AWS Cloud
- Experience with incremental loads and Change Data Capture (CDC)
- SQL (Mandatory)
- Python (Mandatory)
- Git
- Jenkins, GitHub Actions, or AWS CodePipeline
- CSV
- JSON
- XML
- Parquet
- Avro
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
- 2–4 years of experience in Data Engineering
- Hands-on experience with Snowflake and AWS Cloud services
- Strong SQL and Python programming skills
- Experience developing cloud-based ETL pipelines
- Good understanding of data warehousing concepts and dimensional modeling
- Familiarity with Agile/Scrum development methodologies
- Strong analytical and problem-solving skills
- Excellent verbal and written communication skills
- Ability to work independently and within cross-functional teams
- Strong attention to detail and commitment to delivering high-quality solutions
- Ability to manage multiple priorities in a fast-paced environment
- Snowpark
- Amazon Redshift (good to have)
- Amazon EventBridge or Amazon SNS/SQS (preferred)
- Experience scheduling and orchestrating workflows using AWS Glue Workflows or Apache Airflow (preferred)
- PySpark (Preferred)
- AWS CodeCommit (preferred)
- Experience with Apache Airflow or Managed Workflows for Apache Airflow (MWAA)
- Experience with dbt (Data Build Tool)
- Exposure to Apache Spark or PySpark
- Knowledge of Amazon Redshift
- Experience integrating REST APIs
- Familiarity with Power BI, Tableau, or Amazon QuickSight
- Experience implementing CI/CD pipelines for data engineering solutions
- SnowPro Core Certification
- AWS Certified Data Engineer – Associate
- AWS Certified Solutions Architect – Associate
- Experience with Terraform or AWS CloudFormation
- Exposure to Kafka or Amazon Kinesis for streaming data
- Knowledge of DataOps and data governance best practices
Company Overview
Company H1B Sponsorship