Note: The job is a remote job and is open to candidates in USA. Avacend Inc is seeking a Senior Data Engineer to build and maintain scalable data infrastructure. The role focuses on developing Python and Azure Data Factory pipelines, loading and optimizing data in Snowflake, managing data security and quality, integrating external sources, and resolving data issues.
Responsibilities
- Develop and maintain robust data pipelines using Python and ADF
- Design and optimize ELT/ETL processes to load data into Snowflake from various sources
- Implement performance tuning and monitoring on Snowflake queries and tables
- Collaborate with data architects and analysts to understand data requirements and provide scalable solutions
- Automate recurring data ingestion and transformation tasks using scripting in Linux environments
- Manage data security, access controls, and data quality standards in Snowflake
- Schedule and monitor ADF pipelines for production loads and troubleshooting failures
- Integrate external APIs or databases as needed for data enrichment and ingestion
- Perform root cause analysis and provide timely resolutions to data issues
Skills
- 7+ years of experience in data engineering or backend data development
- Strong proficiency in Python for scripting, data manipulation, and API interaction
- Hands-on experience with Snowflake including schema design, SQL development, and performance tuning
- Practical experience with Azure Data Factory: pipelines, triggers, datasets, and linked services
- Proficiency with Linux/Unix command line, shell scripting, and cron jobs
- Experience working with large-scale datasets and cloud data platforms
- Strong understanding of data modeling concepts (star/snowflake schema)
- Familiarity with version control systems such as GitLab
- Experience with Azure services
- Knowledge of CI/CD pipelines in a data engineering context
- Background in data warehousing or big data solutions
Company Overview
Company H1B Sponsorship