Benefits:
Competitive salary
Paid time off
Training & development
Sr Cloud Data Engineer
Remote Position
Position Overview: We are seeking a highly skilled and experienced Sr Cloud Data Engineer to join our team for a Cloud Data Modernization project.
Key Responsibilities:
• Lead the migration of the ETLs from on-premises SQLServer based data warehouse to Azure Cloud and Snowflake.
• Design, develop, and implement data platform solutions using Databricks, Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, Azure Data Lake Storage Gen2 (ADLS Gen2), Blob Storage, and Snowflake.
• Review and analyze existing on-premises ETL processes developed in SSIS and T-SQL.
• Implement DevOps practices and CI/CD pipelines using GitActions.
• Collaborate with cross-functional teams to ensure seamless integration and data flow.
• Optimize and troubleshoot data pipelines and workflows.
• Ensure data security and compliance with industry standards.
Required Qualifications:
• Minimum of 6+ years of experience as a Cloud Data Engineer.
• Hands-on experience with Databricks, Azure Cloud data tools (ADF, SHIR, Logic Apps, ADLS Gen2, Blob Storage) and Snowflake.
• Strong experience in ETL development using on-premises databases and ETL technologies
• Experience with Python or other scripting languages for data processing.
• Proficiency in DevOps and CI/CD practices using GitActions.
• Experience with Agile methodologies.
• Excellent problem-solving skills and ability to work independently.
• Strong communication and collaboration skills.
• Strong analytical skills and attention to detail.
• Ability to adapt to new technologies and learn quickly.
• Experience with the application of AI/ML tools and models to data processing and ETL workloads
• Design and implement AI/ML-enabled data pipelines to improve data quality, anomaly detection, classification, forecasting, and operational insights.
• Leverage Databricks Machine Learning, MLflow, and cloud-native AI services to support machine learning workflows.
• Integrate Generative AI capabilities, Large Language Models (LLMs), and Azure OpenAI services into data engineering processes.
• Develop automated solutions for metadata management, data cataloging, code generation, data validation, and documentation using AI technologies.
• Build scalable feature engineering pipelines to support model training and inference workloads.
• Collaborate with Data Scientists and AI Engineers to operationalize machine learning models within enterprise data platforms.
• Implement MLOps practices for model versioning, deployment, monitoring, governance, and lifecycle management.
Preferred Qualifications:
• Experience with data modeling and database design.
• Knowledge of data governance and data quality best practices.
• Experience with development in Databricks for data engineering and analytics workloads.
• Familiarity with other cloud platforms (e.g., AWS, Google Cloud).
• Certification in Azure or Snowflake.