Note: The job is a remote job and is open to candidates in USA. Suncap Technology is seeking a Data Engineer with strong Azure and Databricks experience. The role involves architecting and designing data warehouse and analytics solutions, implementing ETL processes, and enhancing the core product platform through AI and data engineering practices.
Responsibilities
- Identify opportunities for Data Engineering and AI to enhance the core product platform, select the best machine learning techniques to the specific business problem and then build the models that solve the problem
- Architect and design DW and Analytics solutions and cloud services
- Own the end-end process, from recognizing the problem to implementing the solution
- Establish DataOps principles and best practices
- Design, develop, implement and maintain scalable ETL processes on Azure Cloud using Azure Synapse
- Implement systems and processes to monitor data quality
- Ad hoc analyses troubleshooting data-related issues
- Develop and maintain propriety data warehouse tools
- Offer technical guidance to internal teams and business partners
- Work with offshore team, business analysts and other data engineering team members to ensure alignment of requirements, methodologies and best practices
- Participate in scrum process and deliver stories/features according to the schedule
- Participate in product support activities as needed by the team
- Individual contributor and Technical lead for Data Engineering - Data pipelines, Data modelling and Data warehouse
Skills
- Must have strong Azure and Databricks experience
- Need to validate projects worked on, technologies used, and why/how they used the technologies
- Undergraduate degree or equivalent experience
- BE/BTech or equivalent experience
- BE/BTech or equivalent experience in Data engineering
- Advanced SQL knowledge, capable to write optimized queries for faster data workflows
- Proven work experience in Python, SQL, Any RDBMS
- Experience in designing solutions for multiple large data warehouses with a good understanding of cluster and parallel architecture as well as high-scale or distributed RDBMS
- Must be extremely well versed with handling large volume data and work using different tools to derive the required solution
- To be able to work in a fast-paced agile development environment
- AI/ Client Solution design
- Strong problem solving and troubleshooting skills with the ability to exercise mature judgment
- MLOps
- 5+ years of hands-on software engineering experience
- Strong knowledge of Azure Data Lake, Synapse, ADF, Power BI
- Strong knowledge of Data Modelling techniques like Data Vault 2.0
- Understanding of CI/CD framework is an added advantage
- Experience with Git / Azure DevOps
- Ability to interpret/write custom shell scripts. Python scripting is a plus
- Experience with AzureML
- Expert in Azure Synapse, Azure Container Registry, Azure App services
Company Overview