Data Engineer
Connecticut (Remote)
Contract - 1 Year
Job Description:
Review and translate requirements to end-end technical design with a goal of building data products
Develop pipelines in Databricks using PySpark and DBT to onboard various sources
Working knowledge in Kafka to stream the data into databricks platform
Identify Common approaches and best practices to data integration such as metadata drive ingestion, data validations, complex data type handling
Drive standards for reusability, while developing pipelines
Create or update documentation associated to key decisions on requirements, design and development
Liaise with architects, data owners and users for reviews on day-day basis to ensure the quality of deliverables are high.
Skill Proficiency Requirements:
8+ years of overall IT experience with 6 years as a Data Engineer
Experience in providing data engineering solutions
Working knowledge of below technology toolsets:
Cloud (Azure/AWS)
Databricks, Data Mesh, Data as a Product
Python
Data streaming (Kafka, EventHub, Kinesis, SNS)
Cloud-based API management – good to have
Immuta – good to have
Multi-cloud and on-prem application integration – good to have