Senior Data Engineer
Location: 100% remote, must be eastern time zone since the start time is 6AM Eastern to accommodate for the team being in the UK
Required Skills:
• Azure (Databricks, Data Factory, Synapse, Data Lake) – They do not need to have all the Azure data tools
• Python and Spark
• Strong SQL skills
Key Accountabilities:
• Architects, designs, implements and maintains reliable and scalable data infrastructure
• Writes, deploys and maintains software to build, integrate, manage, maintain, and quality-assure data
• Design, develop, and deliver large-scale data ingestion, data processing, and data transformation projects on Azure
• Mentor and share knowledge with customers as well as provide architecture reviews, discussions, and prototypes
• Work with customers to deploy, manage, and audit best practices for cloud products
• Adheres to and advocates for software engineering best practices (e.g. technical design and review, unit testing, monitoring, alerting, checking in code, code review, documentation)
• Deploy secure and well-tested software that meets privacy and compliance requirements; develops, maintains and improves CI / CD pipeline
• Service reliability and following site-reliability engineering best practices: on-call rotations for services they maintain, responsible for defining and maintaining SLAs. Design, build, deploy and maintain infrastructure as code. Containerizes server deployments
• Actively contributes to improve developer velocity
• Work closely with other data engineers, software engineers, data scientists, data managers and business partners
Desirable Education and Experience:
• BS degree in computer science or related field
Required Criteria
• Deep and hands-on experience (typically 5+ years) designing, planning, productionizing, maintaining and documenting reliable and scalable data infrastructure and data products in complex environments
• Hands on experience with:
• Databricks and using Spark for data processing
• Configuring Delta Lake on Azure Databricks
• Python, Scala, SQL
• C#, ASP.NET, MVC,.NET Core,.Net Framework (4.6) JSON, and API development (optional)
• Azure Data Factory
• Azure Data Lake, Azure SQL DB, Synapse, and Cosmos DB
• Data Management Gateway, Azure Storage Options, Stream Analytics and Event Hubs
• Designing data solutions in Azure incl.