Note: The job is a remote job and is open to candidates in USA. Value Innovation Labs is hiring a Senior Data and Infrastructure Engineer to design, build, and deploy computing resources and data ingestion pipelines. The role applies software development best practices to data pipelines and enterprise data applications, while providing technical guidance and ensuring computational and data resources are documented.
Responsibilities
- Establish best-practice design patterns for data pipelines
- Provision, create, and operate VMs and Kubernetes clusters for load-balanced data processing for large data sets (~10 TB)
- Collaborate with business and data science personnel to define strategy and approach for data pipeline testing
- Define organization structure for code repository
- Establish design patterns for orchestrating pipeline execution
- Troubleshoot complex data-related issues including data reconciliation across sources and governance
- The Senior Data and Infrastructure Engineer provides expert-level knowledge in data integration techniques and computation infrastructure to design, build, and deploy computing resources (VMs, Kubernetes clusters, local Git environments) utilizing on-premises devices and data ingestion pipelines
- This role's focus is applying best practices of software development to the design, implementation, automation, and optimization of data pipelines and enterprise-wide data-centric applications (e.g., data lakes, data warehouses) for large and/or challenging data sets
- This role will provide guidance, oversight, and technical thought leadership in both the computational infrastructure and data pipeline engineering spaces, including ensuring that all computational and data resources are well documented
Skills
- 8+ years of experience in Data Engineering and in delivering complex projects
- Expert-level data engineering knowledge including ELT/ETL, Data Warehousing
- Strong Python experience for data processing
- Experience with computational infrastructure provisioning (VMs and Kubernetes clusters) and deployment, preferably within ecosystems operating on .NET frameworks
- Strong SQL writing and optimizing skills for relational database platforms
- Extensive experience with managing pipelines that process large volumes of data
- Extensive experience with workflow orchestration/scheduling solutions
Company Overview