Note: The job is a remote job and is open to candidates in USA. Zeal, an AI-enabled software consultancy now part of Livefront, helps Fortune 1000 companies achieve business outcomes through technology, talent, and innovative processes. The Data Engineer will build, maintain, monitor, and troubleshoot data pipelines and implement data architectures that support migration, transformation, quality, governance, and business requirements.
Responsibilities
- Gather and understand requirements for data migration and transformation
- Design, build, and maintain performant, resilient, and maintainable data pipelines that meet the business requirements
- Collaborate with stakeholders to ensure that designs and implementations meet requirements and project standards while fulfilling the business needs, providing feedback and proposals for alternative implementations where appropriate
- Implement the required data architectures and designs using the appropriate tools and in a manner that is performant, effective, efficient, and maintainable
- Shepherd the pipelines through the review and deployment process to ensure a proper implementation in production (testing, version control, and CI/CD)
- Monitor new and existing pipelines to ensure that they are meeting requirements and expectations
- Troubleshoot and remediate new and existing pipelines to ensure continued stability and effectiveness
- Document and transfer knowledge as required and appropriate within the project team
- Maintain awareness and understanding of the business and technical requirements and the larger project in order to provide value beyond your specific responsibilities
- Actively engage in project meetings and discussions to build context beyond your immediate tasks
- Provide input and feedback to the architecture teams on the designs with an eye towards improvement
- Ensure data quality, integrity, and availability with upstream and downstream systems
- Contribute to data governance and best practices for data management
Skills
- You specialize in building data pipelines that move and transform data, and you bring basic capabilities across the full spectrum of data skills
- You communicate and interact with stakeholders in a professional manner, and you take pride in pipelines that are performant, resilient, and maintainable
- You are curious about new tools—including AI tools that improve your productivity
- We expect candidates to demonstrate a solid foundation in data engineering and a keen interest in consulting
- Strong SQL skills - including an understanding of performance and parallelism
- At least one year of material experience on a traditional OLTP database (Oracle, SQL Server, Postgres, etc.)
- At least one year of material experience on a modern, MPP, data platform (Databricks, Snowflake, Azure Synapse, Redshift, BigQuery)
- Familiarity with cloud platforms including the basics of storage and security
- Material experience with at least one data transformation tool (Fivetran, Informatica, DataStage, dbt, or the native tools of one of the data platforms) or the native transformation tools of the cloud providers
- A basic understanding of Medallion data architectures
- Basic skills with orchestration (Airflow, etc.), source code control (git) and CI/CD
- Effective writing and communication skills
- The ability to use AI tools to improve productivity
- Data modeling experience on modern data platforms
- Python proficiency
- Experience with queuing and event-driven distribution systems (Kafka, RabbitMQ, native tools of cloud platforms)
- Data presentation and visualization tools (Power BI, Tableau, etc.)
- Specific experience with ELT on Snowflake or Databricks
- Experience with testing methodologies and tools
Company Overview
Company H1B Sponsorship