Note: The job is a remote job and is open to candidates in USA. Pyx Health is a healthcare technology company developing solutions supported by data and analytics. The Healthcare Data Engineer will build and maintain Azure-based data infrastructure, develop reliable pipelines and Airflow workflows, and implement data models, quality monitoring, and governance practices while collaborating with technical and business teams.
Responsibilities
- Build and maintain data pipelines for ingesting, transforming, and cleaning healthcare data in the Azure cloud using Databricks, PySpark, and Delta Lake
- Implement pipeline logic from defined specifications, with guidance from senior engineers on architectural decisions
- Build reusable testing frameworks to ensure pipeline reliability
- Monitor pipelines for failures and performance issues, escalating complex problems appropriately
- Develop and maintain Airflow DAGs with appropriate error handling and retry logic
- Support deployment and configuration of pipelines via Astronomer on Azure using the Astro CLI
- Contribute to improving pipeline reliability and reducing manual intervention
- Implement data models for efficient storage and retrieval using Delta Lake, including merge/upsert patterns
- Design and optimize Delta Lake tables for analytic workloads
- Leverage Unity Catalog for data governance, organization, and security
- Apply Change Data Capture (CDC) patterns under the direction of senior team members
- Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and ETL support
- Work within the Azure ecosystem including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps
- Follow established security and scalability standards when building data infrastructure
- Support infrastructure tasks with guidance from senior engineers on architecture
- Ensure scripts and datasets are well-documented to support enterprise data governance
- Design, implement and continuously improve automated data quality monitoring, reconciliation, and alerting to ensure reliable downstream reporting
- Troubleshoot and resolve pipeline failures, documenting root causes and resolutions
- Work closely with data scientists, analysts, and business stakeholders to understand reporting requirements, troubleshoot issues, and improve data assets
- Participate in code reviews, incorporating feedback to improve code quality
- Document pipelines, processes, and implementation decisions clearly and consistently
- Stay current with data engineering technologies, particularly in the healthcare space
Skills
- Only candidates residing in the USA may apply
- • 2–4 years of experience as a Data Engineer or in a closely related data role
- • Hands-on experience with Azure cloud services (ADLS, Databricks, or similar)
- • Working knowledge of SQL for scripting and data modeling including T-SQL, Spark, and Databricks SQL
- • Ability to contribute to technical projects with moderate oversight
- • Strong communication skills — comfortable asking questions and providing updates to cross-functional partners
- • Familiarity with CI/CD concepts, Git, and version control workflows
- • Solid problem-solving skills with a systematic approach to debugging
- • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience)
- • **Databricks SQL (Mid):**Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts
- • **Python (Mid):**Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience
- • **Databricks Spark Notebooks (Mid):**Notebook-based development, basic cluster usage, Delta table operations
- • **Airflow Python Development (Mid):**Ability to write and maintain DAGs; understanding of error handling and retry patterns
- • **Airflow Astro Configuration (Foundational):**Familiarity with Astronomer or willingness to learn; basic Astro CLI usage
- • **Azure Ecosystem (Foundational):**Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps
- • **T-SQL (Foundational–Mid):**Basic stored procedures, SQL Server querying, and data manipulation
- • **Azure Data Factory (Foundational):**Basic familiarity with pipeline authoring and triggers
- • **Git + Azure DevOps CI/CD (Foundational):**Experience with branching, pull requests, peer review processes, and version control best practices as well as deploying production data pipelines using CI/CD workflows
- • Familiarity with healthcare data standards and regulations (HIPAA, HL7, etc.) is a plus
Company Overview