Note: The job is a remote job and is open to candidates in USA. Humana is a healthcare company focused on improving health outcomes through insurance and healthcare services. The Senior Data Engineer builds and evolves the Databricks Lakehouse data foundation for the NBA platform, developing scalable pipelines, feature engineering workflows, and real-time ingestion patterns. The role delivers governed, reliable data products for decision intelligence, machine learning, reinforcement learning, and AI workloads while partnering with engineering and data science teams.
Responsibilities
- Design, build, and maintain Bronze, Silver, and Gold lakehouse pipelines that power member profiles, clinical signals, engagement data, and decision intelligence use cases
- Develop and maintain Gold-layer feature tables and reusable data products that support model training, model scoring, reinforcement learning, and real-time decisioning
- Build and support batch and real-time ingestion pipelines using Databricks, Spark, Delta Lake, and event-driven data architectures
- Implement data quality validation, reconciliation, monitoring, alerting, and testing controls to ensure trusted and production-ready data assets
- Design scalable data models that balance usability, governance, performance, and long-term maintainability
- Tune Spark workloads, storage strategies, partitioning schemes, and query performance to improve efficiency, scalability, and cost management
- Follow established standards for data lineage, security, PHI handling, auditability, and regulatory compliance
- Partner with Decision Intelligence, AI Engineering, and Platform Engineering teams to ensure high-quality data is available across the NBA ecosystem
- Diagnose and resolve pipeline failures, data quality issues, and production incidents to maintain reliable platform operations
Skills
- 5+ years of data engineering experience building and operating production data platforms
- Strong SQL and Python skills with hands-on experience developing Spark-based data pipelines
- Experience with Databricks, Delta Lake, or comparable lakehouse platforms
- Experience implementing Medallion Architecture (Bronze/Silver/Gold) data patterns
- Experience building batch and streaming data processing pipelines
- Strong understanding of data modeling, pipeline testing, data quality controls, and operational support practices
- Familiarity with modern cloud-based data platforms and distributed data processing systems
- Strong communication skills and the ability to collaborate across engineering, analytics, and business teams
- Occasional travel to Humana's offices for training or meetings may be required
- Very minimal travel might be required for training, meetings, and/or conferences
- Typical business hours are Monday-Friday, 8 hours/day, 5 days/week-- some flexibility might be possible, depending on business needs
- Experience with Databricks Feature Store, Unity Catalog, Delta Live Tables, and Databricks Workflows
- Experience with Spark Structured Streaming and real-time feature engineering
- Experience with Kafka and event-driven data architectures
- Familiarity with machine learning, recommendation systems, reinforcement learning, or decision intelligence platforms
- Experience with Azure Data Factory, Azure Data Lake Storage, Azure Event Hubs, or similar cloud-native data services
- Experience with data observability and quality platforms such as Great Expectations, Monte Carlo, or equivalent tools
- Experience integrating external healthcare, claims, CMS, CDC, consumer, or social determinants of health datasets
- Background in healthcare, insurance, or another regulated industry with PHI/HIPAA handling requirements
- Work Style: Remote/Hybrid - Preferably Boston, MA
Benefits
- This job is eligible for a bonus incentive plan. This incentive opportunity is based upon company and/or individual performance.
- Medical benefits
- Dental benefits
- Vision benefits
- 401(k) retirement savings plan
- Paid time off
- Company holidays
- Personal holidays
- Paid parental leave
- Paid caregiver leave
- Short-term disability
- Long-term disability
- Life insurance
- Remote/Hybrid work arrangement
Company Overview
Company H1B Sponsorship