Note: The job is a remote job and is open to candidates in USA. Optum is hiring a Senior Data Engineer to build, optimize, and maintain scalable data architectures and pipelines for healthcare data solutions. The role delivers governed cloud data platforms, supports analytics and AI/ML use cases, and promotes data quality, security, and reliable production operations.
Responsibilities
- Lead the design and delivery of scalable, governed data engineering solutions, including technical direction, implementation planning, risk management, and production readiness for data products used by business, analytics, and AI/ML stakeholders
- Develop and manage robust ETL/ELT pipelines for data ingestion, transformation, and integration from heterogeneous sources, including relational databases, APIs, file shares, and end-user-computing assets
- Develop Python and PySpark solutions for distributed data processing, automation, integration, and reusable transformation logic
- Optimize data platform performance and cloud consumption through SQL tuning, Spark job optimization, storage format selection, partitioning strategies, cluster configuration, and workload scheduling
- Build and optimize workflows on Azure Cloud services and Databricks - including medallion (bronze/silver/gold) layering and orchestrated pipelines - to support enterprise data needs
- Migrate business logic embedded in legacy and end-user-computing systems into documented, testable transformation logic over governed tables
- Define and implement data quality, validation, reconciliation, and parity testing practices to ensure migrated pipelines are accurate, reliable, auditable, and consistent with the systems they replace
- Establish data lineage, ownership, and cataloguing, and define documented rollup and allocation rules where source systems operate at differing grains
- Implement security and compliance measures for handling PHI/PII and confidential business data in accordance with HIPAA or similar regulations, including de-identification and masking for controlled sharing
- Build and improve CI/CD, version control, and automated testing practices for data pipelines
- Implement monitoring, alerting, logging, and operational runbooks for data pipelines to ensure reliability, timely issue detection, incident response, and continuous improvement of production data platforms
- Support AI/ML and advanced analytical workflows by provisioning trusted data, scalable infrastructure, and governed consumption patterns for data scientists, analysts, and business partners
- Design and publish reusable data products, data services, APIs, and consumption layers that enable downstream applications, analytics platforms, and AI/ML use cases to consume trusted data consistently
- Partner with product owners, business stakeholders, architects, data scientists, analysts, and engineering teams to translate requirements into scalable data solutions and communicate technical tradeoffs clearly
- Design, develop, and deploy AI-enabled data solutions that address complex business challenges, support responsible AI use, and leverage AI-powered development tools to improve engineering productivity and delivery quality
- Mentor data engineers and analysts through design reviews, code reviews, best-practice guidance, and support for improving team delivery quality
Skills
- 6+ years of overall experience in data engineering, data warehousing, distributed data processing, or enterprise data platform delivery
- 6+ years of experience with SQL, data modeling, and performance tuning for large-scale systems
- 3+ years of experience with ETL/ELT frameworks, pipeline orchestration, and heterogeneous source-system integration
- 3+ years of experience with Python for data processing, automation, and integration
- 3+ years of experience with Databricks, Spark/PySpark, and Delta Lake
- 3+ years of experience with public cloud platforms such as Azure or GCP
- 1+ year of experience in integrating data from heterogeneous source systems
- 1+ year of experience with data quality, validation, reconciliation, observability, and production support practices for large-scale data pipelines
- 1+ year of experience with Git, CI/CD platforms, automated testing frameworks, release management, and deployment practices for production data pipelines
- 1+ year of experience in AI-powered development tools responsibly to improve engineering productivity and delivery quality
- Experience with batch and near-real-time data processing patterns, including event-driven ingestion, streaming pipelines, or comparable technologies where business needs require timely data availability
- Experience with Azure Data Factory or comparable orchestration tooling such as Airflow, dbt, or similar
- Experience migrating legacy or end-user-computing (spreadsheet- or macro-based) business logic into governed data pipelines
- Experience with data governance, lineage, and catalog tooling (Unity Catalog, Microsoft Purview, or similar)
- Demonstrated familiarity with AI/ML frameworks and supporting infrastructure for data science workflows
- Experience working with PHI/PII and restricted client data in compliance with HIPAA or similar regulations, including de-identification and masking
- Healthcare payer, actuarial, pricing, or financial-modelling domain experience
Benefits
- A comprehensive benefits package (subject to eligibility requirements)
- Incentive and recognition programs (subject to eligibility requirements)
- Equity stock purchase (subject to eligibility requirements)
- 401k contribution (subject to eligibility requirements)
- Remote work from anywhere within the U.S.; hires in the Minneapolis or Washington, D.C. area are required to work in the office a minimum of four days per week
- Development for other roles you may be interested in
Company Overview
Company H1B Sponsorship