Note: The job is a remote job and is open to candidates in USA. Optum is a global organization that delivers care aided by technology to help millions of people live healthier lives. The Senior Data Engineer will design, build, and optimize scalable data and AI platforms on Azure, developing enterprise data pipelines, real-time processing frameworks, and AI/ML model integrations. The role also focuses on data quality, governance, security, performance optimization, Agile development, and production support.
Responsibilities
- Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure
- Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake
- Collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows
- Develop and maintain feature stores, model input pipelines, and real-time/streaming frameworks
- Ensure data quality, governance, and security across the full data lifecycle
- Build reusable frameworks, accelerators, and automation scripts to improve engineering efficiency
- Optimize performance, scalability, and reliability of data workflows and batch/streaming pipelines
- Participate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelines
- Provide production support and on-call coverage, ensuring system stability and rapid issue resolution
- Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI
Skills
- • Bachelor's degree in Computer Science, Engineering, or IT related field
- • 6+ years of experience in Data Engineering with Python/PySpark
- • 6+ years of experience in building ETL/ELT pipelines using Databricks
- • 6+ years of experience working in Agile environments
- • 5+ years of strong experience in SQL / PL-SQL
- • 4+ years of experience with Azure Databricks and Delta Lake architecture
- • 4+ years of hands-on experience with CI/CD (GitHub Actions, Azure DevOps)
- • 3+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks)
- • 2+ years of experience with Databricks Delta Live Tables (DLT)
- • 2+ years of experience with unit testing, validation, and pipeline testing frameworks
- • Familiarity with medallion architecture and SCD2 implementations
- • AI builder: Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI
- • Experience building enterprise-scale data platforms
- • Strong skills in performance tuning and debugging large-scale pipelines
- • Experience with real-time/streaming frameworks (Structured Streaming)
- • Ability to work in distributed, cross-functional global teams
- • Exposure to GenAI tools (e.g., GitHub Copilot) for engineering productivity
- • Strong understanding of secure coding practices and vulnerability remediation
- • Proven ability to analyze logs, troubleshoot production issues, and optimize performance
- • Demonstrated capability to design and deploy AI-powered solutions responsibly
Benefits
- Flexibility to telecommute from anywhere within the U.S.
- A comprehensive benefits package
- Incentive and recognition programs
- Equity stock purchase
- 401k contribution
- Career development opportunities
Company Overview
Company H1B Sponsorship