Note: The job is a remote job and is open to candidates in USA. Blu Omega is seeking a Senior Data Engineer to support a federal public health analytics program. The role designs, builds, operates, and modernizes enterprise-scale data platforms and pipelines using cloud-native technologies, Databricks, and Palantir Foundry, while providing technical leadership and partnering with stakeholders.
Responsibilities
- Design, build, and maintain enterprise-scale data pipelines and platform components using Databricks, Palantir Foundry, and cloud-native technologies
- Architect robust data solutions through schema design, medallion architecture, partitioning, and distributed processing
- Develop and optimize data transformations with PySpark, Python, and SQL to enhance pipeline reliability and performance
- Build and operate Databricks solutions utilizing Delta Lake, Unity Catalog, and job orchestration capabilities
- Develop data pipelines, transformations, datasets, and data models within Palantir Foundry
- Implement cloud-native data orchestration using Azure Data Factory, Azure Functions, and Azure Storage, or equivalent AWS and GCP services
- Apply CI/CD practices for data services, including automated testing, validation, version control, and deployment
- Lead onboarding and migration efforts for data platforms, facilitating team adoption of modern data solutions
- Modernize existing Python and SQL codebases, promoting reusable engineering patterns and libraries
- Optimize data pipelines, strategies, and queries for performance, reliability, and cost-efficiency
- Establish best practices for data quality, observability, security, and governance
- Partner with stakeholders to gather requirements and translate program needs into scalable data solutions
- Communicate technical decisions and tradeoffs effectively to technical and non-technical audiences
- Provide technical leadership and supervision within cross-functional engineering teams
Skills
- 5+ years of experience with PySpark, SQL, distributed data processing, and enterprise data engineering
- 5+ years of experience designing and maintaining end-to-end data pipelines
- 5+ years of experience with data architecture, including schema design, medallion architecture, and optimization
- 2+ years of hands-on Databricks experience, including Delta Lake, Unity Catalog, and orchestration tools
- 2+ years of cloud platform experience, including Azure Data Factory, Azure Functions, and Azure Storage, or equivalent AWS/GCP services
- Hands-on experience with Palantir Foundry, focusing on pipeline development and data modeling
- Strong skills in Python, PySpark, and SQL development
- Proven ability to lead technical deliverables and mentor team members
- Excellent analytical, troubleshooting, and performance tuning skills
- Effective communication skills for diverse audiences
- Ability to obtain and maintain a Public Trust or Suitability/Fitness determination
- Bachelor's degree in a related field
- Experience implementing CI/CD pipelines using Azure DevOps or similar tools
- Knowledge of generative AI technologies, including LLMs, RAG, and embeddings
- Development or support of AI-driven solutions
- Familiarity with data quality, observability, monitoring, security, and enterprise data governance practices
- Experience supporting federal healthcare, public health, or mission-driven data environments
- Experience leading data platform modernization or migration initiatives
- Knowledge of Agile, DevOps, and DataOps methodologies
Benefits
- Medical, Dental, and Vision coverage through national providers
- 401(k) with company match (eligible after 6 months; vesting applies)
- Company-paid Life and AD&D insurance, voluntary options available
- Short-term and long-term disability options
- Employee Assistance Program (EAP) with 24/7 confidential support
- Telehealth and virtual care services
- Pet insurance, legal services, and identity theft protection options
- Paid Time Off (PTO) and federal holidays for eligible employees
- Wellness programs, discounts, and lifestyle benefits
Company Overview