Note: The job is a remote job and is open to candidates in USA. BlueFlag LLC is hiring Senior Databricks Engineers to help Department of Veterans Affairs teams migrate data workloads to Databricks on Azure Commercial. The role involves assessing and rebuilding on-premises workloads, developing governed data pipelines and machine learning or analytics solutions, and guiding customers through workshops, training, and documentation.
Responsibilities
- Work directly with VA teams to scope use cases, plan migrations, and deliver working solutions on Databricks
- Assess on-premises workloads (SQL Server, SSIS packages, stored procedures, scheduled jobs) and adapt them to Databricks patterns in Azure Commercial
- Build and tune pipelines with PySpark, Spark SQL, Delta Lake, Lakeflow, and Azure Data Factory
- Use Unity Catalog for access, lineage, and sharing, so customer data stays governed as it moves
- Teach customers through workshops, pairing, and clear documentation so they can own their workloads
- Package reusable templates and code (Git, CI/CD, Databricks Asset Bundles) that speed up the next migration
- Build end-to-end ML workflows: feature engineering, training, evaluation, MLflow tracking, the Unity Catalog model registry, and model serving
- Help data scientists move models from local notebooks and on-premises servers into governed, monitored production
- Support generative AI use cases, such as retrieval-augmented generation (RAG) and agents, alongside predictive models
- Convert SSIS and stored procedure ETL into Databricks pipelines, with reconciliation that proves the results match
- Replace on-premises reporting (SSRS reports, SSAS cubes) with Databricks SQL, AI/BI dashboards, and Power BI
- Publish reporting data from Databricks SQL warehouses, including semantic models and query performance tuning
Skills
- 7+ years in data engineering, including 2+ years building production workloads on Databricks
- Strong proficiency in Python, SQL, PySpark, and Spark SQL
- Hands-on experience with Delta Lake, lakehouse (medallion) design, and Spark performance tuning
- Hands-on Unity Catalog experience: catalogs, grants, lineage, and row- and column-level security
- Experience migrating on-premises workloads (e.g., SQL Server, SSIS, stored procedures) to the cloud
- Experience with Azure services, including ADLS Gen2, Azure Data Factory, Entra ID, and Key Vault
- Experience with Git, CI/CD, and infrastructure as code (e.g., Databricks Asset Bundles, Terraform)
- Customer-facing consulting or professional services experience, including requirements sessions, workshops, and training
- Clear written and verbal communication with technical and non-technical audiences
- Bachelor's degree in computer science, information systems, or a related field
- US Citizen: Must be a citizen of the United States
- Security Clearance: Must be able to obtain a public trust clearance
- Must be eligible to work in the United States
- Experience training, evaluating, and deploying models with MLflow and common frameworks (e.g., scikit-learn, XGBoost, PyTorch)
- Experience with Databricks SQL and Power BI data modeling (star schemas, semantic models, DAX)
- Databricks certifications (Data Engineer Professional; Machine Learning Professional or Data Analyst Associate by focus)
- Experience with healthcare data (HL7, FHIR, OMOP) and HIPAA requirements for PHI
- Experience in Azure Government or other FedRAMP High environments
- Experience upgrading Hive metastore workloads to Unity Catalog (e.g., UCX)
- AI / ML focus: Mosaic AI Vector Search, Agent Framework, or model monitoring
- Analytics focus: Microsoft Fabric, Pyramid Analytics, or Genie
Benefits
- Generous annual leave and paid holidays
- Comprehensive group health and dental plans
- 401(k) with company match
- Life insurance and AD&D coverage
- Ongoing training and professional development opportunities
Company Overview