Note: The job is a remote job and is open to candidates in USA. KinderCare Learning Companies is hiring a Senior Databricks Engineer to design, optimize, and govern its Databricks-based medallion lakehouse platform. The role builds reliable data pipelines for enterprise BI and AI/ML products, maintains security and SOX-aligned practices, and mentors engineers while partnering with data and business teams.
Responsibilities
- Design, build, and optimize production-grade ETL/ELT pipelines across the medallion architecture using Delta Lake, Delta Live Tables / Lakeflow Declarative Pipelines, and Databricks Workflows
- Own performance tuning and cost efficiency across the platform — cluster/job sizing, Photon, partitioning and Z-ordering, Liquid Clustering, Auto Loader, and DBU cost governance
- Architect and enforce data models that support enterprise BI (Microsoft Fabric/Power BI) and downstream analytics products
- Build and maintain CI/CD pipelines for Databricks assets (Databricks Asset Bundles, Repos, Git-based deployment) following Agile and Test-Driven Development practices
- Integrate platform pipelines with middleware and source systems (e.g., Boomi) and cloud-native services
- Administer and evolve Unity Catalog: catalogs/schemas, fine-grained access control, lineage, row/column-level security, and workspace-catalog bindings
- Implement and enforce data quality, observability, and reliability practices (expectations/constraints, monitoring, alerting, SLA management) across pipelines
- Partner with security, compliance, and audit teams to maintain SOX ITGC alignment; access reviews, change control, and auditable engineering practices
- Troubleshoot and resolve complex production data pipeline issues, performing root-cause analysis and implementing preventive fixes
- Create clear, durable documentation of architecture, procedures, and operational runbooks
- Evaluate, pilot, and productionize Databricks-native AI/ML capabilities — including Genie for natural-language data access and MLflow for experiment tracking and model lifecycle management, Feature Store, and Model Serving
- Support the build-out of vector search and retrieval-augmented generation (RAG) patterns on top of governed Unity Catalog data for internal AI use cases
- Collaborate with data science and analytics stakeholders to prepare curated, ML-ready Gold-layer datasets and feature pipelines
- Stay current on the Databricks roadmap (Lakehouse AI, Mosaic AI, Agent frameworks) and recommend adoption where it advances platform maturity and business value
- Help define guardrails and human-in-the-loop controls for AI-assisted and agentic engineering workflows introduced to the platform
- Serve as a technical mentor to mid-level data engineers, raising the bar on Databricks best practices, code quality, and architectural rigor
- Partner with the Data Architect and Data Engineering Lead to translate business requirements into technical specifications for BI and AI products
- Collaborate cross-functionally with technical and non-technical stakeholders, including product, security, and business teams
- Contribute to data governance, data security, and data privacy standards across the platform
Skills
- Bachelor's degree in computer science, information systems, engineering, statistics, or related field or equivalent work experience
- 7+ years of experience as a data engineer with 3+ years specifically architecting and operating production workloads on Databricks
- Strong understanding of data governance, data security, and access control best practices
- Experience with Agile development methodologies, CI/CD automation, and Test-Driven Development
- Excellent problem-solving skills and the ability to lead technical troubleshooting independently
- Strong written and verbal communication skills with the ability to explain technical concepts to non-technical stakeholders
- Demonstrated experience owning a lakehouse/medallion architecture at scale, including data modeling for BI consumption
- Experience operating in a governed or regulated environment (SOX, HIPAA, or similar) with formal change control and access governance
- Deep, hands-on expertise with the Databricks Lakehouse Platform: Delta Lake, Unity Catalog, Delta Live Tables / Lakeflow, Workflows, and cluster/job optimization (Photon, Auto Loader, Liquid Clustering)
- Advanced SQL and strong Python (PySpark) development skills
- Experience with Databricks Asset Bundles, Repos, and CI/CD for lakehouse deployments
- Working knowledge of cloud data services (Azure preferred — ADLS, Azure SQL, Synapse/Fabric; AWS/GCP equivalents acceptable) and cloud migration patterns
- Hands-on experience with MLflow for experiment tracking, model registry, and lifecycle management
- Working knowledge of Databricks AI/ML capabilities — Feature Store, Model Serving, Genie, Mosaic AI, or equivalent lakehouse ML tooling
- Exposure to vector search, embeddings, or RAG architectures and an understanding of how governed data feeds AI/ML products
- Comfort partnering with data science teams on ML-ready data pipelines, even without a formal data science background
- Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Machine Learning Associate/Professional) strongly preferred
- Comfort with Scala
- Familiarity with BI integration layers such as Microsoft Fabric/Power BI and enterprise middleware (e.g., Boomi) a plus
Benefits
- Discounted child care benefits
- Medical, dental, and vision benefits for your family (and pets, too!)
- Employee assistance programs
- Health and wellness programs
- Paid time off
- Discounts for work necessities, such as cell phones
- Remote opportunity
Company Overview