Note: The job is a remote job and is open to candidates in USA. AbarVa is hiring a Director, Data & AI Engineering to lead and build its US Databricks engineering practice while remaining hands-on in architecture and implementation. The role covers data ingestion, migration and recovery of legacy data estates, governance, AI-ready data products, and engineering team development.
Responsibilities
- Lead and build: build and lead the US Databricks engineering team — hire, set standards, review code, mentor and grow engineers — while staying hands-on in the hardest problems yourself
- Ingestion framework: design and build a metadata-driven ingestion framework so a new source is onboarded by configuration, not another hand-written pipeline — and hand it to the client’s team as a working asset
- Estate assessment & migration: assess existing SQL, ETL, stored-procedure and warehouse estates with Databricks Lakebridge — inventory code, score complexity and dependencies, accelerate conversion, reconcile legacy-vs-Databricks parity — then turn the assessment into a sized, wave-based build plan
- Lineage & recovery: trace lineage end to end and reconstruct undocumented business logic from analytics estates being decommissioned; validate against source code, job metadata and owners; record the unknowns
- Medallion & data products: design scalable Bronze/Silver/Gold architectures and data products — named owner, contract, quality thresholds, lineage, documented consumers — run as a backlog and release cadence, not project phases
- Governance: design and implement Unity Catalog governance — lineage, security, access control, tenant isolation, data-product patterns
- Engineering velocity: stand up CI/CD and the toolchain for speed — Databricks Asset Bundles, Git workflows, automated testing, environment promotion, IaC, and observability that catches a broken pipeline before the business does
- Data for AI: model and certify Gold data for natural-language consumption through Databricks Genie and Claude on Bedrock — clear semantics, defined metrics, described columns, certified tables
- Partner & scale: partner directly with client CTO, CDAO, architecture, security and engineering leaders, and help build AbarVa’s Data & AI engineering capability as we scale
Skills
- An active Databricks certification — Data Engineer Professional, Solutions Architect, or an equivalent current credential
- Proven experience building and leading data engineering teams — hiring, standards, code review, mentoring — while remaining personally hands-on
- 12–18+ years of hands-on data engineering across modern data architecture, modeling and delivery, with exceptional depth in Databricks, Spark, SQL and Python
- Demonstrated experience building metadata-driven ingestion frameworks — Auto Loader / Lakeflow, CDC, schema evolution, data-quality expectations and reconciliation
- Hands-on Databricks Lakebridge experience (or directly equivalent assessment/transpile/reconcile tooling) to analyze existing data loads, size migrations and prove parity
- Deep Unity Catalog, medallion and data-product architecture; strong command of reconciliation, data quality, identity, master and reference data
- Real CI/CD practice for data
- Strong cloud architecture experience
- Familiarity with legacy platforms — Netezza, Teradata, Exadata, SQL Server, SAS
- Preferably AWS
- You use AI coding assistants daily and can set the team's standard for using them well — including where they're unreliable
- Executive presence that holds a room with senior client leaders while keeping technical credibility with engineers
- MVP or Champion status
Company Overview