Senior Data Architect
FullTIme
Remote (US)
Required Experience : Data Platform, Integration, Azure Databricks & Lakehouse Specialist
Professional Summary
Strong hands-on background in PySpark, Medallion Architecture, Delta Live Tables, Unity Catalog, and MongoDB Atlas integration. Familiar with Databricks Genie for AI-assisted analytics. An effective technical leader who sets architectural direction, mentors teams, and drives delivery. Exposure to Power BI and reporting solutions is a plus, not a core requirement.
Core Skills
Databricks: Medallion Architecture (Bronze/Silver/Gold), Delta Lake, Delta Live Tables (DLT), Auto Loader, Unity Catalog, Databricks Workflows, Databricks Apps, Databricks Genie
Languages: PySpark (expert), Python (PEP 8/PEP 20), SQL
Data Platforms: Azure Databricks, ADLS, Azure Synapse, Snowflake
Databases & Integration: MongoDB Atlas, SQL Server (JDBC), REST APIs, OAuth (SharePoint), cloudfiles
Governance: Unity Catalog - metastores, catalogs, schemas, RBAC, lineage, access policies
DevOps: Git, CI/CD for Databricks workflows and DLT pipelines, JSON/YAML config management
Reporting (nice-to-have): Power BI, DAX, Power Platform
Key Experience
Medallion Architecture & Databricks Pipelines
Designed and implemented end-to-end Lakehouse solutions on Azure Databricks across Bronze/Silver/Gold layers with schema enforcement and data quality checks
Delivered production-grade DLT pipelines and Auto Loader streaming ingestion from ADLS and external sources
Optimised PySpark jobs for performance and cost - partition tuning, caching, modular function-based code design
Unity Catalog & Governance
Implemented Unity Catalog as the foundational governance layer - metastores, catalogs, schemas, fine-grained RBAC, lineage, and audit controls
Standardised metadata management and data discovery across the platform
Data Architecture & Integration
Architected data flows from MongoDB Atlas, JDBC (SQL Server), REST APIs, and OAuth sources into the enterprise data platform
Established architectural standards for ingestion, transformation, and consumption layers; led architectural reviews across squads
Built configuration-driven (JSON/YAML) pipeline frameworks enabling scalable onboarding of new data sources
Databricks Genie
Familiar with Databricks Genie for enabling natural language querying of data assets and AI-assisted analytics for business users
Leadership
Mentored engineers on Databricks, PySpark, and Python best practices (PEP 8/PEP 20, naming conventions, modular design)
Conducted code reviews, enforced coding standards, and drove CI/CD adoption for Databricks workflows and DLT pipelines