Position : Databricks Platform Architect
Location: New York, NY (Remote for now)
Duration: 12+ Months Contract
Job Description:
Key Responsibilities
Platform Engineering & Architecture
• Design, implement, and maintain Databricks platform architecture to support enterprise-scale data workloads
• Establish best practices for workspace setup, cluster management, and job orchestration
• Optimize platform performance, scalability, and cost efficiency
• Define and enforce standards for data engineering and analytics workloads within Databricks
Developer Enablement
• Support development teams by providing guidance, documentation, and reusable frameworks
• Enable developers with CI/CD pipelines, automation, and deployment standards
• Troubleshoot and resolve platform-related issues impacting development teams
• Provide technical mentorship and onboarding support to engineers using Databricks
Governance, Security & Compliance
• Implement and manage data governance, access controls, and security models (Unity Catalog, RBAC, etc.)
• Ensure compliance with enterprise data policies and regulatory requirements
• Monitor and audit platform usage, access patterns, and data lineage
Integration & Ecosystem Management
• Integrate Databricks with cloud platforms (AWS/Azure/Google Cloud Platform), data lakes, and enterprise systems
• Manage connectivity with tools such as Power BI, APIs, data ingestion frameworks, and orchestration tools
• Support ingestion and processing frameworks (e.g., batch, streaming, Delta Lake)
Operational Excellence
• Monitor platform health, performance, and usage metrics
• Implement automation for provisioning, monitoring, and scaling
• Drive continuous improvements in platform reliability and developer experience
Required Qualifications
Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
5+ years of experience in data engineering or platform engineering roles
3+ years of hands-on experience with Databricks (architecture, administration, or platform engineering)
Strong understanding of distributed data processing (Spark)
Experience with cloud platforms (Azure preferred, or AWS/Google Cloud Platform)
Knowledge of data lake architecture, Delta Lake, and modern data platforms
Experience implementing security, governance, and access control frameworks
Familiarity with CI/CD pipelines, DevOps practices, and infrastructure as code (Terraform, etc.)