Note: The job is a remote job and is open to candidates in USA. PCI Professional Services is seeking a Databricks Platform Engineer for an upcoming contract. The role focuses on automating and managing Databricks platform operations, maintaining reliable cloud-based data lakehouse infrastructure, and optimizing platform performance and costs. The engineer will support data teams, implement CI/CD and infrastructure automation, and ensure strong security and governance across Databricks environments.
Responsibilities
- Build, maintain, and optimize the enterprise data lakehouse infrastructure
- Platform provisioning, monitoring, cost optimization, and providing technical support for Databricks workloads
- Build and maintain Databricks workspace infrastructure across development, staging, and production environments
- Design, implement, and maintain Databricks platforms in cloud environments (Azure, AWS, or GCP), ensuring scalability, reliability, and cost-efficiency. Manage and optimize Databricks clusters, including autoscaling, resource allocation, monitoring, and troubleshooting
- Oversee the deployment and configuration of Databricks workspaces, including user management, access controls, and workspace policies
- Collaborate with infrastructure and DevOps teams to implement CI/CD pipelines for deploying Databricks workflows and notebooks. Monitor and manage performance, logging, and system health of the Databricks platform. Implement robust data security and governance measures in compliance with organizational policies and standards
- Work with cross-functional teams to troubleshoot and resolve platform-related issues
- Automate repetitive tasks, such as provisioning, upgrades, and scaling, using Infrastructure-as-Code (IaC) tools like Terraform or CloudFormation
- Enable data engineers and data scientists by providing support for integrating Databricks with various data sources and tools
- Stay up-to-date with the latest Databricks features, updates, and best practices to ensure the platform meets evolving business needs
Skills
- * Bachelor's degree in related discipline
- * 5+ years of experience in platform engineering, cloud infrastructure, or IT operations, with 2+ years of hands-on experience managing Databricks environments
- * Public Trust (or higher) clearance eligibility
- * Strong knowledge of Databricks administration, including cluster setup, workspace management, and job orchestration
- * Proficiency in Apache Spark for distributed data processing
- * Hands-on experience with cloud platforms (Azure Government, AWS GovCloud, or GCP Public Sector)
- * Proficiency in scripting and automation with Python, Bash, or PowerShell
- * Experience with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or ARM templates
- * Solid understanding of cloud networking concepts, such as VPCs, subnets, security groups, and private endpoints
- * Familiarity with federal compliance frameworks, including FedRAMP, FISMA, NIST 800-53, and CMMC
- * Experience implementing data security best practices, such as RBAC, encryption (at rest and in transit), and audit logging
- * Ability to set up and use monitoring and logging tools (e.g., Datadog, AWS CloudWatch, or Azure Monitor) to ensure platform stability
- * Proficiency in diagnosing and resolving cluster performance, job failures, and other technical issues
- * Databricks Certified Associate or Platform Administrator certification
- * Knowledge of Unity Catalog administration and data governance implementation
- * Experience with cost optimization strategies for Databricks, including efficient resource allocation, autoscaling, and workload management
- * Advanced knowledge of performance tuning for Databricks clusters and Spark jobs
- * Implementation of fine-grained access controls and data masking to comply with federal data protection standards
- * Experience implementing single sign-on (SSO), multi-factor authentication (MFA), and data retention policies for secure environments
- * Hands-on experience building automation scripts for platform provisioning, monitoring, and maintenance
- * Familiarity with automation frameworks and tools that integrate with cloud platforms (e.g., AWS Lambda, Azure Logic Apps, Google Cloud Functions)
- * Knowledge of ETL/ELT pipelines and experience integrating Databricks with data warehousing solutions like Redshift, Snowflake, or BigQuery
- * Familiarity with real-time data streaming technologies, such as Kafka, AWS Kinesis, or Azure Event Hubs
- * Federal cloud certifications such as AWS Certified Solutions Architect – Associate/Professional, Azure Administrator Associate, or Google Cloud Associate Cloud Engineer
- * Prior federal client experience
Benefits
- Remote work arrangement
Company Overview