Note: The job is a remote job and is open to candidates in USA. Agility Technologies Inc. is a minority-owned system integrator and cloud implementation company serving federal agencies and commercial customers through digital transformation and emerging technology solutions. The Databricks Platform Engineer will build, maintain, and optimize enterprise data lakehouse infrastructure, including Databricks workspaces, clusters, cloud deployments, monitoring, security, governance, and cost management. The role also supports data teams, automates platform operations, and troubleshoots Databricks workloads across development, staging, and production environments.
Responsibilities
- Build and maintain Databricks workspace infrastructure across development, staging, and production environments
- Design, implement, and maintain Databricks platforms in cloud environments (Azure, AWS, or GCP), ensuring scalability, reliability, and cost-efficiency. Manage and optimize Databricks clusters, including autoscaling, resource allocation, monitoring, and troubleshooting
- Oversee the deployment and configuration of Databricks workspaces, including user management, access controls, and workspace policies
- Collaborate with infrastructure and DevOps teams to implement CI/CD pipelines for deploying Databricks workflows and notebooks. Monitor and manage performance, logging, and system health of the Databricks platform. Implement robust data security and governance measures in compliance with organizational policies and standards
- Work with cross-functional teams to troubleshoot and resolve platform-related issues
- Automate repetitive tasks, such as provisioning, upgrades, and scaling, using Infrastructure-as-Code (IaC) tools like Terraform or CloudFormation
- Enable data engineers and data scientists by providing support for integrating Databricks with various data sources and tools
- Stay up-to-date with the latest Databricks features, updates, and best practices to ensure the platform meets evolving business needs
Skills
- 5+ years of experience in platform engineering, cloud infrastructure, or IT operations, with 2+ years of hands-on experience managing Databricks environments
- Strong knowledge of Databricks administration, including cluster setup, workspace management, and job orchestration
- Proficiency in Apache Spark for distributed data processing
- Hands-on experience with cloud platforms (Azure Government, AWS GovCloud, or GCP Public Sector)
- Proficiency in scripting and automation with Python, Bash, or PowerShell
- Experience with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or ARM templates
- Solid understanding of cloud networking concepts, such as VPCs, subnets, security groups, and private endpoints
- Familiarity with federal compliance frameworks, including FedRAMP, FISMA, NIST 800-53, and CMMC
- Experience implementing data security best practices, such as RBAC, encryption (at rest and in transit), and audit logging
- Ability to set up and use monitoring and logging tools (e.g., Datadog, AWS CloudWatch, or Azure Monitor) to ensure platform stability
- Proficiency in diagnosing and resolving cluster performance, job failures, and other technical issues
- Databricks Certified Associate or Platform Administrator certification
- Knowledge of Unity Catalog administration and data governance implementation
- Experience with cost optimization strategies for Databricks, including efficient resource allocation, autoscaling, and workload management
- Advanced knowledge of performance tuning for Databricks clusters and Spark jobs
- Implementation of fine-grained access controls and data masking to comply with federal data protection standards
- Experience implementing single sign-on (SSO), multi-factor authentication (MFA), and data retention policies for secure environments
- Hands-on experience building automation scripts for platform provisioning, monitoring, and maintenance
- Familiarity with automation frameworks and tools that integrate with cloud platforms (e.g., AWS Lambda, Azure Logic Apps, Google Cloud Functions)
- Knowledge of ETL/ELT pipelines and experience integrating Databricks with data warehousing solutions like Redshift, Snowflake, or BigQuery
- Familiarity with real-time data streaming technologies, such as Kafka, AWS Kinesis, or Azure Event Hubs
- Federal cloud certifications such as AWS Certified Solutions Architect – Associate/Professional, Azure Administrator Associate, or Google Cloud Associate Cloud Engineer
- Public Trust (or higher) clearance eligibility; prior federal client experience
Benefits
- 401(k) matching
- Dental insurance
- Health insurance
- Paid time off
- Parental leave
- Vision insurance
- Fully remote position
Company Overview
Company H1B Sponsorship