Note: The job is a remote job and is open to candidates in USA. Genesis10 is a staffing firm seeking a Senior Cloud Engineer to support a major insurance company’s enterprise data and AI platform. The role owns Azure infrastructure and platform engineering across Databricks, ADLS Gen2, Azure Data Factory, and Azure Machine Learning, including Terraform-based provisioning, networking, security, observability, cost management, and CI/CD. It also establishes Databricks platform standards and provides senior technical partnership on administration, architecture, and complex escalations.
Responsibilities
- Design, implement, and administer the Azure foundation the platform runs on
- Infrastructure as Code (Terraform-first): Author and maintain the reusable Terraform modules that provision the Azure data estate — Databricks workspaces, ADLS Gen2, Data Factory, Key Vault, networking, and Azure ML. Own remote state, module versioning, drift detection, and the pipelines that deploy them
- Network & Security Architecture: Design and implement the private networking and security posture for the platform — private endpoints, hub-and-spoke topology, VNet injection, NSGs and firewall rules, managed identities, Key Vault-backed secret scopes, RBAC, and data exfiltration controls
- Data Lake Infrastructure: Own ADLS Gen2 at the infrastructure layer: storage account architecture, hierarchical namespace, container and ACL strategy, lifecycle and tiering policy, encryption and key management, and access patterns for Databricks and downstream consumers
- Data Factory Platform Operations: Own ADF from the platform side — integration runtime provisioning and scaling (self-hosted and managed VNet), linked service credential and managed identity design, environment promotion, deployment automation, and monitoring. Pipeline development sits with the data engineering team
- AI/ML Platform Infrastructure: Provision and operate the infrastructure supporting machine learning workloads: Azure ML workspaces and compute clusters, GPU quota and capacity planning, MLflow model registry integration, model serving endpoints, and the identity and networking connecting them to Unity Catalog
- Observability & Reliability: Build and maintain the monitoring layer — Azure Monitor, Log Analytics, diagnostic settings, alerting, and operational runbooks across the estate
- FinOps & Capacity Planning: Own cost visibility and optimization across DBU and storage spend — tagging strategy, chargeback and showback, budget alerts, reserved capacity, and forward capacity planning
- CI/CD & Environment Management: Own the deployment pipelines and promotion path across development, test, and production environments
- Day-to-day Databricks operations are supported by a dedicated administration team. This role establishes the platform standards and reference configurations that team works from, and serves as a senior technical partner on architecture, design decisions, and complex escalations. That partnership requires genuine administrative depth across the following areas
- Account & Workspace Administration: Account console configuration, workspace provisioning and topology decisions, workspace-level settings, admin role delegation, and multi-workspace strategy
- Unity Catalog: Metastore design and regional strategy, catalog/schema/table permission models, storage credentials and external locations, Hive metastore migration, lineage and audit, and Delta Sharing configuration
- Identity & Access: Entra ID integration, SCIM provisioning, identity federation, groups and entitlements, service principals, and token policy
- Compute Governance: Cluster policies, instance pools, node type and runtime standards, autoscaling and autotermination enforcement, Photon and serverless evaluation, and SQL warehouse sizing and configuration
- Cost & Usage Analysis: System tables, usage attribution, and DBU forecasting — with the ability to assess a workspace, identify where spend is concentrated, and recommend a remediation path
- Technical Partnership: Review platform configurations, contribute to design decisions, and support the administration and data teams on issues that span infrastructure and Databricks. Publish the standards, reference configurations, and self-service patterns that allow those teams to move quickly and safely
- Cross-Boundary Diagnostics: Investigate job failures, cluster startup issues, permission errors, connectivity faults, and performance concerns, and determine whether the root cause sits in infrastructure, Databricks configuration, or the workload itself
Skills
- Experience: 6+ years in cloud infrastructure or platform engineering, with at least 4 years focused on Microsoft Azure
- Terraform: Expert-level, hands-on. You have authored and maintained production Terraform modules, managed remote state, and run infrastructure as code through CI/CD. This is a core daily skill in this role
- Databricks Administration: Demonstrable hands-on administration experience — account and workspace administration, Unity Catalog, cluster policies, identity federation, and cost governance. You should be able to describe administrative decisions you have made and explain the tradeoffs behind them
- Technical Partnership: A track record as a senior technical resource to adjacent teams, including guiding stakeholders from an initial request to the underlying requirement and building consensus around the recommended approach
- Azure Networking & Security: Deep working knowledge of private endpoints, VNets and hub-spoke design, NSGs, Entra ID, managed identities, RBAC, and Key Vault
- Data Lake Architecture: ADLS Gen2 design and access control at enterprise scale
- Azure Data Factory: Platform-side experience with integration runtimes, managed VNet, credential management, and deployment automation
- Automation & Scripting: Strong Python, plus PowerShell and/or Bash, for platform tooling and automation
- Spark & SQL Literacy: Sufficient working knowledge to diagnose infrastructure and configuration-level performance issues and engage credibly with data teams. Spark development depth is not a requirement for this role
- Only candidates available and ready to work directly as Genesis10 employees will be considered for this position
- Databricks Asset Bundles, the Terraform Databricks provider, and workspace-as-code patterns
- Experience leading a Unity Catalog migration or multi-workspace consolidation
- Azure Machine Learning, MLflow, or model-serving infrastructure experience
- Kubernetes / AKS and containerized workloads
- Policy as code — Azure Policy, OPA, Sentinel, or Checkov
- Event-driven infrastructure — Event Hubs, Kafka, or Stream Analytics
- Formal FinOps practice experience
- Multi-region or multi-tenant Databricks deployments
- Databricks Certified Data Engineer Professional, or a Databricks platform administrator credential
- Microsoft Certified: Azure Solutions Architect Expert (AZ-305)
- Microsoft Certified: Azure Administrator Associate (AZ-104)
- Microsoft Certified: DevOps Engineer Expert (AZ-400)
- HashiCorp Certified: Terraform Associate
Benefits
- Access to hundreds of clients, most who have been working with Genesis10 for 5-20+ years.
- The opportunity to have a career-home in Genesis10; many of our consultants have been working exclusively with Genesis10 for years.
- Access to an experienced, caring recruiting team (more than 7 years of experience, on average.)
- Behavioral Health Platform
- Medical, Dental, Vision
- Health Savings Account
- Voluntary Hospital Indemnity (Critical Illness & Accident)
- Voluntary Term Life Insurance
- 401K
- Sick Pay (for applicable states/municipalities)
- Commuter Benefits (Dallas, NYC, SF)
Company Overview
Company H1B Sponsorship