Note: The job is a remote job and is open to candidates in USA. United Software Group Inc is seeking a highly skilled Cloudera Public Cloud Platform Engineer to operate and manage the end-to-end CDP platform ecosystem, including data services, NiFi, Kafka, AI/ML platforms, and enterprise observability. The role is responsible for ensuring the availability, scalability, security, and performance of platform services supporting data, analytics, and AI workloads across multiple environments.
Responsibilities
- Own end-to-end operational responsibility for Cloudera Public Cloud services across Dev / Stage / UAT / Prod : CDE, CDW, COD, CDL, CDF (NiFi), CDV, CAI, Kafka
- Ensure multi-cluster stability, workload isolation, and SLA adherence
- Support onboarding and operations of multiple applications across environments
- Manage and support multi-environment, multi-cluster deployments with strict isolation, governance, and release coordination across Dev/UAT/Prod
- Operate and support Cloudera AI (CAI) environments: AI Workbenches, AI Studios, Model training and development environments, AI inference endpoints and model serving
- Troubleshoot: Resource contention (CPU/GPU), Model deployment/runtime failures
- Operate CDP services running on Cloudera-managed Kubernetes infrastructure
- Apply strong understanding of containerized workloads and Kubernetes concepts for troubleshooting
- Diagnose and resolve: Pod failures, restarts, and resource contention, Spark job failures in containerized environments (CDE), Service-to-service communication issues
- Analyze logs and metrics to identify runtime failures and performance issues
- Collaborate with Cloudera support for managed service-level issues
- Operate and support: CDF (NiFi) for ingestion pipelines, CDV (Data Visualization) for reporting workloads, Octopai for data lineage and catalog integration
- Ensure reliability and performance of data pipelines and integrations
- Monitor and troubleshoot Kafka environments: Topic configurations, partitions, and replication, Consumer lag and throughput issues, Broker connectivity and performance bottlenecks
- Implement and manage: Kerberos, TLS/SSL, Ranger policies
- Administer SDX for: Centralized security, Metadata and policy enforcement
- Support Atlas and Octopai integration
- Manage and troubleshoot user access and identity mapping across layers, including: Cloud IAM roles and permissions, CDP users/groups and identity providers, Ranger policies for fine-grained data access
- Resolve access-related issues impacting: Data access (S3/ADLS), Query execution (CDW/CDE), Application and service-level permissions
- Troubleshoot: S3 / ADLS storage issues, IAM roles and permissions, VPC, subnets, routing, security groups, Bastion host access and connectivity
- Ensure secure and reliable connectivity across services
- Understand and troubleshoot S3-based data lake patterns, including: Bucket structure, prefix design, and access patterns, Performance issues related to small files, request rates, and throughput limits, Encryption (SSE-S3, SSE-KMS) and access policies
- Manage and troubleshoot cross-account IAM roles and access patterns for CDP environments
- Ensure secure access between: CDP environments and cloud resources, Multiple AWS accounts (dev/prod separation)
Skills
- 12+ years of experience in Big Data Platform Engineering / Cloud Platform Operations / Infrastructure roles
- 6+ years of hands-on experience with Cloudera ecosystem (CDH/CDP/ Cloudera Public Cloud)
- Demonstrated ability to quickly learn and adapt to new technologies and evolving platform capabilities, beyond the currently defined CDP stack
- Strong expertise in:
- End-to-end CDP platform operations (CDE, CDW, CDF, CDL, CAI)
- Advanced troubleshooting across multi-cluster, multi-environment deployments
- Kubernetes-based runtime environments (troubleshooting and diagnostics)
- Observability frameworks, including SLIs/SLOs, alerting, and performance tuning
- Proven experience in:
- Leading P1/P2 incident response, triage, and resolution
- Managing platform upgrades, patching, and lifecycle events
- Supporting large-scale environments (TB/PB scale, high concurrency workloads)
- Strong understanding of:
- Cloud infrastructure (IAM, VPC, networking, storage)
- Security and governance (Ranger, Kerberos, TLS/SSL, SDX)
- Hands-on experience in developing and troubleshooting NiFi (CDF) data flows, including:
- Flow design and configuration
- Processor-level debugging and performance tuning
- Handling backpressure, throughput optimization, and failure recovery
- Strong experience with Cloudera CDP Public Cloud
- Expertise in:
- Cloud platforms (AWS/Azure/GCP)
- Kubernetes concepts (troubleshooting-focused)
- Hands-on with:
- CDE, CDW, CDF (NiFi), CAI
- Knowledge of:
- IAM, networking, observability tools
- Platforms operating at multi-terabyte to petabyte scale with high concurrency workloads
- Hands-on experience with:
- Kafka (or similar streaming platforms) including monitoring, troubleshooting, and performance tuning
- Experience with Cloudera CDP CLI (Command Line Interface) for:
- Platform operations and administration
- Job execution and service management (CDE/CDW/CDL)
- Automation of routine operational tasks
- Strong working knowledge of:
- Cloud IAM (AWS IAM / Azure AD) including roles, policies, and cross-service access
- User and group mapping across CDP, cloud IAM, and Ranger policies
- Troubleshooting access issues across storage (S3/ADLS), CDP services, and data access layers
- Enterprise-scale Cloudera CDP platform supporting data engineering, analytics, and AI workloads across multiple applications
- Modernization of legacy platforms and applications into cloud-native CDP services
- Operational support and scaling of:
- Data services (CDE, CDW, CDF, CDL)
- AI/ML platforms (CAI, inference, workbenches)
- Platform performance optimization, observability, and reliability engineering for mission-critical workloads
- The ideal candidate brings strong expertise in CDP on-prem, public cloud services, cloud infrastructure, Kubernetes-based runtime environments, and platform observability, supporting high-concurrency, mission-critical workloads at multi-terabyte to petabyte scale
- Experience with:
- Modernization of legacy data platforms/applications to Cloudera CDP Public Cloud
- Migration and onboarding of workloads to CDE, CDW, and CAI environments
- Supporting hybrid or multi-environment transitions (on-prem → cloud)
- Familiarity with:
- Cloud platforms (AWS, Azure, GCP) including storage, IAM, and networking concepts
- Kubernetes-based runtime environments (troubleshooting-focused)
- Strong scripting and automation skills (Python, Shell, Terraform) for platform operations
Benefits
- Remote work accepted from anywhere in US
Company Overview
Company H1B Sponsorship