Note: The job is a remote job and is open to candidates in USA. TECHNOGEN, Inc. is a small and woman-owned minority IT services, software development, and solutions provider serving organizations ranging from small businesses to Fortune 50 companies and government agencies. The L3 Cloudera Public Cloud Platform Engineer will operate and manage enterprise-scale CDP services, cloud infrastructure, Kubernetes-based runtimes, data integration, AI/ML platforms, security, observability, incident response, upgrades, disaster recovery, and performance optimization.
Responsibilities
- Own end-to-end operational responsibility for Cloudera Public Cloud services across Dev / Stage / UAT / Prod: CDE, CDW, COD, CDL, CDF (NiFi), CDV, CAI, Kafka
- Ensure multi-cluster stability, workload isolation, and SLA adherence
- Support onboarding and operations of multiple applications across environments
- Manage and support multi-environment, multi-cluster deployments with strict isolation, governance, and release coordination across Dev/UAT/Prod
- Operate and support Cloudera AI (CAI) environments: AI Workbenches, AI Studios, Model training and development environments, AI inference endpoints and model serving
- Troubleshoot: Resource contention (CPU/GPU), Model deployment/runtime failures
- Operate CDP services running on Cloudera-managed Kubernetes infrastructure
- Apply strong understanding of containerized workloads and Kubernetes concepts for troubleshooting
- Diagnose and resolve: Pod failures, restarts, and resource contention, Spark job failures in containerized environments (CDE), Service-to-service communication issues
- Analyze logs and metrics to identify runtime failures and performance issues
- Collaborate with Cloudera support for managed service-level issues
- Operate and support: CDF (NiFi) for ingestion pipelines, CDV (Data Visualization) for reporting workloads, Octopai for data lineage and catalog integration
- Ensure reliability and performance of data pipelines and integrations
- Monitor and troubleshoot Kafka environments: Topic configurations, partitions, and replication, Consumer lag and throughput issues, Broker connectivity and performance bottlenecks
- Implement and manage: Kerberos, TLS/SSL, Ranger policies
- Administer SDX for: Centralized security, Metadata and policy enforcement
- Support Atlas and Octopai integration
- Manage and troubleshoot user access and identity mapping across layers, including: Cloud IAM roles and permissions, CDP users/groups and identity providers, Ranger policies for fine-grained data access
- Resolve access-related issues impacting: Data access (S3/ADLS), Query execution (CDW/CDE), Application and service-level permissions
- Troubleshoot: S3 / ADLS storage issues, IAM roles and permissions, VPC, subnets, routing, security groups, Bastion host access and connectivity
- Ensure secure and reliable connectivity across services
- Understand and troubleshoot S3-based data lake patterns, including: Bucket structure, prefix design, and access patterns, Performance issues related to small files, request rates, and throughput limits, Encryption (SSE-S3, SSE-KMS) and access policies
- Manage and troubleshoot cross-account IAM roles and access patterns for CDP environments
- Ensure secure access between: CDP environments and cloud resources, Multiple AWS accounts (dev/prod separation)
- Support and validate disaster recovery and failover strategies across CDP environments
- Ensure backup, recovery, and environment resiliency for critical workloads
- Participate in DR drills and recovery validation
- Implement and manage end-to-end observability: Metrics, logs, and alerting
- Use: Cloudera observability, Cloudera Manager, Prometheus, Grafana
- Monitor: Cluster health, Workload performance, AI inference endpoints
- Enable proactive issue detection and prevention
- Define and implement SLIs/SLOs and alerting thresholds to ensure platform reliability and performance
- Support high-severity (P1/P2) incident response, triage, and resolution within defined SLAs
- Participate in on-call rotation to support 24/7 platform operations
- Respond to production incidents, alerts, and service disruptions within defined SLAs
- Handle P1/P2 incidents, including triage, troubleshooting, and resolution
- Perform root cause analysis (RCA) and implement preventive measures
- Execute: CDP upgrades and version management, Security patches and hotfixes
- Perform: Rolling upgrades, Validation and rollback strategies
- Optimize: Platform-level performance (Spark, Hive, Impala workloads), Cluster utilization and workload distribution
- Drive: Autoscaling strategies, Cost optimization (FinOps practices)
Skills
- 12+ years of experience in Big Data Platform Engineering / Cloud Platform Operations / Infrastructure roles
- 6+ years of hands-on experience with Cloudera ecosystem (CDH/CDP/ Cloudera Public Cloud)
- Demonstrated ability to quickly learn and adapt to new technologies and evolving platform capabilities, beyond the currently defined CDP stack
- O End-to-end CDP platform operations (CDE, CDW, CDF, CDL, CAI)
- O Advanced troubleshooting across multi-cluster, multi-environment deployments
- O Kubernetes-based runtime environments (troubleshooting and diagnostics)
- O Observability frameworks, including SLIs/SLOs, alerting, and performance tuning
- O Leading P1/P2 incident response, triage, and resolution
- O Managing platform upgrades, patching, and lifecycle events
- O Supporting large-scale environments (TB/PB scale, high concurrency workloads)
- O Cloud infrastructure (IAM, VPC, networking, storage)
- O Security and governance (Ranger, Kerberos, TLS/SSL, SDX)
- Hands-on experience in developing and troubleshooting NiFi (CDF) data flows, including:
- O Flow design and configuration
- O Processor-level debugging and performance tuning
- O Handling backpressure, throughput optimization, and failure recovery
- Strong experience with Cloudera CDP Public Cloud
- O Cloud platforms (AWS/Azure/Google Cloud Platform)
- O Kubernetes concepts (troubleshooting-focused)
- O CDE, CDW, CDF (NiFi), CAI
- O IAM, networking, observability tools
- Platforms operating at multi-terabyte to petabyte scale with high concurrency workloads
- O Kafka (or similar streaming platforms) including monitoring, troubleshooting, and performance tuning
- Experience with Cloudera CDP CLI (Command Line Interface) for:
- O Platform operations and administration
- O Job execution and service management (CDE/CDW/CDL)
- O Automation of routine operational tasks
- O Cloud IAM (AWS IAM / Azure AD) including roles, policies, and cross-service access
- O User and group mapping across CDP, cloud IAM, and Ranger policies
- O Troubleshooting access issues across storage (S3/ADLS), CDP services, and data access layers
- Experience with: o Modernization of legacy data platforms/applications to Cloudera CDP Public Cloud o Migration and onboarding of workloads to CDE, CDW, and CAI environments o Supporting hybrid or multi-environment transitions (on-prem cloud)
- Familiarity with: o Cloud platforms (AWS, Azure, Google Cloud Platform) including storage, IAM, and networking concepts o Kubernetes-based runtime environments (troubleshooting-focused)
- Strong scripting and automation skills (Python, Shell, Terraform) for platform operations
Benefits
- Remote work accepted from anywhere in US
Company Overview