Note: The job is a remote job and is open to candidates in USA. Naveera Technology LLC is seeking a Data Engineering Manager to lead an enterprise data platform migration from AWS to GCP. The role combines data architecture and hands-on engineering with migration leadership, team management, and stakeholder collaboration.
Responsibilities
- Architect and implement scalable enterprise data platforms on GCP
- Design Data Lake and Lakehouse architectures using GCS and BigQuery
- Define Bronze, Silver and Gold/Atomic data layers
- Design scalable data ingestion, transformation and consumption frameworks
- Establish standards for data modeling, partitioning, clustering and storage
- Design multi-tenant and multi-location data architectures
- Define schema-on-read and schema-on-write strategies
- Lead the end-to-end migration of enterprise data platforms from AWS to GCP
- Assess existing AWS architecture, data pipelines, workloads, dependencies and operational processes
- Define the target-state GCP architecture and migration roadmap
- Develop migration strategies for: Amazon S3 → Google Cloud Storage Amazon Redshift → BigQuery AWS Glue → Dataflow / Dataproc / BigQuery AWS Step Functions → Cloud Composer / Workflows AWS DMS → GCP-native CDC solutions Amazon Athena → BigQuery
- Identify opportunities to modernize AWS workloads rather than performing a simple lift-and-shift migration
- Define migration phases, technical dependencies, risks and rollback strategies
- Lead architecture reviews and technical design discussions
- Analyze and optimize existing AWS data platforms before migration
- Work with: Amazon S3 AWS Glue AWS Glue Data Quality Amazon Redshift / Redshift Serverless Amazon Athena AWS Step Functions AWS DMS AWS Lake Formation
- Understand existing AWS ETL/ELT pipelines, data models, workloads and dependencies
- Identify equivalent or improved GCP services for each AWS workload
- Prepare technical mapping and migration plans between AWS and GCP services
- Architect real-time data pipelines using: Google Pub/Sub Dataflow / Apache Beam BigQuery Cloud Storage
- Design high-volume event ingestion, enrichment and transformation pipelines
- Implement event-driven architectures and appropriate delivery guarantees
- Optimize streaming pipelines for latency, throughput and scalability
- Design BigQuery streaming ingestion patterns
- Implement monitoring, logging and alerting for real-time workloads
- Design and implement scalable batch and real-time ETL/ELT pipelines
- Migrate AWS Glue-based pipelines to appropriate GCP services
- Develop transformation frameworks using: Python PySpark SQL Dataflow / Apache Beam BigQuery dbt
- Design CDC pipelines and real-time ingestion patterns
- Build orchestration workflows using Cloud Composer / Airflow
- Optimize data processing jobs and query performance
- Design enterprise data models for analytics and reporting
- Define dimensional, normalized and denormalized data models
- Develop multi-tenant data structures
- Design BigQuery partitioning and clustering strategies
- Optimize BigQuery SQL and query execution
- Design data models supporting both real-time and batch workloads
- Work closely with BI and Analytics teams to create scalable consumption models
- Establish data governance and data quality standards across the GCP platform
- Implement automated data quality checks and validation frameworks
- Establish data lineage, metadata and ownership standards
- Ensure appropriate security controls across all GCP data layers
- Implement: IAM Least-privilege access Encryption Service accounts Network security Data access policies
- Work with governance and security teams to ensure compliance requirements are met
- Lead infrastructure automation using Terraform
- Build repeatable and secure GCP infrastructure deployments
- Implement CI/CD pipelines for data engineering workloads
- Work with: Terraform Git GitHub Cloud Build CI/CD pipelines
- Automate data pipeline deployment, testing and infrastructure provisioning
- Establish Dev, QA, UAT and Production deployment standards
- Lead performance optimization initiatives across GCP data workloads
- Optimize: BigQuery query performance Partitioning and clustering Dataflow pipelines Spark workloads Cloud Storage Streaming workloads
- Analyze AWS workloads and determine the most cost-effective GCP architecture
- Develop cloud FinOps and cost optimization strategies
- Establish performance benchmarks and SLAs for critical workloads
- Lead and mentor a team of Data Engineers, Senior Data Engineers and Technical Leads
- Provide technical direction and establish engineering standards
- Conduct architecture and code reviews
- Define technical roadmaps and engineering priorities
- Break complex migration requirements into actionable deliverables
- Track engineering progress, risks, dependencies and delivery milestones
- Promote best practices around coding, testing, CI/CD, security and documentation
- Mentor engineers on GCP, data architecture and modern data engineering practices
- Act as the primary technical point of contact for US-based stakeholders
- Work closely with Business, Product, Data Science, BI and DevOps teams
- Translate business requirements into scalable technical solutions
- Present architecture decisions, migration strategies and technical roadmaps
- Communicate technical risks, dependencies, timelines and trade-offs
- Collaborate with business teams to define operational and analytical KPIs
Skills
- 15+ Years
- Strong hands-on expertise in AWS and GCP Data Engineering
- Strong experience designing enterprise data platforms on AWS and migrating them to Google Cloud Platform (GCP)
- Strong hands-on experience with AWS services such as S3, Glue, Redshift, Athena, Step Functions and AWS DMS, along with strong GCP expertise across BigQuery, Dataflow, Pub/Sub, Cloud Storage and Cloud Composer
- 15+ years of experience in Data Engineering, Data Architecture or Cloud Engineering
- 5+ years of hands-on GCP Data Engineering experience
- 3+ Yrs Proven experience working on AWS-to-GCP migration projects
- Strong experience designing enterprise Data Lake / Lakehouse platforms
- Experience migrating AWS data workloads to GCP
- Strong knowledge of AWS and GCP service mapping and cloud migration patterns
- Expert-level SQL and strong Python/PySpark skills
- Strong data modeling and data warehousing experience
- Experience with Terraform and CI/CD
- Experience managing and mentoring data engineering teams
- Strong communication skills with experience working with US-based stakeholders
- Amazon S3 AWS Glue AWS Glue Data Quality Amazon Redshift / Redshift Serverless Amazon Athena AWS Step Functions AWS DMS AWS Lake Formation IAM
- BigQuery Google Cloud Storage Pub/Sub Dataflow / Apache Beam Cloud Composer / Airflow Dataproc / Spark Cloud Monitoring Cloud Logging
- Advanced Python Expert-level SQL Strong PySpark / Apache Spark ETL/ELT CDC Batch and streaming data processing Event-driven architecture Data pipeline optimization
- Experience with Dataplex, Data Catalog and data lineage is preferred
Benefits
- Flexible remote work environment.
- Continuous learning and certification opportunities
Company Overview
Company H1B Sponsorship