Note: The job is a remote job and is open to candidates in USA. Naveera Technology LLC is a trusted global engineering partner delivering Data Engineering and IT Infrastructure solutions. They are seeking an experienced Engineering Manager to lead a large-scale AWS-to-GCP data platform migration, requiring a combination of technical architecture, hands-on engineering, and team management.
Responsibilities
- Lead the end-to-end migration of enterprise data platforms from AWS to GCP
- Assess existing AWS architecture, data pipelines, workloads, dependencies and operational processes
- Define the target-state GCP architecture and migration roadmap
- Develop migration strategies for: Amazon S3 → Google Cloud Storage, Amazon Redshift → BigQuery, AWS Glue → Dataflow / Dataproc / BigQuery, AWS Step Functions → Cloud Composer / Workflows, AWS DMS → GCP-native CDC solutions, Amazon Athena → BigQuery
- Identify opportunities to modernize AWS workloads rather than performing a simple lift-and-shift migration
- Define migration phases, technical dependencies, risks and rollback strategies
- Lead architecture reviews and technical design discussions
- Architect and implement scalable enterprise data platforms on GCP
- Design Data Lake and Lakehouse architectures using GCS and BigQuery
- Define Bronze, Silver and Gold/Atomic data layers
- Design scalable data ingestion, transformation and consumption frameworks
- Establish standards for data modeling, partitioning, clustering and storage
- Design multi-tenant and multi-location data architectures
- Define schema-on-read and schema-on-write strategies
- Analyze and optimize existing AWS data platforms before migration
- Work with: Amazon S3, AWS Glue, AWS Glue Data Quality, Amazon Redshift / Redshift Serverless, Amazon Athena, AWS Step Functions, AWS DMS, AWS Lake Formation
- Understand existing AWS ETL/ELT pipelines, data models, workloads and dependencies
- Identify equivalent or improved GCP services for each AWS workload
- Prepare technical mapping and migration plans between AWS and GCP services
- Architect real-time data pipelines using: Google Pub/Sub, Dataflow / Apache Beam, BigQuery, Cloud Storage
- Design high-volume event ingestion, enrichment and transformation pipelines
- Implement event-driven architectures and appropriate delivery guarantees
- Optimize streaming pipelines for latency, throughput and scalability
- Design BigQuery streaming ingestion patterns
- Implement monitoring, logging and alerting for real-time workloads
- Design and implement scalable batch and real-time ETL/ELT pipelines
- Migrate AWS Glue-based pipelines to appropriate GCP services
- Develop transformation frameworks using: Python, PySpark, SQL, Dataflow / Apache Beam, BigQuery, dbt
- Design CDC pipelines and real-time ingestion patterns
- Build orchestration workflows using Cloud Composer / Airflow
- Optimize data processing jobs and query performance
- Design enterprise data models for analytics and reporting
- Define dimensional, normalized and denormalized data models
- Develop multi-tenant data structures
- Design BigQuery partitioning and clustering strategies
- Optimize BigQuery SQL and query execution
- Design data models supporting both real-time and batch workloads
- Work closely with BI and Analytics teams to create scalable consumption models
- Establish data governance and data quality standards across the GCP platform
- Implement automated data quality checks and validation frameworks
- Establish data lineage, metadata and ownership standards
- Ensure appropriate security controls across all GCP data layers
- Implement: IAM, Least-privilege access, Encryption, Service accounts, Network security, Data access policies
- Work with governance and security teams to ensure compliance requirements are met
- Lead infrastructure automation using Terraform
- Build repeatable and secure GCP infrastructure deployments
- Implement CI/CD pipelines for data engineering workloads
- Automate data pipeline deployment, testing and infrastructure provisioning
- Establish Dev, QA, UAT and Production deployment standards
- Lead performance optimization initiatives across GCP data workloads
- Optimize: BigQuery query performance, Partitioning and clustering, Dataflow pipelines, Spark workloads, Cloud Storage, Streaming workloads
- Analyze AWS workloads and determine the most cost-effective GCP architecture
- Develop cloud FinOps and cost optimization strategies
- Establish performance benchmarks and SLAs for critical workloads
- Lead and mentor a team of Data Engineers, Senior Data Engineers and Technical Leads
- Provide technical direction and establish engineering standards
- Conduct architecture and code reviews
- Define technical roadmaps and engineering priorities
- Break complex migration requirements into actionable deliverables
- Track engineering progress, risks, dependencies and delivery milestones
- Promote best practices around coding, testing, CI/CD, security and documentation
- Mentor engineers on GCP, data architecture and modern data engineering practices
- Act as the primary technical point of contact for US-based stakeholders
- Work closely with Business, Product, Data Science, BI and DevOps teams
- Translate business requirements into scalable technical solutions
- Present architecture decisions, migration strategies and technical roadmaps
- Communicate technical risks, dependencies, timelines and trade-offs
- Collaborate with business teams to define operational and analytical KPIs
Skills
- 15+ years of experience in Data Engineering, Data Architecture or Cloud Engineering
- Strong hands-on experience with AWS Data Engineering and Architecture
- 5+ years of hands-on GCP Data Engineering experience
- Proven experience working on AWS-to-GCP migration projects
- Strong experience designing enterprise Data Lake / Lakehouse platforms
- Experience migrating AWS data workloads to GCP
- Strong knowledge of AWS and GCP service mapping and cloud migration patterns
- Expert-level SQL and strong Python/PySpark skills
- Strong data modeling and data warehousing experience
- Experience with Terraform and CI/CD
- Experience managing and mentoring data engineering teams
- Strong communication skills with experience working with US-based stakeholders
- Ability to assess AWS workloads, define the GCP target architecture and lead the migration from strategy through production implementation
- Strong ability to translate complex business and technical requirements into scalable, secure and cost-effective GCP architectures
- Ability to manage, mentor and technically guide Data Engineering teams while remaining hands-on with critical architecture and implementation decisions
- Strong expertise in Data Lakehouse, Data Warehouse, Streaming, Data Modeling and modern data engineering patterns
- Ability to optimize BigQuery, Dataflow and Spark workloads while implementing cloud cost optimization strategies
- Strong understanding of data quality, lineage, metadata, IAM, encryption, access control and enterprise governance
- Excellent communication skills with the ability to work directly with US-based business and technical stakeholders
- Amazon S3
- AWS Glue
- AWS Glue Data Quality
- Amazon Redshift / Redshift Serverless
- Amazon Athena
- AWS Step Functions
- AWS DMS
- AWS Lake Formation
- IAM
- BigQuery
- Google Cloud Storage
- Pub/Sub
- Dataflow / Apache Beam
- Cloud Composer / Airflow
- Dataproc / Spark
- Cloud Monitoring
- Cloud Logging
- Advanced Python
- Expert-level SQL
- Strong PySpark / Apache Spark
- ETL/ELT
- CDC
- Batch and streaming data processing
- Event-driven architecture
- Data pipeline optimization
- Enterprise Data Lake / Lakehouse
- Medallion Architecture
- Data Modeling
- Dimensional Modeling
- Multi-tenant Data Modeling
- Schema-on-Read / Schema-on-Write
- Data Lineage
- Metadata Management
- Data Governance
- Dbt
- Apache Airflow
- Terraform
- Git / GitHub
- Cloud Build
- CI/CD
- Data Quality Frameworks
- Experience with Dataplex, Data Catalog and data lineage is preferred
- OpenLineage is a plus
Benefits
- Flexible remote work environment.
- Continuous learning and certification opportunities.
- Lead large-scale AWS-to-GCP cloud transformation initiatives.
- Work on enterprise Data Lakehouse and analytics modernization projects.
- Exposure to global enterprise customers.
- Collaborative, innovation-driven engineering culture.
Company Overview