Note: The job is a remote job and is open to candidates in USA. The Transformation Group is seeking a Senior Data Engineer to build a new AWS data platform for a client and lead the migration of existing workloads from Azure to AWS. The role will develop a medallion-layer lakehouse, analytics and reporting models, and data foundations for AI solutions.
Responsibilities
- Build the AWS data platform: S3 lakehouse on open table formats (Iceberg/Delta) organized in medallion (bronze/silver/gold) layers, with a Redshift, Snowflake, or Databricks serving layer
- Migrate Azure workloads (Data Factory, Synapse, Azure SQL, ADLS) to AWS: assess, re-platform pipelines and models, validate data parity, and support cutover
- Build batch and event-driven pipelines (Glue, Step Functions, Lambda, Kinesis) from Salesforce and other enterprise sources
- Develop gold-layer star-schema models that support Tableau and self-service analytics
- Build the data foundation for AI: vector pipelines, RAG, and agentic/LLM solutions using Amazon Bedrock, SageMaker, and OpenSearch
- Deliver infrastructure as code and CI/CD (Terraform/CDK, Git) with governance, security, and data-quality controls
- Tune and cost-optimize workloads; document architecture and migration mappings; report progress to stakeholders
Skills
- Excellent communication skills and ability to communicate with our clients in the US
- 7+ years of experience with data engineering
- 5+ years running production data platforms on AWS; with hands-on experience on AWS data stack: S3, Glue, Redshift, Athena, Lambda, Step Functions, Lake Formation, IAM
- 5+ years of Snowflake and/or Databricks experience building medallion-layer lakehouses
- Hands-on Snowflake and/or Databricks development (Spark/SQL transformations, warehouse and lakehouse models)
- Hands-on experience with delivering AI/ML or GenAI solutions to production (e.g., RAG app, AI agent, ML pipeline)
- Experience with Azure data services (Data Factory, Synapse, Azure SQL, ADLS)
- Data modeling: conceptual, logical, and physical design; dimensional (star/snowflake schema) modeling with conformed dimensions and slowly changing dimensions; normalized and Data Vault patterns; defining grain, keys, and semantic-layer models for BI
- Modern techniques: medallion architecture, ELT, change data capture, incremental loads, dbt, data-quality testing
- Cloud architecture: secure, resilient, cost-efficient design (VPC, identity, encryption, multi-account)
- Applied AI/GenAI: Bedrock or SageMaker, LLM integration, RAG, vector databases
- Strong Python (PySpark) and advanced SQL; Parquet, Iceberg, and Delta formats
- Ability to handle sensitive data and communicate technical concepts to executives and non-technical staff
- Bachelor's in Computer Science or related field, or equivalent
- Strong experience with cloud migration; Azure to AWS preferred
- Salesforce integration and Tableau is a plus
- Agent frameworks/MCP a plus
- AWS certification preferred
Company Overview
Company H1B Sponsorship