Duration: 12 months contract with extensions
Shift: Remote role
Key ResponsibilitiesDesign, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions
Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors
Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations
Build serverless data processing workflows using AWS Lambda
Design and optimize queries in AWS Athena for analytical workloads
Manage data warehouse solutions in AWS Redshift including data modeling and performance tuning
Implement monitoring and alerting using CloudWatch for data pipeline health and performance
Provision and manage AWS infrastructure using Terraform
Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements
Implement data governance and security best practices using Lake Formation, IAM, and KMS
Required Technical SkillsPython: Advanced proficiency in Python 3.8+ for data processing, scripting, and automation
PySpark: Extensive experience with PySpark for large-scale data processing and transformations on EMR clusters
AWS Lambda: Hands-on experience building serverless functions for data processing and orchestration
AWS Glue: Expertise in AWS Glue for ETL jobs, Data Catalog management.
AWS Athena: Experience with Athena for SQL-based data analysis and querying data lakes
AWS EMR: Strong experience with EMR for running Spark jobs, EMR Notebooks, and cluster management
AWSRedshift: Knowledge of Redshift for data warehousing, query optimization, and integration with data lakes
Kafka: Experience with Confluent Kafka for real-time data streaming, including Schema Registry, source/sink connectors, and producers/consumers
AWS S3: Deep understanding of S3 for data lake architecture, bucket policies, and data organization (raw, curated layers)
CloudWatch: Proficiency in CloudWatch for logging, monitoring, and alerting on data pipelines
Terraform: Experience with Infrastructure as Code using Terraform for provisioning AWS resources
Additional Technical Experience Knowledge of AWS EventBridge and Step Functions for workflow orchestration
Familiarity with EMR runtime roles and execution roles for secure data access
Experience with DynamoDB for NoSQL data storage
Knowledge of API Gateway for building data APIs
Preferred QualificationsBachelor's degree in Computer Science, Data Engineering, Information Technology, or related field
Strong understanding of distributed computing and big data processing frameworks
Experience with Agile development methodologies
Excellent problem-solving and analytical skills
Strong communication and collaboration abilities
Development and Data ToolsEMR Notebooks or Jupyter for interactive development
VS Code or similar IDE for Python development
Git for version control
AWS CLI and boto3 for AWS service interaction
Terraform for infrastructure provisioning
DBeaver, Data Studio for database management
About US Tech Solutions:
US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit
US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.
By applying, you acknowledge that AI-assisted tools may be used during hiring.
#LI-AS140