Note: The job is a remote job and is open to candidates in USA. Tredence Inc. is seeking a highly skilled GCP Data Engineer with expertise in Scala, Apache Spark, and Google Cloud Platform (GCP). The role involves designing, developing, and optimizing large-scale data processing solutions, building cloud-native data pipelines, and collaborating with data scientists and stakeholders.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Scala and Apache Spark
- Build and optimize batch and real-time data processing solutions on Google Cloud Platform (GCP)
- Develop data ingestion, transformation, and orchestration workflows using GCP services
- Work with large datasets and implement efficient ETL/ELT processes
- Integrate data from various sources including databases, APIs, and streaming platforms
- Monitor, troubleshoot, and improve data pipeline performance and reliability
- Collaborate with data scientists, analysts, and business stakeholders to deliver data solutions
- Implement data quality, governance, and security best practices
- Participate in code reviews and contribute to engineering standards and best practices
- Support production deployments and resolve data pipeline issues
Skills
- Strong hands-on experience with Scala programming
- Extensive experience with Apache Spark (Spark SQL, Spark Streaming, DataFrames, Datasets)
- Strong understanding of distributed computing concepts
- Experience working with Google Cloud Platform (GCP) services such as: BigQuery, Cloud Storage, Dataproc, Dataflow, Pub/Sub, Composer (Airflow), Cloud Functions, Cloud Run
- Experience in ETL/ELT development and data warehousing concepts
- Proficiency in SQL and database optimization
- Experience with CI/CD pipelines and DevOps practices
- Hands-on experience with Git and version control systems
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field
- 4+ years of experience in Data Engineering, Big Data, or Cloud Data Platforms
- Relevant GCP certifications are a plus: Professional Data Engineer, Associate Cloud Engineer
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management skills
- Ability to work independently and collaboratively in a fast-paced environment
- Focus on performance optimization and continuous improvement
- Experience with Kafka or other streaming technologies
- Knowledge of Python in addition to Scala
- Familiarity with containerization technologies such as Docker and Kubernetes (GKE)
- Knowledge of data modeling and cloud data architecture
- Experience with Terraform or Infrastructure as Code (IaC)
- Experience with machine learning data pipelines
- Exposure to multi-cloud environments (AWS/Azure)
- Experience working in Agile/Scrum teams
Company Overview
Company H1B Sponsorship