Note: The job is a remote job and is open to candidates in USA. IBM is hiring a Senior Data Platform Engineer in IBM Consulting to design, build, and maintain scalable data engineering solutions on Google Cloud. The role focuses on batch and real-time data pipelines, data platform operations, and optimizing data layers for migration and processing.
Responsibilities
- Design and develop batch and real-time data pipelines for Data Warehouse and Datalake using Google Cloud services such as DataProc, DataFlow, PubSub, BigQuery, and Big Table
- Build and maintain data engineering solutions using Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, Google DataFlow with Apache Beam or Python, and other open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala
- Schedule and manage the data platform using Google Cloud Scheduler and Cloud Composer (Airflow), ensuring seamless data pipeline operations
- Design and optimize the data layer for efficient data migration and data processing using Google Cloud services
- Ensure scalability and efficiency of data pipelines and data engineering solutions to meet business needs
Skills
- Bachelor's Degree
- Proven experience designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem, including Google DataProc, DataFlow, PubSub, BigQuery, Big Table, Cloud Spanner, CloudSQL, and AlloyDB
- Experience with Apache Beam, Apache Airflow, dbt, Spark/Python, or Spark/Scala, and ability to integrate these technologies with Google Cloud services
- Experience developing and managing batch and real-time data pipelines for Data Warehouse and Datalake using Google Cloud services
- Experience scheduling and managing data platforms using Google Cloud Scheduler and Cloud Composer (Airflow)
- Experience designing and optimizing data layers for efficient data migration and data processing using Google Cloud services
- IBM will not be providing visa sponsorship for this position now or in the future. Therefore, in order to be considered for this position, you must have the ability to work without a need for current or future visa sponsorship
- Experience with Apache Beam, including integrating it with Google Cloud services such as DataFlow, is highly valued
- Ability to optimize Beam pipelines for scalability and efficiency is a plus
- Familiarity with dbt and data modeling concepts, including data warehousing and data lake architecture, is beneficial for designing and optimizing data layers
- Proficiency in Spark and Scala, including integrating them with Google Cloud services such as DataProc, is desirable for building and maintaining data engineering solutions
Benefits
- Healthcare benefits including medical & prescription drug coverage, dental, vision, and mental health & well being
- 401(k)
- Cash balance pension plan
- IBM Employee Stock Purchase Plan
- Financial counseling
- Life insurance
- Short & long-term disability coverage
- Opportunities for performance based salary incentive programs
- 12 holidays
- Minimum 56 hours sick time
- 120 hours vacation
- 12 weeks parental bonding leave in accordance with IBM Policy
- Other Paid Care Leave programs
- Paid family leave benefits to eligible employees where required by applicable law
- Training and educational resources on our personalized, AI-driven learning platform
- Industry-recognized certifications
- Diverse and inclusive employee resource groups
- Giving & volunteer opportunities
- Discounts on retail products, services & experiences
- Hybrid work arrangement
Company Overview