Note: The job is a remote job and is open to candidates in USA. McGraw Hill LLC is hiring a Senior Data Engineer to advance its product eventing, data platforms, and analytics capabilities. The role designs scalable data solutions and end-to-end event pipelines that turn learning interactions into trusted, actionable learning insights.
Responsibilities
- Work closely with product platform teams to define and create meaningful learning events, establish event standards, and build scalable, reliable event pipelines that connect learning interactions to actionable learning insights
- Demonstrate hands-on experience designing and delivering data solutions on Databricks, including Delta Lake, medallion (Bronze/Silver/Gold) architecture, Unity Catalog, and scalable lakehouse design patterns on AWS
- Design, develop, and optimize parallel and distributed ETL/ELT pipelines using Apache Spark (PySpark/Scala) and Databricks. Implement efficient data transformations, joins, aggregations, window functions, and other Spark-native capabilities to support high-volume product and learning event data
- Build and maintain real-time and near-real-time event-driven data pipelines that support the ingestion, processing, and transformation of product and learning interactions. Ensure reliable data flow from event creation through curated data layers and learning insights
- Establish and maintain end-to-end data quality across event schemas, ingestion pipelines, transformations, and curated data products. Implement validation, monitoring, and alerting to ensure data accuracy, consistency, completeness, and timely delivery
- Develop and maintain Databricks Workflows, including dependency management, retry policies, scheduling, monitoring, and alerting. Build maintainable, cloud-native pipeline orchestration solutions
- Apply partitioning, caching, broadcast joins, and other performance optimization techniques to improve pipeline efficiency, scalability, and throughput. Use Delta Lake capabilities to support reliable and maintainable data processing
- Design and implement scalable data models using Delta tables, including fact tables, dimension tables, star and snowflake schemas, and aggregations where appropriate. Ensure curated data assets support product analytics and learning insights
- Apply modern data architecture and governance principles using Unity Catalog, Delta Sharing, and AWS-native lakehouse patterns to support secure, discoverable, and trusted data products
- Use Git-based version control, including Databricks Repos/Git folders, and project management tools such as Jira within Agile/Kanban delivery frameworks
- Translate product and data requirements into technical designs and production-grade solutions, contributing across the full software development lifecycle—from requirements gathering and architecture through implementation, testing, deployment, and ongoing support
- Develop high-quality solution design documentation, including event architecture, data flow diagrams, pipeline specifications, data mappings, and Unity Catalog data asset definitions. Communicate effectively with product, engineering, analytics, and business stakeholders
Skills
- Authorized to work for any employer within the United States
- The ideal candidate brings strong hands-on expertise in AWS, Databricks, and modern event-driven data architectures, with experience designing real-time or near-real-time event pipelines, Delta Lake, and medallion (Bronze/Silver/Gold) data architecture patterns. Proficiency in SQL and Python or Scala for large-scale data transformation is essential
- Deep expertise in modern event-driven data and lakehouse architecture, including product event design, event schemas, real-time and near-real-time event processing, Delta Lake, medallion design patterns, Unity Catalog governance, and cloud-native data solutions on Databricks. Experience connecting product-generated events to reliable data pipelines and actionable learning insights
- 5+ years of experience in Data Engineering, with a focus on the following tools and technologies:
- Databricks — Delta Lake, Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, Databricks SQL, and MLflow, with hands-on experience building scalable event processing and data transformation pipelines
- Eventing & Streaming Architecture — Experience designing and implementing event-driven architectures, event schemas, event ingestion, streaming pipelines, and real-time or near-real-time processing using technologies such as Apache Kafka, cloud-native streaming services, or equivalent eventing platforms
- AWS & Databricks — S3, Redshift, Glue, Lambda, EMR, Athena (with Iceberg), Step Functions, and IAM, integrated with Databricks as the primary compute and transformation layer where applicable
- Scripting and programming languages — Python (PySpark), Scala (Spark), or SQL as primary languages for pipeline development, event processing, and data transformation within Databricks
- 3+ years of experience working with cloud platforms, primarily AWS, architecting and operating Databricks environments, including workspace configuration, cluster policies, instance profiles, security, and cost optimization strategies
- 1+ years of experience with workflow automation and pipeline orchestration using Databricks Workflows, Apache Airflow (with the Databricks provider), or equivalent cloud-native orchestration tools, including dependency management, retries, monitoring, and alerting
- Experience collaborating with product platform teams to define event requirements, establish event schemas, implement event instrumentation, and build reliable pipelines that connect learning interactions to curated data products and learning insights
- Strong understanding of end-to-end event data quality, including schema validation, event completeness, data consistency, pipeline observability, and monitoring across event generation, ingestion, processing, and consumption
- Experience optimizing high-volume event processing and distributed data pipelines using Spark, Delta Lake, and cloud-native technologies to support scalability, reliability, and efficient resource utilization
- Experience with EdTech domain
Benefits
- Remote position open to applicants authorized to work for any employer within the United States.
- An annual bonus plan may be provided as part of the compensation package, in addition to a full range of medical and/or other benefits, depending on the position offered.
Company Overview