Note: The job is a remote job and is open to candidates in USA. Penn Foster Group is an education and workforce training company transforming online learning through accessible, career-focused programs. The Lead Data Engineer will provide technical leadership for the Databricks Lakehouse platform, develop scalable data products and pipelines, establish engineering standards, and mentor Data Engineers. The role also partners across analytics, governance, security, MLOps, and business teams to enable enterprise analytics, machine learning, and generative AI.
Responsibilities
- Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices
- Mentor and develop junior and mid-level Data Engineers, fostering technical growth, knowledge sharing, and continuous learning
- Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence
- Lead technical design discussions, evaluate architectural tradeoffs, and drive adoption of modern engineering practices
- Champion a culture of quality, collaboration, innovation, and continuous improvement
- Serve as the technical lead for Penn Foster Group's Databricks Lakehouse platform
- Design, build, and support scalable enterprise data pipelines using SQL, Python, Apache Spark, and Databricks
- Define best practices for Databricks development, including Workflows, Repos, Jobs, notebooks, reusable Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation
- Design and optimize Delta Lake architectures using Medallion patterns, Delta optimization, Liquid Clustering, and Photon
- Build reusable ingestion frameworks supporting batch, streaming, CDC, and API-based integration patterns
- Optimize Spark workloads for performance, scalability, reliability, and cloud cost efficiency
- Partner with Analytics and business stakeholders to develop trusted semantic data products that power Databricks Genie
- Design and maintain Genie Spaces and Genie Ontologies that accurately represent business entities, relationships, metrics, and terminology
- Establish best practices for semantic modeling, governed metrics, business metadata, and AI-ready datasets
- Evaluate and implement emerging Databricks AI capabilities to improve self-service analytics and business productivity
- Collaborate with Data Science and MLOps teams to enable machine learning, generative AI, and advanced analytics initiatives
- Design scalable Lakehouse architectures supporting analytics, reporting, AI, and operational data products
- Partner with Data Governance teams to implement enterprise governance using Unity Catalog, including data lineage, fine-grained security, metadata management, and access controls
- Champion automated testing, monitoring, observability, data quality, and production reliability
- Serve as the technical escalation point for complex production issues and lead root cause analysis and continuous improvement efforts
- Drive platform modernization initiatives while balancing delivery, scalability, maintainability, and operational excellence
- Collaborate with Business Intelligence, Product, Platform Engineering, Security, and MLOps teams to deliver trusted, scalable data products
- Translate business requirements into modern technical solutions that support enterprise reporting, analytics, AI, and strategic decision-making
- Contribute to technical roadmaps, platform strategy, and the continued evolution of Penn Foster Group's Data & Analytics capabilities
Skills
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience
- 8+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering
- 3+ years of experience leading technical initiatives and mentoring engineering teams
- Demonstrated success developing engineers while driving engineering excellence and technical standards
- Strong communication and collaboration skills with the ability to influence both technical and business stakeholders
- Expert knowledge of the Databricks Lakehouse Platform, including: Apache Spark (PySpark), Delta Lake, Unity Catalog, Databricks Workflows & Jobs, Repos, SQL Warehouses, Cluster Policies, Serverless Compute, MLflow, Lakehouse Monitoring, Auto Loader, Delta Live Tables / Lakeflow
- Expert Knowledge in SQL and Python
- Strong Experience working with BI tools such as PowerBI, Tableau or Business Objects
- Strong Microsoft Azure experience, including ADLS Gen2, Microsoft Entra ID (Azure AD), RBAC, networking, and cloud security
- Experience implementing Medallion Architecture, dimensional modeling, and domain-oriented data products
- Deep understanding of Spark optimization techniques, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization
- Experience implementing CI/CD pipelines, Git-based development workflows, Infrastructure as Code, and automated testing
- Strong understanding of data governance, metadata management, data quality, observability, security, and compliance
- Experience implementing Databricks Genie, Genie Spaces, and Genie Ontologies
- Experience designing semantic models and AI-ready data products
- Experience supporting Power BI, Tableau, or other enterprise BI platforms
- Experience with machine learning platforms, MLOps, or Generative AI applications
- Experience with dbt, Great Expectations, or similar modern data engineering tools
- Databricks Certified Data Engineer Professional and/or Microsoft Azure certifications
Benefits
- Medical coverage
- Dental coverage
- Vision coverage
- Flexible spending
- Generous paid time off
- Sponsored volunteer opportunities
- 401K with a company match
- Free access to all of our online programs
Company Overview
Company H1B Sponsorship