Note: The job is a remote job and is open to candidates in USA. INSPYR Solutions is a national expert in delivering flexible technology and talent solutions. They are seeking a highly motivated Lead Data Engineer responsible for leading the technical design and implementation of enterprise data platforms while remaining hands-on in development.
Responsibilities
- Own the end-to-end technical data engineering for enterprise data platforms
- Lead technical implementation while remaining hands-on with development
- Provide delivery estimates, sprint planning input, and technical guidance to engineering teams
- Mentor and coach data engineers while establishing engineering best practices
- Design and maintain enterprise dimensional data models that support scalable reporting and analytics
- Optimize Gold-layer Delta tables and dimensional models to minimize downstream Power BI DAX complexity through upstream data transformations
- Apply best practices in data warehousing, semantic modeling, and modern lakehouse architecture
- Design and develop modular, reusable, metadata-driven ELT pipelines using Databricks, PySpark, Delta Live Tables (Lakeflow Declarative Pipelines), Unity Catalog, and DBT
- Implement scalable orchestration patterns using Databricks-native architecture
- Build robust, maintainable data pipelines following engineering best practices
- Build high-performance transformations using SQL, PySpark, and DBT
- Implement data quality validations, schema evolution strategies, and automated lineage
- Develop scalable transformation frameworks that support reusable engineering patterns
- Design and implement metadata-driven frameworks, code generation solutions, and agentic engineering patterns to improve engineering productivity and standardization
- Promote reusable architecture patterns across data engineering initiatives
- Partner closely with QA teams to define testing strategies, validation criteria, and automated testing for data accuracy, completeness, and reliability
- Ensure robust quality controls are incorporated throughout the engineering lifecycle
- Proactively communicate technical progress, sprint status, delivery timelines, project dependencies, and technical risks to Project Management
- Raise technical blockers early, escalate issues appropriately, and proactively identify delivery risks without requiring micromanagement
- Integrate data engineering workflows with Git and Azure DevOps for source control, automated testing, and continuous deployment
- Collaborate with analytics, BI, and business stakeholders to ensure data models are optimized for reporting and self-service analytics
- Continuously optimize pipeline performance, storage efficiency, and query execution
- Monitor production environments and ensure high availability, reliability, and scalability
- Stay current on Databricks, Delta Lake, DBT, and modern data engineering best practices, driving continuous improvements across the platform
Skills
- 10+ years of overall experience in data engineering
- 5+ years of hands-on Databricks experience designing and delivering enterprise data platforms
- Demonstrated experience serving as a technical lead responsible for solution architecture, implementation, estimations, delivery planning, and mentoring engineering teams
- Expert-level experience with Databricks, Delta Lake, Unity Catalog, Delta Live Tables (DLT/Lakeflow Declarative Pipelines), and DBT
- Strong SQL and PySpark development skills
- Deep experience developing scalable DBT models and transformation frameworks
- Proven expertise in dimensional modeling, semantic modeling, and enterprise data warehouse design
- Deep experience optimizing Gold-layer Delta tables and dimensional models to simplify downstream Power BI DAX calculations through upstream transformations
- Experience designing metadata-driven frameworks, code generation solutions, or agentic engineering patterns
- Experience implementing enterprise data quality frameworks and partnering with QA teams to define testing strategies and validation criteria
- Experience proactively communicating technical blockers, project dependencies, sprint progress, delivery timelines, and technical risks to Project Management
- Strong understanding of Delta Lake architecture, performance optimization, and modern Lakehouse design principles
- Experience with Git-based source control, CI/CD, and Azure DevOps
- Excellent analytical, communication, leadership, and problem-solving skills
- Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent practical experience)
- Minimum of 10 years of professional data engineering experience
- Minimum of 5 years of hands-on Databricks development experience in enterprise environments
Benefits
- Comprehensive medical benefits
- 401(k) retirement plan
Company Overview