Data Engineer – Insights (AI/ML)
Location: Remote (US)
Department: Insights (AI/ML)
Reports to: Engineering Manager
About the Role
Irth is building a new AI-driven threat and risk management platform for pipeline asset integrity. The platform brings together three capabilities that have historically been separate at Irth:
- A governed, cross-product data platform built on Databricks and Azure
- An AI-powered ingestion layer that normalizes, repairs, and enriches customer data without services-heavy onboarding
- A reusable analytical layer that runs industry-standard, Irth-developed, and customer-built risk models against the data
As a Data Engineer, you will build the pipelines that make this platform real. Pipeline operators hold their integrity data across inline inspection reports, GIS systems, maintenance records, spreadsheets, scanned documents, and enterprise systems of record. Getting that data ingested, cleaned, aligned, and made model-ready is one of the biggest obstacles to adoption in this market—and it is the problem this role exists to solve.
You will implement ingestion and transformation pipelines based on patterns established by the Data Architect, build the AI-assisted ingestion layer in partnership with data scientists, and help operationalize both. This role is well suited to a mid-level engineer who wants to deepen their expertise in Databricks, Spark, and modern lakehouse engineering while helping build a platform from the ground up.
Key Responsibilities
1. Pipeline Development — Primary Responsibility
- Build and maintain ingestion pipelines for structured and semi-structured sources, including GIS, inline inspection data, SCADA, maintenance systems, and enterprise systems of record.
- Implement batch and streaming ingestion using Databricks Workflows, Spark, PySpark, SQL, and declarative pipeline tooling.
- Apply medallion architecture patterns (Bronze, Silver, Gold) for transformation, standardization, and enrichment.
- Implement change data capture (CDC), slowly changing dimensions (SCD), schema evolution, and data-validation rules.
- Normalize third-party and public data feeds, including weather history, soil characteristics, satellite-derived data, and one-call ticket data, into the shared data model.
2. AI-Assisted Ingestion Layer
- Build pipelines that automate normalization of units, schemas, and semantics across inconsistent customer data.
- Implement automated data-quality repair workflows, including gap filling, error correction, and reconciliation, with clear provenance for every synthesized value.
- Work with data scientists to productionize document-extraction pipelines that parse reports, spreadsheets, and field records into the target schema.
- Build human-in-the-loop review and exception workflows so low-confidence extractions are surfaced rather than propagated silently.
3. Platform & Storage Implementation
- Configure and manage Delta Lake tables, partitioning strategies, and optimization routines.
- Implement metadata, lineage, and cataloging standards using Unity Catalog.
- Build and maintain connectors to customer systems of record with configurable refresh cadences.
- Support geospatial data processing, including spatial joins and alignment of results to pipeline centerline geometry.
4. Governance, Quality & Compliance Enablement
- Implement data-quality tests, profiling, and drift monitoring based on standards established by the Data Architect.
- Apply access-control policies, security rules, and classification tags defined by the governance model.
- Implement lineage capture sufficient to support regulatory traceability from ingestion through model output.
5. Orchestration, Automation & Operational Support
- Build, schedule, and monitor data workflows, and own alerting and failure handling for the pipelines you develop.
- Contribute to CI/CD for pipeline code, including version control, automated testing, and environment promotion.
- Troubleshoot production incidents, recover failed pipeline runs, and optimize performance and infrastructure costs.
6. Collaboration & Documentation
- Work closely with the Data Architect to translate architectural designs into production implementations and identify gaps or ambiguities in the design.
- Participate in architecture, design, and code reviews.
- Document pipelines, transformation logic, data dictionaries, job schedules, operational procedures, and runbooks.
Requirements
Required Qualifications
- 3–5 years of experience in data engineering, ETL development, or cloud data platform engineering.
- Hands-on experience with Databricks, Spark, PySpark, or comparable distributed data-processing technologies.
- Strong SQL skills and experience with structured data transformation.
- Experience with at least one major cloud platform; Azure experience preferred.
- Familiarity with data modeling, data-quality practices, schema evolution, and pipeline troubleshooting.
- Experience with workflow orchestration and scheduling frameworks.
- Understanding of core data-security practices, including access control, encryption, and credential management.
- Experience with Git-based development and comfort working within a code-reviewed engineering team.
Preferred Qualifications
- Experience with Delta Lake, medallion architecture, and lakehouse engineering best practices.
- Experience with Unity Catalog, Microsoft Purview, or comparable metadata and data-lineage tooling.
- Experience building pipelines that ingest unstructured or semi-structured documents.
- Experience with geospatial data processing and common GIS data formats.
- CI/CD and DevOps experience for data workloads, including infrastructure as code (IaC).
- Experience preparing and transforming data specifically for machine learning or probabilistic model consumption.
- Cloud or Databricks certifications.
- Experience using AI-assisted coding tools such as Cursor or GitHub Copilot and/or agentic coding tools such as Claude Code as part of a professional development workflow.
Nice to Have
- Experience integrating oil and gas or utility asset data, including pipelines, facilities, and GIS assets, into a data platform.
- Understanding of asset integrity concepts, including inspection data, risk scoring, corrosion, and defect tracking.
- Familiarity with regulatory and compliance reporting requirements for pipeline or asset integrity data.
- Experience migrating customers from legacy or spreadsheet-based systems to modern data platforms.
Success Metrics
Success in this role will be measured by:
- Reliable, well-documented pipelines delivering consistent Bronze, Silver, and Gold data flows.
- Measurable reduction in the manual effort required to onboard new customer data.
- High data-quality pass rates, with failures identified and contained at ingestion rather than downstream.
- Full compliance with established cataloging, lineage, security, and governance standards.
- Low pipeline incident rates, with fast recovery and clear, actionable runbooks when incidents occur.
- Effective collaboration with the Data Architect, data scientists, application engineers, and other cross-functional partners.
Originally posted on