Note: The job is a remote job and is open to candidates in USA. K2 Integrity is seeking a Lead Data Engineer to own and advance scalable, trustworthy data platform capabilities. The role focuses on ingestion, orchestration, observability, lineage, data modeling, data contracts, ETL/ELT pipelines, governance, and mentoring engineers.
Responsibilities
- **Ingestion.** Own how data enters the platform across all channels: automated feeds, batch loads, and source-system extracts from CRM and case-management systems. Design ingestion that validates structure and mandatory elements at the door, handles varying formats and volumes across markets and products. You'll build ingestion to be reusable and configurable, so onboarding a new source or market is a known pattern rather than a one-off, and you'll make sure every inbound record carries the lineage and contract metadata the downstream layers depend on
- **Orchestration.**Design the data flow layer that routes datasets through the correct processing sequence and propagates change to everything that depends on it. You'll build dependency-aware refresh, ordered execution, retry and error handling, and clean rollback, with controls on changes to core routing and strong backup/disaster recovery practices. You'll favor versioned, auditable orchestration-as-code in the existing stack over bolt-on tooling
- **Observability.** Stand up the monitoring that proves the platform is healthy and ensure data integrity. This means data-quality checks at key transformation points, deviation monitoring for volume and expectations, automated flagging of anomalies, and exception reporting on breaches. You'll put monitoring on the seams where systems hand off to each other
- **Lineage.** Build native lineage from ingestion through to reporting, so every element is traceable across transformations. You'll make lineage a first-class capability that supports debugging, impact analysis, audit, and compliance
- **Data modeling.** Own the models that underpin the platform: the medallion layers (raw → validated → interim → canonical, with history/SCD handling), canonical entity and master-ID design, and the linkage between datasets. You'll design master-data and naming conventions and build models flexible enough to absorb differing data requirements by market and product rather than locking to a rigid schema. Sound entity/master-ID modeling is central to this role
- **Data contracts.**Establish declared contracts between producers and consumers so interfaces are explicit and changes are safe
- **Pipelines and integrations.**Design, build, and harden ETL/ELT across ingestion channels into the lake house and out to the operational serving layer. Own reliability, performance, and reusability
- Architect orchestration, observability, lineage, and data contracts as reusable platform capabilities, and document the patterns for the team
- Design and evolve logical and physical data models across the platform's layers, balancing consistency with flexibility to onboard new markets and products
- Build and harden ingestion and transformation pipelines; manage loads into and out of the operational serving layer
- Implement automated data-quality and anomaly detection; define what “healthy” looks like for each key dataset
- Establish data best practices: naming conventions, design standards, entity linkage that other services build on
- Contribute to governance: PII masking, retention rules, and compliance controls that vary by market
- Mentor engineers and raise the bar on rigor, testing, and operational readiness
Skills
- 6+ years in data engineering, with senior-level ownership of production data platforms
- Deep hands-on experience with Databricks (Spark, notebooks, Delta Lake, orchestration/workflows, cluster management) and PostgreSQL (complex SQL, stored procedures, materialized views, performance tuning)
- Proven experience building orchestration for complex, dependency-heavy flows, including dependency-aware refresh, error handling, and rollback
- Demonstrated data observability / data quality work: automated checks, anomaly and deviation detection, alerting — with a clear understanding that job success is not the same as correct data
- Experience implementing or operating data lineage and a grasp of why traceability matters for debugging, impact analysis, and audit
- Strong data modeling across relational and lake-house paradigms: layered/medallion architectures, slowly changing dimensions, and entity/master-data modeling
- Strong Python and SQL; solid software-engineering fundamentals (version control, testing, code review, infrastructure-as-code)
- Experience in regulated or compliance-heavy domains (financial crime, AML, KYC, sanctions screening, risk) preferred
- Experience with data contracts and producer/consumer interface design preferred
- Familiarity with entity resolution and the downstream propagation challenges it creates preferred
- Exposure to data governance and privacy regimes (ISO, GDPR) preferred
- Experience designing platforms intended to scale across multiple markets and product lines preferred
Benefits
- Remote work arrangement
Company Overview
Company H1B Sponsorship