Note: The job is a remote job and is open to candidates in USA. Wilcore Technologies, Inc. is hiring a Data Engineer to support an enterprise cloud-based data and analytics platform for a federal health agency. The role focuses on scalable data ingestion, shared-service data engineering, data quality, observability, governance, federated query tooling, and solution consulting for analytics, reporting, and machine learning.
Responsibilities
- Ingest New and Migrated Sources: Ingest new and migrated data sources into the platform using fit-for-purpose patterns, selected per source characteristics, operational constraints, and approved architecture decisions
- Provide Shared-Service Data Engineering: Transform ingested data into consumption-ready models, including semantic views, conformed dimensions, and performance-oriented partitioning for analytics, reporting, and machine learning
- Operate Delta Lake Patterns and Dedicated Pool: Operate and improve the platform's delta lake patterns and dedicated pool usage, ensuring parity of data and models migrated from other platforms before customer release
- Implement Definition-of-Done Quality Checks: Implement quality checks for each domain: equivalence and parity tests against systems of record and, where applicable, prior platforms; freshness SLAs and anomaly detection tied to pipeline events, publishing pass and fail results and residual risk notes in release artifacts
- Instrument Observability: Wire pipelines to platform monitoring and defect tracking, with playbooks for alerting, triage, rollback, and hot-fix
- Document Ingestion and Modeling Choices: Document pattern, orchestration, lineage, and tuning guidance, and keep links to approvals and decisions current through decision tickets and Architecture Decision Records
- Configure Federated Query Tooling: Configure and implement federated query tooling in alignment with the policies and targets set by the data governance workstream
- Provide Solution Consulting and Provisioning: Assess customer data, analytics, and operational requirements and recommend fit-for-purpose data architecture and ETL solutions. Guide the provisioning of required data infrastructure, tooling, and pipelines aligned with approved enterprise standards
- Define and Monitor KPIs: Define and monitor performance and cost KPIs such as timed loads, parallelism, data quality, and storage and egress for chosen patterns, and publish tuning recommendations for domain teams, documented in a Data Quality and Ingestion Dashboard
Skills
- Experience and expertise with Azure Data Factory, Databricks, and Kafka is required for this role
- Experience with Unity Catalog for unified governance, security, lineage, and data and AI asset management
- Familiarity with Collibra and Immuta, or comparable enterprise data catalog and policy-based access control tooling
- Expertise in the disciplines and technical areas related to the data ingestion, engineering, and quality tasks described above
- Applicants must be authorized to work in the United States
- Must be able to obtain and maintain a federal Public Trust (Tier 2 / Moderate Risk) background investigation
- Must reside and perform all work within the continental United States
- Production system monitoring and support coverage runs 7:00 a.m. to 8:00 p.m. Eastern Time, with participation in a 24/7 on-call escalation process for urgent issues
- Direct experience working in a federal agency environment
Company Overview