Note: The job is a remote job and is open to candidates in USA. Lumenalta partners with global enterprises to design and build large-scale data platforms that power critical products and operations. The Senior Databricks Data Engineer will inventory spreadsheet and flat-file sources, build governed Databricks ingestion pipelines and Delta tables, and integrate the data with engineering systems. The role also includes implementing data quality, lineage, schema evolution, business-rule translation, and documentation.
Responsibilities
- Inventory the Excel workbooks and file exports that business and engineering teams rely on, and document their structure, owners, and downstream use
- Build Auto Loader pipelines to ingest workbooks and flat files landed in S3, with explicit schemas and typed Bronze and Silver tables
- Parse multi-sheet, inconsistently structured workbooks at scale: shifting header rows, merged cells, embedded totals, mixed types in a column, and layouts that differ between versions of the same file
- Configure schema evolution in Databricks declarative pipelines, including new column handling, rescued data columns, and controlled failure on renamed columns
- Implement data quality expectations that catch the failure modes these sources produce, such as duplicate rows, blank keys, and silently changed formats
- Conform ingested data to the platform's part number plus revision keying so it joins to BOM and engineering data from other sources
- Ensure lineage is captured for every pipeline, orchestrate with Databricks Workflows, and document each source for handoff
- Work with the business owners of each workbook to confirm meaning, business rules, and what "correct" looks like
- Translate spreadsheet logic, such as formulas, lookups, and pivot tables, into maintainable SQL and documented business rules
- Validate that platform outputs match the numbers users trust today, and explain differences when they do not
- Document datasets, metrics, and definitions so reporting stays consistent after handoff
Skills
- Production experience building ingestion pipelines on Databricks
- Hands-on experience ingesting Excel and flat-file sources into Spark or Databricks at scale and the judgment to know when each is appropriate
- A track record of making human-maintained data reliable: shifting headers, merged cells, mixed types, and layout drift between versions
- Experience with Auto Loader for incremental file ingestion from S3
- Experience with Databricks declarative pipelines or Delta Live Tables, including schema evolution and data quality expectations
- Experience conforming ingested data to shared keys so it joins with data from other systems
- Strong PySpark and SQL, and working knowledge of Delta Lake and medallion architecture
- Experience working directly with non-technical data owners to pin down definitions and business rules
- US citizenship
- Fluent English, both written and spoken
- This position supports a U.S. Government contract that requires all personnel to possess U.S. Citizenship
- Experience extracting from Microsoft Access databases (ODBC, JDBC, or file-level tooling)
- Prior work with manufacturing or supply chain data, especially BOMs, parts, revisions, and supplier records
- Familiarity with PLM or ERP data, such as Teamcenter or Oracle E-Business Suite
- Prior work in a defense, aerospace, or FedRAMP environment, or with CUI or ITAR-controlled data
- Databricks Data Engineer Professional certification
Benefits
- Be 100% dedicated to one project at a time so that you can innovate and grow.
- Be a part of a team of talented and friendly senior-level developers.
- Work on projects that allow you to use leading tech.
- We sponsor Delivery Partner Program qualification for Databricks practitioners.
- A dedicated Databricks solutions architect as your mentor.
- Access to product teams.
- Confidential roadmap briefings.
- An invitation to the internal Tech Summit.
- We nominate actively and regularly for the Databricks Champion path, and the Champions already on our team mentor candidates through it.
- We pay the exam fee for any Databricks certification you want to take.
- This is a fully remote position.
Company Overview