Note: The job is a remote job and is open to candidates in USA. Lumenalta partners with global enterprises to design and build large-scale data platforms that support core business products and operations. The Senior Databricks Data Engineer will bring Excel workbooks and flat-file sources onto governed data platforms by building ingestion pipelines, parsing inconsistent files, implementing data quality and schema evolution, conforming data to shared engineering keys, and documenting business rules and datasets.
Responsibilities
- Inventory the Excel workbooks and file exports that business and engineering teams rely on, and document their structure, owners, and downstream use
- Build Auto Loader pipelines to ingest workbooks and flat files landed in S3, with explicit schemas and typed Bronze and Silver tables
- Parse multi-sheet, inconsistently structured workbooks at scale: shifting header rows, merged cells, embedded totals, mixed types in a column, and layouts that differ between versions of the same file
- Configure schema evolution in Databricks declarative pipelines, including new column handling, rescued data columns, and controlled failure on renamed columns
- Implement data quality expectations that catch the failure modes these sources produce, such as duplicate rows, blank keys, and silently changed formats
- Conform ingested data to the platform's part number plus revision keying so it joins to BOM and engineering data from other sources
- Ensure lineage is captured for every pipeline, orchestrate with Databricks Workflows, and document each source for handoff
- Work with the business owners of each workbook to confirm meaning, business rules, and what "correct" looks like
- Translate spreadsheet logic, such as formulas, lookups, and pivot tables, into maintainable SQL and documented business rules
- Validate that platform outputs match the numbers users trust today, and explain differences when they do not
- Document datasets, metrics, and definitions so reporting stays consistent after handoff
Skills
- Production experience building ingestion pipelines on Databricks
- Hands-on experience ingesting Excel and flat-file sources into Spark or Databricks at scale and the judgment to know when each is appropriate
- A track record of making human-maintained data reliable: shifting headers, merged cells, mixed types, and layout drift between versions
- Experience with Auto Loader for incremental file ingestion from S3
- Experience with Databricks declarative pipelines or Delta Live Tables, including schema evolution and data quality expectations
- Experience conforming ingested data to shared keys so it joins with data from other systems
- Strong PySpark and SQL, and working knowledge of Delta Lake and medallion architecture
- Experience working directly with non-technical data owners to pin down definitions and business rules
- US citizenship
- Fluent English, both written and spoken
- Candidates must reside in the United States while assigned to our client and be physically located within the United States when performing project work or accessing project systems or data
- Experience extracting from Microsoft Access databases (ODBC, JDBC, or file-level tooling)
- Prior work with manufacturing or supply chain data, especially BOMs, parts, revisions, and supplier records
- Familiarity with PLM or ERP data, such as Teamcenter or Oracle E-Business Suite
- Prior work in a defense, aerospace, or FedRAMP environment, or with CUI or ITAR-controlled data
- Databricks Data Engineer Professional certification
Benefits
- 100% dedicated to one project at a time so that you can innovate and grow.
- A part of a team of talented and friendly senior-level developers.
- Work on projects that allow you to use leading tech.
- Delivery Partner Program qualification; we sponsor it.
- Champion path, including a dedicated Databricks solutions architect as your mentor, access to product teams, confidential roadmap briefings, and an invitation to the internal Tech Summit.
- We pay the exam fee for any Databricks certification you want to take.
- Flexible working hours in a remote environment.
- Health insurance (medical and dental) for W2 Employees.
- 401K Contribution.
- A professional development fund to enhance your skills and knowledge.
- 15 days of paid time off annually.
- Access to soft-skill development courses to further your career.
Company Overview