Note: The job is a remote job and is open to candidates in USA. Lumenalta partners with global enterprises to design and build large-scale data platforms that power core products and operations. The Senior Databricks Data Engineer will build and manage ingestion pipelines that transform Cameo, Teamcenter PLM, and Siemens Capital engineering exports into trusted Delta tables, with resilient schema evolution, data quality, lineage, and orchestration.
Responsibilities
- Establish the S3 landing zone and ingestion framework used for all file-based engineering sources, including folder conventions, file validation, and failure handling
- Build Auto Loader pipelines for incremental ingestion of Cameo, Teamcenter, and Capital exports landed in S3
- Parse engineering export formats (CSV, XML, JSON, and similar) into well-typed Bronze and Silver tables with explicit schemas
- Define and implement the incremental and batch refresh patterns applied across in-scope sources, including nightly batch loads
- Configure schema evolution in Databricks declarative pipelines, including new column handling, rescued data columns, and controlled failure on renamed columns
- Handle bad, late, and re-delivered files without corrupting or double-loading the tables
- Implement baseline data quality checks and ensure lineage is captured for every pipeline
- Orchestrate pipelines with Databricks Workflows
- Work with the source system engineering SMEs to understand export contents, refresh cadence, and how model elements, requirements, and parts are represented in each file
Skills
- 6+ years in data engineering, including production experience building ingestion pipelines on Databricks
- Deep experience with Auto Loader and incremental file ingestion from S3
- Experience parsing structured and semi-structured export formats (CSV, XML, JSON) into typed tables
- Experience with Databricks declarative pipelines or Delta Live Tables, including schema evolution configuration
- A track record of pipelines that handle bad, late, and re-delivered files safely
- Strong PySpark and SQL, and working knowledge of Delta Lake internals such as managed vs. external tables and predictive optimization
- Experience with Databricks Workflows for orchestration
- Working fluency in PLM and MBSE vocabulary, enough to talk to engineering SMEs about parts, BOMs, revisions, and model elements
- Ability to complete the client's background check and onboarding and to work on client-furnished equipment
- US citizenship
- Fluent English, both written and spoken
- Hands-on experience with exports or data from Cameo / Teamwork Cloud, Teamcenter, or Siemens Capital
- Experience ingesting from REST APIs into Databricks
- Experience with Lakeflow Connect or other CDC-based ingestion tools
- Prior work in a defense, aerospace, or FedRAMP environment, or with CUI or ITAR-controlled data
- Databricks Data Engineer Professional certification
Benefits
- 100% dedicated to one project at a time so that you can innovate and grow.
- Be a part of a team of talented and friendly senior-level developers.
- Work on projects that allow you to use leading tech.
- Delivery Partner Program qualification ... we sponsor it.
- Champion path ... a dedicated Databricks solutions architect as your mentor, access to product teams, confidential roadmap briefings, and an invitation to the internal Tech Summit.
- We pay the exam fee for any Databricks certification you want to take.
- Flexible working hours in a remote environment.
- Health insurance (medical and dental) for W2 Employees.
- 401K Contribution.
- A professional development fund to enhance your skills and knowledge.
- 15 days of paid time off annually.
- Access to soft-skill development courses to further your career.
Company Overview