Note: The job is a remote job and is open to candidates in USA. Lumenalta partners with global enterprises to design and build large-scale data platforms that support core products and operations. The company is seeking a Senior Databricks Engineer to own the ingestion layer for a Databricks lakehouse supporting a U.S. defense manufacturer, from source discovery and ingestion-pattern design through implementation, testing, operation, and handoff documentation.
Responsibilities
- Run source discovery across each system in scope, documenting data structures, change patterns, extraction constraints, data quality, and current-state integration topology
- Assess extraction options per source, including change data capture feasibility from PLM, JDBC access to Oracle E-Business Suite under its row-level security model, SQL Server change patterns, and file-based ingestion from S3
- Design the ingestion pattern for each source (Lakeflow Connect, custom JDBC, Auto Loader, or federation) with the tradeoffs written down and defensible
- Model bill-of-materials data across PLM, ERP, and MBSE sources, keyed on part number plus revision, including how conflicting representations of the same part are reconciled into the bronze and silver layers
- Build the ingestion layer: Lakeflow Connect pipelines including change data capture from Teamcenter PLM, custom JDBC connectors against Oracle E-Business Suite with session-level initialization parameter handling, Auto Loader for incremental S3 ingestion, and SQL Server ingestion patterns
- Handle schema evolution deliberately, with automatic handling of new columns, rescued data columns for unexpected shapes, and controlled failure on renamed columns rather than silent data loss
- Operate nightly batch patterns with idempotent reruns and backfill support, quarantining bad source data rather than dropping it
- Apply strong engineering practices including testing, CI/CD, and version control, and write expectations and quality checks into pipelines so failures are attributable to a source and a rule
- Document for handoff, producing each source's ingestion contract and runbook so the client team can own it
Skills
- * 7+ years of production data engineering on Databricks, with strong Python, PySpark, and SQL. You are fluent in Delta Lake, Auto Loader, and declarative pipeline development
- * Hands-on experience with Lakeflow Connect or equivalent managed change data capture ingestion
- * Direct experience building JDBC-based ingestion against Oracle, ideally Oracle E-Business Suite, and familiarity with its security model, session context, and multi-org data structures
- * Experience with SQL Server as an ingestion source
- * Working fluency in manufacturing PLM and MBSE systems: Teamcenter, Cameo, and Siemens Capital. You do not need to be an administrator, but you need to understand their data models well enough to design extraction correctly
- * Experience modeling bill-of-materials data, including revision handling, effectivity, and multi-source reconciliation
- * Solid experience with Git-based workflows, CI/CD pipelines, and testing frameworks such as PyTest
- * Strong AWS experience, particularly S3 and IAM roles
- * Databricks Data Engineer Professional certification
- * Experience in a defense, aerospace, or otherwise regulated data environment
- * Ability to do discovery work: sit with a source system owner, ask the right questions, and document accurately what you find
- * A track record of building pipelines that survive contact with messy real-world sources and that other engineers can maintain
- * US citizenship and ability to pass a background check
- * Fluent English, both written and spoken, with strong collaboration skills in distributed teams
- This is a fully remote position
- This position supports a U.S. Government contract that requires all personnel to possess U.S. Citizenship
- Availability to work overlapping Pacific, Central, or Eastern U.S. time zones
- * Prior Teamcenter or other PLM database extraction experience, including direct schema knowledge
- * Familiarity with Unity Catalog governance from the producer side: tagging, documentation, and lineage-friendly pipeline design
- * Experience with Infrastructure as Code using Terraform, and with Databricks Asset Bundles
- * Experience integrating AI-native tooling into engineering workflows
Benefits
- Be 100% dedicated to one project at a time so that you can innovate and grow.
- Be a part of a team of talented and friendly senior-level developers.
- Work on projects that allow you to use leading tech.
- Delivery Partner Program qualification is sponsored for Databricks practitioners.
- Champion path, including a dedicated Databricks solutions architect as your mentor, access to product teams, confidential roadmap briefings, and an invitation to the internal Tech Summit.
- We offer the exam fee for any Databricks certification.
- This is a fully remote position.
Company Overview