Note: The job is a remote job and is open to candidates in USA. Rubicon is a company with an Information Technology department supporting analytics, business, and AI initiatives. The Data Engineer will build and operate production data pipelines on Databricks, collaborating with cross-functional teams to deliver reliable, documented, and governed data solutions from requirements through production. The role also owns pipeline quality, monitoring, CI/CD, governance, and on-call operations.
Responsibilities
- Build batch and streaming ingestion pipelines from databases, APIs, SaaS applications, event streams, and file drops using Lakeflow Declarative Pipelines for CDC and SCD Type 2 patterns
- Build and maintain medallion architecture (bronze, silver, gold) with clear contracts at each layer, applied as part of every pipeline you deliver
- Write and tune complex SQL and PySpark transformations against large Delta Lake tables, and model gold-layer datasets for analytics
- Own operational health of your pipelines, including data quality checks, monitoring and alerting, Unity Catalog governance, Continuous Integration/Continuous Development (CI/CD) promotion, and on-call
- Ability to travel and/or work onsite as needed
- Performs other duties as assigned or apparent
- This job has no direct supervisory responsibilities
- May assist with onboarding and training new hires on Information Technology processes, systems, and operational procedures
- Serves as a resource for internal team members regarding IT/Data operational processes and standards
- Supports knowledge sharing, continuous improvement initiatives, and best practices across the Information Technology and Data Engineering functions
Skills
- High School Diploma required
- 5+ years of experience in Data Engineering or adjacent roles with a track record of owning and fixing pipelines in production
- Strong knowledge of Production Apache Spark (PySpark) and Python (modular, testable code with unit tests)
- Demonstrated understanding of data modeling and pipeline mechanics, including dimensional modeling, grain and keys, slowly changing dimensions, CDC, schema drift, and formats such as Parquet and Avro
- Proficiency with Delivery fundamentals (orchestration with Lakeflow Jobs, Airflow, or Dagster; Git and code review; Continuous Integration/Continuous Deployment (CI/CD); cloud object storage on a major cloud provider)
- Advanced SQL skills, including window functions, CTEs, complex joins, aggregations over large datasets, and identifying why queries are slow through query plan analysis
- An in-depth understanding of incremental processing, idempotency, and late-arriving data
- Experience with debugging correctness issues by reasoning through joins, grain, and duplication
- Hands on Databricks experience required, including:
- Delta Lake at scale: MERGE-based upserts, OPTIMIZE, time travel, schema evolution
- Lakeflow Declarative Pipelines and Lakeflow Jobs. Prior Delta Live Tables and Workflows experience counts
- Unity Catalog for governance, access control, and lineage
- Databricks SQL warehouses and compute selection, including cost awareness
- Ability to communicate with stakeholders at all levels, ensuring action can be taken from written reports without follow-up meetings
- Works well with analysts, product owners, and source system teams to turn ambiguous requirements into workable specifications
- Scopes work into shippable increments
- Strong organizational and project management skills with the ability to manage multiple priorities and deadlines
- A proactive, can-do attitude with a willingness to take ownership of tasks and drive them to completion
- Ability to work independently while also being a team player who thrives in a collaborative environment
- Discretion and trustworthiness in handling sensitive information and supporting high-level strategic initiatives
- Ability to travel and/or work onsite as needed
- While performing the duties of this job in a home office setting, the employee is regularly required to work on a computer for extended periods of time
- Frequent use of a computer requires fine motor skills and hand-eye coordination
- Ability to sit for extended periods while working from home or a designated workspace
- Ability to perform tasks that require sustained attention and focus
- Occasional lifting of materials up to 25 pounds
- Travel to attend team meetings may be required
- To facilitate working from home, and as a requirement for this role, the employee must provide reliable internet connection with sufficient bandwidth to execute all job functions and technology setup conducive to remote work
- A quiet, distraction-free workspace is required for maintaining productivity
- Bachelor's degree in Business, Information Systems, Data Science, or related field preferred
- Databricks Certified Data Engineer Associate or Professional certifications are highly valued
- Lakeflow Connect or another managed ingestion or CDC tool, plus Databricks Asset Bundles, Terraform, or Docker
- Data quality and observability tooling, BI tools and reverse ETL, and PII masking and retention practices
- RAG, vector search, or document processing workloads, and MLflow tracing and evaluation
- No prior AI experience is required, but GenAI Engineering experience (document ingestion pipelines, vector search indexes, and MLflow tracing and evaluation) is valued
Benefits
- Remote/hybrid work arrangement in the USA
- The company laptop will be provided
Company Overview
Company H1B Sponsorship