Note: The job is a remote job and is open to candidates in USA. Bitcruit Solutions is recruiting for an unnamed client seeking a Senior Databricks Data Engineer with pharmaceutical and scientific data experience. The role focuses on building scalable Databricks data solutions for scientific datasets and working with scientists and business stakeholders to gather requirements, model data, and resolve data quality and integration issues.
Responsibilities
- Design, develop and optimize data pipelines, transformations, views and data solutions within Databricks
- Work directly with scientists and business stakeholders to gather, clarify and document requirements
- Design data models and mappings for complex scientific datasets, particularly laboratory, assay and clinical data
- Establish relationships between scientific entities such as compounds, assays, experimental results and associated identifiers
- Integrate data from multiple scientific systems, laboratory sources and external vendors
- Develop processes for handling corrected or revised scientific data, including pipeline reprocessing and maintaining data accuracy
- Translate scientific and functional requirements into robust, scalable Databricks solutions
- Demonstrate technical approaches, explain design decisions and guide stakeholders toward appropriate solutions
- Troubleshoot data quality, integration and modeling issues
Skills
- Extensive hands-on experience with Databricks
- Strong Python, PySpark and SQL skills
- Experience building and maintaining data ingestion and transformation pipelines
- Strong data modeling, mapping and integration experience
- Experience managing pipeline reruns, incremental processing, corrections and data quality
- Proven experience working with pharmaceutical, biotechnology or life sciences data
- Hands-on experience with laboratory and assay datasets
- Understanding of scientific data structures, experimental results, compounds and associated identifiers
- Experience integrating, transforming and modeling scientific data for analytics
- Experience gathering technical and functional requirements directly from scientists or scientific business users
- Ability to ask relevant questions to uncover business needs, data relationships and quality issues
- Strong analytical and problem-solving capabilities
- Excellent verbal and written communication skills
- Ability to explain technical solutions, trade-offs and recommendations to nontechnical stakeholders
- Capable of working independently and taking ownership of requirements through implementation
- Familiarity with clinical data is valuable
- Experience working in pharmaceutical R&D, drug discovery or scientific research environments
- Familiarity with laboratory data sources and scientific data workflows
- Experience developing data solutions supporting scientific analysis and decision-making
- Databricks certifications
Benefits
- 100% Remote - US-based candidates only (Onshore)
Company Overview