Description
Vi manages a petabyte scale data lakehouse that drives data and ML pipelines across all our products. Vi is looking for an engineer with deep expertise building and scaling big-data and ML pipelines. The role will be a hybrid between an FDE that owns integrations of customer data with Vi’s data lakehouse and a platform engineer responsible for packaging Vi’s products into repeatable, scalable, automated pipelines.
Tech Stack
- Apache PySpark, Apache Iceberg, AWS EMR, Glue, S3
- Python, PyTorch, sklearn, xgboost, catboost, mlflow
- AWS SageMaker, Airflow
Key Responsibilities
- Own the design and implementation of data and ML pipelines based on a combination of customer first party data and Vi’s internal data lakehouse.
- Prototype predictive insight products and run rapid feasibility studies to assess new commercial opportunities for Vi’s capabilities built on petabyte scale data lakehouse.
- Identify commonalities between customer engagements to synthesize reusable data capabilities.
What We’re Looking For
- Deep expertise building large-scale data analytics pipelines with Apache Spark.
- Comfortable using Python and AWS technologies for data engineering and ML.
- Strong communication skills to interact with technical counterparts across customer accounts to coordinate data integrations.
Nice to Have
- Experience in the healthcare and life sciences domain.
- Familiarity with ML and statistical modeling methodologies and experimental design.
Originally posted on