Note: The job is a remote job and is open to candidates in USA. Brilliant Infotech Inc. is seeking a Data Scientist / ML Engineer to migrate and modernize Dataiku data science and machine learning workflows to Databricks. The role focuses on scalable data pipelines, ML lifecycle management, MLOps, CI/CD, monitoring, performance optimization, and collaboration with data and platform teams.
Responsibilities
- Migrate Dataiku projects, recipes, scenarios, and models to Databricks
- Rebuild data pipelines, feature engineering, model training, and scoring workflows using Databricks, Spark/PySpark, Python, and SQL
- Develop scalable Lakehouse, Delta Lake, and ML pipelines
- Implement MLflow for experiment tracking, model registry, and deployment
- Implement CI/CD, automated retraining, monitoring, and data quality practices
- Optimize distributed processing for performance and cost
- Collaborate with Data Scientists, Data Engineers, and platform teams; document and provide knowledge transfer
Skills
- • 5+ years Data Science / Analytics / ML Engineering
- • Strong hands-on experience with Dataiku DSS and Databricks
- • Strong Python, PySpark, SQL, and Apache Spark skills
- • Experience with MLflow / MLOps and data/ML platform migration
- • Experience with AWS, Azure, or GCP
- • Delta Lake, Feature Stores, Streaming
- • Git, GitHub, CI/CD, Azure DevOps
- • Airflow / Databricks Workflows
- • Data governance, security, and enterprise data platforms
Company Overview
Company H1B Sponsorship