Note: The job is a remote job and is open to candidates in USA. ControlRooms.ai is the AI troubleshooting platform for chemical, energy, and materials manufacturers, designed to detect plant issues earlier and resolve them faster. The role involves hands-on ML Engineering / MLOps responsibilities, including owning pipelines, ensuring model reliability at scale, and collaborating across teams to deliver production-ready machine learning solutions.
Responsibilities
- Design and build ML training and inference pipelines for time-series prediction (anomaly detection and forecasting)
- Build distributed processing services with Prefect and Ray (Ray Serve on Kubernetes) and package them as FastAPI microservices for our full-stack applications
- Deploy models for real-time scoring on Azure ML managed online endpoints, including blue/green rollout and autoscaling, integrated with the services that orchestrate and call them
- Orchestrate distributed training and batch scoring with Prefect and Azure ML, including parallel hyperparameter search (Optuna) across per-tenant compute pools
- Familiarity with tuning Linear Regression, Neural Networks, Tree Methods, and other standard predictive methods
- Familiarity with tuning unsupervised anomaly detection models
- Own model reliability at scale: diagnose and fix production failure modes (numerical instability, data-quality edge cases, memory/serialization issues) and harden pipelines against them
- Build validation rigor — time-series cross-validation / backtesting, champion-challenger comparison against production baselines, and experiment tracking with MLflow
- Implement monitoring, logging, distributed tracing, and alerting (e.g. OpenTelemetry) so model and pipeline health is observable
- Collaborate across data science, front-end, product, and QA to ship production-ready ML, and recommend improvements from model-performance analysis
Skills
- 3+ years in machine learning engineering, shipping models to production
- Strong Python and the ML stack: pandas, NumPy, scikit-learn, and a deep-learning framework (PyTorch, TensorFlow, or Keras)
- Hands-on experience training, debugging, and improving ML/DL models — able to diagnose why a model misbehaves, not just train it
- MLOps in practice: pipeline orchestration, automated monitoring/logging/alerting, experiment tracking, and CI/CD (Azure DevOps / Azure Pipelines or similar)
- Production model deployment, including online serving and safe-rollout strategies (blue/green)
- Distributed processing with Ray or a similar framework, exposed as FastAPI (or comparable) microservices
- Building and running secure services in Kubernetes (Docker; service mesh such as Istio a plus)
- Strong SQL and experience with databases at scale (PostgreSQL; TimescaleDB / time-series a strong plus)
- Strong problem-solving and the communication skills to explain technical trade-offs
- Bachelor's or Master's in Computer Science, Electrical Engineering, Mathematics, or related field (or equivalent experience)
- Git and software development methodologies like Agile, Scrum, or Kanban
- Industrial / IoT / sensor or other high-volume time-series data experience
- Orchestration and MLOps tooling: Prefect, MLflow, Kubeflow, BentoML
- Hyperparameter optimization (Optuna) and time-series modeling / ensembles / forecasting
- Observability tooling (OpenTelemetry, Grafana / Loki / Tempo)
Company Overview