Note: The job is a remote job and is open to candidates in USA. Hollstadt Consulting is seeking a Senior Consultant, AI/ML Ops Engineer to operationalize machine learning and LLM workloads across NMDP. The role focuses on building and maintaining AWS SageMaker-based training and inference platforms, Terraform infrastructure, GitLab CI/CD pipelines, monitoring, security controls, and automated model deployment and remediation.
Responsibilities
- Own the ML lifecycle end-to-end — build and operate SageMaker training pipelines and inference endpoints, model registry, versioning, promotion (validation → live), and blue/green deployment with automated rollback
- Build MLOps infrastructure as code — author and review Terraform for SageMaker, S3, KMS, IAM, CloudWatch, and cross-account roles across multiple AWS accounts (dev/eng/prod)
- Run the CI/CD — design GitLab CI/CD pipelines covering testing, IaC security scanning (Checkov), SAST (SonarQube), dependency checks, Terraform plan/apply gates, and automated model/artifact promotion
- Ensure train/serve parity — maintain shared feature-encoding pipelines so preprocessing is identical in training and inference; catch drift before it ships
- Instrument for production — CloudWatch dashboards, alarms, model/data drift monitoring, and endpoint-level observability. Own the on-call story for ML services
- Harden and govern — least-privilege cross-account IAM, KMS encryption, model artifact signing, secrets management, and cost visibility for GPU/inference and Bedrock spend
- Operationalize LLM workloads — support Bedrock-based pipelines (batch and real-time), including throughput/cost management and guardrails
- Partner with data scientists — turn experimental models (XGBoost, survival models, etc.) into reproducible, tested, deployable services, and coach the team on MLOps best practices
- Bring AIOps to the platform — add anomaly detection on endpoint, drift, and cost signals, and event-driven auto-remediation (auto-rollback, scaling) so operational issues are caught and handled before they escalate
- Reduce operational toil with AI — use LLM-assisted log/trace analysis and alert correlation to speed up incident triage and root-cause analysis, cutting MTTR and on-call noise
Skills
- 5+ years in MLOps / ML platform / ML infrastructure engineering, with production ownership of deployed models
- Deep AWS expertise , especially SageMaker (training jobs, pipelines, model registry, endpoints), plus S3, IAM, KMS, CloudWatch, Lambda, Step Functions, and multi-account architectures
- Strong DevOps foundation : Infrastructure as Code with Terraform , CI/CD pipeline design (GitLab CI, GitHub Actions, or similar), Docker, and Git-based workflows
- Strong data science fundamentals — you understand model training, evaluation metrics (e.g., C-index, AUC, calibration), feature engineering, and can reason about why a model behaves as it does in production
- Proficient in Python (production-grade, tested code) for both ML tooling and services
- Experience with model monitoring, drift detection, and the operational realities of models degrading over time
- Solid grasp of security best practices: least-privilege IAM, secrets management, encryption at rest/in transit
Benefits
- Remote
Company Overview