Note: The job is a remote job and is open to candidates in USA. EPAM Systems is developing an AI Reliability Engineering capability within its Operational Intelligence practice to support production AI, ML, and LLM systems. The Senior AI Reliability Engineer will instrument, monitor, and harden AI applications, establish AI-native reliability objectives, improve incident and cost management, and create reusable accelerators and reference implementations.
Responsibilities
- Add AI telemetry to production LLM, RAG, and agentic applications using OpenTelemetry and APM-native AI monitoring tools
- Deploy distributed tracing across multi-model chains, agent workflows, and retrieval-augmented generation pipelines to identify systemic latency and failure points
- Establish and track AI-native SLIs and SLOs, including time to first token (TTFT), throughput, error and refusal rates, cost per request, semantic drift, hallucination boundaries, and contextual accuracy
- Implement structured semantic logging and prompt/response monitoring to support quality analysis
- Create and maintain evaluation loops for output quality and safety using golden sets, LLM-as-a-judge methods, and Ragas/DeepEval-style frameworks, integrating them into CI/CD and runtime environments
- Monitor token-based cloud spend, model API rate limits, and quota usage while driving AI cost optimization efforts
- Set up AI gateways to manage API load balancing, failover, and fallback models across multiple LLM providers
- Build guardrails covering prompt injection and jailbreak filtering, output compliance, and bias and safety constraints
- Develop detection, triage, restoration, and problem management workflows for AI incidents, incorporating autonomous AI agents into root cause analysis to parse logs, generate hypotheses, and correlate state changes
- Enable rollback, canary, and fail-safe patterns for model, prompt, and configuration releases, while maintaining reproducibility through versioning of data, code, prompts, and models
- Develop practice accelerators, reference architectures, and internal training materials, and contribute to presales activities and client assessments
Skills
- 4+ years of experience in SRE, DevOps, platform, or observability engineering, with hands-on exposure to production AI/ML or LLM workloads
- Strong grasp of SRE fundamentals, including Golden Signals, SLI/SLO definition, error budgets and burn rate, incident lifecycle, and ITIL basics
- Strong Python skills for building instrumentation, automation, and evaluation tools
- Hands-on experience with OpenTelemetry and at least one APM/observability platform such as New Relic, Datadog, Grafana LGTM stack, Splunk, or Elastic
- Production experience with at least one cloud platform (Azure preferred, AWS or GCP also acceptable) and Kubernetes
- Solid understanding of LLM application architecture, including prompts, embeddings and vector stores, RAG, and agent orchestration tools like LangChain/LangGraph or similar
- Awareness of MLOps concepts, including model lifecycle (training vs. inference), model endpoints, containerization, and deployment/rollback patterns
- Experience with Infrastructure as Code using Terraform, along with CI/CD tools such as Azure DevOps, GitLab CI, or GitHub Actions
- B2+ English proficiency, since the role involves direct client interaction and requires clear technical communication in both writing and speech
- Familiarity with AI-specific observability and evaluation tools such as Traceloop/OpenLLMetry, Langfuse, Arize Phoenix, Ragas, DeepEval, or MLflow
- Experience with distributed inference serving at scale, including vLLM, KServe, Ray Serve, Kubernetes-native LLM orchestration, and GPU capacity planning
- Knowledge of AI security practices, including the OWASP LLM Top 10, prompt injection defense, and guardrail frameworks like NeMo Guardrails or Llama Guard
- Experience with Databricks (including Mosaic AI/MLflow) or Azure AI Foundry
- Relevant certifications such as Anthropic Claude, Azure AI Engineer, AWS ML Specialty, or Databricks GenAI
- Background in FinOps for AI workloads, particularly token and GPU cost modeling
- Experience in Data Reliability Engineering, covering data quality and pipeline SLOs, since AI reliability depends on data reliability
- Prior mentoring or team lead experience
Benefits
- Flexible schedule with remote work or work from EPAM offices or coworking spaces in Ukraine
- Necessary equipment to perform work tasks
- Ability to change projects and technology stacks within EPAM
- Relocation opportunities may be available for eligible candidates, depending on the role and openings at other EPAM locations
- Volunteer and charity programs and communities
- Individual career-path planning with a manager
- Regular feedback from colleagues
- Free English improvement with certified teachers, including Speaking Clubs and client interview preparation courses
- Free training and certification in AWS, GCP, or Azure Clouds
- Access to the internal E-learn training program with 18,200+ specialized training and mentoring programs
- Corporate accounts on LinkedIn Learning, Get Abstract and other partner resources
- Access to EPAM Solution Architecture School with practicing architect instructors
- Leadership development opportunities, including Delivery Management, Resource Management, and Leadership Essentials schools
- Internal communities, including meetups, technical discussions, brainstorming sessions, online events and conferences
- Vacation and sick leave, including sick leave without a medical certificate
- Voluntary Medical Insurance programs providing medical treatment and preventive options, including sports activities
- Medical insurance for family members at corporate rates
- Company support during significant life events, including childbirth or adoption and marriage
- Discounts on services from mental health specialists or coaches and thematic training
- E-kids program providing free programming language training for EPAMers' children
Company Overview
Company H1B Sponsorship