Note: The job is a remote job and is open to candidates in USA. EPAM is a global leader in AI transformation engineering and integrated consulting. The company is seeking a Senior ML / Evaluation Engineer to build evaluation capabilities for an enterprise agent development platform, including custom evaluators, quality metrics, and automated checks. The role focuses on validating LLM-based applications and AI agents and integrating evaluations into CI/CD and production monitoring.
Responsibilities
- Design and implement evaluation frameworks for LLM-based applications and AI agents
- Develop LLM-as-a-Judge and deterministic, code-based evaluators
- Build custom Python evaluators for quality and behavioral checks
- Define evaluation criteria, metrics, thresholds, and acceptance rules
- Evaluate agent behavior across individual responses, tool calls, and complete workflows
- Work with OpenTelemetry traces and spans as evaluation data
- Integrate evaluations into CI/CD pipelines and automated deployment gates
- Enable continuous quality monitoring of solutions in production
- Establish reusable evaluation patterns and engineering standards
- Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process
Skills
- 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering
- Strong Python development experience
- Hands-on experience with LLM/GenAI evaluation
- Experience designing and implementing evaluation frameworks
- Experience developing custom or deterministic evaluators
- Experience integrating AI/ML quality checks into CI/CD
- Good understanding of LLM and AI agent architectures
- Hands-on experience with AWS AgentCore Evaluation
- Experience with AWS Bedrock Guardrails, including PII detection
- Knowledge of CloudWatch metrics and production monitoring
- Experience with OpenTelemetry
- Familiarity with LangGraph, Strands Agents, or similar agent frameworks
Benefits
- Georgia: Enjoyable working environment, whether it is the vibrant office or the comfort of your own home.
- Georgia: Opportunity to work abroad for up to two months per year.
- Georgia: Relocation opportunities within our offices in 55+ countries.
- Georgia: Corporate and social events.
- Georgia: Leadership development, career advising, soft skills and well-being programs.
- Georgia: Certifications, including GCP, Azure and AWS.
- Georgia: Unlimited access to EPAM's internal learning database.
- Georgia: Free English classes with certified teachers.
- Georgia: Participation in the Employee Stock Purchase Plan.
- Georgia: Monetary bonuses for engaging in the referral program.
- Georgia: Comprehensive medical & family care package.
- Georgia: Five trust days per year (sick leave without a medical certificate).
- Georgia: Benefits package including sports activities and a variety of stores and services.
- Kazakhstan: Enjoyable working environment, whether it is the vibrant office or the comfort of your own home.
- Kazakhstan: Opportunity to work abroad for up to two months per year.
- Kazakhstan: Relocation opportunities within our offices in 55+ countries.
- Kazakhstan: Corporate and social events.
- Kazakhstan: Leadership development, career advising, soft skills and well-being programs.
- Kazakhstan: Certifications, including GCP, Azure and AWS.
- Kazakhstan: Unlimited access to EPAM's internal learning database.
- Kazakhstan: Free English classes with certified teachers.
- Kazakhstan: Discounts in local language schools, including online courses for the Kazakh language.
- Kazakhstan: Participation in the Employee Stock Purchase Plan.
- Kazakhstan: Monetary bonuses for engaging in the referral program.
- Kazakhstan: Medical & family care package.
- Kazakhstan: Six trust days per year (sick leave without a medical certificate).
- Kazakhstan: Coverage of psychology sessions of your choice.
- Kazakhstan: Benefits package including sports activities and a variety of stores and services.
- Kazakhstan: Housing support program with preferential mortgage access via Otbasy Bank partnership.
- Kyrgyzstan: Enjoyable working environment, whether it is the vibrant office or the comfort of your own home.
- Kyrgyzstan: Opportunity to work abroad for up to two months per year.
- Kyrgyzstan: Relocation opportunities within our offices in 55+ countries.
- Kyrgyzstan: Corporate and social events.
- Kyrgyzstan: Leadership development, career advising, soft skills and well-being programs.
- Kyrgyzstan: Certifications, including GCP, Azure and AWS.
- Kyrgyzstan: Unlimited access to EPAM's internal learning database.
- Kyrgyzstan: Free English classes with certified teachers.
- Kyrgyzstan: Monetary bonuses for engaging in the referral program.
- Kyrgyzstan: Medical & family care package.
- Kyrgyzstan: Six trust days per year (sick leave without a medical certificate).
- Kyrgyzstan: Coverage of psychology sessions of your choice.
- Kyrgyzstan: Discounts for fitness clubs and sports programs.
- Kyrgyzstan: Benefits package including sports activities and a variety of stores and services.
- Turkiye: Diversity of tasks and projects.
- Turkiye: Assessment center for objective review of competency level.
- Turkiye: Personal development plan.
- Turkiye: Mentoring programs and leadership development.
- Turkiye: Certification and professional development support.
- Turkiye: Access to learning platforms including more than 2,500 internal courses.
- Turkiye: English courses taught by certified teachers.
- Turkiye: Extra leave days.
- Turkiye: Referral bonuses.
- Turkiye: Regular salary and performance reviews.
- Turkiye: Private health insurance.
- Turkiye: Well-being events.
- Turkiye: Recreation areas and kitchens.
- Turkiye: Tea, coffee and snacks.
- Turkiye: Sports equipment and game consoles.
- Turkiye: IT Equipment.
- Turkiye: Microsoft’s Software Assurance Home Use Program (HUP).
- Uzbekistan: Enjoyable working environment, whether it is the vibrant office or the comfort of your own home.
- Uzbekistan: Opportunity to work abroad for up to two months per year.
- Uzbekistan: Relocation opportunities within our offices in 55+ countries.
- Uzbekistan: Corporate and social events.
- Uzbekistan: Leadership development, career advising, soft skills and well-being programs.
- Uzbekistan: Certifications, including GCP, Azure and AWS.
- Uzbekistan: Unlimited access to EPAM's internal learning database.
- Uzbekistan: Free English classes with certified teachers.
- Uzbekistan: Discounts in local language schools, including offline courses for the Uzbek language.
- Uzbekistan: Monetary bonuses for engaging in the referral program.
- Uzbekistan: Medical & family care package.
- Uzbekistan: Four trust days per year (sick leave without a medical certificate).
- Uzbekistan: Discounts for fitness clubs, dance schools and sports programs.
- Uzbekistan: Benefits package including sports activities and a variety of stores and services.
- Armenia: Enjoyable working environment, whether it is the vibrant office or the comfort of your home.
- Armenia: Opportunity to work abroad for up to two months per year.
- Armenia: Relocation opportunities within our offices in 55+ countries.
- Armenia: Corporate and social events.
- Armenia: Leadership development, career advising, soft skills and well-being programs.
- Armenia: Certifications, including GCP, Azure and AWS.
- Armenia: Unlimited access to EPAM's internal learning database.
- Armenia: Free English classes with certified teachers.
- Armenia: Participation in the Employee Stock Purchase Plan.
- Armenia: Monetary bonuses for engaging in the referral program.
- Armenia: Comprehensive medical & family care package.
- Armenia: Four trust days per year for personal needs.
- Armenia: Discounts for fitness clubs.
- Armenia: Benefits package including hotels, restaurants, stores and services.
- Georgia: Remote work from anywhere in Georgia, including home, EPAM offices in Tbilisi and Batumi, or a coworking space in Kutaisi.
- Kazakhstan: Remote work from home or access to EPAM offices in Astana, Almaty or Karaganda.
- Kyrgyzstan: Remote work from anywhere in Kyrgyzstan, including home or the EPAM office in Bishkek.
- Uzbekistan: Remote work from anywhere in Uzbekistan, including home or the EPAM office in Tashkent.
- Armenia: Fully remote work from any location in Armenia, including home or offices in Yerevan or Gyumri.
Company Overview
Company H1B Sponsorship