Note: The job is a remote job and is open to candidates in USA. Acunor is seeking an experienced AI Evaluation Engineer to establish quality standards for production AI and LLM systems built using Amazon Bedrock. The role owns centralized evaluation frameworks, golden datasets, evaluation thresholds, testing audits, safety and accuracy assessments, and continuous production monitoring.
Responsibilities
- Design and build centralized LLM/AI evaluation frameworks and reusable evaluation templates
- Establish evaluation standards that can be consistently adopted across multiple AI delivery teams
- Build and maintain a central evaluation harness that individual delivery pods can use for application-specific testing
- Create and maintain golden datasets in partnership with business, domain, and clinical subject-matter experts
- Develop evaluation suites covering functional quality, accuracy, safety, reliability, medical accuracy, and domain-specific requirements
- Define measurable pass/fail thresholds for production AI systems
- Audit evaluation suites developed by individual engineering teams for quality and rigor
- Implement and assess LLM-as-a-judge evaluation pipelines
- Identify and mitigate LLM-as-a-judge failure modes including position bias, verbosity bias, and self-preference
- Calibrate automated evaluation results against human labels and perform inter-rater checks
- Evaluate RAG and retrieval quality , including groundedness and hallucination rates
- Evaluate AI applications using Bedrock Guardrails, Bedrock Knowledge Bases, and multiple foundation models
- Establish continuous production monitoring for model drift and regression
- Integrate evaluation processes with Amazon Bedrock invocation logs, CloudWatch metrics/alarms, and automated re-evaluation pipelines
- Track and report evaluation pass rates and production AI quality incidents to leadership
Skills
- 4+ years of experience in ML/LLM evaluation, AI-focused QA engineering, or applied research engineering
- Hands-on experience with LLM evaluation frameworks such as Promptfoo, Ragas, DeepEval, OpenAI Evals, or equivalent custom evaluation frameworks
- Strong Python skills with the ability to build evaluation harnesses from scratch
- Deep understanding of LLM-as-a-judge methodologies and associated failure modes
- Experience developing golden datasets, regression evaluation suites, and measurable AI quality thresholds
- Strong statistical understanding, including confidence intervals, significance testing, and evaluation of pass-rate differences using small samples
- Ability to independently establish and defend AI quality standards under delivery pressure
- Working knowledge of AWS and the ability to automate evaluations against Amazon Bedrock endpoints
- Experience integrating evaluation and monitoring with CloudWatch or equivalent observability systems
- Ability to work full Pacific Time (PST/PDT) business hours
- Healthcare or safety-critical AI evaluation experience
- Experience evaluating clinical or medical accuracy
- Red-teaming experience involving adversarial testing, jailbreak testing, and prompt-injection testing
- Experience configuring or evaluating Amazon Bedrock Guardrails or equivalent AI safety layers
- Data annotation program experience, including rubric design, labeler training, and annotation quality control
Benefits
- Remote work – United States, Canada, or Latin America
Company Overview
Company H1B Sponsorship