Note: The job is a remote job and is open to candidates in USA. Claritev is a healthcare technology company seeking a Senior AI Agent Quality Engineer to serve as the quality gatekeeper for agentic AI systems powering its next-generation healthcare products. The role designs and runs test suites, benchmarks, and evaluation pipelines; monitors agent quality and safety; investigates failures; and collaborates with engineering, data science, and product teams to improve reliability and compliance.
Responsibilities
- Design, build, and maintain automated test suites and evaluation pipelines for agentic AI systems, covering single-turn, multi-turn, and end-to-end workflow scenarios
- Develop and curate golden datasets, test scenarios, and simulation environments that reflect real healthcare workflows and edge cases
- Execute and automate agent benchmarking, including regression testing across model, prompt, and tool changes, with clear pass/fail quality gates in CI/CD
- Measure and report agent quality metrics: task completion, tool-call accuracy, grounding/hallucination rates, response quality, latency, cost per task, and safety compliance
- Implement LLM-as-judge and rubric-based scoring approaches, and validate automated scores against human review
- Triage, reproduce, and root-cause agent failures by analyzing traces of LLM calls, tool invocations, retrieval steps, and orchestration paths; file actionable defect reports
- Perform adversarial and negative testing, including prompt-injection probes, malformed inputs, boundary conditions, and guardrail verification
- Monitor production agent behavior, investigate quality incidents and drift, and feed findings back into offline test coverage
- Contribute to evaluation dashboards, runbooks, and quality documentation used by engineering and product teams
- Verify PHI/PII handling, auditability, and human-in-the-loop controls behave as designed, supporting HIPAA and data-governance compliance
- Collaborate with AI engineers and data scientists to make agents more testable, observable, and reliable by design
Skills
- Bachelor's degree in Computer Science, Engineering, Data Science, a quantitative discipline, or a related field required
- 5+ years of hands-on experience in software engineering, QA/test automation engineering, ML engineering, or a related technical discipline
- 2+ years of experience testing, operating, or supporting production ML or AI systems
- 1+ years of hands-on experience with generative AI, LLMs, RAG, and/or agentic AI systems (professional projects preferred; substantial personal or open-source work considered)
- Track record of building test automation or evaluation capabilities that measurably improved software or model quality
- Strong Python skills, including test frameworks (e.g., pytest) and building automation/tooling; comfort working with APIs and distributed systems
- Hands-on experience with LLM/agent evaluation concepts: eval datasets, LLM-as-judge, rubric-based scoring, and regression detection
- Familiarity with evaluation and observability tooling such as LangSmith, Langfuse, Arize Phoenix, Ragas, DeepEval, promptfoo, or equivalent
- Working knowledge of agentic AI frameworks and patterns (e.g., LangGraph, LangChain, AutoGen, CrewAI), including tool use, orchestration, and guardrails
- Understanding of RAG pipelines, embeddings, and vector search, and how retrieval quality affects agent behavior
- Experience with CI/CD pipelines and integrating automated tests and quality gates into them
- Familiarity with cloud environments, containers, and log/trace analysis for debugging distributed systems
- Solid grasp of statistics fundamentals for interpreting evaluation results (sampling, variance, significance)
- Strong analytical and debugging skills, with high attention to detail and a healthy skepticism toward “it works on my machine.”
- Clear written communication, especially in defect reports, test plans, and quality summaries
- Ability to operate effectively in a fast-moving, cross-functional environment
- Master's degree a plus
- Experience with Oracle Cloud Infrastructure (OCI), including its generative AI, data science, database, and observability capabilities
- Experience in healthcare, health technology, insurance, claims, payment integrity, or other regulated industries
- Experience testing systems that process sensitive data, including PHI or PII, in HIPAA-regulated environments
- Exposure to public agent/LLM benchmarks (e.g., SWE-bench, GAIA, AgentBench, tau-bench) or red-teaming and safety evaluation of LLM systems
- Experience with Kubernetes, infrastructure-as-code (e.g., Terraform), or SRE practices
- Prior QA/SDET background applied to ML or data-intensive systems
Benefits
- This position is also eligible for health insurance, 401k and bonus opportunity.
- Remote work arrangement
- Medical, dental, and vision coverage
- Life and disability coverage
- 401(k) with company match
- Employee stock purchase plan
- Flexible spending accounts and health savings accounts
- Paid time off and company holidays
- Paid parental leave
- Mental health resources
- Tuition reimbursement
- Financial planning resources
- Professional development opportunities
Company Overview
Company H1B Sponsorship