Note: The job is a remote job and is open to candidates in USA. Codefeast is seeking experienced AI Benchmark Quality Reviewers to assess the quality, correctness, and fairness of AI benchmark tasks and grading systems. The role focuses on reviewing task instructions, reference solutions, execution traces, automated graders, and evaluation criteria to ensure AI model performance is assessed accurately.
Responsibilities
- Validate task instructions, source materials, reference solutions, and evaluation criteria
- Review AI agent execution traces, tool calls, and generated deliverables
- Identify hidden requirements, missing information, and inconsistencies in benchmark tasks
- Distinguish genuine AI model limitations from task defects, grader issues, environment failures, or tool errors
Skills
- 5+ years of relevant professional experience
- Comfortable reading and understanding: • Python • SQL • Shell scripts
- Ability to provide specific, reproducible, evidence-backed feedback
Company Overview