Key responsibilities
Create expert-level technical questions: Develop challenging problems that require advanced reasoning, domain knowledge, and professional judgment rather than simple factual recall.
Design evaluation rubrics: Define clear, measurable criteria for correct answers, including required concepts, reasoning steps, acceptable alternative approaches, and critical errors.
Establish partial-credit standards: Distinguish between essential steps, minor mistakes, acceptable alternatives, and critical failures.
Evaluate AI-generated solutions: Assess correctness, reasoning quality, completeness, methodology, and technical precision.
Identify hidden reasoning errors: Detect flawed assumptions, invalid logic, missed edge cases, and incorrect methods, even when the final answer is correct.
Review other experts' rubrics: Identify ambiguity, missing criteria, redundancy, technical inaccuracies, and inconsistent scoring.
Develop reference content: Produce expert answers, explanations, critiques, and gold-standard examples.
Support AI model training: Create evaluation questions, rubrics, preference judgments, and reference-answer pairs for reinforcement learning (RL) and supervised fine-tuning (SFT).
Maintain technical rigor: Ensure all content meets appropriate professional or academic standards