We are sharing a specialised consulting opportunity for PhD-level researchers, professors, and advanced subject-matter experts to contribute to an AI evaluation project focused on developing rigorous, research-grade test material for advanced AI systems.
Selected professionals will create original, high-difficulty questions within their areas of expertise, develop thoroughly sourced reference answers, and test whether advanced AI systems can solve problems requiring genuine specialist reasoning. The work requires strong research judgement, source triangulation, methodological precision, and the ability to produce defensible, unambiguous evaluation material. No prior experience in AI is required.
Key Responsibilities
Advanced Research Question Development
Create original, high-difficulty question-and-answer pairs at the frontier of your academic or research discipline
Design questions requiring advanced reasoning, methodological nuance, or synthesis across multiple sources
Develop problems that cannot be solved reliably through superficial pattern matching or simple lookup
Ensure questions reflect genuine specialist-level knowledge and research complexity
Maintain originality, technical depth, and intellectual rigour across submitted tasks
Source Verification & Research Triangulation
Research and verify answers using primary sources and authoritative references
Triangulate information across multiple high-quality sources when necessary
Provide clear citations and supporting reasoning for reference answers
Distinguish well-supported conclusions from uncertain, contested, or insufficiently evidenced claims
Ensure all supporting material is accurate, traceable, and appropriate for expert-level evaluation
AI Model Testing & Difficulty Calibration
Test developed questions against advanced AI systems
Identify questions that are insufficiently challenging or vulnerable to shortcut solutions
Iteratively increase difficulty while preserving factual accuracy and methodological validity
Analyse where AI systems succeed or fail when addressing specialist research problems
Refine questions to better measure genuine reasoning and subject-matter capability
Precision, Review & Quality Standards
Write questions and answers with exceptional clarity and precision
Eliminate ambiguity that could prevent objective or defensible evaluation
Incorporate reviewer feedback and revise deliverables accordingly
Follow project guidelines, formatting requirements, and quality standards consistently
Maintain reliable quality across independent, remote research workflows
Ideal Profile
Completed PhD, active PhD candidacy, or equivalent advanced research experience as a specialist, researcher, or professor
Demonstrated record of scholarly research and deep subject-matter expertise
Strong familiarity with primary literature and authoritative research sources within your field
Excellent analytical reasoning and ability to engage with methodologically complex problems
Experience sourcing, verifying, and triangulating information across authoritative references
Exceptional attention to detail and written precision
Strong written English communication skills
Ability to create challenging, original, and methodologically sound research questions
Proven self-direction and reliability when completing expert-level work independently
Comfortable incorporating reviewer feedback and iterating on technical material
Previous experience in AI training, model evaluation, or related research workflows is advantageous but not required
Engagement Details
Independent contractor engagement
Fully remote
Compensation: $40–$90/hour
Compensation is output-based, with payment made for tasks that meet project specifications
Minimum weekly submission requirements apply
Work will involve advanced research question development, source triangulation, reference-answer authoring, AI model testing, and iterative difficulty calibration
Task completion time may vary depending on discipline, research complexity, and individual workflow
Selected professionals should be prepared to begin their first tasks within approximately 24–48 hours of completing onboarding
Roles are typically filled within approximately 48 hours
Project scope, workload, and evaluation standards may evolve depending on project requirements
Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, research group, or other third party
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy