Note: The job is a remote job and is open to candidates in USA. 24-MAG LLC offers this independent consulting opportunity to evaluate how advanced AI systems handle data science, statistics, machine learning, and quantitative problems. The role includes assessing model outputs, developing prompts, datasets, and reference solutions, and providing clear feedback to support model improvement.
Responsibilities
- Evaluate AI-generated responses to data science, statistics, machine learning, and quantitative problems
- Assess whether analytical methods, assumptions, and conclusions are technically sound
- Identify flawed methodology, incorrect statistical reasoning, and unsupported interpretations
- Review model outputs for numerical accuracy, logical consistency, and appropriate use of evidence
- Apply professional data-science judgement across complex analytical scenarios
- Assess AI reasoning involving machine learning models, statistical methods, and predictive analysis
- Evaluate whether modelling choices are appropriate for the data and problem context
- Identify issues involving overfitting, leakage, bias, weak validation, or inappropriate metrics
- Compare alternative analytical approaches and explain relevant trade-offs
- Determine whether conclusions are adequately supported by quantitative evidence
- Create expert-level prompts that test advanced quantitative and analytical capability
- Develop datasets and scenarios suitable for rigorous model evaluation
- Produce high-quality reference solutions grounded in sound statistical and data-science practice
- Define assumptions, calculations, and methodology clearly
- Ensure reference materials are reproducible, technically defensible, and appropriately challenging
- Evaluate experiment design, hypothesis testing, and interpretation of results
- Review analytical workflows involving Python, pandas, and related data-science tools
- Assess data preprocessing, feature construction, and data-modelling approaches
- Identify weaknesses in experimental setup, measurement, or causal interpretation
- Apply strong quantitative reasoning when reviewing ambiguous or incomplete data scenarios
- Provide structured feedback on errors, weaknesses, and opportunities for improvement
- Explain statistical and technical issues in clear written language
- Identify recurring patterns of flawed reasoning across model outputs
- Contribute high-quality evaluation material that improves model capability
- Maintain consistent standards across diverse data-science and quantitative tasks
Skills
- At least 1 year of professional experience working in data science, quantitative analysis, machine learning, or a closely related role at a top-tier or highly regarded organisation
- Relevant experience may come from technology, finance, research, or comparable quantitative environments
- At least part of the relevant professional experience should have been gained within approximately the past 7 years
- Strong proficiency in Python
- Strong knowledge of statistics, experimentation, and quantitative analysis
- Practical experience with machine learning and data modelling
- Familiarity with pandas and modern data-analysis workflows
- Strong ability to identify methodological errors and weak statistical reasoning
- Excellent written communication and ability to explain technical concepts clearly
- Undergraduate degree from a highly ranked university is preferred
- Candidates based in English-speaking countries are preferred
Benefits
- Fully remote
Company Overview