Note: The job is a remote job and is open to candidates in USA. Weekday AI is recruiting for an unnamed client for a Junior Data Scientist contractor role focused on evaluating and improving AI systems. The role involves assessing AI-generated data science responses, creating prompts, datasets, and reference solutions, and providing structured feedback on accuracy and reliability.
Responsibilities
- AI Model Evaluation: Evaluate AI-generated outputs on data science, statistics, machine learning, experimentation, and quantitative analysis tasks
- Problem and Prompt Development: Create challenging, well-defined prompts and technical problems that assess analytical reasoning and practical data science skills
- Dataset Preparation: Prepare and validate relevant datasets for testing AI models across statistical and machine learning use cases
- Reference Solution Development: Develop accurate reference answers, calculations, and analytical solutions to establish reliable evaluation benchmarks
- Statistical Validation: Identify statistical errors, incorrect assumptions, flawed methodologies, and inappropriate interpretations of data
- Quantitative Reasoning Assessment: Assess whether AI-generated solutions demonstrate sound mathematical reasoning, appropriate analytical methods, and evidence-based conclusions
- Structured Feedback: Provide clear, actionable feedback highlighting errors, missing steps, and opportunities to improve the accuracy and reliability of AI-generated outputs
- Quality Assurance: Validate analytical results, review solution consistency, and ensure that evaluation criteria are objective and reproducible
- Technical Documentation: Explain statistical concepts, analytical approaches, and evaluation findings in a clear and structured manner
Skills
- At least one year of professional experience at a well-established company in technology, finance, research, or a related field
- At least part of the relevant professional experience should have been gained within the past three years
- Foundational knowledge of data science, statistics, machine learning, and quantitative analysis
- Strong analytical and problem-solving skills, with the ability to evaluate data-driven conclusions critically
- Understanding of statistical methods, experimental design, data interpretation, and model evaluation
- Ability to identify incorrect calculations, methodological weaknesses, and unsupported conclusions
- Strong written communication skills, particularly when explaining technical and statistical concepts
- Ability to work independently, follow evaluation guidelines, and deliver accurate, high-quality assessments
- Data Science Fundamentals
- Statistics and Probability
- Machine Learning Fundamentals
- Quantitative Reasoning
- Data Analysis and Interpretation
- Statistical Methodology
- Analytical Problem-Solving
- Technical Writing and Documentation
- Python Programming
- SQL and Data Manipulation
- Pandas, NumPy, and scikit-learn
- Experimental Design and Hypothesis Testing
- Predictive Modeling and Model Evaluation
- Dataset Creation and Validation
- AI Model Evaluation and Benchmarking
- Experience in Technology, Finance, or Research Organizations
- Experience working at a recognized technology company, financial institution, or research organization
- Professional exposure to statistical analysis, machine learning projects, quantitative research, or data-driven decision-making
- Familiarity with evaluating AI-generated answers or validating analytical solutions
- Ability to communicate complex quantitative concepts in clear, concise English
- Candidates based in English-speaking countries are preferred
Company Overview