Note: The job is a remote job and is open to candidates in USA. Weekday AI is a YC W21 company supporting work for a leading AI research organization developing agentic evaluation benchmarks for advanced AI models. The QA/Test Engineer will evaluate benchmark tasks, design test cases, debug technical environments, identify evaluation gaps, and collaborate with researchers and task authors to maintain rigorous quality standards.
Responsibilities
- Design Test Cases: Develop comprehensive test scenarios to verify that benchmark tasks function correctly, including edge cases, failure conditions, and unexpected inputs
- Review Task Quality: Thoroughly review task descriptions, requirements, expected outcomes, and reference solutions to identify ambiguity, inconsistencies, missing requirements, and potential quality issues
- Debug & Troubleshoot: Use Python and other development tools to investigate and resolve issues within task environments, validation logic, and automated checks
- Develop QA Processes: Create practical, repeatable testing procedures, quality checklists, and review frameworks that can be consistently applied across benchmark tasks
- Identify Evaluation Gaps: Examine grading logic and AI-agent execution results to detect loopholes, shortcuts, inconsistent scoring, or other factors that could compromise benchmark reliability
- Collaborate with Researchers: Work closely with researchers and task authors to communicate issues clearly, recommend improvements, and ensure fixes are properly validated
- Maintain Quality Standards: Help establish and enforce rigorous quality standards across complex technical evaluation datasets
Skills
- * **Education:** MSc or PhD in a STEM discipline, or equivalent practical experience in a research-intensive or engineering-focused environment
- * **Experience:** 1+ years of experience in **QA, test engineering, software engineering, research engineering**, or a comparable role involving significant ownership of quality
- * Proven ability to design effective test cases, develop QA processes, and investigate complex technical systems end-to-end
- * Working proficiency in **Python and Git**, with the ability to understand unfamiliar codebases, environments, and technical workflows
- * Strong debugging and analytical skills with an ability to systematically isolate and resolve issues
- * Exceptional attention to detail and a strong habit of maintaining clear, structured technical documentation
- * Ability to identify subtle failures, inconsistencies, edge cases, and unintended behaviors that may be overlooked by others
- * Comfortable working independently on ambiguous and open-ended technical problems
- * Strong written communication skills and the ability to provide actionable feedback to technical stakeholders
- * A quality-focused mindset with the creativity and persistence to uncover problems others may miss
- * Ability to commit reliably to approximately **35 hours per week**
- The ideal candidate combines strong **software testing and debugging capabilities** with exceptional analytical judgment. You should enjoy breaking complex systems, investigating unexpected behavior, and asking whether a test genuinely proves what it claims to prove
- * Previous experience with **AI evaluation, AI training, model testing, benchmark development, or reviewing AI-generated outputs** is highly desirable
- Experience working with AI systems, evaluation frameworks, automated grading, or complex technical benchmarks will be particularly valuable
Benefits
- Fully Remote — United States
- Approximately 35 hours per week
- Full-Time
Company Overview