About the Company
This contract opportunity supports a high-volume project focused on evaluating technical AI outputs and improving the quality of AI systems. The work combines software quality assurance expertise with human-in-the-loop AI evaluation to help ensure AI-generated responses meet established standards for accuracy, completeness, and reliability.
About the Role The role is suited to an experienced software quality assurance professional who has previously completed paid human-in-the-loop AI evaluation work. You will assess AI-generated technical responses, design and execute test cases, review testing documentation, and identify defects that may affect output quality.
This is a remote contract position that does not require previous AI industry experience beyond the required paid AI evaluation background. Practical software testing experience is prioritized over formal education, and no degree is required.
Key Responsibilities
Evaluate and score AI-generated technical outputs against established quality criteria and evaluation rubrics.
Identify responses that appear correct but contain inaccuracies, omissions, or incomplete information.
Design and assess functional, regression, edge-case, negative, and boundary test cases.
Review bug reports and test documentation for reproducibility, completeness, and appropriate severity ratings.
Identify, isolate, and document defects with precise steps that allow others to reproduce the issue.
Maintain structured defect-tracking records and provide clear, actionable feedback to developers.
Write detailed annotations that explain technical findings without requiring additional clarification.
Collaborate with project teams to refine evaluation guidelines and improve testing standards and methods.
Required Qualifications
Professional experience as a QA Engineer, SDET, Test Engineer, QA Analyst, or in a comparable software quality assurance role.
Strong understanding of software quality assurance, test case design, bug tracking, regression testing, and manual and automated testing strategies.
Hands-on experience with automation frameworks and test management tools such as Selenium, Playwright, Cypress, Appium, Postman, Jira, TestRail, Zephyr, BrowserStack, or similar platforms.
Previous paid experience in human-in-the-loop AI evaluation, AI training support, human data annotation, data labeling, RLHF, AI response or model evaluation, or rubric-based grading. Software QA experience alone does not satisfy this project requirement.
Strong analytical and problem-solving skills, with close attention to detail.
Ability to communicate complex technical findings clearly in written English at B2 level or above.
Reliable internet access and readiness to begin promptly.
Preferred Qualifications
Demonstrable practical software testing experience is prioritized.
Experience applying structured evaluation rubrics and documenting technical issues in a clear, reproducible format is relevant to the work.
Location and Work Arrangement
Remote contract engagement.
Compensation
$90–$175 per hour.