About the role
AI Labs is seeking a Software Engineering Evaluator for a remote, hourly contract supporting the assessment of AI systems that write, debug, and review production code. The role requires 2–3 years of professional software engineering experience, strong command of at least one listed programming language, sound software engineering fundamentals, code review experience, excellent written English, and the ability to evaluate complex code with independent, evidence-based judgment.Key Responsibilities
Design task-specific grading criteria for feature implementations, bug fixes, refactors, tests, and code reviews.
Evaluate AI-generated code for correctness, including failures that pass the code's own tests.
Assess edge-case handling, security, performance, and readability in coding deliverables.
Write clear, evidence-based justifications for every evaluation score.
Apply consistent judgment across software engineering evaluation tasks.
Review code that may appear correct and run without failing while identifying subtle defects.
Reason through ambiguity and distinguish clearly between known facts and inferences during evaluations.
Iterate on evaluation work based on feedback from senior reviewers.
Required Qualifications
2–3 years of professional software engineering experience shipping production code.
Strong command of at least one of Python, JavaScript/TypeScript, Java, Go, C++, C#, or a comparable language.
Solid grounding in data structures, algorithms, testing, debugging, and version control.
Experience reviewing other engineers' code.
Excellent written English.
Ability to follow complex instructions, reason through ambiguity, and spot subtle errors in code that looks correct and runs without failing.
Ability to clearly separate what you know from what you are inferring.
Ability to work independently, learn new tools quickly, and take feedback well.
Based in and working from the US, Canada, or UK.
Available for 10+ hours per week for the full 12 weeks.
Benefits and Perks
$50–$125 per hour, paid weekly via Stripe or Wise.
Remote work on your own schedule within the 10+ hour weekly engagement.
Initial 12-week contract with the possibility that projects may be extended based on needs and performance.
Opportunity to evaluate AI systems for leading AI research organizations.
Reasonable accommodations available upon request.
Skills
Python
JavaScript/TypeScript
Java
Go
C++
C#
Data Structures
Algorithms
Testing
Debugging
Version Control
Code Review
Production Code
Grading Criteria Design
Evaluating AI-Generated Code
Correctness Evaluation
Edge-Case Handling
Security
Performance
Readability
Backend Systems
Distributed Systems
Infrastructure
Open-Source Contributions
Competitive Programming
Evaluating AI Outputs
Grading Technical Work
Stripe
Wise