Note: The job is a remote job and is open to candidates in USA. 24-MAG LLC is offering an independent contractor opportunity to evaluate and improve AI coding systems. The role involves reviewing AI-generated code in real-world repositories, developing evaluation rubrics and infrastructure, and sharing findings with research and engineering teams.
Responsibilities
- Review code produced by AI coding agents
- Assess implementations for correctness, robustness, and maintainability
- Determine whether agents selected appropriate technical approaches
- Identify subtle implementation errors and engineering weaknesses
- Explain clearly why generated solutions succeed or fail
- Design and refine technical evaluation rubrics
- Define criteria for assessing coding-agent performance
- Review and label preference and evaluation data
- Apply consistent quality standards across repeated assessments
- Capture nuanced differences between competing implementations
- Work within substantial real-world software repositories
- Analyse code changes in their broader architectural context
- Evaluate repository-level behaviour rather than isolated snippets
- Review implementation quality using senior engineering judgement
- Identify recurring failure patterns across coding tasks
- Build and maintain pipelines supporting data generation and evaluation
- Improve infrastructure for collecting and reviewing model outputs
- Support scalable evaluation and feedback workflows
- Contribute to reliable processes for repeated technical assessment
- Help translate qualitative engineering judgement into structured systems
- Collaborate with research and engineering teams
- Summarise evaluation findings in clear written reports
- Provide actionable recommendations based on observed model behaviour
- Help improve evaluation methodologies and coding-agent benchmarks
- Communicate complex technical issues in concise, structured language
Skills
- 5+ years of hands-on software engineering experience
- Strong proficiency in Python, TypeScript, JavaScript, Go, Java, or another major production language
- Experience working in substantial real-world codebases
- Strong code-review and technical-analysis skills
- Ability to assess implementation correctness and maintainability
- Strong understanding of modern software-engineering practices
- Experience with GitHub-based development workflows
- Familiarity with CI/CD and production engineering processes
- Comfortable analysing unfamiliar repositories and code changes
- Ability to identify subtle technical issues others may overlook
- Strong written communication and structured technical reasoning
- Experience using modern LLMs or AI-assisted coding tools
- Familiarity with open-source development is valuable
- LLM evaluation or coding-agent experience is advantageous
- Experience with RLHF, preference data, rubric design, or post-training is useful but not required
Benefits
- Remote — North America only
- Flexible commitment of approximately 20–40 hours per week
Company Overview