Note: The job is a remote job and is open to candidates in USA. RJI Search is seeking an experienced Senior Software Engineer - Technical Advisor to evaluate how frontier AI coding models perform on real, production-grade engineering work. The role involves reviewing model-generated code, designing evaluation benchmarks, analyzing AI coding-agent sessions, and collaborating with AI researchers to document technical failures and reasoning.
Responsibilities
- Evaluate Agent Sessions: Analyze how AI coding agents behave over entire coding sessions—tracking what the agent investigates, verifies, assumes, and leaves undone
- Review Model-Written Pull Requests: Audit model-generated pull requests against real production repositories, documenting every identified issue alongside its severity and detailed technical rationale
- Build Evaluation Benchmarks: Design and construct hard, container-based problems that serve as rigorous test benchmarks for frontier models
- Collaborate with AI Researchers: Work directly alongside AI researchers on frontier problems, producing clear written analyses that explain root-cause failures and boundary-condition gaps
- Maintain High Written Rationale Standards: Author original, clear technical rationale for all evaluations (all written deliverables must be independently authored without AI text generation, though AI tools are welcomed for codebase exploration and running test suites)
- Review model-generated pull requests against real production repositories, documenting every issue you find along with its severity and your technical rationale
- Evaluate full AI coding-agent sessions, tracking what the model investigated, verified, assumed, or skipped
- Design and build container-based (Docker) test problems that serve as benchmarks for evaluating frontier models
- Write clear, original technical rationale explaining why code fails. All written deliverables must be self-authored, no AI text generators for your written analysis, though AI tools are welcome for exploring codebases or running test suites
- Collaborate asynchronously with Anthropic's AI researchers to share findings and refine evaluation criteria
- Solve difficult engineering problems across real, production-grade codebases
- Identify where model-generated code fails, and articulate precisely why
- Work directly with researchers on problems they are actively investigating
- Hold a technical bar that others build on
Skills
- 1. United States
- Must have 8+ years of hands-on production engineering experience (Senior, Staff, Principal, Tech Lead, or Open-Source Maintainer backgrounds all fit well)
- Real experience in codebases with a genuine code review culture, whether at a startup, big tech, or open source
- Comfortable dropping into unfamiliar languages and codebases on short notice
- Genuinely comfortable with Docker, git, and the command line to reproduce and debug issues locally
- Can work across backend, frontend, APIs, data, testing, or dev tooling
- Write clearly enough to explain why something is broken, not just how to fix it fast
- Contract type: W2 Contractor. This role cannot support C2C arrangements or visa sponsorship, so you'll need to confirm US remote work eligibility
- Are comfortable dropping into unfamiliar codebases and languages you don't use every day?
- Are comfortable with containers and reproducing results locally?
- Can explain your reasoning as clearly as you can write the solution?
- Are more interested in why a solution is right than in shipping it quickly?
- Want to spend a focused period on this rather than committing to a permanent role?
- Have deep, demonstrated expertise as a software engineer - we care about the depth of your judgment, not your years!
- Write code that other strong engineers learn from
- Have reviewed a lot of other people's code, and are known for catching what CI and the author both missed
Company Overview