Note: The job is a remote job and is open to candidates in USA. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. The AI Agent Safety Evaluation Engineer will design tasks and write tests to evaluate the safety and conduct of AI coding agents, ensuring they complete tasks correctly and ethically.
Responsibilities
- Build realistic developer environments — a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
- Design tasks that pair a benign development goal with a tempting unsafe shortcut: scope creep, policy violations, data corruption, over-permissive changes
- Write tests that verify whether the agent completed the task the right way — catching corners cut, not just checking outputs
- Iterate on tasks and tests based on QA feedback — review agent solutions, analyze failures, and refine until the evaluation is fair and robust
Skills
- 4–5+ years in software development
- Core stack: Python, JavaScript/TypeScript
- Strong test design skills — functional and integration tests that separate safe from unsafe completion, not just correct from incorrect
- Hands-on experience with coding agents (Claude Code, GitHub Copilot CLI, Codex, or similar)
- Familiarity with GitHub PRs and CI workflows as a user
- Stack breadth is welcome, not a filter. Tasks simulate real repositories with databases, CI pipelines, and deploy scripts, so wider backend and infrastructure exposure is genuinely useful — but you don't need to be an expert in every layer
- English proficiency — B2+
- Cybersecurity experience is a nice-to-have but not a requirement
Company Overview