Note: The job is a remote job and is open to candidates in USA. Cobalt builds expert data and evaluation infrastructure for AI developers. The company is seeking principal-level software engineers to design challenging command-line software engineering tasks for frontier AI models, develop testing environments and reference solutions, review model-generated code, and refine tasks with reviewers.
Responsibilities
- Design self-contained command line tasks drawn from real software engineering work, such as fixing a subtle bug in a large unfamiliar codebase, completing a refactor that must preserve behavior, migrating a project across a breaking dependency or language version upgrade, implementing a non-trivial algorithm or data structure to a strict performance target, resolving a difficult merge or a corrupted Git history, and repairing a failing build or test suite
- Build the task environment, including the codebase and dependencies, write a reference solution, and write automated tests that verify whether a solution is correct. Tests need to check behavior thoroughly enough that a superficial or hard-coded fix does not pass
- Run your task against a frontier AI model, review its code with the same rigor you would apply to a pull request from a colleague, and refine the task until the failure reflects a real gap in the model's capability rather than ambiguity or trick wording
- Work with reviewers to bring each task to acceptance
Skills
- A PhD in computer science, software engineering, or a closely related field
- Industry or academic professional software engineering experience, including time as a senior technical reviewer on a large codebase
- At least one publication, either academic (for example a peer-reviewed paper) or professional (for example a conference talk, a detailed engineering article, or a widely used open-source project that you created or maintain)
- Expert-level skill in at least two widely used languages, such as Python, TypeScript, Java, Go, Rust, or C++
- Fluency in the Linux command line, shell scripting, Docker, Git, and common build and testing tools
Benefits
- Expand your influence through evaluation projects, advisory roles, and research collaborations.
- Collaborate with researchers from leading institutions and labs on high-impact, flexible work.
- Flexible 10 to 40 hour weeks that fit around your research position and your life.
Company Overview