Note: The job is a remote job and is open to candidates in USA. Cobalt is seeking experienced software engineers to produce expert reasoning, task environments, and evaluation data used to train and assess frontier AI models on real software work. The role involves creating engineering tasks, evaluating model-generated code, designing scoring rubrics, and documenting debugging and software engineering reasoning.
Responsibilities
- Produce written reasoning traces on real engineering tasks, capturing how you locate a bug, what you rule out and why, and how you arrive at a fix, rather than only the final diff
- Author non-trivial engineering problems and task environments with test suites that verify success automatically, including multi-file and multi-step tasks
- Evaluate model-generated code, ranking solutions, explaining what makes the stronger one stronger, and identifying the specific point at which the approach goes wrong
- Assess whether a proposed solution actually satisfies the requirement, and identify failures that pass the tests but break the intent, introduce regressions, or would not survive code review
- Design rubrics and partial-credit criteria for scoring multistep engineering tasks, and classify observed failures into a consistent taxonomy
Skills
- Several years of professional software engineering experience building and maintaining production systems, with a track record you can point to
- Strong proficiency in at least one of Python, TypeScript or JavaScript, Go, Rust, Java, C, or C++, and comfort reading unfamiliar codebases in more than one language
- Depth in at least one area, for example distributed systems, backend and API development, infrastructure and cloud, databases and data engineering, compilers and tooling, or performance engineering
- Solid testing and debugging discipline, including the ability to write tests that actually verify intent rather than restate the implementation
- Ability to explain each step of your reasoning clearly in writing, and to produce work another engineer could reproduce and review
- Open source maintenance, code review at scale, or interview and technical assessment design are useful but not required
Benefits
- Advance frontier AI where it counts. Apply your engineering judgment to data that frontier labs cannot obtain any other way, where your reasoning directly shapes how the next generation of models writes and debugs software.
- Grow professionally. Expand your influence through evaluation projects, advisory roles, and research collaborations, while developing a working understanding of how frontier models are trained and assessed.
- Work with a top-tier network. Collaborate with engineers and researchers from leading organizations on high-impact, flexible work.
- Set your own schedule. Flexible 10 to 40 hour weeks that fit around your existing role and your life.
Company Overview