Note: The job is a remote job and is open to candidates in USA. Cobalt builds expert data and evaluation infrastructure for AI developers. The company is seeking a Staff Machine Learning Engineer and AI Training Data Expert to design rigorous machine learning research tasks, evaluate frontier AI model performance, and refine tasks with reviewers for use in training and evaluation.
Responsibilities
- Design self-contained command line tasks drawn from the practice of machine learning research, such as implementing a method from a paper's description alone, locating the error in a flawed experimental setup, reproducing a reported result and explaining a discrepancy, implementing a custom loss function or optimizer correctly, and running a sound statistical comparison between two models
- Build the task environment, including code, data, and dependencies, write a reference solution, and write automated tests that verify whether a solution is correct. Because results can vary between runs, tests need to be deterministic or use well-justified tolerances
- Run your task against a frontier AI model, review its attempt with the same rigor you would apply to a paper under peer review, and refine the task until the failure reflects a real gap in the model's capability rather than ambiguity or trick wording
- Work with reviewers to bring each task to acceptance
Skills
- A PhD in machine learning, computer science, statistics, or a closely related field
- Industry or academic experience in machine learning research, in academia or industry. Years spent on doctoral research count toward this total
- At least one publication, ideally peer-reviewed work at a recognized machine learning venue or journal
- Deep working knowledge of Python and at least one major framework, such as PyTorch or JAX
- Fluency in the Linux command line, shell scripting, Docker, and Git
- Experience as a peer reviewer, or in designing course assignments and automated grading, is a strong advantage, because the work draws on the same skills
Benefits
- Expand your influence through evaluation projects, advisory roles, and research collaborations.
- Collaborate with researchers from leading institutions and labs on high-impact, flexible work.
- Flexible 10 to 40 hour weeks that fit around your research position and your life.
Company Overview