Note: The job is a remote job and is open to candidates in USA. Contra Labs is a human-centered AI lab focused on creative and multimodal outputs, building datasets for benchmarking, evaluation, and post-training of AI models. The Research Scientist will independently lead rigorous research projects spanning human-data collection, evaluation methodology, experimentation, qualitative and quantitative analysis, and client-ready insights across creative and multimodal domains.
Responsibilities
- Translate ambiguous client questions into clear research questions, hypotheses, and study designs
- Develop research proposals and methodologies from scratch rather than relying on a predefined approach
- Identify promising new research directions and push the team toward ideas we have not already considered
- Help define what Contra Labs should measure, study, and build next
- Design human-data collection strategies, evaluation protocols, rubrics, annotation schemes, and experiments
- Develop strong prompt strategies for eliciting, evaluating, and comparing model behavior
- Determine appropriate sampling, evaluator selection, calibration, quality-control, and reliability methods
- Design methodologies that work across creative and multimodal domains such as design, image, video, and agentic workflows
- Analyze qualitative and quantitative evaluation data
- Move beyond reporting results to explain why patterns occur and what they imply
- Synthesize findings into clear recommendations, research narratives, and new hypotheses
- Produce client-ready reports, presentations, benchmarks, and research publications
- Independently own research engagements from initial scoping through final delivery
- Work closely with project leads, creative experts, research engineers, and data/engineering teams
- Maintain a high bar for research rigor and data quality while operating on fast client timelines
- Potentially support multiple concurrent projects at different stages
- Be comfortable working on research driven by real client problems and constraints
- Participate in scoping and proposal development and help establish credibility with sophisticated AI/ML research teams
- Direct client-facing experience is valuable, but this is not primarily an account-management role. You will collaborate with Strategic Project Leads on client delivery
Skills
- • 3–5+ years of relevant research experience, with a Master's, PhD, or equivalent industry research depth in HCI, Human-Centered AI, ML, design research, or a related field
- • Strong qualitative and quantitative research skills, with experience in human evaluation, experimentation, data collection, UX/HCI, annotation, or model behavior research
- • Proven ability to independently design and execute rigorous research, including deciding what data to collect, why, and how to analyze it
- • Ability to turn data and observations into new insights, research directions, and actionable recommendations
- • Comfortable analyzing data with Python and/or SQL
- • Strong written communication and ability to clearly explain methodological tradeoffs and research decisions
- • Comfortable operating independently through ambiguity and balancing research rigor with startup speed and pragmatism
- • Based in San Francisco, California
- • Familiarity with modern ML and generative AI systems
- • Understanding of RLHF / post-training / preference data / model evaluation
- • Experience studying image, video, multimodal, creative, or agentic systems
- • Experience designing prompts or systematically studying prompting/model behavior
- • Experience working with large-scale human-data or annotation programs
- • Experience working directly with AI research or model teams
- • Strong publication record or other evidence of consistently generating original research ideas
- • Startup / 0-to-1 experience
Benefits
- Equity
- Medical, Dental, Vision Benefits
- 401k Matching
- We will provide you with a company laptop on your start date
- Paid Case Study and Presentation (60 minutes)
Company Overview