Note: The job is a remote job and is open to candidates in USA. NewtonX is a B2B insights company trusted by the world's most innovative companies to make high-stakes decisions with confidence. In this role, you will be the technical owner of data quality, partnering with the Program Lead to ensure that the data produced meets the customer's training and evaluation needs, while also designing task structures and validating datasets before delivery.
Responsibilities
- Own the judgment of whether task designs and rubrics produce a useful training signal for the consuming method (SFT, RLHF, RLVR, agentic RL, CoT, evals). Catch mismatches between what the data rewards and what the customer is actually training for
- Design the task, environment, and rubric structure for agentic workflows
- Own the ML-level validation of the dataset before delivery — not just whether individual submissions meet spec
- Partner with the Program Lead to convert customer requirements into concrete technical specs: expert profiles, screener trees, task interfaces, task templates, QC rubrics, statistical thresholds
- Partner with the Program Lead to define and defend quality metrics: inter-annotator agreement targets, gold-standard injection rates, statistical power thresholds — providing the statistical and methodological grounding
- Calibrate the ops team on what 'good' looks like per engagement; run alignment sessions when standards shift
- Serve as the technical counterpart to the customer's ML, applied science, and product teams. Hold your ground on the technical questions that decide data quality across training and evaluation — task and reward design, data quality for SFT and preference methods, RL and RLVR, contamination, statistical rigor, and agentic workflows
- Help diagnose data-quality questions when a customer reports the data underperformed — reason through whether the issue is the data, the quantity, or something in their training setup, and make the case defensibly
Skills
- Deep applied ML experience centered on post-training human data — you've owned a human-data workstream as an applied scientist or ML engineer
- At least 2 years working hands-on with RL, including how tool-use trajectories are rewarded and evaluated. If you're not fluent in RL, this isn't the role — it's the foundation of the core judgment you'll be making
- Working fluency across modern LLM post-training and evaluation: SFT, RLHF/preference data quality, RLVR, chain-of-thought, eval harness construction, contamination handling, statistical significance, and agentic/tool-use evaluation
- Genuine understanding of how training data becomes model behavior — you can reason about what a model will learn from a given dataset, not just whether the data meets spec
- Strong programming foundation: read and reason about an eval harness, write Python comfortably, work with model APIs, prototype scoring pipelines. Not a production engineer, but not hands-off
- Statistical fluency: you know when an effect is real vs. noise and can defend a sample size or significance threshold
- Client-facing presence: you've defended technical design choices in real time to skeptical audiences and adjusted scope without losing rigor. Range matters — you can talk to a Series B CTO and a Fortune 100 AI lead in the same week
- Strong written communication: methodology sections, technical reports, and specs that hold up to expert review
Benefits
- Offers Equity
- Excellent medical, dental, and vision insurance.
- 401k match with immediate vesting.
- Health savings/flexible savings account, and pre-tax commuter benefits.
- Paid time off: vacation, holidays, sick, and parental leave.
- A diverse, collaborative, and positive culture where we invest in and celebrate each other's success (happy hours, team projects, and retreats).
Company Overview