Note: The job is a remote job and is open to candidates in USA. CENTRL Inc. is a risk and compliance technology company providing AI-powered enterprise solutions for risk, due diligence, cybersecurity, and privacy management. The Technical Product Manager will own the accuracy, reliability, evaluation, and cost efficiency of the AI powering CentrlX, building evaluation systems, benchmarking models, optimizing costs, and driving improvements into the product. The role partners with engineering, product, design, sales, professional services, and domain practitioners to turn AI quality findings into shipped changes.
Responsibilities
- Build and own CentrlX's evaluation foundation from the ground up: golden datasets, grading rubrics, LLM-as-judge pipelines calibrated against human labels, and regression suites that run before prompt or model changes ship
- Define what "good" means for each core workflow — document digitization and extraction, retrieval and groundedness, Smart Summary, Smart Response, Smart Evaluation, and full multi-step agent runs — and set a measurable quality bar for each
- Evaluate agent behavior, not just single responses: tool selection, retrieval quality, step sequencing, and whether the finished deliverable holds up to a practitioner's review
- Turn every real client failure into a permanent eval case, so the same class of error does not come back
- Continuously benchmark models across providers — OpenAI, Anthropic, Google, open-weight, and specialized document models — on accuracy, latency, and cost for each workflow, and make the call on what we run where
- Own model migrations end to end, including our in-flight move off GPT-4.1 in document digitization, where current alternatives are materially faster, cheaper, and more accurate
- Track and manage AI spend by workflow, and use routing, model tiering, caching, and context strategy to hold quality while bringing cost down
- Maintain a working view of the model landscape — releases, pricing changes, deprecations — and turn it into a recommendation with evidence attached, not a newsletter
- Define the logging and tracing we need — prompt inputs, retrieved context, prompt text, outputs, tool calls, token counts, latency — and write the stories to get it built
- Partner with Product and Design to build in-app feedback capture (ratings, corrections, structured reason codes) so labeled data accumulates as a byproduct of normal use instead of a periodic collection project
- Build and maintain the datasets yourself: pull the data, label it, curate the hard slices, and keep the sets honest with holdouts and rotation
- Serve as the single point of contact for AI quality escalations from Client Success, Sales Engineering, and Professional Services — triage, reproduce, root-cause, and close the loop
- Write the user stories and acceptance criteria that turn findings into shipped changes, and make the prompt, configuration, and model changes yourself where that is the fastest path
- Publish a regular quality and cost readout that leadership, engineering, and client-facing teams all treat as the same version of the truth
- Work with CENTRL's Manager Research, Investor Relations, and diligence practitioners to encode domain judgment into rubrics — in our market, accuracy is defined by industry expertise, not by a generic benchmark
Skills
- Must have work authorization in the USA
- 3+ years across product management, data science, or applied AI, including at least 2 years working on LLM-based products in production
- Hands-on Python and SQL. You are comfortable in a notebook pulling data, running batch inference, and computing metrics. This role writes code; it does not only specify it
- Demonstrated experience building evaluation datasets and harnesses for LLM systems: golden sets, rubric design, LLM-as-judge with human calibration, and regression testing against prompt and model changes
- Working fluency with at least one eval or LLM observability platform: Braintrust, LangSmith, Langfuse, Arize Phoenix, W&B Weave, Inspect, Promptfoo, or a comparable in-house harness
- Practical understanding of RAG systems: retrieval quality, groundedness and faithfulness, hallucination detection, and chunking and context strategy
- Statistical literacy: you can size a comparison, judge significance, and say plainly when a difference is not real
- Ability to write clear user stories and acceptance criteria and work inside an agile engineering process
- Strong written communication. This role produces recommendations that executives act on
- Experience with document AI: OCR and vision-language extraction, table and layout parsing, and structured output from complex PDFs
- Experience evaluating agentic systems — multi-step trajectories, tool-use correctness, and long-run failure modes
- Experience in financial services or investment management: due diligence, manager research, investor relations, or DDQ/RFP workflows
- Experience reducing inference cost at scale through routing, tiering, batching, caching, or context compression
- Experience designing in-product feedback mechanisms that generate labeled evaluation data
- Familiarity with agent frameworks and MCP
- Degree in a quantitative or technical field
Benefits
- Remote (any location)
Company Overview