Note: The job is a remote job and is open to candidates in USA. Uber AI Solutions is seeking a Finance Domain Expert to build evaluation frameworks for financial generative AI applications. The role involves creating rubrics and datasets, evaluating model outputs, training annotators, and partnering with product teams and client engineers to improve model alignment.
Responsibilities
- Design and document end-to-end evaluation rubrics, taxonomies, and style guides to measure LLM accuracy, reasoning, compliance, and hallucination rates in financial contexts
- Establish granular evaluation dimensions tailored to finance (e.g., distinguishing between GAAP vs. IFRS standards, validating financial statement linkage, identifying subtle calculation errors in discounted cash flow models)
- Build "gold standard" comparison benchmarks (side-by-side criteria, rank-ordering scales, and rating frameworks) that ensure consistent scoring across diverse global teams
- Create high-quality, synthetic financial prompt-and-response datasets (synthetic data creation) representing real-world user behaviors
- Design adversarial prompts to stress-test financial LLMs on regulatory compliance, risk modeling, tax code nuances, and strategic investment analysis
- Provide expert-level preference ratings, edits, and reasoning justifications to drive Reinforcement Learning from Human Feedback (RLHF) pipelines
- Design and deliver training materials, masterclasses, and calibration bootcamps for globally dispersed annotator cohorts via our eLearning platforms
- Perform second-tier spot checks and consensus resolution on labeling queues, serving as the ultimate arbiter for complex financial disputes
- Partner with Uber AI Solutions product teams and client engineers to turn metric trends into actionable model-alignment strategies
Skills
- 5+ years of professional experience in Corporate Finance, Investment Banking, Equity Research, Quantitative Finance
- Proven experience working on RLHF pipelines, data labeling platforms, prompt engineering, or LLM system evaluations
- Demonstrated success building qualitative evaluation rubrics, QA frameworks, or scoring methodologies from scratch in corporate or academic settings
Benefits
- Remote US (excluding California)
Company Overview
Company H1B Sponsorship