Job Description:
• Own the reliability and quality bar for an AI copilot embedded in a trading platform used by sophisticated investors
• Design and build evaluation systems measuring correctness, safety, latency, and regression risk across market analysis, portfolio/risk reasoning, and trading workflows including order placement
• Develop and maintain benchmarks including curated golden sets, scenario suites, stress/adversarial cases, and refreshed market/regime-based test corpora
• Build automated quality gates and regression workflows that block releases when key metrics degrade
• Partner with engineering and product to define safe tool/action contracts with deterministic previews, confirmations, and auditability
• Own model improvement loops tied to evaluations, including data collection/labeling strategies, error taxonomy, prompt/tooling changes, and appropriate fine-tuning or preference optimization
• Design and operate AI monitoring and incident response, including telemetry, alerting, root-cause analysis, and fix-forward processes
• Develop deep understanding of trading concepts such as margin, shorting, portfolio margin, risk, and execution, and express them accurately to users
• Work with the technology stack: Rust, TypeScript, Postgres, React, React Native, observability/telemetry tooling, LLM APIs, model serving, and evaluation/training pipelines
Requirements:
• At least Eight (8) years of experience shipping production software
• Strong proficiency with any programming language
• Strong knowledge of computer science fundamentals, testing methodology, and systems design
• Experience building evaluation frameworks, test harnesses, and benchmark suites for complex systems (LLMs/agents/search/retrieval/ranking/recommenders)
• Experience running model improvement cycles: dataset curation, labeling/QA, offline experimentation, and deploying changes with measurable impact on benchmarks
• Ability to define metrics, build measurement pipelines, and drive engineering/product decisions from data
• Comfort working across the stack: debugging model/tooling failures, instrumenting services, and partnering with frontend/product on UX patterns that improve safety and trust
• High degree of self-motivation and willingness to jump into unfamiliar areas to solve problems
• Bonus: Experience with fine-tuning, preference optimization, distillation, or prompt/compiler-style techniques for improving tool-use reliability
• Bonus: Experience creating domain-specific benchmarks and adversarial suites for high-stakes applications
• Bonus: Deep experience with trading across asset classes, margin types, etc.
• Bonus: Experience with Rust and performance-sensitive services
• Bonus: Experience designing incident response and SLOs for ML/AI systems
Benefits:
• Company equity
• 401k matching
• Gender neutral parental leave
• Full medical, dental and vision insurance
• Lunch stipends
• Fully stocked kitchens
• Happy hours
• Great location
• Amazing views
• Equal opportunity workplace