Note: The job is a remote job and is open to candidates in USA. SoTalent is seeking a Data Scientist to lead evaluation, benchmarking, and reliability efforts for an AI copilot embedded in a trading platform. The role builds quality frameworks, testing and monitoring systems, and improvement processes to support safe, reliable AI behavior in financial workflows.
Responsibilities
- Design and build evaluation frameworks for AI-powered trading assistants
- Establish quality metrics for:
- Correctness
- Accuracy
- Safety
- Reliability
- Latency
- Hallucination Risk
- Regression Detection
- Create repeatable evaluation methodologies that measure AI performance across financial workflows
- Define acceptance criteria and release quality standards for production AI systems
- Develop and maintain benchmark datasets and evaluation suites
- Create:
- Golden Datasets
- Scenario-Based Test Suites
- Adversarial Test Cases
- Edge Case Libraries
- Market Regime Validation Sets
- Continuously refresh benchmark corpora to reflect evolving market conditions and trading scenarios
- Validate AI performance across diverse trading and portfolio management use cases
- Build automated quality gates that prevent releases when critical metrics degrade
- Implement regression testing frameworks for LLMs, AI agents, retrieval systems, and tool-using assistants
- Evaluate model updates, prompt changes, workflow modifications, and tool integrations before deployment
- Establish testing methodologies for production AI systems operating in regulated financial environments
- Own AI improvement cycles through structured evaluation and experimentation
- Develop:
- Error Taxonomies
- Data Collection Strategies
- Labeling Frameworks
- Quality Assurance Processes
- Identify root causes of model failures and implement corrective improvements
- Support:
- Prompt Engineering
- Fine-Tuning
- Preference Optimization
- Retrieval Improvements
- Tool-Use Optimization
- Measure and validate performance improvements against benchmark standards
- Partner with Product and Engineering teams to establish safe AI interaction patterns
- Design safeguards for:
- Trading Recommendations
- Order Placement Workflows
- Portfolio Analysis
- Risk Management Functions
- Support deterministic previews, user confirmations, auditability, and safe action execution
- Ensure AI outputs are predictable, explainable, and trustworthy
- Build monitoring and observability solutions for AI systems
- Establish telemetry, alerting, and operational dashboards
- Detect performance degradation, reliability issues, and production anomalies
- Lead root cause investigations and incident response activities
- Develop "fix-forward" processes to rapidly remediate AI-related production issues
- Develop expertise in:
- Trading Workflows
- Portfolio Management
- Order Execution
- Margin Accounts
- Short Selling
- Risk Management
- Portfolio Margin
- Market Structure
- Ensure AI systems accurately communicate financial concepts and trading risks
- Validate AI-generated explanations and recommendations within financial contexts
- Partner with:
- AI Engineers
- Software Engineers
- Product Managers
- Quantitative Teams
- UX Teams
- Platform Engineers
- Influence product decisions using evaluation data and model performance insights
- Support development of safe and intuitive AI-powered user experiences
- Drive continuous improvement in AI engineering practices
Skills
- 8+ years of experience delivering production software
- Strong software engineering fundamentals and systems design expertise
- Experience building:
- Evaluation Frameworks
- Testing Infrastructure
- Benchmark Suites
- Automated Quality Systems
- Experience working with:
- LLMs
- AI Agents
- Search Systems
- Retrieval Systems
- Recommendation Systems
- Ranking Systems
- Proven experience improving model performance through structured experimentation
- Strong debugging, troubleshooting, and analytical problem-solving skills
- Ability to work across backend systems, AI infrastructure, and product workflows
- Highly self-motivated with strong ownership and execution capabilities
- Large Language Models (LLMs)
- Agentic AI
- AI Evaluation
- Prompt Engineering
- Model Benchmarking
- Fine-Tuning
- RLHF / Preference Optimization
- Retrieval-Augmented Generation (RAG)
- Systems Design
- Distributed Systems
- API Development
- Production Software Development
- Testing Frameworks
- Performance Engineering
- AI Monitoring
- Observability
- Telemetry
- Incident Response
- Model Performance Tracking
- Evaluation Pipelines
- Experimental Design
- A/B Testing
- Benchmark Analysis
- Error Analysis
- Metrics Frameworks
- Data Labeling Pipelines
- Trading Platforms
- Portfolio Analysis
- Market Data
- Order Management
- Risk Analytics
- Financial Workflows
- Rust
- TypeScript
- PostgreSQL
- React
- React Native
- LLM APIs
- Model Serving Platforms
- AI Evaluation Platforms
- AI Evaluation
- LLM Quality Assurance
- Agentic AI
- Benchmark Development
- AI Safety
- Prompt Engineering
- RAG
- Testing Frameworks
- MLOps
- Observability
- Production AI Systems
- Trading Technology
- Quantitative Analysis
- Systems Design
- Rust
- TypeScript
- PostgreSQL
- Model Optimization
- Incident Management
- AI Governance
Benefits
- Remote work (US)
Company Overview