Note: The job is a remote job and is open to candidates in USA. Delta Exchange is reimagining and rebuilding the financial system through cryptocurrency derivatives and AI-powered products. The Senior QA Engineer - AI will own the quality evaluation of AI products, including chatbots, trading-script copilots, and an MCP server, by building datasets, designing scoring methodologies, analyzing failures, and operating release gates.
Responsibilities
- Build and maintain golden datasets for our AI products, trace mined from production or generated, and verified against a documented source of truth
- Design layered scoring for AI outputs: deterministic rules where behaviour can be asserted, and model-graded evaluation where it cannot
- Manage evaluation runs end to end: schedule and execute them against release candidates, compare results against prior baselines, and triage failures into product defects, incorrect expectations, and platform issues
- Own and operate the release gate that determines whether a prompt or model change is approved for production
- Identify failure modes ahead of customers, including hallucination, incorrect tool selection, loss of context across conversation turns, and PII exposure
- Investigate production traces to distinguish retrieval failures from generation failures from tool failures
- Convert findings into actionable engineering evidence and into permanent regression coverage
- Define and report quality metrics for AI surfaces, and drive improvement against them
Skills
- At least 1 year owning quality for AI products in a lead or primary-owner capacity. This includes chatbots and conversational assistants, code or content generation products, and agentic systems. You should have worked directly with prompts, tool and function schemas, and model behaviour, rather than testing around them
- Demonstrated hands-on experience building or operating an evaluation harness for an AI system, whether in-house or using a framework such as DeepEval, RAGAS or Promptfoo. You should be able to describe what the harness measured, the results it produced, and the decisions those results informed
- 4-6 years of QA / SDET experience
- Working proficiency in Python. Our evaluation platform, trace analysis and internal tooling are Python-based
- Ability to analyse traces and tool calls, using observability tooling such as Opik, Langfuse or LangSmith or an equivalent, to determine root cause rather than reporting the symptom
- Strong API testing experience. The majority of the surface under test is API-level
- Sufficient engineering ability to build your own tooling and automation
- Clear written communication. Findings must be documented in a form engineering can act on directly
- Experience with trading or exchange platforms, including familiarity with futures and options, crypto derivatives, margin, or settlement flows. Domain knowledge can be picked up on the job, but arriving with it shortens the ramp considerably
Benefits
- Remote work arrangement
- Work directly with founders and senior leadership on high-impact strategic initiatives
- Significant growth opportunities
- Be part of a mission-driven, fast-growing organisation at the forefront of financial innovation
Company Overview