Note: The job is a remote job and is open to candidates in USA. Dealstitch AI is a technology company focused on private-market applications and agent systems. The AI Quality Engineer will own automated evaluation pipelines for AI agents, assess performance across reasoning, tools, and retrieval, maintain production datasets and observability, and continuously improve the quality standards for shipped systems.
Responsibilities
- Own the evaluation pipelines that gate every change we ship, keeping them automated and running in CI/CD rather than leaning on manual spot-checks
- Keep agents scored at the step level, not just the final answer, across tool selection, planning and reasoning chains, and retrieval quality, so subtle regressions surface before clients do
- Extend the eval toolkit as new failure modes appear: LLM-as-a-judge, RAG faithfulness and relevance, hallucination detection, prompt and agent regression suites, and red-teaming for safety
- Maintain the golden datasets that mirror real production inputs and the tracing and observability that watch our agents once they are live
- Own the quality bar for what ships and raise it over time with the rest of the engineering team
Skills
- Strong software engineering with Python, and a feel for testing non-deterministic systems, where classic assertion-based tests are not enough
- Experience evaluating LLM and agent systems: agent graphs (e.g. LangGraph), eval frameworks (DeepEval, RAGAS, G-Eval, LangSmith, or similar), and LLM-as-a-judge methods
- Fluency with RAG and retrieval quality, plus observability and tracing for agents in production
- A bias for measurement: you define what "good" means, then build the harness that proves it
- Bonus: data-pipeline testing experience, or exposure to private markets, financial services, or enterprise data
Company Overview