Note: The job is a remote job and is open to candidates in USA. KLDiscovery is hiring a senior AI practitioner to set the scientific and architectural direction for generative AI and machine learning across its eDiscovery platform and operations. The role builds and ships AI systems, retrieval and agent infrastructure, and rigorous evaluation methods for legal data use cases where accuracy and defensibility are critical.
Responsibilities
- Own the science and the architecture, and build both personally. Design the experimentation and evaluation methodology that determines whether our AI's outputs are accurate, consistent, and defensible — then define the end-to-end gen AI architecture across Nebula and CS & Operations to deliver on it: LLMs, agent harnesses, RAG, vector search, embeddings, and model selection and triage. Build the hardest parts personally: prototype agent loops, design and run the evaluations that validate them, tune retrieval, and ship the shared infrastructure that powers AI Case Explorer (case overviews, timelines, key people and themes, PII surfacing, and Agent chat), AI Agent Review (pre-classifying relevance, privilege, and key issues, shipping MLP), and CS & Ops tech-enablement as part of our central work orchestration system
- Own evaluation rigor, AI/MLOps, and telemetry end-to-end. Design the statistical and experimental methodology behind our evaluation harnesses — the standard our outputs have to meet to hold up under legal scrutiny — and own the infrastructure that enforces it: model deployment and versioning, eval pipelines, drift and quality monitoring, cost and latency telemetry, and prompt and agent observability. Define and implement how we select, triage, and route across models (Azure OpenAI, Anthropic, open-source, fine-tuned), manage vector databases and retrieval, and evolve our evaluation methodology and agent harness as the frontier moves
- Lead the practice from the front. Set the technical and scientific bar by building, not by reviewing. Partner with Engineering, Product, and Data Science leadership to translate that rigor into shipped product. Raise the bar on both AI engineering and evaluation discipline, mentor senior ICs through hands-on technical leadership, build and maintain relationships with model and infrastructure vendors, and represent KLD's AI strategy directly with customers, partners, and at industry events
Skills
- 7+ years in data science, applied AI/ML, or ML engineering, with recent hands-on experience as a senior or principal-level practitioner in the gen AI era
- Real training in statistics, experimentation, or applied research
- You know how to design an evaluation that actually tests what you think it tests, not just one that looks reasonable
- Proven track record architecting and personally building enterprise gen AI systems in production with measurable customer impact
- Experience with consumer-facing or B2B customer-facing AI products — systems that real external users or customers depend on, not only internal tooling
- Builder at heart: still writes code, runs experiments, ships, and tunes prompts and evals, and wants to keep doing so as a leader
- Deep expertise across the modern gen AI stack: LLMs, agents, RAG, vector databases, embeddings, search, and evaluation harnesses
- Hands-on experience designing system-of-systems AI pipelines spanning search, retrieval, agent harnesses, and model selection/triage
- Strong proficiency with the Microsoft AI stack: Azure OpenAI, Azure AI Foundry, Azure AI Search, and supporting Azure infrastructure
- Excellent technical leadership skills; demonstrated ability to influence architecture and methodology decisions across product, engineering, and data science
- Strong communication skills, including explaining evaluation results and architecture trade-offs to executive and customer audiences
- Career experience spanning both a larger, established technology company and a smaller company or startup — comfortable operating with both institutional rigor and startup pace
- Mentor team members and contribute to a culture of continuous improvement
- Advanced degree (MS or PhD) in Statistics, Machine Learning, Computer Science, or a related quantitative field
- Peer-reviewed publications, technical writing, conference talks, or other evidence of contributing to the field, not just building within it
- Background building agentic systems with tool use, planning, and multi-step reasoning in production
- Prior experience setting up AI governance and evaluation harnesses in a regulated or high-stakes domain
- Open-source contributions or other evidence of being a recognized builder in the AI community
- A demonstrated track record of sticking with hard, ambiguous problems over long timeframes rather than pivoting away when the first approach doesn't work
Benefits
- Bonus potential
- Equity
- Inclusive benefits
- Wellness programs
- Perks
Company Overview