Note: The job is a remote job and is open to candidates in USA. Rightsline is seeking a Senior Data Analyst – LLMOps to make the behavior and quality of its AI-powered features measurable. The role owns end-to-end observability, evaluation suites, telemetry pipelines, dashboards, experimentation, cost monitoring, and quality investigations while partnering with engineering, product, and data teams.
Responsibilities
- Instrument Rightsline’s LLM and agent workflows end to end with tracing and structured logging, using tools such as LangSmith, Langfuse, or equivalent
- Define and maintain the metrics that describe AI system health — latency, token and cost per request, error and fallback rates, tool-call success, and retrieval hit rates
- Build and maintain dashboards and alerting so quality and cost regressions surface in hours rather than in customer tickets
- Design and maintain offline evaluation suites — golden datasets, scoring rubrics, and LLM-as-judge grading — for prompts, retrieval pipelines, and agent flows
- Run online evaluations and A/B experiments on production traffic, and report results with enough rigor to support a ship / no-ship decision
- Track quality across model, prompt, and retrieval changes, and maintain the benchmark history that shows whether the product is actually improving
- Partner with engineers on human review and annotation workflows, including labeling guidelines and inter-rater agreement
- Build and maintain the pipelines that move trace, evaluation, and usage telemetry into the analytics stack using SQL and Python
- Analyze how customers actually use AI features — adoption, drop-off, and failure patterns — and translate findings into prioritized recommendations
- Produce recurring reporting on AI quality, cost, and usage for engineering, product, and leadership audiences
- Monitor token spend and model usage, and identify where prompt, model, or caching changes cut cost without hurting quality
- Investigate quality incidents — bad or unsafe outputs, hallucinations, tool failures, latency spikes — from trace to root cause
- Follow and help extend Rightsline’s development best practices and standards in the pipelines, queries, and evaluation code you ship
- Attend and contribute to team meetings, and present telemetry and evaluation findings in a way non-specialists can act on
- Raise the team’s LLMOps literacy — help engineers instrument their own features, write their own evals, and read the dashboards
Skills
- 5+ years in data analytics, analytics engineering, or data science, including hands-on work with LLM-based systems in production
- Strong SQL and Python (including pandas and notebook-based analysis), with the discipline to build repeatable pipelines rather than one-off queries
- Hands-on experience with LLM observability and evaluation tooling such as LangSmith, Langfuse, LangChain/LangGraph, or equivalent
- Practical understanding of LLM application architecture — prompting, retrieval-augmented generation, tool calling, and agent workflows — and how each of them fails
- Experience designing evaluation methodology: golden datasets, scoring rubrics, LLM-as-judge grading, human annotation, and A/B testing on production traffic
- Working knowledge of experiment design and statistics, enough to say confidently whether a difference is real
- Ability to work independently, meet deadlines, and be accountable for your work, while operating within established standards
- Excellent written and verbal communication skills, and a strong sense of ownership, urgency, and initiative — you can explain a quality regression to an engineer and to an executive in the same afternoon
- BS in Computer Science, Statistics, Data Science, or equivalent knowledge
- Experience with AWS services such as Lambda, SQS/SNS, API Gateway, Step Functions, and Amazon Bedrock
- Familiarity with dbt, Airflow or a similar orchestrator, Git, and BI tools such as Looker, Power BI, or Tableau
- Working knowledge of MS SQL, Redis or another NoSQL store, and vector stores used for retrieval
- Exposure to model providers and platforms (Anthropic, OpenAI, Amazon Bedrock), guardrail and PII-redaction tooling, and prompt versioning and release workflows
- Experience with coding agents or AI developer tools (Claude Code, GitHub Copilot, Cursor, or similar) in day-to-day analysis work
Company Overview
Company H1B Sponsorship