Note: The job is a remote job and is open to candidates in USA. Digivance Solutions is building a global team to evaluate a personalization feature for a leading AI assistant. The LLM AI Quality Analyst will design multi-turn prompts, assess model responses for grounding, integration, and helpfulness, compare outputs, verify debug information, and provide structured feedback to improve personalization quality.
Responsibilities
- Design and execute multi-turn conversational prompts (1–5 turns) that require the AI to use your personal information and experiences
- Evaluate model responses for: • Grounding : Are personal claims supported by evidence, or are they flawed inferences/hallucinations? • Integration : Is personal data woven naturally into the response, without robotic “overnarrating”? • Helpfulness : Which response is more useful, easy to use, and enjoyable?
- Perform side-by-side (SxS) comparisons of two model responses and rank them
- Write clear, structured rationales for your rankings, explicitly referencing turn numbers and specific issues or strengths
- Extract and verify debug information to confirm the model used the correct chat summaries and data sources
- Maintain strict data hygiene by deleting evaluation conversations to prevent contamination of future personalization
- Provide constructive feedback and annotations to help improve personalization guidelines and model behavior
- Work independently in a remote, global, 24-hour operations team
Skills
- • Excellent English proficiency (reading and writing) – English is the focus language for this project
- • 1+ year of experience in data annotation, AI quality evaluation, content moderation, LLM evaluation, or a related analytical role (strongly preferred)
- • Exceptional analytical thinking : Ability to evaluate nuanced, ambiguous AI responses and assess personalization quality
- • Creative prompt engineering : Experience designing multi-turn prompts based on personal context to thoroughly test model capabilities
- • Strong evaluation acumen : Understanding of personalization concepts, including incorrect personalization, poor inferences, and forced connections
- • Meticulous attention to detail : Ability to review SxS responses and spot subtle differences in naturalness and over-narrating
- • Excellent written communication : Superior ability to write clear, concise, and structured rationales
- • Self-motivated and independent : Comfortable working remotely with minimal supervision
- • Technical setup : Desktop/laptop with a reliable internet connection
- • Personal account usage : Willingness to use your primary personal Google account (not a testing account) and enable personal data sources for genuine assessment
- • Schedule flexibility : Full-time availability in your local time zone, with 4 hours overlap with PST (for UK/US candidates)
- • Must be available to start immediately for a 4-month contract
- • 1+ year of experience in data annotation, AI quality evaluation, content moderation, LLM evaluation, or a related analytical role (strongly preferred)
- • BS/BA degree or equivalent experience in Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field
- • Prior experience with AI model evaluation , RLHF , prompt engineering , or content safety/policy work
- • Familiarity with personalization concepts , data grounding , and hallucination detection
Benefits
- Remote (UK / US)
Company Overview