About The Role
Senior data annotation jobs in Chicago focus on building and evaluating training datasets for modern AI systems, including LLM training pipelines, RLHF, and multimodal annotation. This is a remote, full-time role aligned to Chicago/US hiring requirements.
What You Will Do
Plan and lead data labeling projects across NLP, computer vision, and content safety labeling
Execute RLHF and LLM evaluation tasks (pairwise ranking, rubric-based grading, prompt evaluation)
Create and iterate annotation guidelines with clear edge cases and decision rules
Run QA evaluation programs (gold sets, audits, inter-annotator agreement) to ensure guidelines compliance
Perform error analysis to reduce label noise and improve model performance
Collaborate with engineering on tooling, schema design, and dataset versioning for LLM training pipelines
Mentor annotators, run calibration sessions, and drive consistent labeling outcomes across distributed teams
Document dataset limitations, bias risks, and safety considerations to support responsible AI development
Required Qualifications
3+ years in data annotation, data labeling, or AI evaluation with measurable QA ownership
Experience with RLHF, prompt evaluation, or LLM response rating
Strong knowledge of annotation guideline design and compliance practices
Familiarity with NLP tasks (e.g., named entity recognition, classification) and/or CV annotation (boxes, polygons, segmentation)
Comfort working in remote workflows with ticketing, documentation, and dataset version control practices
Compensation
Competitive hourly base pay:
$30–$50 per hour
, depending on scope, domain complexity, and demonstrated leadership in training data quality and QA evaluation.
How To Apply
Apply through Rex.zone with a resume highlighting senior data annotation leadership, RLHF or prompt evaluation experience, and examples of training data quality improvements.