Note: The job is a remote job and is open to candidates in USA. The client is a leader in AI testing and Generative AI solutions. They are seeking an AI Safety and Evaluations Engineer to design evaluation frameworks, benchmark AI models, measure safety and reliability metrics, monitor harmful outputs, and translate policy into technical safety constraints.
Responsibilities
- Designing and building rigorous evaluation frameworks to measure model bias, hallucinations, and toxicity
- Creating automated "Eval" datasets to benchmark new models before they are promoted to production
- Developing metrics for "Grounding" and "Faithfulness" in RAG-based systems
- Building monitoring tools that flag harmful or non-compliant AI outputs in real-time
- Partnering with legal and ethics teams to translate policy into technical safety constraints
Skills
- 3+ years of experience in AI Research or Quality Engineering
- Deep expertise in model evaluation techniques and NLP metrics (ROUGE, BLEU, BERTScore). This isn't a learning role—you need to be a subject matter expert
- Demonstrated ability to work autonomously and manage your own time effectively to meet project goals
- Experience with Python, data analysis tools, and LLM-as-a-Judge frameworks
- Strong communication skills to provide clear and concise status updates to the project team
Benefits
- Remote work
Company Overview