Note: The job is a remote job and is open to candidates in USA. Skill connects professional, IT, engineering, financial, and administrative talent with major brands. The Applied AI Safety Engineer will identify, evaluate, and mitigate safety risks in conversational and agentic AI products by developing threat models, adversarial evaluations, data pipelines, and monitoring systems. The role partners with Engineering and Trust & Safety to integrate safety practices into product development and communicates findings to technical and non-technical stakeholders.
Responsibilities
- Develop product-specific threat models and harm taxonomies for conversational, recommender, and tool-using AI systems
- Design and run single- and multi-turn adversarial evaluations, combining expert red teaming, automated attack generation, synthetic data, and sampled production data
- Build reusable Python evaluation pipelines, LLM-as-a-judge workflows, regression tests, dashboards, and curated golden datasets
- Validate evaluators against human labels and quantify coverage, judge reliability, false positives, false negatives, and safety–utility trade-offs
- Turn findings into practical mitigations, including policy and prompt changes, context engineering, classifiers, data improvements, preference tuning, and system-level controls
- Work directly with Engineering and Trust & Safety to embed evaluations into product-development and monitoring loops
- Communicate results clearly to technical and non-technical stakeholders
- Full-time availability is preferred, although part-time arrangements may be considered
Skills
- • You have personally delivered safety evaluations or mitigations for a real AI or machine-learning product, using automation to scale processes
- • You have strong coding agent and practical data-analysis skills, with sufficient programming and SQL to independently judge code and queries
- • You have experience designing adversarial tests, evaluation datasets, taxonomies, rubrics, and metrics
- • You can work autonomously when the risk, success criteria, and methodology are initially unclear
- • You have experience working across research, engineering, product, policy, or Trust & Safety
- • You communicate clearly in writing and have a record of turning research findings into action
- • Experience evaluating multi-turn or tool-using agents
- • Experience calibrating LLM judges or building human-in-the-loop evaluations
- • Experience with multilingual or multimodal evaluation
- • Experience with preference tuning or other model-alignment techniques
- • An MSc or PhD in an AI/ML-related field
- Full-time availability is preferred, although part-time arrangements may be considered
Benefits
- Health benefit contributions
- Retirement plans with match
- Flexible spending accounts
- Remote position
- Inclusive environment where different backgrounds, experiences, and perspectives are valued
- Opportunities to contribute, grow your career, and thrive
Company Overview