Note: The job is a remote job and is open to candidates in USA. Microsoft's Cloud & AI organization works to secure digital technology platforms, devices, clouds, and AI systems. The AI Security Engineer will implement and execute cyber-capability evaluations of frontier AI models and agentic systems, build realistic evaluation environments, analyze model behavior, and communicate security findings.
Responsibilities
- Design, implement, and operate offensive cyber-capability evaluations for frontier, preview, production, and open-weight AI models
- Build and maintain realistic evaluation tasks covering vulnerability analysis, exploit development, application and system exploitation, attack-path reasoning, post-exploitation activities, and multi-step offensive workflows
- Execute controlled experiments using AI models; define success criteria and baselines; collect reliable telemetry; and document results
- Analyze model trajectories and investigate unexpected behavior to determine whether it reflects genuine capability, task leakage, environmental flaws, scoring errors, or other evaluation artifacts
- Develop and maintain cyber ranges, vulnerable applications, exploit-development targets, and evaluation harnesses, and apply security controls for sandboxing, secrets management, telemetry, access, and containment when testing autonomous AI systems
- Partner with red-team operators, researchers, engineers, and Responsible AI stakeholders to translate evaluation findings into security insights, and communicate results through reports, documentation, and briefings
Skills
- * Master's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 1+ year(s) experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection
- + OR Bachelor's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 2+ years experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection
- + OR equivalent experience
- * Doctorate in Statistics, Mathematics, Computer Science, Computer Security, or related field OR Master's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 3+ years experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection
- + OR Bachelor's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 5+ years experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection
- + OR equivalent experience
- * Programming ability, particularly in Python, with experience building automation, scripts, or security tooling
- * Familiarity with large language models, generative AI systems, coding models, AI agents, or model APIs
- * Ability to follow experimental methodology, record results accurately, and document technical findings clearly
- * Exposure to security labs, cyber ranges, capture-the-flag environments, or vulnerable applications
- * Experience contributing to cyber-capability evaluations or model-evaluation work for AI or agentic systems
- * Coursework, internship, competition, or project experience in exploit development, vulnerability research, penetration testing, application security, or red teaming
- * Experience working with coding agents, tool-using models, or autonomous agent frameworks
- * Experience building capture-the-flag challenges, vulnerable applications, or exploit-development targets
- * Familiarity with PyRIT, adversarial-testing tools, or comparable model-evaluation frameworks
- * Familiarity with containers, sandboxed execution, or cloud-based test infrastructure
- * Hands-on cybersecurity experience with systems, networks, applications, vulnerability analysis, penetration testing, red teaming, or exploitation in authorized environments
- * Experience working with large language models, generative AI systems, coding models, AI agents, model APIs, or model-evaluation frameworks
- * Experience designing controlled experiments, defining success criteria, analyzing results, and documenting technical findings
- * Familiarity with common evaluation risks, including contamination, task saturation, unreliable scoring, weak baselines, environmental leakage, and limited reproducibility
Benefits
- Certain roles may be eligible for benefits and other compensation.
Company Overview