Note: The job is a remote job and is open to candidates in USA. OpenTeams is hiring a Senior AI/ML Test and Evaluation Engineer to build and operate benchmarking and evaluation capabilities for an AI platform. The role develops evaluation methodologies and tooling, documents model limitations and failure modes, and produces reports to inform decisions about deploying capabilities.
Responsibilities
- Design and implement a platform for test & evaluation of AI models and Agentic systems
- Develop evaluation methodologies combining human and AI expert judging and multiple input and output modalities
- Design and enhance user interfaces such that platform customers can express workflows in an intuitive and generic manner
- Collaborate closely with subject matter experts, Model/Agent developers, and evaluation designers to ensure coverage over existing and to be discovered requirements
- Architect and implement robust experiment provenance and result tracking systems
- Ensure performance across integrated tools and scaling systems
- Provide technical mentorship, code review, design guidance, and clear documentation to support team knowledge transfer and user onboarding
Skills
- U.S. citizenship required
- Candidates must be eligible and willing to obtain and maintain a U.S. security clearance
- An active TS/SCI clearance is required
- Proven experience building and deploying software platforms with complex integration surface, preferably handling Machine Learning or agentic workloads
- Deep experience using python, including API frameworks and agentic harnesses
- High level knowledge of Computer vision, language modeling, and/or task specific agent evaluation
- Hands on experience with common frameworks and tooling, eg PyTorch and Hugging Face ecosystem
- Strong communication skills and experience collaborating effectively with cross functional teams including external stakeholders
- Demonstrated experience or potential for leadership, especially in making architectural decisions, coordinating the efforts of other engineers, and developing standards and best practices
- An active TS/SCI clearance is preferred
- Candidates may also be considered if they previously held a TS/SCI with CI polygraph or currently hold an active TS or Secret clearance
- Currently hold or eligible to hold U.S. Security Clearance (Secret or higher)
- Prior experience designing test and evaluation systems for AI
- Experience deploying complex software systems in IL 4 or higher environments
- Experience participating in security assessment and authorization, hardening systems, and remediating vulnerabilities
- Previous experience contributing to open source projects and participating in open source communities
- Exposure to the intelligence community or department of defence subject matter experts
Benefits
- 100% employer paid medical premiums for employees
- Self-managed PTO with a minimum time off requirement
- One U.S.-remote position: candidates may work remotely from anywhere in the United States
- One hybrid position in Washington, DC; Denver, CO; or Colorado Springs, CO
- A career framework that provides a pathway and recognition of increased impact
- Supported to learn fast
Company Overview