Note: The job is a remote job and is open to candidates in USA. TestPros, Inc. delivers independent IT assessment solutions for government and commercial clients. The company is seeking a Senior Artificial Intelligence (AI) System Test Engineer to evaluate AI systems and software for functional alignment and AI-specific qualities including robustness, data integrity, security, safety, reliability, and performance. The role develops and executes test plans, documents defects and remediation, and reports assessment results to senior management and stakeholders.
Responsibilities
- Develop and execute test plans and evaluate software for compliance with the OWASP AI Testing Guide and other guidance, create defect and remediation records, capture test evidence and document defects with clear reproduction steps and severity ratings, report results to senior management and stakeholders
Skills
- * Bachelor's degree in computer science, engineering or a related field
- * 10+ years of experience in Software/System Testing and Quality Engineering (QE), including enterprise-scale transformation programs
- * Solid understanding of and experience with the OWASP AI Testing Guide, and the NIST AI Risk Management Framework (RMF)
- * 4+ years hands-on experience defining the testing strategy for large scale programs and ensuring requirements traceability throughout the system development lifecycle. Experience should include leading the testing efforts, including experience with all phases of testing: system integration, user acceptance, and performance testing
- * A deep understanding of testing planning and associated delivery challenges
- * A strong understanding of the various approaches used in testing techniques - white-box, black-box, load testing, coverage testing and regression testing
- * Proven experience in defining and executing QE transformation strategy
- * Hands-on expertise in Agentic AI / GenAI frameworks (e.g., LangChain, AutoGen, CrewAI, Cursor, etc.)
- * Experience working in Cloud, Web and Mobile Testing platforms - Saucelabs, Browerstack are preferred
- * Hands-on experience with machine learning models
- * Experience in Python NLTK or similar natural language processing technology
- * Autonomous test generation
- * Self-healing automation
- * Defect prediction and intelligent triaging
- * Expertise in test automation tools (Selenium, Playwright, Cypress, etc.)
- * Strong experience with CI/CD pipelines and DevOps integration (GitHub)
- * Solid understanding of API, microservices, and cloud-native architectures
- * Experience evaluating or testing Artificial Intelligence (AI) Large Language Model (LLM)-based application features (e.g., chatbots, copilots, RAG-based systems), including prompt-response validation and hallucination/edge-case testing
- * Exposure to AI-assisted defect triage or root-cause analysis tools that reduce manual investigation time
- * Experience with cloud platforms (AWS, Azure, GCP) in QE/AI context
- * Exposure to self-healing test automation frameworks (AI-driven locator repair, visual AI testing)
- * Understanding of responsible AI/testing governance principles (bias detection, explainability, model validation) as applied to QE
- * Experience contributing to or influencing AI-adoption roadmaps within a testing/QE organization
- * Experience implementing testing best practices specific to the test planning and execution processes and gaining buy-in for changes from executives
- * Familiarity with JIRA project/test management system
- * Knowledge of other IT compliance such as WCAG/Accessibility, IT Security, IT Privacy
- * Professional certifications in the software testing discipline
- * Experience with test automation tools, and AI assisted test tools
Benefits
- Remote-friendly with occasional on-site or hybrid work as needed.
Company Overview