Note: The job is a remote job and is open to candidates in USA. SGA is a technology and resource solutions provider delivering IT and professional services. The company is seeking an AI QA and Test Engineer to test, evaluate, and validate AI-powered agent experiences, execute evaluation frameworks, manage defects, and monitor AI observability. The role partners with product managers, software engineers, and business stakeholders to improve agent reliability, accuracy, and user experience.
Responsibilities
- Test and evaluate existing prompts, system instructions, and agent workflows for accuracy, consistency, groundedness, and user satisfaction
- Conduct structured prompt experiments and A/B test result analysis (execution and analysis, not prompt authorship)
- Document evaluation findings, failure patterns, and recommended refinements for the engineering team to implement
- Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services
- Develop evaluation frameworks and scoring methodologies for AI output quality
- Create test cases covering: - Accuracy - Hallucination detection - Safety - Tone and style adherence - Instruction following - Context retention - Response consistency
- Perform regression testing for prompt and model changes
- Track, prioritize, validate, and retest defects using Azure DevOps
- Utilize Arize or similar observability platforms to monitor model and agent performance
- Analyze user interactions and evaluation results to identify trends and improvement opportunities
- Develop quality metrics and dashboards for: - Accuracy - Relevance - Latency - User satisfaction - Defect rates - Prompt performance
- Investigate production issues and recommend prompt, workflow, or architectural improvements
- Maintain documentation for evaluation methodologies, testing procedures, and defect logs
- Participate in sprint planning, backlog refinement, and Agile ceremonies
- Communicate findings and recommendations to technical and non-technical stakeholders
- Share best practices and lessons learned across the team
Benefits
- This role will be remote.
Company Overview
Company H1B Sponsorship