Note: The job is a remote job and is open to candidates in USA. XM is seeking a Senior AI QA Test Automation Engineer to lead quality assurance and test automation for its AWS Bedrock AgentCore-based CX Agent Suite. The role involves designing AI evaluation pipelines, integrating AI-driven testing into CI/CD, analyzing quality data, evaluating agent-orchestration frameworks, and establishing technical standards across QA, Data Science, and Engineering.
Responsibilities
- Act as the primary technical enabler for QA, building scalable AI/ML frameworks, libraries, and tooling for the CX Agent Suite and beyond
- Design and build AI evaluation pipelines that assess the CX Agent Suite's multi-agent responses (Intent Detector, FAQ Agent, Missing-Deposits Agent) for accuracy, relevance, tone, hallucination rate, safety/guardrail compliance, and task completion — extending or replacing the current DeepEval-based evaluation harness
- Evaluate and prototype Strands Agents (or comparable agent-orchestration frameworks) for building self-evolving, autonomous QA agents, and drive the go/no-go decision on adoption
- Collaborate with QA, Data Science, and Engineering to integrate AI-driven testing into the existing GitLab CI/CD pipeline (dev → test → staging → prod), alongside Terraform-provisioned, EKS/AgentCore-hosted services
- Build resilient and adaptive automation that can detect and respond to changes in agent behavior, Bedrock Guardrails configuration, and routing logic
- Develop data-driven quality analytics, including root-cause analysis, quality trends, and intelligent test prioritization, leveraging existing observability and evaluation data from OpenTelemetry, CloudWatch, X-Ray traces, and LangFuse offline evaluation runs
- Lead research into emerging AI testing methodologies and mentor engineers through code reviews, workshops, and architectural guidance
Skills
- BSc/MSc in Computer Science, AI, or a related discipline
- 6+ years of hands-on experience in QA/Test Automation, with strong experience designing and maintaining automation frameworks
- 1+ years of experience applying AI/ML in software testing or QA process improvement, with a proven track record of bringing AI/ML solutions into production workflows
- Strong Python skills, with experience working in Python/AWS-centric technology environments
- Hands-on experience with LLM evaluation techniques, including LLM-as-a-judge, human-in-the-loop evaluation, RAG, and multi-agent orchestration patterns
- Practical experience with DeepEval or comparable evaluation frameworks, along with familiarity with agent-orchestration SDKs such as Strands Agents
- Knowledge of LLM tooling, vector databases, and MLOps pipelines
- Experience integrating AI tooling into enterprise CI/CD environments, particularly GitLab, and working with containerized cloud-native environments such as Docker and Kubernetes/EKS
- Strong communicator, able to influence technical decisions and drive engineering standards across teams
- Java and/or TypeScript is a plus
- Experience with autonomous QA agents or agentic orchestration frameworks for self-evolving test suites
- Experience with LLM observability tools such as LangFuse, LangSmith, or Arize, particularly for measuring probabilistic and adversarial robustness
- Knowledge of AI ethics, fairness, and bias detection for guardrail and model validation
- Experience with gRPC, WebSockets, and/or HTTP/2
- Experience with AWS Bedrock, plus familiarity with GCP Vertex AI and/or Azure AI
Benefits
- Attractive remuneration package
- Intellectually stimulating work environment
- Continuous personal development and international training opportunities
- Hybrid work arrangement
Company Overview