Note: The job is a remote job and is open to candidates in USA. Phase2 is a digital agency focused on customer experience, data and insights, and technology delivery. The Lead AI Quality Engineer will own the evolution of the company’s quality assurance strategy, establish standards and tooling for AI-era software delivery, lead client quality engagements, and manage and mentor the quality team.
Responsibilities
- Set the vision and roadmap for software quality. Define the standards for how Phase2 ensures quality in the AI era, from how we write tickets to how we test our work. Evaluate our current suite of AI-enabled and traditional testing tools, and our current QA practices and methods, and maintain a roadmap that outlines planned improvements to our tools and methods
- Define testing standards and defaults. Set the standards and defaults for what gets tested manually by humans, what is handled through automation, and what human oversight of that automation looks like. Lead this work in partnership with Engineering leaders, supported by AI Quality Specialists
- Innovate approaches to accelerate quality assurance testing. Identify and implement tools and methodologies to enable testing to keep pace with AI-accelerated software development
- Lead the use of agentic AI to write tests. Teach people across a variety of roles how to use agentic AI to write automated tests
- Recommend improvements to execution of projects and the SDLC. Identify how quality should be integrated through the project lifecycle, including when and how QA specialists should be involved at each stage of a project
- Socialize the quality assurance approach across the company. Educate and enroll other departments in the company about our evolving approach to QA, ensuring everyone has a shared vision and understanding
- Oversee agentic QA tooling. Build, run, maintain, and enhance agentic AI tools that do QA work and contribute those to Phase2’s AI delivery framework
- Test Phase2-built AI tools. Evaluate Phase2's internally built agentic AI tools (both for internal use and client use) and identify areas for improvement with clear, specific detail
- Set approaches for testing AI features. Set guidelines for designing and executing test plans for AI-powered features, including LLM-based applications, agentic workflows, and ML-driven components
- Develop AI evaluation frameworks. Develop evaluation frameworks for AI outputs, covering accuracy, relevance, consistency, hallucination detection, and bias across protected attributes
- Own integration of testing with automated development workflows. Define and help implement improvements to our CI/CD testing tools, including traditional unit, functional, integration, and VRT tests, as well as agentic AI-enabled tests
- Advise clients on quality best practices. Coach client engineering teams to adopt a quality ownership mindset, acting as a quality consultant to client stakeholders
- Lead the quality team. Manage, mentor, and coach other Quality Specialists and Quality Engineers
- Follow the cutting edge of quality engineering and share knowledge back to Phase2. Stay current on AI testing methodologies, responsible AI principles, and emerging tooling, bringing insights back to the team proactively
Skills
- 8+ years of software QA experience, preferably with demonstrated experience testing AI/ML-powered features
- Experience managing, mentoring, and coaching direct report employees
- Experience recommending and then executing improvements to QA methods and tools
- Proficiency in writing automated tests in at least one framework
- Working knowledge of AI testing concepts, including hallucination detection, output evaluation, prompt engineering for test generation, and bias testing
- Strong verbal and written communication skills, with the ability to present quality outcomes both to technical and to non-technical stakeholders
- Experience with WCAG 2.1/2.2 accessibility standards
- Experience working in a digital agency, technology consultancy, or professional services environment
Benefits
- Remote-first work environment
Company Overview