QA Engineer - Remote

About the Company BMC helps organizations accelerate business value through technology that connects people, systems, and data. Its solutions support business growth, innovation, and operational efficiency, serving nearly 80% of the Forbes Global 100. The IZOT product line focuses on Intelligent Z Optimization & Transformation, helping large enterprises monitor and manage mainframe systems. Its products support mainframe modernization by improving developer experience, integration, application development speed, code quality, and security while reducing operational costs and risks. About the Role BMC is seeking a Principal Quality Automation Engineer to develop and maintain automated evaluation systems for generative AI, retrieval-augmented generation (RAG), and agentic AI applications. This role focuses on determining whether AI systems are reliable, consistent, safe, and ready for release. You’ll build evaluation frameworks, golden datasets, scoring rubrics, and automated testing pipelines that measure AI quality before changes reach customers. Working with data science, AI engineering, and product teams, you’ll identify failure patterns such as hallucinations, model drift, and inconsistent or unsafe outputs, then use evaluation results to guide improvements. The role requires independent ownership of defined evaluation initiatives, from planning through delivery, with a direct impact on product quality and release decisions. Key Responsibilities Define and implement evaluation strategies and frameworks for generative AI, RAG, and agentic AI applications Create golden datasets, evaluation rubrics, and scoring methods, including LLM-as-a-judge approaches, to measure quality consistently Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices Build automated evaluation harnesses within CI pipelines to assess model, prompt, and agent changes before release Establish quality metrics and release gates to distinguish production-ready solutions from prototypes Design testing strategies for multi-agent workflows and AI systems that use tools, APIs, MCP, and enterprise applications, including Playwright where relevant Evaluate end-to-end agent behavior, including planning, tool usage, response generation, and task completion Use observability and tracing to identify failure modes and recommend improvements to system reliability Collaborate with data science and AI engineering teams to turn evaluation findings into model and prompt improvements Promote responsible AI practices and human-in-the-loop principles while contributing to governance and model validation Produce actionable quality reports and integrate evaluation workflows into release decisions for a product or team Required Qualifications Bachelor’s or master’s degree in Computer Science, Artificial Intelligence, or a related field, or equivalent experience Experience with generative AI, large language model applications, or AI/ML systems and quality evaluation frameworks Understanding of failure modes associated with LLMs, RAG, and agentic AI systems Knowledge of vector databases and embedding-based retrieval, as relevant to the role Proficiency in Python, including building evaluation datasets, scoring rubrics, and automated evaluation workflows Experience integrating evaluation processes with CI/CD pipelines Familiarity with LLM observability and tracing tools such as Langfuse or OpenTelemetry (OTel), and analytics platforms such as OpenSearch or similar tools, as relevant to the role A rigorous, data-driven approach to quality assessment and problem-solving Ability to collaborate effectively with data science, engineering, and product stakeholders Demonstrated ownership of an evaluation suite or quality workflow that influenced an AI feature’s release decision Experience developing repeatable datasets and scoring rubrics rather than relying solely on manual spot checks Ability to independently manage defined quality initiatives from requirements through delivery Preferred Qualifications Experience with evaluation frameworks such as Ragas, DeepEval, or promptfoo Knowledge of AI safety and governance practices Experience working in enterprise or regulated environments Exposure to OpenShift or AWS Experience with performance, load, and reliability testing for AI applications Knowledge of MLOps practices Experience with open-weight models or local model serving Familiarity with MCP and Agile or Atlassian tools Location and Travel Work Arrangement: Remote within the United States Travel: Occasional business travel may be required Compensation Estimated Annual Base Salary Range: $152,925–$254,875. Actual compensation depends on factors such as skills, experience, training, certifications, and business needs. The base salary range represents one component of BMC’s overall compensation package. Additional rewards may include variable compensation and country-specific benefits. Offers are not typically made at or near the top of the salary range. Equal Employment Opportunity BMC is committed to equal employment opportunity regardless of race, age, sex, creed, color, religion, citizenship status, sexual orientation, gender, gender expression, gender identity, national origin, disability, marital status, pregnancy, or veteran status. Reasonable accommodations are available during the application and hiring process.