AI Product Analyst

AI Product Analyst Pay Rate: $50-$65/hour a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } About the Team and the Environment This role sits within a centralized AI Engineering and Automation Center of Excellence responsible for delivering enterprise-scale AI and automation solutions across a large, distributed North American organization. The AI work is real, deployed, and actively supporting business operations. This is not a pilot program or innovation lab. Current capabilities include: Reusable AI skills and plug-ins packaged into governed applications used across multiple business functions Retrieval-Augmented Generation (RAG), embeddings, and multi-step workflows operating on governed enterprise data A governed AI asset marketplace and Model Context Protocol (MCP) catalog covering intake, publishing, versioning, lifecycle management, and maintenance Enterprise-grade evaluation frameworks, golden-set testing, guardrails, monitoring, and logging designed to ensure reliable AI outputs at scale The organization also operates a mature automation ecosystem alongside its AI initiatives, including: Approximately 125 unattended production robots More than 200 production automations More than 19,000 automation jobs executed every 30 days Platforms including UiPath and Microsoft Power Platform For projects requiring significant engineering investment, such as new backend services, infrastructure implementation, or highly complex integrations, a dedicated Enterprise Engineering organization partners with the AI Engineering team to build and deliver solutions. Once completed, those assets are cataloged, governed, and managed through established operational processes. What This Role Actually Is This role serves as the Product Owner for the AI Engineering delivery pipeline. The successful candidate will lead and direct solution development efforts while partnering closely with engineering teams. This is not a hands-on software engineering role, and the individual in this position will not be writing production code. Key responsibilities include: Managing intake, qualification, prioritization, and opportunity assessment for AI initiatives Developing business cases and defining value measurement strategies, including establishing baselines, success metrics, attribution methodologies, and measurable outcomes Defining requirements and acceptance criteria for AI and LLM-enabled solutions Authoring evaluation specifications, curating golden datasets, and establishing acceptance thresholds used to determine production readiness Leading user validation activities, adoption tracking, and value realization assessments after deployment Monitoring portfolio health across AI assets, including ownership, lifecycle management, versioning, deprecation, and retirement planning Managing governance processes for AI skills, plug-ins, and MCP catalog assets, translating security, privacy, compliance, and regulatory requirements into actionable product requirements Building strong partnerships with business leaders, operational stakeholders, and executive teams while maintaining disciplined intake and prioritization processes Important Distinction Engineering teams own: Evaluation harness implementation Retrieval pipelines Model serving infrastructure Monitoring instrumentation Technical delivery This role owns: Defining what success looks like Defining evaluation criteria Defining acceptance thresholds Determining production readiness Making the final product recommendation for release Candidates who view evaluation criteria, testing strategy, or acceptance definitions as purely engineering responsibilities are unlikely to be successful in this position. Must Have The following are required screening criteria. Candidates who do not meet all requirements should not be considered a fit. 1. Product Management Experience Three or more years of experience in: Product Management Product Ownership Product Analysis Including ownership of at least one AI or LLM-enabled product successfully deployed to production. Pilot projects, demonstrations, prototypes, and proof-of-concepts do not satisfy this requirement. 2. AI Evaluation Ownership Demonstrated ownership of acceptance criteria and evaluation processes for AI or LLM-enabled functionality. Candidates should be able to clearly explain: Evaluation specifications Golden datasets Acceptance thresholds that they personally defined. 3. RAG and LLM Evaluation Knowledge Working knowledge of: Retrieval-Augmented Generation (RAG) Faithfulness Groundedness Retrieval Precision Retrieval Recall Answer Relevance Regression Testing across prompt and model versions Enough depth to guide engineering teams, evaluate outputs, and challenge results without requiring direct coding responsibilities. 4. Business Case Development Experience creating business cases that include: Defined baseline Success metric Attribution methodology Measured outcome Simply reporting hours saved is not sufficient. 5. Executive and Operational Stakeholder Management Experience partnering with: Operational leaders Business stakeholders Executive leadership Including: Intake management Opportunity scoring Portfolio reporting 6. Work Authorization Must be: Authorized to work in the United States on a W-2 basis without sponsorship Physically located within the United States for the duration of the engagement Preferred The following strengthen a candidate profile but are not required. Do not reject otherwise qualified candidates solely because these items are absent. Familiarity with Model Context Protocol (MCP), AI plug-ins, AI skills, AI asset catalogs, or internal capability marketplaces Experience with AI evaluation platforms such as: RAGAS DeepEval Braintrust LangSmith Arize Phoenix Weights & Biases Weave OpenAI Evals Similar evaluation frameworks Familiarity with AI governance frameworks, including: NIST AI Risk Management Framework NIST Generative AI Profile ISO 42001 Colorado AI Act Industry-specific AI governance requirements Experience operating in regulated environments where auditability, access controls, compliance, and logging are standard requirements Insurance, financial services, healthcare, or similarly regulated industry experience Familiarity with enterprise platforms such as: Applied Epic Vertafore Microsoft Dynamics 365 Salesforce Orion Workday Oracle Fusion ACORD data standards Experience working within an AI or Automation Center of Excellence (CoE) or similar centralized operating model Experience supporting demand governance, portfolio management, and ongoing product rationalization efforts Exposure to RPA, UiPath, Microsoft Power Platform, or intelligent automation ecosystems Experience using Excel, Power BI, or similar tools for value analysis and portfolio reporting Jira or Azure DevOps experience for backlog management and delivery planning Bachelor's degree or equivalent practical experience Certifications No certification is required for this role, and demonstrated experience delivering successful AI-enabled products carries significantly more weight than certifications. The following may be viewed as minor positives: Reforge Product School Pragmatic Institute Scrum.org Product Owner Certifications Microsoft AI Fundamentals (AI-900) Certification Guidance Do not reference, prioritize, or screen candidates based on: PL-500 PL-600 PL-200 These certifications are being retired and are not considered target credentials. There is currently no broadly recognized individual certification for Anthropic Claude expertise, so candidates should not be evaluated based on possession of one.