Note: The job is a remote job and is open to candidates in USA. Angi is a global company that has powered the future of the home services industry for over 30 years. The Staff AI Product Analyst will define AI-native product analytics, ensuring alignment between human teams and AI agents while building evaluation frameworks and a trusted data ecosystem.
Responsibilities
- Flagship Journey Measurement: Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomes
- Causal Framing & Experimentation: Align closely with data science partners on test design, interpretation, and causal framing when new AI products ship
- Data Layer Systems Thinking: Map how agents interact with metrics, raw tables, documentation, and APIs to ensure requirements for data and analytics engineering are explicit, structured, and testable
- Eval Framework Architecture: Design and maintain formal evaluation sets, "golden datasets," rubrics, and regression checks for agent outputs to guarantee accuracy, completeness, policy safety, and usefulness
- Taxonomy & Failure Mode Analysis: Systematically categorize wrong answers, tool misuse, context gaps, and data edge cases, feeding those insights back into prompts, retrieval (RAG), tools, and data modeling pipelines
- Performance Monitoring: Monitor live usage of Agentic AI analytics workflows—tracking success, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlog
- Playbooks & Narrative: Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty. Support enablement and leadership storytelling around agentic AI analytics
- Strategic Collaboration: Partner with business leaders, the Principal Analytics Engineer, and data engineering teams to translate complex AI capabilities into measurable business outcomes
- Defining Success: Deliver recurring measurement artifacts (dashboards, reviews, or written readouts) that leadership uses to prioritize roadmaps, while continuously reducing high-severity incidents where agents produce incorrect metrics
Skills
- Bachelor's in computer science or a quantitative field (e.g., Computer Science, Statistics, Mathematics, or related fields)
- 10+ years of experience across product analytics, data science, or technical product management within a complex, high-scale operating environment
- 1+ years of direct, hands-on experience working with AI/LLM systems in an applied context, with a specific focus on evaluating, trusting, and deploying them responsibly
- Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices
- Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data discipline
- Exceptional communication skills with a demonstrated ability to translate highly technical AI insights and tradeoffs into clear narratives for non-technical business stakeholders
- Experience building or evaluating agentic analytics workflows, orchestration frameworks, and familiarity with multi-agent systems
- Demonstrated ability to operate fluidly in ambiguity, run structured experiments or pilots, and default to fast action and iteration
- A systems-oriented mindset with a healthy skepticism of AI hype, focusing strictly on organizational outcomes and driving technical investments that yield true business value
Benefits
- Eligible for a competitive year end performance bonus & equity package.
- Full medical, dental, vision package to fit your needs
- Flexible vacation policy; work hard and take time when you need it
- Pet discount plans & retirement plan with company match (401K)
- #LI-Remote
Company Overview
Company H1B Sponsorship