Note: The job is a remote job and is open to candidates in USA. Health Note is building AI-powered digital assistants for healthcare workflows. The Senior AI Product Engineer will build and improve production AI product systems for the Access Agent, using evaluations, analytics, experiments, and operational data to measure performance, identify issues, and drive reliable improvements.
Responsibilities
- Own improvements to the Access Agent from investigation through production rollout, measurement, and follow-up
- Build practical ways to measure agent performance, including evals, dashboards, review workflows, logs, and production metrics
- Analyze transfer patterns, containment rates, task success, and operational outcomes to identify improvement opportunities
- Build tooling and workflows that connect prompts, configurations, interventions, customer workflows, and outcomes
- Create checks that help us catch regressions before and after changes go live
- Partner with Product, Operations, and Engineering to prioritize and validate improvements
- Design lightweight experiments and measurement approaches for model, STT, prompt, workflow, and configuration changes
- Build visibility into agent performance through dashboards, analytics, and reporting
- Use AI-assisted development tools and agentic workflows to accelerate experimentation and delivery
- Own a product improvement from problem identification through production rollout and measured impact
- Establish practical checks for evaluating changes before production deployment
- Create visibility into key Access Agent performance metrics and breaking points
- Ship measurable improvements with documented before-and-after impact
- Help Health Note build a disciplined, scalable system for continuous AI improvement
Skills
- 5+ years of software engineering experience building production systems
- Experience shipping Agentic systems into production and measuring whether they worked
- Strong analytical and problem-solving skills with an ability to translate ambiguous problems into measurable outcomes
- Ability to use metrics, logs, qualitative review, or experiments to understand whether product changes worked
- Strong product instincts and ability to connect technical decisions to business outcomes
- Demonstrated ability to independently own initiatives from problem identification through measurable impact
- Experience using modern AI-assisted engineering workflows (e.g., Claude, Cursor, Codex, or similar tools)
- Experience building eval frameworks for AI products
- Familiarity with Langfuse, observability platforms, analytics systems, or experimentation tooling
- Experience with RAG, agent orchestration, prompt management, or workflow automation systems
- Experience measuring and improving customer-facing AI experiences
- Healthcare technology or regulated-industry experience
- Experience working in startup environments with significant ambiguity
Benefits
- Remote work arrangement
Company Overview