Note: The job is a remote job and is open to candidates in USA. Arcadia is a healthcare platform that transforms complex healthcare data into trusted intelligence for providers, payers, and life sciences organizations. The Staff Applied AI Engineer will own product-layer decisions shaping agent behavior, including prompting, retrieval, context, memory, evaluation, and escalation, while improving the safety, reliability, transparency, and performance of agentic workflows.
Responsibilities
- Design and iterate on agent behavior across real, live workflows, including long-horizon, multi-turn agentic tasks
- Design retrieval and context architecture so the right source data reaches a model in the right structure and agents remain grounded in real data rather than filling gaps with assumptions
- Design memory and state handling across multi-turn and multi-agent flows, determining what is carried forward, summarized, or dropped and why
- Create context and prompt templates that combine few-shot examples, structured formatting, and reasoning scaffolding for consistent agent behavior
- Improve performance through prompting, tool-use strategy, and context construction, validated through direct experimentation rather than guesswork
- Build and run evaluations against real production conditions to measure performance, regressions, failure modes, and edge cases
- Author evaluation rubrics, quality heuristics, and thresholds that weight failures by severity and cost, not just frequency, and monitor those measures against production behavior
- Design and validate escalation paths that route agents to human review based on confidence and uncertainty while preserving safety and consistency under adversarial and edge-case conditions
- Design for cost-aware performance alongside latency, reliability, and accuracy through efficient context construction and tool-call economy
- Evaluate and sign off on model changes by baselining current behavior, running comparative evaluations, and making the go/no-go call before a change reaches a customer
- Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes, so customer-facing teams work from actual agent behavior
- Partner closely with Product and product managers to ensure agents are not just capable, but steerable, trustworthy, and ready to scale
Skills
- We value equivalent practical experience that demonstrates the depth required for this staff-level role
- 8+ years of production software engineering experience, including 3+ years of hands-on ownership of ML, LLM, or agentic systems in production, with direct experience in healthcare, finance, or another regulated industry
- Demonstrated ability to diagnose why an agent failed, correctly attribute the fix to instruction, retrieval, context, or memory design, and weigh failures by severity and cost rather than frequency alone
- Hands-on experience with RAG architecture, production-grounded evaluation frameworks, and fallback or human-in-the-loop logic for automated systems
- Working familiarity with AWS AI/ML services, including Bedrock and SageMaker, sufficient to build and evaluate effectively in Arcadia's environment
- Evidence-led judgment and the credibility to push back on launch decisions, paired with a builder's instinct to run the experiment and move from a production failure to a fix
- Experience applying AI to healthcare data or workflows where safety, transparency, and calibrated uncertainty directly affect care teams or patients
- Experience with long-horizon, multi-turn or multi-agent workflows and product-level AI documentation such as model cards
Benefits
- A flexible, remote-friendly culture with personality and heart
- Employee-driven programs and initiatives for personal and professional development
- Membership in the talented, energized, diverse, and purpose-driven Arcadian community
Company Overview
Company H1B Sponsorship