Note: The job is a remote job and is open to candidates in USA. Impiricus is hiring a Director, AI Automation Engineering to own the quality and reliability layer for AI systems across its internal platform and customer-facing products. The role builds automated testing infrastructure, evaluation frameworks, and release gates, while partnering with AI and product teams and developing the automation function.
Responsibilities
- Own the infrastructure that continuously verifies whether autonomous agents are taking the actions we intend, before and after they reach production
- Stage, test, and lifecycle-manage bots and agent workflows before they reach production
- Establish baseline quality metrics and coverage standards for AI initiatives
- Build the infrastructure to run automated research chains and agent evals at scale
- Create automated evaluation frameworks for LLM outputs: accuracy, safety, bias, and regression
- Implement RAG evaluation, prompt regression testing, and golden-set benchmarking
- Define and enforce quality thresholds that AI Engineers and builders must meet before ship
- Partner with AI Researchers and Engineers on eval design for new model and agent patterns
- Build the release-gating infrastructure that governs when an agent is trusted to act in production, including staged rollouts and rollback authority
- Develop AI automation solutions to ensure that agentic coding delivers highly functioning software while maintaining quality across the product portfolio
- Integrate testing into GitHub Actions (or equivalent) for AI model and agent releases
- Maintain existing test suites and keep them current as models, prompts, and tools change
- Work with AI Solutions Architects on eval criteria for workflow automations they build in the field
- Partner with Principal AI Product on launch-readiness and QA for AI projects across the org
- Support product teams (Ascend, DocUpdate, Platform) on AI feature validation without owning their roadmaps
- Document standards, runbooks, and patterns so non-engineers building in Sage can self-serve basic testing
Skills
- 5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety
- Direct experience evaluating LLM outputs: RAG eval, prompt regression, safety and accuracy benchmarking
- Experience building CI/CD pipelines with integrated test gates (GitHub Actions or similar)
- Comfortable working across AI platform infrastructure and product-facing AI features
- Operates independently as a senior IC and communicates quality standards to engineers and non-engineers
- Proficiency in Python and standard testing frameworks (pytest or equivalent)
- Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus
- Familiarity with LLM observability and eval tooling (LangSmith, Braintrust, custom harnesses, etc.) is a strong plus
- Background in healthcare, pharma, or regulated environments where quality gates matter is a strong plus
- Prior experience standing up a QA or automation function from scratch, with appetite to grow into leading a team
Benefits
- Medical, dental, and vision coverage for you and your dependents + on-demand healthcare concierge
- Pre-tax savings options for healthcare and dependent care, with monthly employer contributions to HSA (if enrolled in a high-deductible plan)
- 100% paid short- and long-term disability, plus life and AD&D insurance
- Flexible vacation policy
- Paid parental leave to spend time with your newborn, adopted, or foster child (available after 6 months)
- If you're close to an office, we encourage spending some time in-person to collaborate and connect. If not, remote is always an option.
- 401(k): Save for your future with tax advantages (and company match!)
Company Overview