AI Engineer
Department: Technology
Location: Remote
FLSA Status: Exempt
People Leader: No
Travel: Less than 10%
Summary
Company Summary
Accuity partners with hospitals and health systems through a technology-enabled, physician-led model that improves clinical documentation integrity, coding accuracy, reimbursement optimization, and quality outcomes.
Job Summary
The AI Engineer develops and operates the AI systems that support automation across Accuity’s clinical documentation and revenue cycle workflows. Reporting to the Vice President, Data Science and AI, this role builds production services using large language models, intelligent agents, retrieval systems, knowledge graphs, and traditional machine learning.
The AI Engineer owns the technical systems that allow applications to use AI reliably and securely. This includes model integration and adaptation, agent orchestration, retrieval-augmented generation, vector and graph-based knowledge systems, Model Context Protocol services, evaluation pipelines, deployment automation, observability, and production support.
The role works across proprietary and open-source models and selects the appropriate approach for each problem. Depending on the use case, this may involve prompt and context engineering, retrieval, fine-tuning, tool integration, classical machine learning, or a combination of these methods.
The AI Engineer partners with Data Science to move validated approaches into production, Platform Management to use approved infrastructure and delivery pipelines, Data Architecture to access governed data, and application engineering teams to deliver AI capabilities through stable and reusable services.
Success in this role is measured by the reliability, security, maintainability, and measurable workflow impact of the AI systems delivered across the Accuity ecosystem.
Responsibilities
AI and Agent Engineering
- Build and maintain production systems using large language models, machine learning models, and intelligent agents.
- Develop agent workflows that use tools, retrieval, structured outputs, state, and human approval controls.
- Select and adapt models through prompting, retrieval, fine-tuning, or other appropriate techniques.
- Integrate AI systems with internal applications, data sources, APIs, and approved external services.
- Build reusable AI services with clear, stable interfaces for application teams.
Retrieval and Knowledge Systems
- Develop retrieval-augmented generation systems using approved enterprise content and data.
- Build and maintain document ingestion, embedding, indexing, search, reranking, and citation pipelines.
- Implement vector databases, hybrid search, knowledge graphs, and graph-based retrieval where appropriate.
- Maintain data provenance, access controls, and processes for keeping indexed information current.
- Measure retrieval quality and correct missing, outdated, duplicated, or irrelevant results.
Model Context Protocol and Tool Integration
- Build and maintain Model Context Protocol servers, clients, tools, and resources.
- Define secure tool interfaces with clear schemas, permissions, validation, and failure handling.
- Connect agents to approved applications, databases, APIs, and business systems.
- Apply authentication, authorization, audit logging, and least-privilege access to agent tools.
- Develop reusable integrations that can support multiple AI applications.
AI and Machine Learning Operations
- Deploy and operate LLMs, agents, retrieval services, and machine learning models in production.
- Maintain automated pipelines for testing, versioning, deployment, promotion, retraining, and rollback.
- Track versions and lineage across models, prompts, tools, datasets, indexes, and configurations.
- Implement monitoring for quality, reliability, latency, usage, and cost.
- Investigate production failures and implement lasting corrective actions.
Evaluation, Quality and Security
- Build automated evaluation and regression testing for models, prompts, agents, retrieval systems, and tools.
- Work with Data Science to convert evaluation methods and production thresholds into release controls.
- Test systems against representative, unusual, adversarial, and high-risk scenarios.
- Implement safeguards against prompt injection, data leakage, unauthorized tool use, and unsafe outputs.
- Ensure AI systems comply with HIPAA, HITRUST, PHI protection requirements, and Accuity policy.
- Preserve human review, decision traceability, and audit logging where clinical or coding judgment is involved.
Collaboration and System Ownership
- Partner with Data Science to move validated AI approaches into production.
- Partner with Platform Management and Data Architecture on infrastructure, deployment, data access, and governance.
- Partner with application teams to deliver AI capabilities through reliable and documented services.
- Assume ownership of systems delivered by internal teams or external development partners.
- Write clean, secure, version-controlled code and maintain documentation that another engineer can use to operate the system.
- Communicate system behavior, limitations, risks, and tradeoffs clearly to technical, clinical, and business audiences.
Other Duties as Assigned
• Perform additional responsibilities as needed to support data science, technology, and organizational goals.
Qualifications
Education and Credentials
- Master’s degree in computer science, artificial intelligence, machine learning, or a related technical field, or equivalent practical experience, required.
- PhD degree in computer science, artificial intelligence, machine learning, or a related field preferred.
Experience
Required Experience
- 5+ years of experience in software, AI/ML, or platform engineering, including ownership of production systems.
- Hands-on experience developing and operating LLM applications, AI agents, and retrieval-augmented generation (RAG) solutions.
- Strong Python proficiency, including experience with AI/ML libraries, vector search, tool integration, and human-in-the-loop workflows.
- Experience deploying and maintaining AI/ML systems using automated testing, evaluation, CI/CD, monitoring, and production controls.
- Experience implementing AI observability, explainability, security, and governance practices.
Preferred Experience
- Experience with Azure AI Foundry, Azure ML, Databricks, vector databases, or comparable AI platforms.
- Familiarity with knowledge graphs, agent orchestration, Model Context Protocol (MCP), or model fine-tuning.
- Healthcare data experience, particularly clinical coding, CDI, claims, or revenue cycle; experience taking ownership of externally developed systems preferred.
Core Competencies
- AI Systems Engineering: Understands how models, agents, tools, retrieval, knowledge systems, and application services work together in production.
- Language Model Engineering: Selects, adapts, evaluates, deploys, and maintains language models based on measurable requirements.
- Agent Engineering: Builds controlled, observable agents that use tools safely and complete defined tasks reliably.
- Retrieval Engineering: Designs retrieval systems that return relevant, current, authorized, and traceable information.
- Knowledge Representation: Models entities and relationships in ways that improve retrieval, reasoning, and system interoperability.
- Production Ownership: Treats deployment, monitoring, support, documentation, and recovery as part of engineering the system.
- Evaluation Discipline: Uses repeatable evidence to determine whether changes improve or degrade system performance.
- Reliability: Anticipates failure, implements recovery behavior, and resolves the underlying causes of production problems.
- Security and Compliance: Treats data protection, access control, traceability, and approved routing as core architecture requirements.
- Technical Judgment: Selects an architecture based on the problem, operating constraints, and measurable value.
- Reusability: Builds shared services and patterns that reduce duplicate work across AI initiatives.
- Communication: Explains system behavior, limitations, risk, and tradeoffs clearly.
- Collaboration: Works effectively across data science, platform, data architecture, application engineering, clinical, and external partner teams.
- Adaptability: Evaluates new AI methods and tools without introducing them before their value, reliability, and risks are understood.
- Remote Work Effectiveness: Communicates proactively, documents decisions, and maintains visibility into work in a fully remote organization.
Additional Requirements
1) Physical Requirements: The requirements described here are representative of those that must be met by an employee to successfully perform the essential functions of this job with or without reasonable accommodations. Unless otherwise indicated, Accuity positions require interaction with people and technology while either sitting or standing. Employees must be able to communicate via phone, email, etc. and sit for extended periods of time, with or without reasonable accommodations. Physical effort and exposure to physical risk are limited to that of an office role / environment.
2) Position and Employment Statement: While this job description is intended to be an accurate reflection of the job requirements, management reserves the right to modify, add or remove duties from a job and to assign other duties as necessary and at any time. All positions at Accuity Delivery Systems, LLC, are at-will employment, and a position description is not a guarantee of a job or of job responsibilities.
While this job description is intended to be an accurate reflection of the job requirements, management reserves the right to modify, add or remove duties from a job and to assign other duties as necessary and at any time. All positions at Accuity are at-will employment, and a position description is not a guarantee of a job or of job responsibilities.
Master's degree in computer science, AI, or related field required; 5+ years software/AI/ML engineering experience; Python proficiency; experience with LLM applications, RAG, CI/CD, and AI observability; remote work.
Key Responsibilities
- building AI systems
- developing agent workflows
- maintaining retrieval systems
Skills & Tools
Python, Azure AI Foundry, Azure ML, Databricks, Model Context Protocol (MCP)
Job Details
- Category: Software Development
- Seniority: Senior Level
- Commitment: Full Time
- Workplace: Remote — United States
- Languages: English
About Accuity
A technology-enabled, physician-led company improving clinical documentation integrity, coding accuracy, reimbursement optimization, and quality outcomes. — Industry: Information Technology