Note: The job is a remote job and is open to candidates in USA. Vytalize Health is a company focused on leveraging AI to enhance healthcare delivery. The AI Engineer will design and maintain systems that automate healthcare workflows and support clinical decision-making, collaborating with various teams to implement AI-driven solutions.
Responsibilities
- Design, build, and maintain agentic systems and LLM-powered applications that automate healthcare workflows, data pipelines, and clinical decision support — from conception through production deployment
- Build and orchestrate agents using LLM APIs (OpenAI, Anthropic, etc.) and agentic frameworks (LangChain, LangGraph, CrewAI, or custom orchestration) to solve complex, multi-step healthcare problems
- Develop prompt libraries, agent instructions, and reusable 'skills' that improve agent accuracy, consistency, and reliability across different use cases and data domains
- Build validation and confidence-scoring layers that flag low-confidence agent decisions for human review before production deployment; establish guardrails and review workflows for agent-authored code and outputs
- Own end-to-end delivery of AI-automated systems — from problem scoping and requirements gathering through agent development, testing, and validated production deployment
- Implement rigorous evaluation and QA frameworks for agentic systems — including golden datasets, test cases, output validation, hallucination detection, and regression testing
- Establish and maintain evaluation metrics for agent performance, reliability, and clinical appropriateness; measure agent accuracy, hallucination rates, clinical validity, and real-world impact
- Implement observability, evaluation, and regression testing frameworks specific to agentic systems — decision tracing, lineage logging, and performance tracking
- Collaborate with data engineering and platform teams to integrate agent-built outputs (dbt models, transformation logic, recommendations) into existing data architectures and clinical workflows
- Ensure all agentic systems comply with healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices — including explainability, auditability, and clinician trust
- Continuously evaluate new LLM models, agent frameworks, prompt engineering techniques, and tooling; recommend adoption or migration based on healthcare-specific requirements (accuracy, cost, latency, regulatory alignment)
- Partner with data engineering to establish robust data validation and input validation layers for agents — agents are only as good as the data they operate on
- Lead experimentation and measurement of AI-automated systems impact on speed, quality, compliance, and cost across healthcare workflows
- Document agent architectures, prompt strategies, evaluation frameworks, and best practices for both technical and non-technical stakeholders
- Mentor AI Connector Engineers and other team members on agentic development patterns, LLM-powered application design, and responsible AI practices
- Work on-call as needed to support production agentic systems, troubleshoot agent issues, and respond to performance degradation or hallucination detection
Skills
- 3+ years of professional experience in data engineering, backend engineering, machine learning, or a related field
- 1+ years of hands-on experience building with LLM APIs and agentic orchestration frameworks — not just using AI coding assistants, but architecting agentic systems
- Strong Python and SQL proficiency
- Experience with cloud data platforms (AWS, Databricks)
- Solid understanding of data modeling, ETL/ELT patterns, and medallion architecture (Bronze/Silver/Gold)
- Experience building and consuming APIs
- Demonstrated experience with prompt engineering, agent evaluation, and validating LLM outputs
- Experience designing evaluation frameworks, test cases, and quality assurance for AI/ML systems
- Demonstrated ability to measure and track AI system performance through metrics and KPIs (accuracy, precision, recall, hallucination rates)
- Strong debugging and analytical skills, especially in ambiguous or novel technical territory
- Excellent written and verbal communication skills — this role requires documenting agent reasoning, decisions, and limitations clearly for both technical and non-technical audiences
- Comfortable working in a fast-moving environment with incomplete information and rapidly evolving AI/ML capabilities
- Experience with dbt or similar data transformation frameworks
- Familiarity with orchestration tools (Airflow, Databricks Workflows) and workflow automation
- Experience with agent evaluation and observability tooling (LangSmith, Langfuse, or custom frameworks)
- Background in healthcare, fintech, or another regulated/high-stakes domain where AI reliability is critical
- Experience building internal developer tooling, platform capabilities, or developer-facing products
- Hands-on experience with RAG (retrieval-augmented generation) or other grounding techniques for LLMs
- Familiarity with healthcare data formats and standards (FHIR, HL7, claims data, clinical NLP)
- Experience with model evaluation, fairness assessment, or bias detection in ML/AI systems
- Understanding of healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices
- Experience establishing QA frameworks, test plans, and quality metrics for ML/AI systems
- Startup or high-growth environment experience with rapid iteration and learning
- Published research, open-source contributions, or demonstrated thought leadership in AI/agentic systems
Company Overview
Company H1B Sponsorship