The Principal AI Operations Engineer reports to the Sr. Director, AI Center of
Excellence, in the Technology Department. This role defines how AI systems at
Inspira Financial are instrumented, measured, monitored, and evidenced, and
drives adoption of those standards across the enterprise, spanning both
vendor-provided and internally built AI capabilities. Primary duties include
building the enterprise telemetry and observability standard for AI systems;
designing evaluation, drift detection, and agent guardrail frameworks;
translating AI risk and compliance frameworks into operational controls and
audit evidence; maintaining the enterprise AI system inventory; and teaching
other engineering teams to adopt these standards independently.
Duties & Responsibilities:
1. Define the enterprise telemetry standard for AI systems, covering traces,
prompts, tool calls, retrieval, tokens, latency, and cost, built on a
vendor-neutral schema such as OpenTelemetry GenAI semantic conventions.
2. Build the logging and observability layer that provides the AI Center of
Excellence, business units, and the AI Governance Committee with a single
view of AI activity across custom applications, agent platforms, and
vendor-embedded AI.
3. Integrate AI telemetry with enterprise log aggregation, application
performance monitoring, security monitoring, alerting, and incident
response platforms to support operational visibility, investigation, and
audit retention.
4. Establish retention, redaction, masking, and access patterns for AI
conversation and trace data that satisfy privacy, records retention, and
legal hold obligations.
5. Publish reference implementations and starter kits so teams adopting the
telemetry standard inherit a working path.
6. Design the evaluation framework for AI systems, including golden datasets,
offline evaluation harnesses, automated and LLM-as-judge scoring, and
regression gates in CI.
7. Define drift criteria for each class of AI system and build detection for
output quality degradation, retrieval decay, input distribution shift, and
silent model updates from providers or vendors.
8. Build runtime guardrails and boundary enforcement for agents, including
tool and action authorization, scope and spend limits, escalation paths,
human-in-the-loop triggers, and hard-stop conditions.
9. Establish the AI incident model, including severity taxonomy, detection,
alerting, rollback, postmortem process, and conversion of incidents into
new test cases.
10. Translate NIST AI RMF, ISO/IEC 42001, model risk management practice, and
applicable privacy and recordkeeping obligations into operational controls
engineers can implement.
11. Build the evidence layer that automates capture of artifacts needed for
internal audit, external examination, and client due diligence.
12. Integrate AI operational data with governance, risk, and compliance
platforms to support continuous control monitoring, automated evidence
collection, risk reporting, and audit readiness.
13. Maintain the operational AI inventory across custom builds, agents,
integrations, and vendor AI features, including an owner, risk
classification, and control coverage for each entry.
14. Define the assurance pattern for vendor AI, including required telemetry,
attestations, detection of vendor model changes, and compensating
monitoring.
15. Build metric frameworks connecting AI usage, quality, cost, and risk
posture to business outcomes, and report these to CoE leadership, business
unit leaders, and the AI Governance Committee.
16. Own token and platform cost visibility and attribution by team and use case
and lead resulting optimization work.
17. Run working sessions, document patterns, review designs, and coach teams
through their first fully instrumented AI deployment.
18. Partner with Risk, Compliance, Legal, Privacy, and Security to incorporate
feedback from the risk triad and ensure adequate detection, response, and
mitigation measures are in place.
19. Perform other duties as assigned.
Qualifications
Education & Experience:
Minimum required:
Bachelor of Science in Engineering or Computer Science
Relevant certifications are a plus: Datadog Certified Professional, Google
Cloud Professional Data Engineer for BigQuery ML, or PagerDuty’s AI Ops
8 or more years of telemetry-driven engineering experience, including 2 years
operating LLM, ML, or agentic systems in production, with hands-on ownership
of instrumentation and monitoring.
Skills & Abilities:
Production-grade proficiency in Python and SQL, including building tooling,
pipelines, and services other teams depend on.
Demonstrated experience in AI evaluation and observability, including eval
harnesses, tracing, scoring, drift detection, and guardrails.
Working command of at least one AI risk or governance framework, such as NIST
AI RMF, ISO/IEC 42001, EU AI Act, or model risk management practice, and the
ability to convert it into controls and code.
Experience operating in a regulated environment where logging, retention,
evidence, and access control are enforced rather than aspirational.
Ability to influence without formal authority across engineering teams, legal
counsel, and executive stakeholders.
Financial services, healthcare, benefits administration, or other regulated
industry experience preferred, particularly environments handling PHI, PII,
or fiduciary data.
Depth of knowledge with telemetry platforms and related metric and
measurement practices preferred.
Hands-on experience integrating and operating enterprise platforms for AI and
application observability, centralized logging, governance, risk, continuous
compliance monitoring, AI system inventory, evaluation and testing, incident
management, and cloud cost management preferred.
FinOps or cloud cost management experience applied to token and inference
spend preferred.
AI security fluency preferred, including prompt injection defense, OWASP LLM
and agentic risk categories, red teaming, tool sandboxing, and
least-privilege design for agent identities.
Track record of standing up a capability that other teams adopted voluntarily
preferred.
Other Requirements:
Ability to occasionally work outside normal business hours to meet project
deadlines as needed.
Prolonged periods of sitting at a desk and working on a computer.