Note: The job is a remote job and is open to candidates in USA. OnBoard is a board intelligence platform trusted by organizations to simplify governance and deliver AI-powered product experiences. The Senior AI Engineer will build, evaluate, instrument, and improve reliable, measurable, cost-effective, and safe AI systems using LLMs, RAG, semantic search, and agentic workflows. The role also establishes quality standards, observability practices, evaluation pipelines, and reusable engineering patterns across product teams.
Responsibilities
- Partner with product engineers to design, build, and improve AI features using LLMs, RAG, semantic search, and agentic workflows
- Contribute directly to production codebases in Python, C#/.NET, or related technologies
- Improve prompt, retrieval, context assembly, ranking, grounding, and response-generation patterns
- Help teams make practical architecture tradeoffs across quality, latency, cost, privacy, and maintainability
- Support model and provider evaluations, migrations, fallback strategies, and rollout plans
- Design and implement evaluation pipelines for LLM-powered features, and build scoring methodologies from first principles rather than reaching for the nearest metric
- Build and maintain versioned golden datasets covering real-world use cases, edge cases, failure modes, and customer-critical workflows
- Implement LLM-as-judge, heuristic, human-feedback, and task-specific quality scoring approaches
- Establish the criteria that determine whether an existing evaluation is sufficient for a given feature and risk profile — and identify gaps before they become production issues
- Establish prompt and retrieval regression testing as part of the development lifecycle
- Define quality gates and thresholds that help teams know when an AI feature is ready to ship
- Instrument LLM interactions, RAG pipelines, tool calls, and agent workflows using observability platforms (OnBoard currently uses Arize; comparable tools include Langfuse, LangSmith, W&B, and OpenTelemetry-based stacks)
- Track latency, token usage, cost, retrieval quality, groundedness, failure modes, safety signals, and user feedback
- Build dashboards and alerts that surface meaningful product and engineering signals, not just raw telemetry
- Analyze production traces to identify quality issues, cost spikes, regressions, and improvement opportunities
- Create runbooks and response patterns for common LLM and AI-product failure modes
- Build reusable libraries, SDKs, templates, and reference implementations that make correct AI instrumentation and evaluation easy
- Document standards for tracing, metadata, prompt/version tracking, evaluation, cost reporting, and incident response
- Coach product teams on AI quality, evaluation design, observability, and reliable release practices
- Help establish shared patterns that let OnBoard scale AI development across product lines
- Ensure AI systems handle sensitive data appropriately and align with security, privacy, SOC 2, ISO 27001, and data-residency requirements
- Monitor guardrails, policy enforcement, content safety signals, and safety-related anomalies as a distinct observability concern
- Support auditability and traceability of AI interactions where required
Skills
- 5+ years of software engineering experience building production systems
- Hands-on experience building or operating LLM-powered features, RAG systems, AI workflows, or similar AI applications
- Strong engineering ability in Python, C#/.NET, or both
- Practical understanding of prompts, embeddings, vector search, retrieval quality, orchestration patterns, and LLM application architecture
- Experience designing evaluations, quality metrics, or regression frameworks for AI or software systems — with the judgment to assess whether an evaluation actually measures what matters
- Strong observability fundamentals: tracing, logging, metrics, alerting, and production debugging
- Experience with CI/CD, git workflows, cloud environments, and production release practices (Azure DevOps preferred)
- Ability to communicate clearly with engineering, product, QA, security, and business stakeholders
- Strong product judgment and genuine curiosity about how AI systems behave with real users
- Proficiency with AI-assisted development tools (e.g., Claude Code, PlayerZero)
- Experience with LLMOps or AI observability tools such as Arize, Langfuse, LangSmith, W&B, Humanloop, or Helicone
- Experience with OpenTelemetry, Azure Monitor, Application Insights, or similar observability platforms
- Experience with Azure AI Search, Pinecone, Qdrant, Weaviate, pgvector, or other vector search platforms
- Experience with Semantic Kernel, LangChain, LlamaIndex, AutoGen, or related frameworks
- Experience with LLM-as-judge evaluation, RAG evaluation, semantic similarity metrics, hallucination detection, groundedness scoring, or human-feedback workflows
- Experience with dedicated evaluation frameworks (e.g., DeepEval, LangTest) and benchmarking approaches for LLM outputs
- Experience with A/B testing, online experimentation, or product analytics for AI features
- Experience in regulated environments with SOC 2, ISO 27001, PII handling, or data-residency requirements
- Background in QA, ML, or data engineering that informs a rigorous approach to quality measurement
Benefits
- Fully remote work with company provided equipment (laptop, software, etc.)
- Employment with a growing, casual, fun, philanthropic minded company
- US-based employees: Comprehensive, high-quality medical/prescription drug plan options, as well as dental and vision plan offerings.
- US-based employees: An employer contribution to your Health Savings Account (HSA) if you participate in a High Deductible Healthcare Plan.
- US-based employees: Medical Flexible Spending Accounts available.
- US-based employees: Dependent Care Flexible Spending Accounts available.
- US-based employees: Basic life insurance in the amount of $50,000 or 1 X’s your salary (whichever is higher).
- US-based employees: Short and long-term disability and Accidental Death and Dismemberment benefits at no cost to you.
- US-based employees: 401K Retirement Savings Plan with automatic enrollment at the first of the month following 60 days of employment at 5% to help you secure your financial freedom.
- US-based employees: A generous company match that starts on the first of the month following 60 days of employment. The company match is dollar for dollar on the first 3% of your pay that you contribute and $0.50 on the dollar on the next 2%, for a total match of 4%.
- US-based employees: Paid Time Off (PTO)/Holiday
- CAN-based employees: Employer paid Life and Accidental Death Insurance
- CAN-based employees: Contribution to Health Care Spending Account
- CAN-based employees: Dependent Life Insurance
- CAN-based employees: Optional Life Insurance
- CAN-based employees: LTD Insurance
- CAN-based employees: Drug and Paramedical Coverage
- CAN-based employees: Dental Insurance
- CAN-based employees: Vision Insurance
- CAN-based employees: EAP
- AUS-based employees: Superannuation rate of 12%
- AUS-based employees: Monthly stipend of $400 AUD to purchase private medical insurance
- UK-based employees (via EPG): Pension - Aegon
- UK-based employees (via EPG): Passageways/OnBoard contributes 8% of the employee's basic salary
- UK-based employees (via EPG): Enrolled from Day 1 of employment
- UK-based employees (via EPG): Private Medical Insurance
- UK-based employees (via EPG): Life Assurance
- UK-based employees (via EPG): Income Protection
- UK-based employees (via EPG): Critical Illness
- UK-based employees (via EPG): Employee Assistance Programme
- UK-based employees (via EPG): Serious Illness Benefit
- UK-based employees (via EPG): Help@Hand
- UK-based employees (via EPG): Cashplan
Company Overview
Company H1B Sponsorship