Note: The job is a remote job and is open to candidates in USA. ServiceTitan is a cloud-based software company that helps businesses operate more efficiently. The Senior Site Reliability Engineer will own and improve the reliability, observability, scalability, and efficiency of cloud applications and infrastructure, including Kubernetes platforms and Azure/AWS environments. The role also focuses on building AI agents and automation to resolve infrastructure and reliability work, managing incidents, and partnering with engineering teams on architecture and delivery.
Responsibilities
- Participate in an on-call rotation, using runbooks and playbooks to diagnose and resolve production issues (e.g., adjusting Horizontal Pod Autoscaler rules in response to load)
- Design, build, and maintain observability dashboards and alerting grounded in Service Level Indicators (SLIs) and Service Level Objectives (SLOs)
- Operate and improve our Kubernetes-based compute platform, which runs the large majority of our infrastructure
- Work across cloud networking and infrastructure (Azure/AWS) to support reliable, scalable systems
- Investigate and resolve production incidents, including root-cause analysis and follow-up remediation work
- Build and operate AI agents and automation that take on manual, repetitive SRE work directly — not just tools that assist a human doing the work
- Partner with product engineering teams to review architecture and infrastructure decisions before they ship
- Write and maintain runbooks and documentation so on-call knowledge is shared across the team, not siloed with one person
- Help define non-functional requirements — scalability, availability, performance — for new systems as they're designed
- Collaborate across engineering teams to adopt best practices in reliability and observability
- Contribute to CI/CD pipelines and help teams ship changes safely and quickly
Skills
- Kubernetes (must-have): strong, hands-on understanding of Kubernetes as a system
- AI-native SRE practice (must-have): hands-on, personal experience using AI tools and agents (e.g., Claude Code, GitHub Copilot Workspace, Cursor, or custom agents built on Claude/MCP servers) to diagnose, automate, and resolve infrastructure and reliability work
- You've built or operated agents that autonomously monitor infrastructure and take action (e.g., an agent that watches system load and scales, remediates, or escalates without a human in the loop) — not just general AI coding assistance
- You can speak concretely to how these agents are actually built and operated (MCP protocol, what a harness is, how to differentiate agent-design strategies rather than just naming tools), to context management (e.g., progressive-disclosure strategies for surfacing the right information without dumping everything into context), and to securing agent actions (scoping authorization, guardrails, what's available in the AI infra ecosystem to enforce it)
- That agentic work is pointed at infrastructure and reliability problems specifically — you use AI to move at a materially faster pace in the SRE domain, not as a general-purpose coding aid
- SRE principles: practical experience with SLIs, SLOs, and error budgets — able to speak to how you've defined and monitored these on real systems, not just definitions
- Cloud engineering & networking: solid grounding in AWS or Azure, including networking fundamentals (subnetting, IP addressing)
- Observability: deep experience with at least one modern observability stack (OpenTelemetry, Prometheus, Grafana, Datadog, or Elasticsearch) and the ability to translate that understanding across tools
- CI/CD: strong understanding of a CI/CD system — GitHub Actions preferred, but TeamCity, Azure DevOps, or GitLab CI experience is acceptable
- Programming: strong programming skills with the ability to build web applications — ideally with solid working knowledge of .NET and ASP.NET
- We're also open to strong Python (Flask, FastAPI) or Java (Spring) backgrounds
- Experience with distributed systems and their common failure modes (retries, timeouts, cascading failures)
- Strong production troubleshooting skills — comfortable diagnosing issues under pressure
- 8-10+ years of relevant hands-on experience
- GitHub Actions preferred
- Solid working knowledge of .NET and ASP.NET
- Nice-to-have: database experience (not mandatory — databases are monitored by the same team, not owned individually)
Benefits
- Flexible time off with ample learning and development opportunities to continue growing your career.
- A comprehensive onboarding program.
- Leadership training for Titans at all levels, and other programs and events.
- Great work is rewarded through Bonusly, peer-nominated awards, and more.
- Company-paid medical, dental, and vision, with 100% employer paid options and 90% coverage for dependents.
- FSA and HSA.
- 401k match.
- Telehealth options including memberships to One Medical.
- Parental leave and support.
- Up to $20k in fertility services (i.e. IUI and IVF).
- Surrogacy and adoption reimbursement.
- On-demand maternity support through Maven Maternity.
- Free breast milk shipping through Maven Milk.
- Pet insurance.
- Legal advisory services.
- Financial planning tools.
- An annual bonus.
- Equity.
Company Overview
Company H1B Sponsorship