About the role:
We run a distributed, camera-based video monitoring and AI alerting platform. The system spans the full spectrum of modern and legacy infrastructure: an AWS-hosted fleet of Java microservices and workers, a GPU-backed computer-vision inference pipeline, a real-time streaming and presence layer, and thousands of on-premise edge "media boxes" deployed in the field that ingest camera feeds, serve HLS video, and stream events back to the cloud.
This is a reliability-first role. Your primary job is to keep a large, mixed operational estate healthy at scale: meaningful service objectives, trustworthy alerting, sound capacity, tested disaster recovery, and fast, calm incident response. Delivery pipelines matter, but here they exist in service of reliability, not the other way around. We are not hiring a pipeline-and-self-service DevOps engineer who treats operations as a side concern. We are hiring an SRE who owns uptime and operational quality, and who can write the software to make that uptime measurable and automatic.
If you think in SLOs, error budgets, and blameless postmortems, and you are happiest when a noisy, fragile system becomes quiet and predictable on your watch, this is your role.
What you'll keep reliableYou will own the operational health of all of the following:
- Computer-vision / AI models: frame-based inference services running on GPU EC2 (g4dn-class, AWS Deep Learning AMIs), fed by camera frames from S3 and a Kafka (Amazon MSK) event bus, with Redis (ElastiCache) for state. Outputs flow through an alerts pipeline (OutgoingInferenceMessage to SNS/IoT to notification workers).
- Python services: the AI/alerts inference tier and supporting tooling.
- Legacy Java services: ~140 Java 8 services and libraries (REST APIs, SQS/SNS workers, Lambda functions) running on Jetty 9.4, deployed to Elastic Beanstalk, ECS, and Lambda.
- Edge appliances ("media boxes"): Ubuntu 22.04 / Docker Compose appliances managed remotely over AWS IoT Core secure tunnelling, with Cloudflare tunnels for egress. Includes an active CentOS 7 to containerized-stack migration effort.
- Data and messaging backbone: PostgreSQL (RDS) across many schemas, TimescaleDB for analytics, Redis, DynamoDB, Amazon MSK (Kafka), SQS/SNS, and Kinesis.
What you'll do Reliability and operations (the core of the role)
- Own service objectives. Define SLIs and SLOs for the services that matter, manage error budgets, and use them to drive prioritization and change-rate decisions. Make reliability a measured number, not a feeling.
- Make observability trustworthy. Own alert quality end-to-end: high signal-to-noise, alarms that reliably catch the incidents they are meant to catch, and the discipline to turn off a misleading alarm until it is fixed properly. Build the dashboards and the custom metrics, exporters, and instrumentation (CloudWatch, OpenTelemetry) needed to see the system clearly.
- Lead incident response. Run incidents calmly, drive mean-time-to-recovery down, and produce blameless postmortems with action items that actually get closed. Improve and own the on-call rotation and its health.
- Plan capacity and performance. Forecast and right-size compute (especially GPU), Kafka/MSK throughput and partitioning, RDS/TimescaleDB load, and Redis. Catch saturation before customers do.
- Own business continuity and disaster recovery. Backups, replication, failover, and recovery for RDS, MSK, Redis, and the edge fleet. Define RPO/RTO and prove them with regular, tested game days, not assumptions.
- Keep the edge fleet healthy. Remote diagnosis and recovery over AWS IoT, container auto-update over systemd timers, and the ongoing CentOS 7 migration to the containerized media stack (Ubuntu 22.04).
- Engineer away toil. Write real software (Python, Golang, Bash) to automate operational work, self-heal common failures, and make reliability repeatable instead of heroic.
- Govern production change safely. Enforce collaborative, reviewed change management; protect the system from risky, unilateral changes (topology, instance-count, scaling, and config).
Delivery and platform (in support of reliability)
- Keep CI/CD healthy and safe: CircleCI with Bazel/Gradle builds, OIDC-based AWS auth, container builds to ECR, and EB/ECS/Lambda deploys, tuned so that releases are safe, observable, and reversible.
- Maintain Terraform for the AWS estate (compute, networking, IAM, databases, messaging, monitoring), following our module-based conventions and S3-backed state.
- Harden security and compliance: IAM least-privilege, Secrets Manager/KMS, TLS and certificate management (including IoT device certificates), CloudTrail, and AWS Config.
Requirements
- AWS certification is mandatory. A current AWS Certified DevOps Engineer – Professional or AWS Certified Solutions Architect – Professional is strongly preferred.
- 10+ years in Site Reliability Engineering or production operations at scale. We do not expect mastery of every area below on day one. We expect real depth in several and the ability to ramp quickly on the rest.
- Demonstrated SLO/error-budget practice. You have defined SLIs and SLOs, run against an error budget, and used it to make real decisions.
- Strong production observability skills. Deep with metrics, logs, alarming, and dashboards (CloudWatch, OpenTelemetry, and/or Prometheus/Grafana/Datadog), and able to build the instrumentation when it does not exist.
- Proven incident command. You have led incidents, owned an on-call rotation, and written postmortems that changed how a system behaved.
- Capacity planning and performance experience across compute, databases, and a messaging or streaming system (Kafka/MSK ideal).
- Disaster recovery ownership: backups, replication, failover, and tested RPO/RTO.
- Software engineering ability for automation. Comfortable writing Python, Golang, and Bash to build reliability tooling, not just configure off-the-shelf tools.
- Expert with Terraform (or equivalent IaC) and strong Linux administration (shell plus Linux GNU utils), comfortable from cloud to bare-metal/edge.
- Database operations experience with PostgreSQL (time-series a plus).
- A reliability mindset: you instrument before you guess, and you write the runbook.
- Operating GPU workloads and serving computer-vision or ML models in production (CUDA, Deep Learning AMIs, inference scaling).
- Apache MSK / Kafka and streaming-data operations (Kinesis, Kinesis Video Streams).
- AWS IoT Core at scale: device provisioning, certificates, secure tunnelling.
- Managing a fleet of edge / on-premise devices (golden images, remote update, systemd).
- Operating and modernizing legacy systems (Java 8, Jetty, CentOS).
- Chaos engineering / game-day practice, and capacity modeling.
- Familiarity with Bazel in a monorepo; Cloudflare, Cognito/Auth0, API Gateway.
How we work, and what we expect from this hire:
These principles reflect lessons we take seriously. The right candidate already operates this way:
- Observability must be trustworthy. Alarms that cry wolf are worse than no alarms. You own alert quality end-to-end: high signal-to-noise, alarms that reliably catch the incidents they are meant to catch, and the judgment to turn off a misleading alarm until it is fixed properly rather than leave a noisy one firing. We have felt the cost of unreliable alerting and want someone who closes that gap and keeps it closed.
- Change management is collaborative, never unilateral. Any infrastructure change that affects a service is made together with the engineer who owns and understands that service. Scaling, instance-count, topology, and config changes are reviewed with that owner before they ship. Single-instance vs. multi-instance assumptions, in particular, are never changed on a guess.
- Follow-through over activity. We measure outcomes, not effort or hours. When something is asked for, we expect it driven to a correct, durable resolution on a reasonable timeline, not left half-solved for months. Find what is actually needed, focus on it, and finish it.
- Prefer the right fix to a quick patch. Solve the root cause. Do not paper over it and move on.
Originally posted on