Case study: an AI SRE read every alert for 25 days

Understanding one alert properly can take an engineer an afternoon: the logs, the metrics, the recent deploys, the config. So every on-call team rations it. Alerts that arrive on fire get investigated. The rest get acknowledged, snoozed or closed, and nobody finds out what they were saying.

The question for this trial was simple: what happens when every alert gets read?

Nobody wrote rules, mapped alerts to runbooks or tuned anything first.

• Kubernetes: one kubectl apply per cluster installed RubixKube's observer. It reads pods, events, logs and rollouts, and changes nothing.
• Datadog: one webhook, so every monitor that fires opens an incident and its recovery resolves it. Datadog was also connected as a tool, so investigations could read its events, monitors and dashboards.
• Read-only throughout. Every Fix went to people as a proposal.

The first root cause arrived four minutes after the first alert.

Several of the best-known AI SRE products start with a demo, a sales process and a deployment scoped for your stack before you see a single answer (compare them here). This team saw what their own alerts were saying on the first afternoon.

Half the stress of on-call is not knowing whether anyone is looking. Every incident showed where its investigation stood: detected, triaged, waiting, investigating, written, each step with how long it took, and the incidents list showed the same on every row.

Nobody had to ask in a channel whether someone was on it, and nobody started a second investigation of the same thing. When an incident was waiting on purpose, to see whether it cleared on its own, the page said so and said for how long. Teams on Slack get the same updates in the incident's thread.

The expensive part of an incident is the stretch between "something is wrong" and "I know why". That is the part RubixKube took off the team.

• A median investigation took 4.2 minutes, and nine in ten finished within about 9.
• From alert to finished root cause, the median was 11 minutes, including deliberate waits so alerts that clear on their own are left alone.
• Each confident root cause carried a median of 10 pieces of evidence: the exact query, what came back, and a link from every claim to the result behind it.

On-call opened an incident and found an answer to check, not a blank screen and a dashboard.

Reading every alert only helps if it means less work for people, not more. Most of what RubixKube did was take work away before anyone saw it.

• Related alerts became one incident. Alerts about the same service, or with the same likely cause, were grouped. Over a third fewer investigations were needed, and an alert storm produced one answer.
• Triage sorted before anyone was asked. Hold, escalate, or mark as a known alert, each with its reason. 548 incidents were sorted this way.
• Blips were left alone. A short wait by severity meant an alert that recovered on its own was never investigated.
• Warnings stayed warnings. Datadog Warn alerts opened one severity lower, with the investigation held until the alert was clearly real.
• Nothing was left in someone's head. Each root cause came with a Fix and Follow-ups as Tasks, each with a check that says when it is done. 648 were proposed.

Each of these removes a moment where an engineer would stop, look, and decide it was nothing. Those moments are what alert fatigue is made of.

Read every alert, and a pattern appears that is invisible one alert at a time: 27 investigations ended in a false alarm or test traffic, and each one named the monitor behind it. Noise that looks random from the on-call seat comes from a short list of monitors, and a short list is something a team can fix in an afternoon.

The false alarms fell into patterns every on-call team will recognise:

RubixKube adapts to a team, not the other way round, and it got better the longer it ran.

• Their knowledge, in plain words. The team wrote down how their best engineer investigates their most common alert, as a skill in plain language. RubixKube follows the skill that applies and says in the report which one it used.
• Where their data lives. One short skill told it where each kind of the team's data lives and how to query it, so investigations went straight to the right place.
• Their ratings. Every Right or Wrong on a root cause shapes how confident similar investigations are next time.

By the last week, 65 of 94 investigations named a root cause at 80% confidence or more, nearly seven in ten. Confidence here is earned by cited evidence and capped, so nothing claims certainty it has not shown. See how skills work.

A team that starts today starts where this trial finished. Everything above is in the product, ready on the first afternoon:

Reading every alert a large company sends, for 25 days, showed us things that are hard to see from inside one on-call shift.

1. An alert is rarely alone. 476 alerts were about only 165 problems, about three alerts for every real one. At this scale, counting alerts overstates the trouble, and reading them one at a time multiplies the work.
2. Volume comes in storms. Most days brought a few dozen incidents at most. A couple of days brought around 200. No rota is sized for the storm days, and those are the days that matter most.
3. Much of the noise is self-inflicted. Load tests and planned maintenance set off alarms meant for real failures. The bigger the company, the more of its alerts are about its own activity.
4. Noise comes from a few monitors. 27 false alarms traced back to a short list. Fix those few and most of the noise goes, without raising a single threshold elsewhere.
5. Real problems do not page. The finding that mattered most was a job running at twice its limit. Nothing had broken yet, so its alert was the easiest one to dismiss.
6. No single tool holds the answer. The evidence was spread across the cluster, Datadog and the logs. Knowing where to look is half of every investigation, and writing it down once paid off on every investigation after.

When understanding an alert costs minutes instead of an afternoon, rationing stops making sense. The noisy monitors get fixed, the slow problems get caught early, and the person on call starts with an answer.

The docs have each step: connect a Kubernetes cluster, connect Datadog, and read and rate a root cause.