One incident for an alert storm

Use caseUpdated November 20266 min read

Alerts from Datadog, PagerDuty and Grafana are triaged and grouped, so one cause becomes one incident.

On this page

Quick answer

When one failure sets off alerts across your tools, RubixKube folds them into one incident with one root cause analysis. Alerts from its own observers and from Datadog, PagerDuty, Grafana, Sentry and other tools are triaged, and incidents on the same resource or with the same cause become one group.

Your on-call engineer reads one incident, with every related alert inside it, instead of fifty pages.

When orders-db runs out of connections, payments-api crash-loops, Datadog fires a latency monitor on checkout, PagerDuty pages, and Sentry fills with errors. Four tools, one cause. The first minutes of the incident go to working out which page is real and which ones are echoes.

RubixKube does that sorting before anyone is woken up.

How it works

  1. Every source reports in one place. RubixKube's observers report what breaks in your clusters, cloud accounts and machines. Datadog, PagerDuty, Grafana, Sentry, Jira, Linear and your own tools send alerts by webhook.
  2. Repeats are not new incidents. An alert that fires again while its incident is open counts as another sighting. A closed incident that comes back opens a new one, linked to the old.
  3. Triage decides what needs a person. The Triage incident automation can change severity and impact, hold the RCA, start it early, or mark a known alert, such as a worker that restarts after a weekly job, so no RCA runs for it.
  4. Related incidents become one group. Incidents on the same resource, or with the same cause, within 30 minutes are grouped. Resources widen through the topology first, so a Pod's alert meets its Deployment's and its Service's. When two sources describe the same system, a Datadog alert on orders-db and the observer's crash loop on payments-api land in the same group.
  5. One investigation per group. The RCA starts once the group has settled: straight away for critical, after 2 minutes for high, 5 for medium and 10 for low. You can change the waits in Settings > Incidents.

What your team sees

  • One row in Incidents, marked Group with a stack icon, with every member alert listed inside it.
  • One RCA and one set of Tasks for the whole failure.
  • Recovery without noise. A problem that clears shows Verifying, and counts as recovered after 10 minutes healthy.
  • Repeat offenders flagged. Three RCAs in seven days marks an incident Recurring, and Rubix stops re-investigating until you choose Snooze, Ignore or Run RCA again.
  • Closing a group closes its open members.

Works with PagerDuty, Datadog and the rest

Alert sources

ToolHow it connectsWhat it does in RubixKube
PagerDutyGeneric Webhooks (v3)Triggered and reopened incidents open one, resolved closes it. P1 is critical, P2 high
DatadogWebhook on your monitorsA firing monitor opens an incident, recovery resolves it. One monitor and one scope is one incident
Grafana, Sentry, ObserveWebhookAlerts open incidents in the same shape
Jira, LinearWebhookIssues labelled incident open incidents

The connection is one way: RubixKube never acknowledges or resolves anything in PagerDuty. Each source has an Events tab with its last 20 messages, so you can see what a tool really sent.

Measure the noise you removed

Analytics shows Noise removal: how many alerts did not need their own investigation. It is one minus investigations divided by alerts, for the period and environments you pick.

Connect your first alert source from Environments > Connect. Start free, or read PagerDuty and how incidents work.

Frequently asked questions

Does grouping hide alerts?

No. Every alert stays a member of the group, listed inside it with its source. Grouping changes how many investigations run, not what you can see.

How does RubixKube decide alerts belong together?

Same resource or same cause, within 30 minutes. Resources are matched through the topology, so a Pod, its owner and its Service count as one thing, and sources that describe the same system are linked.

Does it replace PagerDuty?

No. It reads PagerDuty's incidents and investigates them. Paging stays in PagerDuty, or in Hubble if you want on-call from RubixKube.

What happens when a fixed problem comes back?

If it breaks again within 24 hours of its Fix, the Fix reopens as one that did not hold, and the member leaves the old group so it gets its own look.

See how it works.

Book a 30-minute demo. No slides, just your stack.

Download Whitepaper