Investigate a failing pod

Use caseUpdated November 20266 min read

OOMKilled, CrashLoopBackOff, ImagePullBackOff: ask Rubix what happened and get the cause with the evidence.

On this page

Quick answer

When a Kubernetes pod keeps failing, ask Rubix in plain words: "Why is payments-api crash-looping in prod-us-east-1?" Rubix reads the pod's status, events, logs, resource limits and the recent changes in that cluster, and answers with the cause and the evidence behind it.

It works for OOMKilled, CrashLoopBackOff, ImagePullBackOff and pods stuck in Pending, and it only reads unless you allow a change.

A pod that keeps restarting tells you something is wrong, not what. Finding out means kubectl get, describe, logs --previous, checking limits, checking the last rollout, then guessing which command to run next. Often the real cause is not in the failing pod at all, but in a service it depends on.

Rubix does that reading for you, across the whole cluster, and keeps going until it can show you why.

How it works

  1. Connect your cluster once. In Environments > Connect > Kubernetes, copy one kubectl apply command. It installs a read-only observer in the rubixkube-system namespace. It works on EKS, GKE, AKS, self-managed clusters and KIND, and usually connects within a minute.
  2. The observer reports what breaks. A failing pod becomes an incident once the problem holds, after three checks or five minutes by default (two minutes for a Pending pod), so a passing blip does not page anyone.
  3. Rubix investigates. Each incident gets a root cause analysis on its own. You can also ask in chat at any time, about any pod, with or without an incident.
  4. You get the cause and the fix. The answer names the cause, cites the log line, event, metric or change behind each step, and proposes a Fix as a Task.

Example: the cause was in another service

In the demo organization, payments-api starts crash-looping in prod-us-east-1. Rubix reads the pod's logs and events and the recent changes in the payments namespace. It rules out a bad payments-api image and a node problem. Then it confirms that orders-api v2.4 raised DB_POOL_SIZE from 10 to 50 a few minutes earlier, and orders-db reached its connection limit, so new payments-api pods could not connect and crash-looped.

The Fix it proposes: set DB_POOL_SIZE on orders-api back to 10. kubectl describe on the failing pod would never have shown that.

Common pod failures, and what Rubix shows you

Kubernetes pod failures

What you seeWhat it usually meansWhat Rubix shows you
OOMKilled (exit code 137)The container went past its memory limitThe limit, memory use before the kill, and the change that pushed it up
CrashLoopBackOffThe process exits soon after it startsThe error in the last container's logs, and the config, image or dependency behind it
ImagePullBackOff, ErrImagePullThe image cannot be pulledWhether the tag does not exist, the registry refused, or a pull secret is missing
PendingThe pod cannot be scheduledWhich resource request, node rule, taint or volume is blocking it

Questions to try

  • "Why is payments-api crash-looping in prod-us-east-1?"
  • "Show me all failing pods in prod-us-east-1."
  • "What changed in the payments namespace since the last deploy?"
  • "Explain CrashLoopBackOff in simple terms, using payments-api as the example."

Pin the incident, the environment or a time window with @ so Rubix starts from exactly what you are looking at.

Safe on production

  • The observer only reads. It watches pods, workloads, Services, Ingresses, nodes and events, and reads ConfigMaps and Secrets as metadata only. It runs as a non-root pod with a read-only root filesystem.
  • Investigations only read, in every mode.
  • You choose what Rubix may change. New chats start in Observe, which only reads. Assist asks before every change. Deleting, scaling and restarting always ask, even in Auto.
  • Personal data is masked before it reaches the model, on by default.

Try it on a test cluster

Break a pod on purpose with kubectl run brokenpod --image=nginx:doesnotexist -n default, wait for the incident, and read what Rubix found. Clean up with kubectl delete pod brokenpod -n default.

Start free: the Individual plan covers one environment and 10 investigations a month. Setup steps are in Connect a Kubernetes cluster and Your first incident.

Frequently asked questions

Does RubixKube need write access to my cluster?

No. The observer reads pods, workloads, events and resource metadata. Investigations only read. Any change Rubix proposes goes through your mode: Observe never changes anything, and Assist asks first.

Which Kubernetes clusters does it support?

EKS, GKE, AKS, self-managed clusters and KIND. The observer is one pod that requests 256Mi of memory and 250m of CPU, sized for about 50 nodes, and can run up to five replicas.

Can I ask about a pod that has no incident yet?

Yes. Ask Rubix in chat about any pod or namespace. It reads the live state of the cluster through the observer.

How is this different from running kubectl myself?

Rubix reads everything kubectl would show you, across related services and recent changes, and connects it into one cause with evidence. In the demo case the failing pod was fine. The cause was a config change in another service.

See how it works.

Book a 30-minute demo. No slides, just your stack.

Download Whitepaper