Lesson 10 of 11 · 13 min · Troubleshooting Kubernetes

CrashLoopBackOff, OOMKilled and probes

Covers: Kubernetes 1.34 (k3s), kubectl 1.34. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • What CrashLoopBackOff is and how to find the cause
  • Exit codes and OOMKilled, and how to size memory limits
  • The three probes (liveness, readiness, startup) and their different effects
  • Probe design and the mistakes that cause outages
  • The restart policy and back-off timing

In simple words

A container that starts, crashes, starts and crashes again is in a crash loop. Kubernetes waits a little longer before each new try. The cause is usually a bug, a missing setting or too little memory. Probes are the health questions Kubernetes keeps asking each container: are you alive, and are you ready?

Words to know

  • CrashLoopBackOff: the status of a container that keeps crashing while Kubernetes waits longer between restarts.
  • OOMKilled: stopped for using more memory than its limit.
  • Liveness probe: a check that asks "are you still alive?". If it fails, the container is restarted.
  • Readiness probe: a check that asks "can you take requests now?". If it fails, traffic is held back.

CrashLoopBackOff

CrashLoopBackOff is not an error itself; it is the kubelet saying "this container keeps exiting, so I am waiting longer between restart attempts". The delay doubles (10 s, 20 s, 40 s...) up to five minutes, and resets after the container runs for ten minutes. The real cause is whatever makes the process exit. We reproduced it with a container that logs a fatal error and exits with code 1:

kubectl run crash --image=busybox:1.36 --restart=Always --command -- sh -c 'echo starting; echo "fatal: cannot connect to db" >&2; exit 1'
kubectl get pod crash
crash   0/1   CrashLoopBackOff   restarts=2

Find the cause in this order:

  1. Logs of the crashed instance: kubectl logs crash --previous printed fatal: cannot connect to db. Most crashes announce themselves here: missing configuration, a dependency that is not reachable, a bad command, an exception at start-up.
  2. kubectl describe pod: the container's Last State.
State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
Restart Count:  2
  1. Events: Warning BackOff ... Back-off restarting failed container crash in pod crash_default.

Reading exit codes

Exit code Meaning Typical cause
0 success a process that finishes immediately (not a service), wrong command
1 general error the application's own failure: read the log
2 misuse of a command wrong arguments
126 found but not executable missing execute permission
127 command not found wrong command, missing binary or shell in the image
128+n killed by signal n 137 = SIGKILL (often OOM), 139 = segmentation fault (SIGSEGV), 143 = SIGTERM

(Linux and Docker courses: exit 137 and signals.) Common root causes of crash loops:

  • application configuration or environment missing or wrong (a Secret key, a database URL),
  • the dependency (database, API) is not available yet and the app does not retry,
  • permissions: the non-root user cannot write to a directory the app needs,
  • wrong command or entrypoint, or a script with Windows line endings,
  • a failing liveness probe kills a healthy-but-slow container (below),
  • out of memory (next section),
  • the process is not a long-running one (a container that runs a script and exits 0 is "crashing" if the restart policy is Always; use a Job).

To debug a container that exits too fast to inspect, override the command with sleep (kubectl debug --copy-to or a temporary change) and run the real command by hand inside.

OOMKilled

When a container uses more memory than its limit the kernel kills it. Kubernetes shows OOMKilled and exit code 137. We ran a Python process allocating 300 MB in a Pod with a 64 MiB limit:

hungry   0/1   OOMKilled   restarts=1
reason=OOMKilled exit=137

kubectl describe pod hungry showed Last State: Terminated, Reason: OOMKilled, Exit Code: 137 and the limit under Limits. Because the Pod restarts, repeated OOM kills end up as CrashLoopBackOff.

Fixing it:

  1. Confirm with data: kubectl top pod (metrics server) or your monitoring: does memory grow steadily (leak) or spike at a certain action (large request, cache warm-up)?
  2. Fix the application if it leaks or loads too much into memory (streaming instead of loading whole files, bounded caches).
  3. Raise the limit to cover legitimate peak plus headroom (and the request accordingly). Runtime settings can matter: the JVM needs -XX:MaxRAMPercentage (or container-aware defaults) so the heap fits inside the limit; Node's --max-old-space-size; Python workers multiplied by process count.
  4. Remember non-heap memory (threads, buffers, page cache attributable to the cgroup) counts against the limit.
  5. Node-level OOM (the node itself running out) evicts BestEffort/Burstable Pods first (QoS, namespaces lesson). A Pod Evicted for memory says the node, not the container's own limit, was the problem; fix with proper requests and capacity.

Probes

Kubernetes can test your containers. There are three probes, each with a different consequence:

Probe Question If it fails Typical check
Startup "Has the app finished starting?" the container is killed and restarted; other probes wait until it passes slow-starting apps (JVM, migrations)
Readiness "Can it serve traffic now?" the Pod is removed from Service endpoints (not restarted) HTTP /ready, dependency checks
Liveness "Is it stuck beyond recovery?" the container is killed and restarted HTTP /healthz that only checks the process itself

The distinction is the most important thing in this lesson: readiness controls traffic, liveness controls restarts. They are often confused, with painful results.

Probe types: httpGet (status 200-399), tcpSocket (port opens), exec (command exits 0), grpc. Tuning: initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, failureThreshold.

Readiness failure, reproduced

A Deployment whose readiness probe requests /ready, a path nginx does not serve:

readinessProbe:
  httpGet: {path: /ready, port: 80}
  periodSeconds: 3
  failureThreshold: 2
probed-5df4d86c66-4hk95   0/1   Running   restarts=0
probed-5df4d86c66-t94nx   0/1   Running   restarts=0
Warning  Unhealthy  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 404

The Pods are Running (the process is alive) but 0/1 ready, so the Service's endpoint list shows them with ready: false and receives no traffic. There were no restarts. After we created the file /ready in one Pod, that Pod became 1/1 within seconds and joined the Service; the other stayed out. This is exactly how a rolling update protects users from a bad version (previous lesson), and also how a wrong readiness path silently takes your whole service offline: "all Pods Running but the Service has no endpoints" means check readiness.

Good readiness endpoints answer "can I handle a request right now?": the server is up, caches warmed, and essential dependencies reachable. Be careful: if readiness checks a shared database and the database is down, all Pods go unready at once and you serve nothing, even for requests that do not need the database. Decide deliberately which dependencies count.

Liveness failure, reproduced

A container creates /tmp/healthy, removes it after 12 seconds, and the liveness probe runs cat /tmp/healthy:

Warning  Unhealthy  kubelet  Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
Normal   Killing    kubelet  Container c failed liveness probe, will be restarted

The kubelet killed and restarted the container; the RESTARTS counter went up. Use liveness probes sparingly and only for conditions a restart can fix (deadlock, stuck event loop). Common liveness disasters:

  • A probe that depends on a database or another service: when the dependency is down, every Pod is restarted, in a loop, making the outage worse. Liveness should check only the process itself.
  • A probe that is too strict (timeoutSeconds: 1, failureThreshold: 1) during GC pauses or load: healthy Pods get killed under load.
  • No startup allowance: a slow app is killed before it finishes starting. Use a startup probe (generous failureThreshold × periodSeconds) or initialDelaySeconds.
  • Using the same endpoint and thresholds for readiness and liveness. They should differ: readiness is quick to fail and recover, liveness slow to fail.

A well-formed trio for a slow-starting web app:

startupProbe:
  httpGet: {path: /healthz, port: 8080}
  periodSeconds: 5
  failureThreshold: 30          # up to 150 s to start
readinessProbe:
  httpGet: {path: /ready, port: 8080}
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet: {path: /healthz, port: 8080}
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

Shutdown

When a Pod stops, Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (we saw the default: 30), then SIGKILL. Handle SIGTERM and finish in-flight requests (previous lessons). A Pod that takes the full grace period to die usually has PID 1 not forwarding signals (Docker lesson).

A debugging recipe for restarting Pods

  1. kubectl get pod → RESTARTS rising?
  2. kubectl describe pod → Last State reason and exit code, probe events.
  3. kubectl logs --previous.
  4. Reason OOMKilled → memory section. Reason Error + code → application. Completed with code 0 and restartPolicy: Always → wrong workload type. Killing events with failed liveness probe → probe design.
  5. Reproduce locally with the same image, command and environment (docker run), because that is often faster.

Common mistakes

  • Treating CrashLoopBackOff as the problem instead of the symptom.
  • Never reading --previous logs.
  • Memory limits copied from a template, below the app's real need.
  • Liveness probes that check dependencies or share settings with readiness.
  • No startup probe for slow applications.
  • Readiness endpoints that always return 200, or that depend on everything.
  • Probe ports/paths that don't exist (copy-paste).

Try it yourself

  1. Create a Pod that exits 1 and diagnose it using logs --previous and describe. Then make it exit 127 by a typo in the command and compare.
  2. Reproduce OOMKilled with a low memory limit and confirm the exit code 137.
  3. Add a readiness probe on a missing path to a Deployment and observe 0/1 ready, then fix it.
  4. Write a liveness probe that fails after a while and watch the restart events.

Check yourself

What is the difference between a readiness and a liveness probe?

Readiness decides whether the Pod receives traffic (failure removes it from Service endpoints). Liveness decides whether the container is restarted (failure kills it).

A Pod shows OOMKilled with exit code 137. What do you do?

Check actual memory use (metrics), fix leaks or excessive loading, and set a memory limit with headroom above real peak (and runtime settings such as JVM heap sized to the limit).

Why is a liveness probe that checks the database dangerous?

When the database fails, every Pod fails liveness and is restarted continuously, worsening the outage. Liveness should test only the process's own health.

What does CrashLoopBackOff mean?

The container repeatedly starts and exits, and Kubernetes is delaying each restart with an exponential back-off (up to five minutes). The cause is in the exit code and the previous logs.

Interview questions

Explain the three types of probe and give a use for each.

Startup protects slow starts (restart if it never starts); readiness gates traffic (checks that the instance can serve now); liveness restarts stuck processes (deadlock detection).

A Deployment's Pods are all Running but users get errors and the Service shows no endpoints. What happened?

The Pods are not Ready: the readiness probe fails (wrong path or port, or an unavailable dependency), so they are removed from the endpoints. Check probe events with describe and test the probe path from inside the Pod.

How do you pick a memory limit for a service?

Measure real usage under realistic and peak load, set the request near typical use and the limit above observed peaks with a safety margin, align runtime settings (heap sizes) with the limit, and monitor for OOM kills and throttling.

Summary

  • CrashLoopBackOff is a symptom: read logs --previous, the exit code and Last State.
  • OOMKilled (exit 137) means the memory limit was exceeded: measure, fix or raise, and align runtime memory settings.
  • Readiness gates traffic, liveness restarts, startup protects slow starts; keep liveness simple and independent of dependencies.
  • Probe mistakes cause as many outages as the bugs they were meant to catch.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Kubernetes for DevOps Engineer and scores every answer, with what to fix.