Lesson 10 of 11 · 13 min · Troubleshooting Kubernetes
CrashLoopBackOff, OOMKilled and probes
What you will learn
- What CrashLoopBackOff is and how to find the cause
- Exit codes and OOMKilled, and how to size memory limits
- The three probes (liveness, readiness, startup) and their different effects
- Probe design and the mistakes that cause outages
- The restart policy and back-off timing
In simple words
A container that starts, crashes, starts and crashes again is in a crash loop. Kubernetes waits a little longer before each new try. The cause is usually a bug, a missing setting or too little memory. Probes are the health questions Kubernetes keeps asking each container: are you alive, and are you ready?
Words to know
- CrashLoopBackOff: the status of a container that keeps crashing while Kubernetes waits longer between restarts.
- OOMKilled: stopped for using more memory than its limit.
- Liveness probe: a check that asks "are you still alive?". If it fails, the container is restarted.
- Readiness probe: a check that asks "can you take requests now?". If it fails, traffic is held back.
CrashLoopBackOff
CrashLoopBackOff is not an error itself; it is the kubelet saying "this container keeps exiting, so I am waiting longer between restart attempts". The delay doubles (10 s, 20 s, 40 s...) up to five minutes, and resets after the container runs for ten minutes. The real cause is whatever makes the process exit. We reproduced it with a container that logs a fatal error and exits with code 1:
kubectl run crash --image=busybox:1.36 --restart=Always --command -- sh -c 'echo starting; echo "fatal: cannot connect to db" >&2; exit 1'
kubectl get pod crash
crash 0/1 CrashLoopBackOff restarts=2
Find the cause in this order:
- Logs of the crashed instance:
kubectl logs crash --previousprintedfatal: cannot connect to db. Most crashes announce themselves here: missing configuration, a dependency that is not reachable, a bad command, an exception at start-up. kubectl describe pod: the container'sLast State.
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Restart Count: 2
- Events:
Warning BackOff ... Back-off restarting failed container crash in pod crash_default.
Reading exit codes
| Exit code | Meaning | Typical cause |
|---|---|---|
| 0 | success | a process that finishes immediately (not a service), wrong command |
| 1 | general error | the application's own failure: read the log |
| 2 | misuse of a command | wrong arguments |
| 126 | found but not executable | missing execute permission |
| 127 | command not found | wrong command, missing binary or shell in the image |
| 128+n | killed by signal n | 137 = SIGKILL (often OOM), 139 = segmentation fault (SIGSEGV), 143 = SIGTERM |
(Linux and Docker courses: exit 137 and signals.) Common root causes of crash loops:
- application configuration or environment missing or wrong (a Secret key, a database URL),
- the dependency (database, API) is not available yet and the app does not retry,
- permissions: the non-root user cannot write to a directory the app needs,
- wrong command or entrypoint, or a script with Windows line endings,
- a failing liveness probe kills a healthy-but-slow container (below),
- out of memory (next section),
- the process is not a long-running one (a container that runs a script and exits 0 is "crashing" if the restart policy is
Always; use a Job).
To debug a container that exits too fast to inspect, override the command with sleep (kubectl debug --copy-to or a temporary change) and run the real command by hand inside.
OOMKilled
When a container uses more memory than its limit the kernel kills it. Kubernetes shows OOMKilled and exit code 137. We ran a Python process allocating 300 MB in a Pod with a 64 MiB limit:
hungry 0/1 OOMKilled restarts=1
reason=OOMKilled exit=137
kubectl describe pod hungry showed Last State: Terminated, Reason: OOMKilled, Exit Code: 137 and the limit under Limits. Because the Pod restarts, repeated OOM kills end up as CrashLoopBackOff.
Fixing it:
- Confirm with data:
kubectl top pod(metrics server) or your monitoring: does memory grow steadily (leak) or spike at a certain action (large request, cache warm-up)? - Fix the application if it leaks or loads too much into memory (streaming instead of loading whole files, bounded caches).
- Raise the limit to cover legitimate peak plus headroom (and the request accordingly). Runtime settings can matter: the JVM needs
-XX:MaxRAMPercentage(or container-aware defaults) so the heap fits inside the limit; Node's--max-old-space-size; Python workers multiplied by process count. - Remember non-heap memory (threads, buffers, page cache attributable to the cgroup) counts against the limit.
- Node-level OOM (the node itself running out) evicts BestEffort/Burstable Pods first (QoS, namespaces lesson). A Pod
Evictedfor memory says the node, not the container's own limit, was the problem; fix with proper requests and capacity.
Probes
Kubernetes can test your containers. There are three probes, each with a different consequence:
| Probe | Question | If it fails | Typical check |
|---|---|---|---|
| Startup | "Has the app finished starting?" | the container is killed and restarted; other probes wait until it passes | slow-starting apps (JVM, migrations) |
| Readiness | "Can it serve traffic now?" | the Pod is removed from Service endpoints (not restarted) | HTTP /ready, dependency checks |
| Liveness | "Is it stuck beyond recovery?" | the container is killed and restarted | HTTP /healthz that only checks the process itself |
The distinction is the most important thing in this lesson: readiness controls traffic, liveness controls restarts. They are often confused, with painful results.
Probe types: httpGet (status 200-399), tcpSocket (port opens), exec (command exits 0), grpc. Tuning: initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, failureThreshold.
Readiness failure, reproduced
A Deployment whose readiness probe requests /ready, a path nginx does not serve:
readinessProbe:
httpGet: {path: /ready, port: 80}
periodSeconds: 3
failureThreshold: 2
probed-5df4d86c66-4hk95 0/1 Running restarts=0
probed-5df4d86c66-t94nx 0/1 Running restarts=0
Warning Unhealthy kubelet Readiness probe failed: HTTP probe failed with statuscode: 404
The Pods are Running (the process is alive) but 0/1 ready, so the Service's endpoint list shows them with ready: false and receives no traffic. There were no restarts. After we created the file /ready in one Pod, that Pod became 1/1 within seconds and joined the Service; the other stayed out. This is exactly how a rolling update protects users from a bad version (previous lesson), and also how a wrong readiness path silently takes your whole service offline: "all Pods Running but the Service has no endpoints" means check readiness.
Good readiness endpoints answer "can I handle a request right now?": the server is up, caches warmed, and essential dependencies reachable. Be careful: if readiness checks a shared database and the database is down, all Pods go unready at once and you serve nothing, even for requests that do not need the database. Decide deliberately which dependencies count.
Liveness failure, reproduced
A container creates /tmp/healthy, removes it after 12 seconds, and the liveness probe runs cat /tmp/healthy:
Warning Unhealthy kubelet Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
Normal Killing kubelet Container c failed liveness probe, will be restarted
The kubelet killed and restarted the container; the RESTARTS counter went up. Use liveness probes sparingly and only for conditions a restart can fix (deadlock, stuck event loop). Common liveness disasters:
- A probe that depends on a database or another service: when the dependency is down, every Pod is restarted, in a loop, making the outage worse. Liveness should check only the process itself.
- A probe that is too strict (
timeoutSeconds: 1,failureThreshold: 1) during GC pauses or load: healthy Pods get killed under load. - No startup allowance: a slow app is killed before it finishes starting. Use a startup probe (generous
failureThreshold × periodSeconds) orinitialDelaySeconds. - Using the same endpoint and thresholds for readiness and liveness. They should differ: readiness is quick to fail and recover, liveness slow to fail.
A well-formed trio for a slow-starting web app:
startupProbe:
httpGet: {path: /healthz, port: 8080}
periodSeconds: 5
failureThreshold: 30 # up to 150 s to start
readinessProbe:
httpGet: {path: /ready, port: 8080}
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet: {path: /healthz, port: 8080}
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
Shutdown
When a Pod stops, Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (we saw the default: 30), then SIGKILL. Handle SIGTERM and finish in-flight requests (previous lessons). A Pod that takes the full grace period to die usually has PID 1 not forwarding signals (Docker lesson).
A debugging recipe for restarting Pods
kubectl get pod→RESTARTSrising?kubectl describe pod→Last Statereason and exit code, probe events.kubectl logs --previous.- Reason
OOMKilled→ memory section. ReasonError+ code → application.Completedwith code 0 andrestartPolicy: Always→ wrong workload type. Killing events withfailed liveness probe→ probe design. - Reproduce locally with the same image, command and environment (
docker run), because that is often faster.
Common mistakes
- Treating CrashLoopBackOff as the problem instead of the symptom.
- Never reading
--previouslogs. - Memory limits copied from a template, below the app's real need.
- Liveness probes that check dependencies or share settings with readiness.
- No startup probe for slow applications.
- Readiness endpoints that always return 200, or that depend on everything.
- Probe ports/paths that don't exist (copy-paste).
Try it yourself
- Create a Pod that exits 1 and diagnose it using
logs --previousanddescribe. Then make it exit 127 by a typo in the command and compare. - Reproduce OOMKilled with a low memory limit and confirm the exit code 137.
- Add a readiness probe on a missing path to a Deployment and observe 0/1 ready, then fix it.
- Write a liveness probe that fails after a while and watch the restart events.
Check yourself
What is the difference between a readiness and a liveness probe?
Readiness decides whether the Pod receives traffic (failure removes it from Service endpoints). Liveness decides whether the container is restarted (failure kills it).
A Pod shows OOMKilled with exit code 137. What do you do?
Check actual memory use (metrics), fix leaks or excessive loading, and set a memory limit with headroom above real peak (and runtime settings such as JVM heap sized to the limit).
Why is a liveness probe that checks the database dangerous?
When the database fails, every Pod fails liveness and is restarted continuously, worsening the outage. Liveness should test only the process's own health.
What does CrashLoopBackOff mean?
The container repeatedly starts and exits, and Kubernetes is delaying each restart with an exponential back-off (up to five minutes). The cause is in the exit code and the previous logs.
Interview questions
Explain the three types of probe and give a use for each.
Startup protects slow starts (restart if it never starts); readiness gates traffic (checks that the instance can serve now); liveness restarts stuck processes (deadlock detection).
A Deployment's Pods are all Running but users get errors and the Service shows no endpoints. What happened?
The Pods are not Ready: the readiness probe fails (wrong path or port, or an unavailable dependency), so they are removed from the endpoints. Check probe events with describe and test the probe path from inside the Pod.
How do you pick a memory limit for a service?
Measure real usage under realistic and peak load, set the request near typical use and the limit above observed peaks with a safety margin, align runtime settings (heap sizes) with the limit, and monitor for OOM kills and throttling.
Summary
- CrashLoopBackOff is a symptom: read
logs --previous, the exit code andLast State. - OOMKilled (exit 137) means the memory limit was exceeded: measure, fix or raise, and align runtime memory settings.
- Readiness gates traffic, liveness restarts, startup protects slow starts; keep liveness simple and independent of dependencies.
- Probe mistakes cause as many outages as the bugs they were meant to catch.
Going deeper
- Liveness, readiness and startup probes: concepts and tuning.
- Configure liveness, readiness and startup probes (task): examples.
- Pod lifecycle: container restart policy and back-off: the exponential back-off.
- Assign memory resources to containers: requests, limits and OOM behaviour.
See how much of this you can explain
A free mock interview asks you new questions on Kubernetes for DevOps Engineer and scores every answer, with what to fix.