Lesson 9 of 11 · 11 min · Troubleshooting Kubernetes
Debugging with kubectl describe, logs and events
What you will learn
- A repeatable debugging sequence
- Reading
kubectl get,describe,eventsandlogsoutput - Logs of crashed containers, multi-container Pods and Deployments
- Getting inside:
exec, ephemeral debug containers, port-forward - What the common STATUS values mean and where to look next
In simple words
When something doesn't work in Kubernetes, don't guess. Ask the cluster in a fixed order: what is the status, what happened to it recently, and what did the program itself print? Three commands answer those questions. This lesson turns them into a routine you can follow every time.
Words to know
- kubectl get: lists objects and their current status.
- kubectl describe: shows the details of one object and its recent history.
- Events: short notes from Kubernetes about what it did or could not do.
- Logs: what the program inside the container printed.
The method
Debugging Kubernetes means walking from the outside in, and from the object that failed down to the one that explains it. Resist guessing; ask the cluster, which records an event for almost everything it does.
1. What is the symptom? get pods / get deploy (STATUS, READY, RESTARTS)
2. Why is it in that state? describe pod (Events, State, Last State, Conditions)
3. What did the app say? logs (--previous)
4. What does it see inside? exec / debug / port-forward
5. What does the platform say? events, node status, controller logs
1. get: the first screen
kubectl get pods -o wide
kubectl get deploy,rs,pods,svc,ingress -l app=web
The key columns:
| Column | Meaning |
|---|---|
READY |
containers ready / containers in the Pod (1/2 = one not ready) |
STATUS |
phase or the reason for trouble: Pending, ContainerCreating, Running, Completed, Error, CrashLoopBackOff, ImagePullBackOff, OOMKilled, Terminating |
RESTARTS |
container restarts; a growing number means a crash loop |
AGE |
how long it has existed |
NODE, IP (with -o wide) |
where it runs; useful for node-specific problems |
Use -A for all namespaces, -w to watch changes, --show-labels to check selectors, -o yaml for the full object including status.
2. describe: the story in the Events
kubectl describe pod <name> collects the object's state and the events concerning it. The sections that matter:
- Status / Conditions:
PodScheduled,Initialized,ContainersReady,Readyand which isFalse. - Containers → State / Last State / Exit Code / Reason / Restart Count: what each container is doing now and why the previous run ended (
OOMKilled,Errorwith an exit code). - Limits / Requests / Probes / Environment / Mounts: the spec as understood by the cluster.
- Events (at the bottom): the timeline, from the scheduler, kubelet and controllers.
The events of a healthy start:
Normal Scheduled default-scheduler Successfully assigned default/web to node1
Normal Pulling kubelet Pulling image "nginx:1.27-alpine"
Normal Pulled kubelet Successfully pulled image ...
Normal Created kubelet Created container: nginx
Normal Started kubelet Started container nginx
When something goes wrong the failing step shows up as a Warning: FailedScheduling (scheduler could not place it), Failed / ErrImagePull (image problem), FailedMount (missing Secret/volume), Unhealthy (probe failed), BackOff (restart back-off), Killing. Read the Warning events first.
Events are kept only for about an hour and are not logs: capture them quickly. List everything in a namespace sorted by time:
kubectl get events -n shop --sort-by=.lastTimestamp
kubectl get events -A --field-selector type=Warning
Describe other objects the same way: a Service (selector, endpoints), Deployment (conditions such as ProgressDeadlineExceeded), Node (capacity, pressure conditions, taints), Ingress (backend errors), PVC (binding events).
3. logs
kubectl logs web-57b5dddfb4-kzt9d # current container's stdout/stderr
kubectl logs web-57b5dddfb4-kzt9d -c sidecar # a specific container in a multi-container Pod
kubectl logs web-57b5dddfb4-kzt9d --previous # the PREVIOUS container instance (the one that crashed)
kubectl logs -f web-57b5dddfb4-kzt9d --tail=100 # follow, last 100 lines
kubectl logs -l app=web --all-containers --since=10m --prefix # all Pods with a label (with Pod prefix)
kubectl logs deploy/web # picks one pod of the deployment
The --previous flag is essential for crash loops: the running instance has just started, and its log is nearly empty; the reason it died is in the last instance's log. Our crash-looping Pod:
kubectl logs crash -> starting / fatal: cannot connect to db
kubectl logs crash --previous -> fatal: cannot connect to db
Logs are the container's stdout/stderr only; if the app writes to files inside the container, kubectl logs shows nothing (log to stdout). They are rotated and deleted with the Pod, so production needs a log pipeline (Loki, Elasticsearch/OpenSearch, CloudWatch) that ships logs off the node; the observability course returns to this.
4. Getting inside
kubectl exec -it web-57b5dddfb4-kzt9d -- sh # a shell (only if the image has one)
kubectl exec web-57b5dddfb4-kzt9d -- env | sort # what the app actually sees
kubectl exec web-57b5dddfb4-kzt9d -- wget -qO- http://orders:8080/health # test connectivity from the Pod
kubectl port-forward svc/web 8080:80 # reach a Service/Pod from your laptop
kubectl cp web-...:/var/log/app.log ./app.log # copy a file out
port-forward is the easy way to hit an internal service from your laptop for a test, without exposing it.
Minimal and distroless images have no shell, no curl, no ps. Use an ephemeral debug container that shares the target Pod's namespaces:
kubectl debug -it web-57b5dddfb4-kzt9d --image=busybox:1.36 --target=nginx
kubectl debug node/node1 -it --image=ubuntu # a pod on the node with the host filesystem at /host
kubectl debug web-57b5dddfb4-kzt9d --copy-to=web-debug --image=nicolaka/netshoot # a copy of the Pod with extra tooling
(Lesson 1 mentioned the same idea for Docker with --network container:.) A temporary tool Pod is also useful: kubectl run tmp --rm -it --image=busybox:1.36 -- sh, to test DNS and connectivity from a Pod's point of view.
5. Beyond the Pod
If the Pod's story is not enough:
- Node:
kubectl get nodes,kubectl describe node <n>(conditionsMemoryPressure,DiskPressure,PIDPressure,Ready; allocated resources; taints; recent events),kubectl top node/pod(needs the metrics server). Node problems may need the Linux tools from the first course on the node itself (journalctl -u kubelet,dmesg). - Controllers and control plane:
kubectl -n kube-system get pods, logs ofkube-controller-manager/kube-scheduler/CoreDNS on self-managed clusters; on managed services, the provider's control-plane logs. - Networking: Service endpoints, DNS tests, NetworkPolicies, the ingress controller's logs (previous lessons).
- Config and RBAC:
kubectl auth can-i create pods --as=system:serviceaccount:ns:sa,kubectl get ... -o yamlto compare with what you think you applied,kubectl diff -f.
STATUS cheat sheet
| STATUS | Meaning | Where to look |
|---|---|---|
Pending |
not scheduled or waiting for volume | describe events: FailedScheduling (lesson 11) |
ContainerCreating |
image pulling, volume or secret mounting | events: pull progress, FailedMount |
ImagePullBackOff / ErrImagePull |
cannot get the image: wrong name/tag, private registry without credentials, rate limit | events show the registry's error (tested next) |
CreateContainerConfigError |
a referenced ConfigMap/Secret/key is missing | events name the missing object |
CrashLoopBackOff |
container starts, exits, restarts with growing delay | logs --previous, exit code (lesson 10) |
OOMKilled (Last State) |
killed for exceeding the memory limit | raise limit or fix leak (lesson 10) |
Running but not Ready |
readiness probe failing | probe events (lesson 10) |
Completed |
finished successfully (Jobs) | normal |
Error |
exited with non-zero status | logs, exit code |
Terminating (stuck) |
waiting on finalizers or a node that vanished | describe, finalizers, node state |
Evicted |
node ran out of resources | node conditions, resource requests |
A real image error, from our lab:
badimg 0/1 ErrImagePull
Warning Failed kubelet Failed to pull image "nginx:no-such-tag": rpc error: code = NotFound ...
Warning Failed kubelet Error: ErrImagePull
(ErrImagePull then becomes ImagePullBackOff as the kubelet retries with a delay.)
Habits that make debugging fast
- Always know the namespace and context (
kubectl config current-context): debugging production while thinking you're in staging is a classic mistake. - Look at events before logs for Pods that never started; logs before events for Pods that start then fail.
- Change one thing at a time and re-check; use
kubectl diffbefore applying. - Keep manifests in Git so you can compare "what should be" with "what is".
- Label everything consistently (
app,version,env) so selectors and-lqueries work. - Install aliases/tools:
k,kubectx/kubens,stern(tail logs of many Pods),k9s(terminal UI).
Common mistakes
- Reading only
kubectl getoutput and guessing. - Forgetting
--previouswhen the container restarted. - Wrong namespace or context.
- Looking at logs of a Pod that was replaced (use the new name, or label selectors).
- Deleting the broken Pod before capturing its description, events and logs, destroying the evidence.
- Adding debugging tools to the production image instead of using ephemeral debug containers.
Try it yourself
- Create a Pod with a bad image tag and read the status, the events and what the registry error says.
- Create a Pod that exits with an error; read
logsandlogs --previousand its exit code fromdescribe. - Reference a ConfigMap that does not exist in a Pod's env and find the
CreateContainerConfigError. - Attach an ephemeral debug container to a running Pod (or use
kubectl debugto copy one) and test connectivity from inside.
Check yourself
A Pod is in CrashLoopBackOff. Which two commands do you run first?
kubectl describe pod (exit code, reason, events) and kubectl logs <pod> --previous (what the crashed instance logged).
Where does Kubernetes record why a Pod could not be scheduled or started?
In events, shown at the bottom of kubectl describe and via kubectl get events.
How do you debug a container whose image has no shell?
Use kubectl debug with an ephemeral container sharing the Pod's namespaces (or a debug copy of the Pod) using an image that has the tools.
Why are events not enough for long-term troubleshooting?
They expire after about an hour and are not logs. Ship logs and metrics to a central system.
Interview questions
Walk me through debugging an application that is unreachable after a deployment.
Check the Deployment rollout and Pod status; for failing Pods read events and logs; verify readiness; check the Service selector and endpoints; test from another Pod by Service name; then Ingress rules and controller logs, NetworkPolicies, and DNS. Compare with the previous working revision.
What does kubectl logs --previous give you?
The logs of the previous container instance in the Pod, which is where the crash reason lives once a container has restarted.
How would you find out whether a node problem is causing Pod failures?
kubectl describe node for conditions and events, kubectl get pods -o wide to see if failures cluster on one node, kubelet and system logs on the node, and resource pressure metrics.
Summary
- Debug outside-in:
getfor symptom,describefor state and events,logs(and--previous) for the app's voice,exec/debugfor the inside view. - Warning events name the failing step: scheduling, image pull, mount, probe, back-off.
- Know the STATUS values and where each points; capture evidence before deleting Pods.
- Use ephemeral debug containers instead of fattening images; mind context and namespace.
Going deeper
- Debug Pods: the official walkthrough.
- Debug running Pods (kubectl debug and ephemeral containers): copies and ephemeral containers.
- Troubleshooting applications: the index of debugging guides.
- kubectl cheat sheet: common commands with examples.
See how much of this you can explain
A free mock interview asks you new questions on Kubernetes for DevOps Engineer and scores every answer, with what to fix.