Lesson 9 of 11 · 11 min · Troubleshooting Kubernetes

Debugging with kubectl describe, logs and events

Covers: Kubernetes 1.34 (k3s), kubectl 1.34. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • A repeatable debugging sequence
  • Reading kubectl get, describe, events and logs output
  • Logs of crashed containers, multi-container Pods and Deployments
  • Getting inside: exec, ephemeral debug containers, port-forward
  • What the common STATUS values mean and where to look next

In simple words

When something doesn't work in Kubernetes, don't guess. Ask the cluster in a fixed order: what is the status, what happened to it recently, and what did the program itself print? Three commands answer those questions. This lesson turns them into a routine you can follow every time.

Words to know

  • kubectl get: lists objects and their current status.
  • kubectl describe: shows the details of one object and its recent history.
  • Events: short notes from Kubernetes about what it did or could not do.
  • Logs: what the program inside the container printed.

The method

Debugging Kubernetes means walking from the outside in, and from the object that failed down to the one that explains it. Resist guessing; ask the cluster, which records an event for almost everything it does.

1. What is the symptom?        get pods / get deploy   (STATUS, READY, RESTARTS)
2. Why is it in that state?    describe pod            (Events, State, Last State, Conditions)
3. What did the app say?       logs (--previous)
4. What does it see inside?    exec / debug / port-forward
5. What does the platform say? events, node status, controller logs

1. get: the first screen

kubectl get pods -o wide
kubectl get deploy,rs,pods,svc,ingress -l app=web

The key columns:

Column Meaning
READY containers ready / containers in the Pod (1/2 = one not ready)
STATUS phase or the reason for trouble: Pending, ContainerCreating, Running, Completed, Error, CrashLoopBackOff, ImagePullBackOff, OOMKilled, Terminating
RESTARTS container restarts; a growing number means a crash loop
AGE how long it has existed
NODE, IP (with -o wide) where it runs; useful for node-specific problems

Use -A for all namespaces, -w to watch changes, --show-labels to check selectors, -o yaml for the full object including status.

2. describe: the story in the Events

kubectl describe pod <name> collects the object's state and the events concerning it. The sections that matter:

  • Status / Conditions: PodScheduled, Initialized, ContainersReady, Ready and which is False.
  • Containers → State / Last State / Exit Code / Reason / Restart Count: what each container is doing now and why the previous run ended (OOMKilled, Error with an exit code).
  • Limits / Requests / Probes / Environment / Mounts: the spec as understood by the cluster.
  • Events (at the bottom): the timeline, from the scheduler, kubelet and controllers.

The events of a healthy start:

Normal  Scheduled  default-scheduler  Successfully assigned default/web to node1
Normal  Pulling    kubelet            Pulling image "nginx:1.27-alpine"
Normal  Pulled     kubelet            Successfully pulled image ...
Normal  Created    kubelet            Created container: nginx
Normal  Started    kubelet            Started container nginx

When something goes wrong the failing step shows up as a Warning: FailedScheduling (scheduler could not place it), Failed / ErrImagePull (image problem), FailedMount (missing Secret/volume), Unhealthy (probe failed), BackOff (restart back-off), Killing. Read the Warning events first.

Events are kept only for about an hour and are not logs: capture them quickly. List everything in a namespace sorted by time:

kubectl get events -n shop --sort-by=.lastTimestamp
kubectl get events -A --field-selector type=Warning

Describe other objects the same way: a Service (selector, endpoints), Deployment (conditions such as ProgressDeadlineExceeded), Node (capacity, pressure conditions, taints), Ingress (backend errors), PVC (binding events).

3. logs

kubectl logs web-57b5dddfb4-kzt9d                 # current container's stdout/stderr
kubectl logs web-57b5dddfb4-kzt9d -c sidecar       # a specific container in a multi-container Pod
kubectl logs web-57b5dddfb4-kzt9d --previous       # the PREVIOUS container instance (the one that crashed)
kubectl logs -f web-57b5dddfb4-kzt9d --tail=100    # follow, last 100 lines
kubectl logs -l app=web --all-containers --since=10m --prefix   # all Pods with a label (with Pod prefix)
kubectl logs deploy/web                            # picks one pod of the deployment

The --previous flag is essential for crash loops: the running instance has just started, and its log is nearly empty; the reason it died is in the last instance's log. Our crash-looping Pod:

kubectl logs crash            -> starting / fatal: cannot connect to db
kubectl logs crash --previous -> fatal: cannot connect to db

Logs are the container's stdout/stderr only; if the app writes to files inside the container, kubectl logs shows nothing (log to stdout). They are rotated and deleted with the Pod, so production needs a log pipeline (Loki, Elasticsearch/OpenSearch, CloudWatch) that ships logs off the node; the observability course returns to this.

4. Getting inside

kubectl exec -it web-57b5dddfb4-kzt9d -- sh        # a shell (only if the image has one)
kubectl exec web-57b5dddfb4-kzt9d -- env | sort    # what the app actually sees
kubectl exec web-57b5dddfb4-kzt9d -- wget -qO- http://orders:8080/health   # test connectivity from the Pod
kubectl port-forward svc/web 8080:80               # reach a Service/Pod from your laptop
kubectl cp web-...:/var/log/app.log ./app.log      # copy a file out

port-forward is the easy way to hit an internal service from your laptop for a test, without exposing it.

Minimal and distroless images have no shell, no curl, no ps. Use an ephemeral debug container that shares the target Pod's namespaces:

kubectl debug -it web-57b5dddfb4-kzt9d --image=busybox:1.36 --target=nginx
kubectl debug node/node1 -it --image=ubuntu           # a pod on the node with the host filesystem at /host
kubectl debug web-57b5dddfb4-kzt9d --copy-to=web-debug --image=nicolaka/netshoot   # a copy of the Pod with extra tooling

(Lesson 1 mentioned the same idea for Docker with --network container:.) A temporary tool Pod is also useful: kubectl run tmp --rm -it --image=busybox:1.36 -- sh, to test DNS and connectivity from a Pod's point of view.

5. Beyond the Pod

If the Pod's story is not enough:

  • Node: kubectl get nodes, kubectl describe node <n> (conditions MemoryPressure, DiskPressure, PIDPressure, Ready; allocated resources; taints; recent events), kubectl top node/pod (needs the metrics server). Node problems may need the Linux tools from the first course on the node itself (journalctl -u kubelet, dmesg).
  • Controllers and control plane: kubectl -n kube-system get pods, logs of kube-controller-manager/kube-scheduler/CoreDNS on self-managed clusters; on managed services, the provider's control-plane logs.
  • Networking: Service endpoints, DNS tests, NetworkPolicies, the ingress controller's logs (previous lessons).
  • Config and RBAC: kubectl auth can-i create pods --as=system:serviceaccount:ns:sa, kubectl get ... -o yaml to compare with what you think you applied, kubectl diff -f.

STATUS cheat sheet

STATUS Meaning Where to look
Pending not scheduled or waiting for volume describe events: FailedScheduling (lesson 11)
ContainerCreating image pulling, volume or secret mounting events: pull progress, FailedMount
ImagePullBackOff / ErrImagePull cannot get the image: wrong name/tag, private registry without credentials, rate limit events show the registry's error (tested next)
CreateContainerConfigError a referenced ConfigMap/Secret/key is missing events name the missing object
CrashLoopBackOff container starts, exits, restarts with growing delay logs --previous, exit code (lesson 10)
OOMKilled (Last State) killed for exceeding the memory limit raise limit or fix leak (lesson 10)
Running but not Ready readiness probe failing probe events (lesson 10)
Completed finished successfully (Jobs) normal
Error exited with non-zero status logs, exit code
Terminating (stuck) waiting on finalizers or a node that vanished describe, finalizers, node state
Evicted node ran out of resources node conditions, resource requests

A real image error, from our lab:

badimg 0/1 ErrImagePull
Warning  Failed  kubelet  Failed to pull image "nginx:no-such-tag": rpc error: code = NotFound ...
Warning  Failed  kubelet  Error: ErrImagePull

(ErrImagePull then becomes ImagePullBackOff as the kubelet retries with a delay.)

Habits that make debugging fast

  • Always know the namespace and context (kubectl config current-context): debugging production while thinking you're in staging is a classic mistake.
  • Look at events before logs for Pods that never started; logs before events for Pods that start then fail.
  • Change one thing at a time and re-check; use kubectl diff before applying.
  • Keep manifests in Git so you can compare "what should be" with "what is".
  • Label everything consistently (app, version, env) so selectors and -l queries work.
  • Install aliases/tools: k, kubectx/kubens, stern (tail logs of many Pods), k9s (terminal UI).

Common mistakes

  • Reading only kubectl get output and guessing.
  • Forgetting --previous when the container restarted.
  • Wrong namespace or context.
  • Looking at logs of a Pod that was replaced (use the new name, or label selectors).
  • Deleting the broken Pod before capturing its description, events and logs, destroying the evidence.
  • Adding debugging tools to the production image instead of using ephemeral debug containers.

Try it yourself

  1. Create a Pod with a bad image tag and read the status, the events and what the registry error says.
  2. Create a Pod that exits with an error; read logs and logs --previous and its exit code from describe.
  3. Reference a ConfigMap that does not exist in a Pod's env and find the CreateContainerConfigError.
  4. Attach an ephemeral debug container to a running Pod (or use kubectl debug to copy one) and test connectivity from inside.

Check yourself

A Pod is in CrashLoopBackOff. Which two commands do you run first?

kubectl describe pod (exit code, reason, events) and kubectl logs <pod> --previous (what the crashed instance logged).

Where does Kubernetes record why a Pod could not be scheduled or started?

In events, shown at the bottom of kubectl describe and via kubectl get events.

How do you debug a container whose image has no shell?

Use kubectl debug with an ephemeral container sharing the Pod's namespaces (or a debug copy of the Pod) using an image that has the tools.

Why are events not enough for long-term troubleshooting?

They expire after about an hour and are not logs. Ship logs and metrics to a central system.

Interview questions

Walk me through debugging an application that is unreachable after a deployment.

Check the Deployment rollout and Pod status; for failing Pods read events and logs; verify readiness; check the Service selector and endpoints; test from another Pod by Service name; then Ingress rules and controller logs, NetworkPolicies, and DNS. Compare with the previous working revision.

What does kubectl logs --previous give you?

The logs of the previous container instance in the Pod, which is where the crash reason lives once a container has restarted.

How would you find out whether a node problem is causing Pod failures?

kubectl describe node for conditions and events, kubectl get pods -o wide to see if failures cluster on one node, kubelet and system logs on the node, and resource pressure metrics.

Summary

  • Debug outside-in: get for symptom, describe for state and events, logs (and --previous) for the app's voice, exec/debug for the inside view.
  • Warning events name the failing step: scheduling, image pull, mount, probe, back-off.
  • Know the STATUS values and where each points; capture evidence before deleting Pods.
  • Use ephemeral debug containers instead of fattening images; mind context and namespace.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Kubernetes for DevOps Engineer and scores every answer, with what to fix.