Lesson 11 of 11 · 11 min · Troubleshooting Kubernetes
Pending Pods and scheduling
What you will learn
- How the scheduler decides: filtering and scoring
- The common reasons for
Pending, with the exact messages - Node selectors, affinity, taints and tolerations
- Volumes, quotas and image pulls as other causes of "not starting"
- Fixing capacity: requests, autoscaling, topology spread
In simple words
A Pod that stays in Pending is waiting for a seat: Kubernetes has not found a machine that can hold it. Perhaps no machine has enough free memory, or the Pod has conditions that nothing matches. This lesson shows how the choice is made and how to read the message that says what is missing.
Words to know
- Scheduler: the part of Kubernetes that picks a machine for each new Pod.
- Pending: the status of a Pod that has not been placed on a machine yet.
- Taint: a mark on a machine that keeps Pods away unless they are allowed there.
- Toleration: the permission on a Pod that lets it run on a tainted machine.
What the scheduler does
When you create a Pod it has no node. The kube-scheduler picks one in two steps:
- Filter: remove nodes that cannot run the Pod: not enough free CPU/memory (compared with requests, not actual use), a nodeSelector or affinity that doesn't match, an untolerated taint, a volume in another zone, a needed port already used.
- Score: among the remaining nodes, prefer those with better spread, resource balance and affinity preferences, then choose the best.
If no node passes the filters, the Pod stays Pending and the scheduler records a FailedScheduling event saying why. The scheduler retries as the cluster changes (new node, freed resources). The message always has the form 0/N nodes are available: <count> <reason>, .... Read it as "of N nodes, here is why each was rejected".
Pending can also mean the Pod was scheduled but is waiting for a volume to bind, or (displayed as ContainerCreating) waiting for an image or mount. First question: is there a node assigned? kubectl get pod -o wide shows NODE; <none> means a scheduling problem.
1. Insufficient resources
We asked for 100 CPUs and 1 GiB on a small node:
big 0/1 Pending
Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 Insufficient cpu. ...
The scheduler compares the Pod's requests with each node's allocatable capacity minus the requests of Pods already there. Check:
kubectl describe node <node> # "Allocated resources" and "Allocatable"
kubectl top nodes # actual use (metrics server)
kubectl get pods -A -o wide | grep <node>
Notice the node may be nearly idle in real CPU use while "full" in requests: over-requesting wastes capacity (the commonest reason clusters look full). Remedies:
- Reduce requests to realistic values (measure real use).
- Add capacity: more nodes, bigger nodes, or let the Cluster Autoscaler / Karpenter (cloud) add nodes automatically when Pods are unschedulable; they react to exactly these
FailedSchedulingevents. Managed node pools need a max size and quota high enough. - Remove idle workloads, set sensible replica counts, use quotas and limit ranges.
- Large Pods (many CPUs) may fit no single node even if the cluster has total free capacity: fragmentation.
Related: Insufficient memory, Too many pods (the node's pod limit, ~110 by default; on AWS EKS with the VPC CNI the limit depends on the instance type's network interfaces), Insufficient ephemeral-storage, Insufficient nvidia.com/gpu (extended resources).
2. nodeSelector, affinity
spec:
nodeSelector:
disktype: ssd
No node carries that label, so:
picky 0/1 Pending
Warning FailedScheduling 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. ...
After labelling the node (kubectl label node <node> disktype=ssd) the Pod was scheduled and became Running within seconds: the scheduler retries on cluster changes, you do not need to recreate the Pod. Check labels with kubectl get nodes --show-labels. Node affinity (requiredDuringSchedulingIgnoredDuringExecution) is the more expressive form of the same hard rule, and preferredDuringScheduling... is a soft preference that never causes Pending. Pod affinity/anti-affinity places Pods near or away from other Pods (spread replicas across nodes: podAntiAffinity or, better, topologySpreadConstraints across topology.kubernetes.io/zone). A required anti-affinity with more replicas than nodes leaves the extras Pending.
3. Taints and tolerations
A taint on a node repels Pods that do not tolerate it: used to reserve nodes for GPU jobs, system components, or to cordon problem nodes. We tainted our only node and created an ordinary Pod:
kubectl taint node <node> dedicated=gpu:NoSchedule
tainted 0/1 Pending
Warning FailedScheduling 0/1 nodes are available: 1 node(s) had untolerated taint {dedicated: gpu}. ...
To run there, the Pod needs a matching toleration:
spec:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
Remove a taint with a trailing dash: kubectl taint node <node> dedicated=gpu:NoSchedule-. Effects: NoSchedule (do not place new Pods), PreferNoSchedule (try to avoid), NoExecute (also evict running Pods). Kubernetes itself adds taints such as node.kubernetes.io/not-ready and unreachable, and node.kubernetes.io/unschedulable for a cordoned node (kubectl cordon/drain during maintenance). "All my nodes are cordoned or tainted" is a frequent cause of cluster-wide Pending.
4. Quotas, limit ranges and admission
If a ResourceQuota is exhausted, the Pod is not even created: the API returns Forbidden: exceeded quota (namespaces lesson). A Deployment then shows fewer Pods than desired, and kubectl describe replicaset has the error under events. Always look at the ReplicaSet's events when a Deployment lacks Pods: kubectl describe rs <name>.
5. Volumes
A Pod using a PersistentVolumeClaim stays Pending if the claim is Pending: no matching PersistentVolume or StorageClass, the cloud disk is in another availability zone than any schedulable node, or a WaitForFirstConsumer volume cannot be provisioned. Check kubectl get pvc, kubectl describe pvc (events name the provisioner error), the StorageClass, and zone constraints. Note that a zonal disk pins the Pod to that zone for life; a StatefulSet with zonal disks plus a full zone means Pending.
6. Not Pending, but not starting
Once scheduled, the next failures show in ContainerCreating/ErrImagePull (events from the kubelet):
- Image problems: wrong name or tag, missing
imagePullSecretsfor a private registry, registry rate limits, no network from the node to the registry. We saw:Failed to pull image "nginx:no-such-tag": ... NotFoundandErrImagePull, laterImagePullBackOff. - Mount failures: a ConfigMap, Secret or PVC that does not exist →
FailedMountand the Pod waits. - CNI/network failures:
FailedCreatePodSandBox: the network plugin could not set up the Pod (IP address exhaustion, plugin crash).
A scheduling debug checklist
kubectl get pod -o wide: node assigned? If<none>, it is scheduling.kubectl describe pod: theFailedSchedulingmessage; count the reasons against N nodes.- Resources → compare requests with
describe nodeallocatable; labels →get nodes --show-labels; taints →describe node | grep Taints; cordon →get nodesshowsSchedulingDisabled. - Volumes →
get pvc,describe pvc. - For Deployments with missing Pods →
describe rsfor quota/admission errors. - Is an autoscaler present and why did it not scale (max size reached, quota, unsatisfiable constraints such as a Pod too large for any node type)?
- Fix the cause, not the symptom: shrink requests, add capacity, fix labels or tolerations.
Priority and preemption
PriorityClasses let important Pods preempt (evict) lower-priority ones when the cluster is full. Without them all Pods are equal, and a flood of low-priority batch Pods can crowd out a critical service. Define classes for system-critical, production and batch workloads. In our small test the message even mentioned "Preemption is not helpful for scheduling": the scheduler checked whether evicting Pods would help and concluded no.
Common mistakes
- Requesting far more CPU/memory than used, leaving the cluster "full" and idle.
- Forgetting nodes are tainted/cordoned after maintenance.
- Hard affinity rules with fewer eligible nodes than replicas.
- Labels typed differently on the node and in the Pod.
- Expecting the autoscaler to fix an impossible Pod (request larger than the biggest node).
- Debugging the Pod when the ReplicaSet never created it.
- Zonal disks and a full zone.
Try it yourself
- Create a Pod requesting more CPU than your node has and read the FailedScheduling message.
- Add a nodeSelector with a label no node has, then label a node and watch the Pod start.
- Taint a node, observe an ordinary Pod go Pending, then add a toleration.
- Create a Deployment in a namespace with a tight ResourceQuota and find the error in the ReplicaSet's events.
Check yourself
A Pod is Pending with "0/3 nodes are available: 3 Insufficient memory." What does that mean and what can you do?
No node has enough unreserved memory for the Pod's memory request. Reduce the request if it's too high, free capacity, add nodes (cluster autoscaler), or use bigger nodes.
What is the difference between a taint and a node selector?
A taint on a node repels Pods that do not tolerate it (node-side rule); a node selector in the Pod only allows nodes with specific labels (Pod-side rule). Together they reserve and target nodes.
Does the scheduler consider actual CPU usage?
No, it uses requests versus allocatable capacity. A node can be idle in reality but full in requests.
A Deployment has 3 desired Pods but only 2 exist and no Pending Pod. Where do you look?
At the ReplicaSet's events (kubectl describe rs): the Pod creation was probably rejected, for example by a ResourceQuota or an admission policy.
Interview questions
How would you troubleshoot a Pod stuck in Pending?
Check whether a node is assigned; read the FailedScheduling event; examine resource requests against node allocatable, nodeSelector/affinity vs node labels, taints and cordons, PVC binding, and quotas; then fix by adjusting requests, labels, tolerations, storage or adding capacity.
How do you make sure replicas are spread across nodes and zones?
Use topologySpreadConstraints (or pod anti-affinity) on kubernetes.io/hostname and topology.kubernetes.io/zone, with whenUnsatisfiable: ScheduleAnyway for soft spreading or DoNotSchedule for strict.
What does the cluster autoscaler do and what does it need?
It adds nodes when Pods are unschedulable due to insufficient resources, and removes underused nodes. It needs node groups with min/max sizes, cloud permissions and Pods with realistic resource requests.
Summary
- The scheduler filters nodes by requests, selectors, affinity, taints and volumes, then scores; no passing node means Pending with a
FailedSchedulingmessage0/N nodes are available: .... - Typical causes: insufficient requested resources, missing labels, untolerated taints or cordons, unbound volumes, quota errors.
- Check
NODE, then events, node allocatable and labels/taints; remember the scheduler goes by requests, not usage. - Fix with right-sized requests, correct labels or tolerations, storage fixes, and autoscaling.
Going deeper
- Kubernetes scheduler: filtering and scoring.
- Assigning Pods to nodes: selectors, affinity and topology spread.
- Taints and tolerations: effects and built-in taints.
- Troubleshooting Pods that won't schedule: the official checklist.
See how much of this you can explain
A free mock interview asks you new questions on Kubernetes for DevOps Engineer and scores every answer, with what to fix.