Kubernetes Cheatsheet

kubectl commands, resource configuration, symptom-driven troubleshooting, production operations and Helm — and how the scheduler, Services, operators and etcd work underneath. Each entry gives the problem, how it works, YAML and kubectl examples, real-world notes and trade-offs.

Updated: 2026-10-08

29 entries

Quick tables

Reference and comparison tables for quick scanning.

kubectl recipes: task to command

The commands you reach for most often. Add `-n <namespace>` (or `-A` for reads) as needed.

kubectl recipes: task to command
TaskCommandNote
Pods with node and IP`kubectl get pods -o wide`Shows the node and Pod IP; add `--show-labels` to see labels
Logs of the crashed container`kubectl logs <pod> --previous`Add `-c <container>` for multi-container Pods
Follow logs of every Pod of an app`kubectl logs -l app=web -f --tail=50 --prefix`With a selector, `-f` fails beyond 5 Pods by default (`--max-log-requests`) and `--tail` defaults to 10
Shell into a container`kubectl exec -it <pod> -c <container> -- sh`Distroless images have no shell; use `kubectl debug`
Why is it not running?`kubectl describe pod <pod>`Read the Events section at the bottom first
Recent cluster events`kubectl get events --sort-by=.lastTimestamp`Events expire after about an hour by default
Scale a Deployment`kubectl scale deploy/web --replicas=5`Conflicts with a manifest or HPA that sets replicas
Roll out a new image`kubectl set image deploy/web web=<image>:<tag>`Format is `container=image`; follow with `rollout status`
Wait for a rollout`kubectl rollout status deploy/web --timeout=120s`Non-zero exit if the rollout does not finish; good for CI
Roll back`kubectl rollout undo deploy/web`Add `--to-revision=N` after `rollout history`
Restart without changing the spec`kubectl rollout restart deploy/web`Picks up changed env-var ConfigMaps and Secrets
Preview a change`kubectl diff -f app.yaml`Exit code 1 means differences, greater than 1 means an error
Reach a Service locally`kubectl port-forward svc/web 8080:80`Forwards to one backing Pod, not load balancing
Can this account do it?`kubectl auth can-i create deployments -n <ns>`Impersonate with `--as <user>` to test someone else

Prefer `deploy/web` style names over Pod names: Pod names change on every rollout.

Label selectors vs field selectors

Both accept comma-separated terms that are ANDed together. There is no OR in either; run two queries instead.

Label selectors vs field selectors
FormExampleMeaning / limit
Label equality`-l app=web` or `-l app==web`Label `app` equals `web`
Label inequality`-l tier!=db`Also matches objects without the label
Label set-based in`-l 'env in (prod,staging)'`Quote it: the shell treats parentheses specially
Label set-based notin`-l 'env notin (dev)'`Also matches objects without the label
Label exists / absent`-l canary` or `-l '!canary'`Key presence only; the value is ignored
Field equality`--field-selector status.phase=Running`Operators are `=`, `==` and `!=` only
Field on any resource`--field-selector metadata.name=web,metadata.namespace=prod`Only `metadata.name` and `metadata.namespace` are universal
Pod fields`--field-selector spec.nodeName=<node>`Pods also support `status.phase`, `status.podIP` and `spec.restartPolicy`, among others
Event fields`--field-selector type=Warning,involvedObject.name=web`Events support `reason`, `type` and `involvedObject.*`
Not supported in fields`--field-selector 'status.phase in (Running)'`No set-based syntax; arbitrary fields such as `spec.replicas` are rejected for most kinds

Service `spec.selector` only supports plain equality maps; set-based forms work in `kubectl -l`, and in Deployment and Job `matchExpressions`.

QoS classes compared

QoS classes compared
ClassConditionEviction priorityTypical use
GuaranteedEvery container (init included) has CPU and memory limits, and requests equal to limits (or unset, so they default to limits)Last; lowest `oom_score_adj` (-997)Databases, brokers, latency-critical services
BurstableAt least one request or limit set, but not GuaranteedMiddle; Pods using more than their requests go first, then by priorityMost stateless services (memory request = limit, CPU request only)
BestEffortNo requests and no limits on any containerFirst; highest `oom_score_adj` (1000)Throwaway jobs, dev experiments

Node-pressure eviction ranks by usage above requests, then Pod priority, so the class is only a guide, not a rule the kubelet applies directly. Check it with `kubectl get pod <pod> -o jsonpath='{.status.qosClass}'`.

Probe types compared

Probe types compared
ProbeOn failureWhen it runsTypical configPitfall
livenesskubelet kills the container; `restartPolicy` decides the restartThroughout the container's life, after startup probe succeedsCheap local check, `periodSeconds: 10`, `failureThreshold: 3`+Checking databases or downstream services turns a dependency outage into a restart storm
readinessPod is marked not Ready and removed from Service endpoints; no restartThroughout the container's life, after startup probe succeedsCan include light dependency checks, `periodSeconds: 5`-10Too strict or flapping probes drain every replica at once and cause a full outage
startupContainer is killed and restarted; liveness and readiness stay disabled until it succeedsOnly at boot, until the first success`failureThreshold` x `periodSeconds` covers the worst-case boot (for example 30 x 10s)Skipping it and inflating `initialDelaySeconds` on liveness instead slows failure detection forever

FailedScheduling event messages

Scheduler messages look like `0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint {...}.` Each clause is a count of nodes rejected by one filter; the counts add up to N. Fix the clause that covers the most nodes, or every clause if you need a node that passes them all.

FailedScheduling event messages
Event messageMeaningFix
`Insufficient cpu` / `Insufficient memory`The Pod's requests exceed what is still unreserved on every node (allocatable minus the requests of Pods already there). Live usage is irrelevant.Lower the requests if they are inflated, scale the Deployment down or evict low-priority Pods, or add nodes (autoscaler). Compare with `kubectl describe node | grep -A8 Allocated`.
`node(s) didn't match Pod's node affinity/selector``nodeSelector` or required `nodeAffinity` matches no node, usually a label typo or a label that exists only on some pools.Compare `kubectl get nodes --show-labels` with the Pod spec; fix the selector or label the nodes.
`node(s) had untolerated taint {key: value}`Nodes carry a taint (dedicated pool, control plane, `node.kubernetes.io/not-ready`) that the Pod does not tolerate. The braces show the key and value of one untolerated taint (v1.35+ prints only `untolerated taint(s)`).Add a matching `tolerations` entry, or schedule elsewhere. Inspect with `kubectl describe node | grep Taints`.
`pod has unbound immediate PersistentVolumeClaims`A PVC the Pod uses is not Bound: no matching PV, missing or wrong StorageClass, or the provisioner failed.`kubectl describe pvc <name>`; fix the `storageClassName`, the provisioner, or the capacity and access mode.
`node(s) had volume node affinity conflict`The bound PV is pinned to a zone or node (for example an EBS volume in one zone) that no candidate node satisfies. Releases from about v1.33 word this `node(s) didn't match PersistentVolume's node affinity`.Allow nodes in the volume's zone, or use `volumeBindingMode: WaitForFirstConsumer` so the volume is created where the Pod lands.
`node(s) didn't match pod topology spread constraints`A `topologySpreadConstraints` entry with `whenUnsatisfiable: DoNotSchedule` would exceed `maxSkew` on every node.Add capacity in the lagging zone, raise `maxSkew`, or switch to `ScheduleAnyway`.
`node(s) didn't match pod anti-affinity rules`Required `podAntiAffinity` forbids every node (one Pod per node or zone, and all are taken). The variant `didn't satisfy existing pods anti-affinity rules` means an existing Pod's rule blocks this one.Add nodes or zones, reduce replicas, or make the rule `preferredDuringSchedulingIgnoredDuringExecution`.
`Too many pods`Every node already runs its maximum number of Pods (`maxPods`, 110 by default, or a lower value set by your node or cloud setup).Add nodes, raise `maxPods` where the CNI allows it, or use larger nodes.
`node(s) were unschedulable`The nodes are cordoned (`spec.unschedulable: true`), often mid-drain or mid-upgrade.`kubectl uncordon <node>` once maintenance is done, or add nodes.
`exceeded quota: <name>, requested: ..., used: ..., limited: ...`Not a scheduler event. The API server rejected the Pod create because a ResourceQuota in the namespace would be exceeded. The Pod never exists; the message is on the ReplicaSet or Job as `FailedCreate`.`kubectl describe quota -n <ns>`; free quota, raise it, or reduce requests. Look at the ReplicaSet events, not the Pod.

Exact wording shifts between Kubernetes versions (for example a trailing `preemption: ...` clause, or the taint and volume messages noted above). Match on the key phrase, not the whole line. Scheduler events expire after about an hour by default, so describe the Pod early.

Container exit codes

Read the code from `kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'` (or `Last State` in `describe`). Codes above 128 mean the process died from signal `code - 128`.

Container exit codes
CodeMeaningTypical cause in containers
`0`Clean exit.A Job or init container that finished. For a long-running service it means the main process returned (daemonised itself, or the script ended), so the kubelet restarts it under `restartPolicy: Always`.
`1`Generic application error.Unhandled exception, failed startup check, bad config or missing env var, unreachable dependency. Read `logs --previous`.
`2`Misuse of a shell builtin, or an app-defined usage error.Wrong CLI flags, a syntax error in an entrypoint script, or an argument the binary rejects.
`126`Command found but cannot be executed.Missing execute bit, a directory instead of a binary, or a `noexec` mount; sometimes the wrong architecture.
`127`Command not found.Wrong `command` or `args`, binary absent from the image (distroless or multi-stage build), or a script whose interpreter is missing.
`128+N`Terminated by signal N.See 137, 139 and 143 below; `130` is SIGINT and `134` is SIGABRT (abort, common in native crashes and failed assertions).
`137`128+9: SIGKILL.The OOM killer (`reason: OOMKilled`), or the kubelet killing the container after `terminationGracePeriodSeconds` expired, or a failed liveness probe on a process that ignored SIGTERM.
`139`128+11: SIGSEGV.Segmentation fault: native-code bug, incompatible native library (for example glibc binary on musl/Alpine), or corrupted memory.
`143`128+15: SIGTERM.Normal graceful shutdown during rollout, scale-down, drain or eviction. Only a problem if it appears when nobody asked for a stop, or if the app does not exit cleanly.

Codes are only conventions of the process; a runtime can use any 0-255 value. Container runtimes also report `reason` (`Error`, `OOMKilled`, `Completed`; start failures get a runtime-specific reason such as `ContainerCannotRun`) next to the code, so read both.

Pod STATUS column decoder

The STATUS column of `kubectl get pods` is a display string, not the API `phase`: kubectl derives it from container states, reasons and deletion timestamps. Start from the value, run the next command, then go to the matching entry.

Pod STATUS column decoder
STATUSMeaningNext command
`Pending`Accepted by the API server but not running: not scheduled yet, or waiting for volumes or images.`kubectl describe pod <pod>` and read Events (see pod-pending).
`ContainerCreating`Scheduled; the kubelet is pulling the image, mounting volumes or setting up the sandbox and network.`kubectl describe pod <pod>`; look for `FailedMount`, `FailedCreatePodSandBox` or slow pulls.
`Running`At least one container is running. It does not mean ready or healthy.Check the READY column; `kubectl describe pod <pod>` for probe failures.
`Completed`All containers exited with 0 (normal for Jobs and CronJobs).`kubectl logs <pod>`; if it is a service that should keep running, fix why the process exits.
`Error`A container exited non-zero (container reason `Error`). It stays this way when it will not be restarted (`restartPolicy: Never`, or Job Pods out of retries); under `Always` or `OnFailure` it shows briefly between restarts.`kubectl logs <pod>` and read the exit code under `Last State`.
`CrashLoopBackOff`The container keeps exiting; the kubelet is waiting out an exponential delay before restarting it.`kubectl logs <pod> --previous` (see crashloopbackoff).
`ImagePullBackOff`Image pull failed and the kubelet is backing off between retries.`kubectl describe pod <pod>`, read the pull error (see imagepullbackoff).
`ErrImagePull`The most recent pull attempt failed; shows briefly before turning into `ImagePullBackOff`.Same as above: the pull error text is in Events.
`OOMKilled`The container was killed for exceeding its memory limit; shown until the next restart state replaces it.`kubectl describe pod <pod>` for `Reason: OOMKilled` (see oomkilled).
`Terminating`The Pod has a deletion timestamp; it is shutting down or stuck on a finalizer or an unreachable node.`kubectl get pod <pod> -o yaml` (check `finalizers`) and `kubectl get nodes`.
`Evicted`The kubelet evicted the Pod under node pressure (memory, disk, PID); the Pod object stays as `Failed`.`kubectl describe pod <pod>` for the message; `kubectl describe node <node>` for conditions.
`Init:N/M`N of M init containers have finished. Variants: `Init:Error`, `Init:CrashLoopBackOff`.`kubectl logs <pod> -c <init-container>`; init containers run in order and block the app.
`CreateContainerConfigError`The kubelet cannot build the container config, typically a missing Secret or ConfigMap, or a missing key referenced by `secretKeyRef` or `configMapKeyRef`.`kubectl describe pod <pod>` names the missing object; then `kubectl get secret,configmap -n <ns>`.
`Unknown`The Pod is being deleted and its node is unreachable (`status.reason: NodeLost`), so the control plane cannot confirm its state.`kubectl get nodes`; `kubectl describe node <node>` and check kubelet and network on it.

RBAC verbs

The verbs you can list in a Role rule and what they map to on the API.

RBAC verbs
VerbAPI meaningNotes
getHTTP GET on a single objectReading one named object; `kubectl get pod <name>`
listHTTP GET on a collectionReturns full objects, so `list secrets` exposes every Secret value in scope
watchHTTP GET with `watch=true`Streams changes; `kubectl get -w`, controllers and informers need it with `list`
createHTTP POSTTop-level create cannot be limited by `resourceNames` (the name may not be known at authorization time)
updateHTTP PUT (replace)Full-object replace (`kubectl replace`); the client normally sends the object with its current `resourceVersion` so conflicting writes are rejected
patchHTTP PATCHTypically used by `kubectl apply` (on existing objects), `edit` and `patch`; usually granted together with `get`
deleteHTTP DELETE on one objectRemoves a named object
deletecollectionHTTP DELETE on a collection`kubectl delete pods --all` or by label; cannot be limited by `resourceNames`; grant sparingly
escalateCreate or update a Role/ClusterRole with permissions you do not holdNormally blocked by escalation prevention; effectively a path to more privilege
bindCreate a binding to a Role/ClusterRole whose permissions you do not holdGranted on `roles`/`clusterroles`, ideally with `resourceNames`; treat as privileged
impersonateAct as another user, group or ServiceAccount (`--as`)Granted on `users`, `groups`, `serviceaccounts`; powerful, restrict with `resourceNames`

Subresources are separate: `pods/log`, `pods/exec` and `deployments/scale` each need their own rule. `exec` into a Pod is often equivalent to reading everything that Pod can read.

Helm hook events

Set the event in the `helm.sh/hook` annotation; several can be comma-separated.

Helm hook events
HookWhen it runsTypical usePitfall
pre-installAfter templates render, before any release resource is createdCreate a prerequisite Secret or run a one-time bootstrap JobAnything it needs from the chart (ConfigMap, ServiceAccount) does not exist yet unless it is a hook too
post-installAfter all resources are loaded into KubernetesSeed data, register the service somewhere, notifyFailure marks the install failed even though the workloads are already running
pre-upgradeAfter templates render, before any resource is updatedDatabase schema migration JobA failure blocks the upgrade; the migration must be backward compatible with the old Pods still serving
post-upgradeAfter all resources are upgradedCache warm-up, smoke check, cleanup of old dataFailure marks the release failed although the new version is live
pre-deleteBefore any resource is deleted on uninstallExport or back up data, deregisterA stuck hook makes `helm uninstall` hang until the timeout
post-deleteAfter all release resources are deletedRemove external state such as DNS records or cloud resourcesRuns after the chart's ServiceAccount and ConfigMaps are gone
pre-rollback / post-rollbackBefore or after resources are restored by `helm rollback`Reverse a migration, re-register the old versionHook side effects (migrations) are not undone by a rollback; you must write the reverse step yourself
testOnly when you run `helm test <release>`Smoke test Pod that calls the ServiceNever runs during install or upgrade; do not rely on it as a deploy gate unless CI calls it

Order inside one event: ascending `helm.sh/hook-weight` (default 0), then resource kind, then name per the Helm docs; set explicit weights rather than relying on tie-breaking. Helm waits for hook Jobs and Pods to complete before continuing.

Taint effects

Taint effects
EffectNew Pods without a tolerationPods already running
NoScheduleNot scheduled onto the nodeNot evicted
PreferNoScheduleScheduler tries to avoid the node (soft)Not evicted
NoExecuteNot scheduled onto the nodeEvicted immediately, or after tolerationSeconds if the Pod tolerates it for a while

Which placement mechanism to use

Which placement mechanism to use
GoalUseNotes
Run only on labelled nodesnodeSelector / required nodeAffinityHard filter; Pod stays Pending if no node matches
Prefer some nodespreferred nodeAffinityAdds score only
Keep a node for special workloadstaint + tolerationThe node repels everything else; pair with nodeAffinity to attract
Co-locate with another PodpodAffinityNeeds a topologyKey; costly on large clusters
Spread replicas aparttopologySpreadConstraintsmaxSkew over zones or nodes; preferred over podAntiAffinity for even spread
Decide who yields under pressurePriorityClass (+ preemptionPolicy)Higher priority can preempt lower; Never makes it queue instead

kubectl commands

List, inspect, exec into, tail logs from, copy files to, port-forward to and delete Pods without guessing flags.

The problem

Most day-to-day Kubernetes work is looking at Pods: what is running where, what it printed, how to get a shell in it, and how to reach it from your laptop. Knowing a dozen flags saves the trial and error during an incident.

How it works

  • `kubectl get pods -o wide` adds the node and Pod IP; `-w` watches changes; `--show-labels` helps when a selector matches nothing.
  • `kubectl logs <pod> -c <container>` picks a container; `--previous` shows the last terminated instance and is the first command for a restarting container; `-f`, `--tail=N` and `--since=10m` bound the output.
  • `kubectl logs -l app=web --all-containers --prefix` reads many Pods at once; with a selector, the last 10 lines per container are shown unless you set `--tail`, and with `-f` it fails beyond `--max-log-requests` (default 5) Pods.
  • `kubectl exec -it <pod> -c <container> -- sh` opens a shell; put `--` before the command so kubectl does not parse its flags. Images without a shell need `kubectl debug` instead.
  • `kubectl cp <pod>:/path ./local -c <container>` needs `tar` inside the image, so it fails on distroless images; for big or repeated transfers use a volume or object storage.
  • `kubectl port-forward pod/<pod> 8080:80` binds localhost by default and is a debugging tool, not an ingress path; it dies when the Pod is replaced.
  • `kubectl delete pod <pod>` is safe for controller-managed Pods (a replacement is created), but it respects `terminationGracePeriodSeconds`; `--grace-period=0 --force` skips graceful shutdown and should be a last resort.

Examples

Find and read the logs of a misbehaving Pod.

bash
kubectl get pods -n shop -l app=checkout -o wide
kubectl logs checkout-6c9f7d8b5-x2k4p -n shop --tail=100
# Container restarted? Read the previous instance
kubectl logs checkout-6c9f7d8b5-x2k4p -n shop --previous
# Pick a container in a multi-container Pod
kubectl logs checkout-6c9f7d8b5-x2k4p -n shop -c sidecar --since=15m
# All Pods of the app, prefixed with the Pod name
kubectl logs -n shop -l app=checkout --all-containers --prefix --tail=20

Get a shell, copy a file out and forward a port.

bash
kubectl exec -it checkout-6c9f7d8b5-x2k4p -n shop -c app -- sh
# One-off command without a TTY
kubectl exec checkout-6c9f7d8b5-x2k4p -n shop -- env
# Copy a file out (needs tar in the image)
kubectl cp shop/checkout-6c9f7d8b5-x2k4p:/tmp/heap.hprof ./heap.hprof -c app
# Local 8080 to container port 80; Ctrl-C to stop
kubectl port-forward -n shop pod/checkout-6c9f7d8b5-x2k4p 8080:80

Delete Pods: gracefully, by selector, or forced as a last resort.

bash
kubectl delete pod checkout-6c9f7d8b5-x2k4p -n shop
# Recreate every Pod of an app one by one: prefer rollout restart
kubectl rollout restart deploy/checkout -n shop
# Stuck Terminating (for example a lost node); skips graceful shutdown
kubectl delete pod checkout-6c9f7d8b5-x2k4p -n shop --grace-period=0 --force

In practice

  • stern / kubetail — Community tools that tail logs from many Pods and containers with colour and regex filtering, a common upgrade over `kubectl logs -l`.
  • kubectl port-forward — Handy for reaching an internal admin UI or database; it tunnels through the API server, so it is not for production traffic.
  • In practice — Logs of a deleted Pod are gone with it unless a node agent (for example Fluent Bit, Vector or Promtail) ships them elsewhere; do not rely on `kubectl logs` for history.

Trade-offs

Pros

  • Fast, no extra tooling or cluster setup
  • Works the same on any conformant cluster

Cons

  • Logs and exec sessions are ephemeral and not audited by default
  • Single-Pod focus; fleet-wide questions need metrics or log aggregation

Scale, change images, watch and undo rollouts, and know when to use apply, create or replace, with dry-run and diff first.

The problem

Shipping a change means scaling, swapping an image, watching the rollout and being able to undo it. The confusing part is choosing between apply, create and replace, and previewing what a manifest will actually change before you run it.

How it works

  • `kubectl scale deploy/web --replicas=5` changes replicas imperatively; if an HPA owns the Deployment it will overwrite you, and if a manifest in Git sets `replicas` the next apply resets it.
  • `kubectl set image deploy/web web=registry.example.com/web:1.4.3` uses `container=image`; a changed Pod template starts a rollout. Pin a tag or digest, not `latest`.
  • `kubectl rollout status deploy/web --timeout=120s` blocks until done and exits non-zero on failure or timeout, which makes it the right CI gate; `rollout history` lists revisions.
  • `kubectl rollout undo deploy/web [--to-revision=N]` restores an old Pod template; `rollout restart` recreates Pods with the same spec by bumping an annotation; `rollout pause` and `resume` batch several edits into one rollout.
  • `create` fails if the object exists, `replace` overwrites the whole object (and fails if it is missing), `apply` computes a three-way merge from the last-applied state; use `apply` for anything kept in Git and `--server-side` when several tools or controllers co-own fields.
  • `--dry-run=client` only prints locally; `--dry-run=server` runs validation and admission on the API server without persisting, so it catches webhook and schema rejections.
  • `kubectl diff -f app.yaml` shows what apply would change against live state; it exits 0 for no change, 1 for differences and greater than 1 for errors, so it is scriptable.
  • Record why a revision exists with `kubectl annotate deploy/web kubernetes.io/change-cause='bump to 1.4.3'`; the old `--record` flag is deprecated.

Examples

Ship an image, gate on the rollout, and undo if it fails.

bash
kubectl set image deploy/web web=registry.example.com/web:1.4.3 -n shop
kubectl annotate deploy/web -n shop kubernetes.io/change-cause="bump web to 1.4.3" --overwrite
if ! kubectl rollout status deploy/web -n shop --timeout=180s; then
  kubectl rollout undo deploy/web -n shop
  exit 1
fi
kubectl rollout history deploy/web -n shop

Preview before you apply: diff, then server-side dry run.

bash
# Exit 0 = no change, 1 = differences, >1 = error
kubectl diff -f deploy.yaml
# Validation and admission webhooks run, nothing is stored
kubectl apply -f deploy.yaml --dry-run=server
kubectl apply -f deploy.yaml
# Several controllers touch the same object: let the server track field owners
kubectl apply -f deploy.yaml --server-side --field-manager=ci-pipeline

Generate a manifest from a command instead of writing YAML by hand.

bash
kubectl create deployment web --image=registry.example.com/web:1.4.3 --replicas=3 --dry-run=client -o yaml > deploy.yaml
# Batch several edits into one rollout
kubectl rollout pause deploy/web
kubectl set resources deploy/web -c web --requests=cpu=250m,memory=256Mi
kubectl set env deploy/web LOG_LEVEL=debug
kubectl rollout resume deploy/web

In practice

  • Argo CD / Flux — GitOps controllers run the equivalent of apply from Git; manual `kubectl scale` or `set image` shows up as drift and is usually reverted.
  • kubectl diff in CI — Commonly run on pull requests so reviewers see the real effect of a manifest change against the live cluster.
  • Server-side apply — Tracks field ownership in `managedFields`, which is why conflicts are reported instead of silently overwritten.

Trade-offs

Pros

  • Imperative commands are quick for incidents; apply keeps Git as the source of truth
  • Dry-run and diff make changes reviewable

Cons

  • Imperative changes drift from Git and are overwritten by the next apply or GitOps sync
  • Client-side apply can silently clobber fields set by other controllers

Expose workloads, forward to a Service, and inspect Endpoints, EndpointSlices and Ingress to see where traffic really goes.

The problem

A Service looks healthy in YAML but sends traffic nowhere if its selector matches no ready Pods. You need to expose workloads quickly and then check, layer by layer, where the request would actually go.

How it works

  • `kubectl expose deploy/web --port=80 --target-port=8080` creates a ClusterIP Service whose selector is copied from the Deployment; add `--type=NodePort` or `LoadBalancer` if needed.
  • `kubectl get endpoints web` lists ready backend addresses; an empty list means the selector matches no Pods, or they are not Ready (readiness probe failing).
  • `kubectl get endpointslices -l kubernetes.io/service-name=web` is the current API behind Services; the older `Endpoints` object is capped at 1000 addresses and deprecated since v1.33, so check slices on big Services.
  • `kubectl describe svc web` shows selector, ports and endpoints together; compare `Selector` with `kubectl get pods --show-labels` for the typo-class of bugs.
  • `kubectl port-forward svc/web 8080:80` picks one backing Pod and tunnels to it; it proves the Pod works, not that Service load balancing or DNS works.
  • `kubectl create ingress web --class=nginx --rule="shop.example.com/*=web:80"` generates an Ingress; add `,tls=<secret>` to a rule for TLS. `kubectl describe ingress web` shows the backend resolution and events.
  • An Ingress needs a controller (for example Traefik, an Envoy-based controller or your cloud's own) and a matching `ingressClassName`; without one the object exists and nothing happens.

Examples

Expose a Deployment and check that the Service has backends.

bash
kubectl expose deploy/web -n shop --port=80 --target-port=8080
kubectl get svc web -n shop -o wide
# Empty ENDPOINTS column = selector mismatch or no Ready Pods
kubectl get endpoints web -n shop
kubectl get endpointslices -n shop -l kubernetes.io/service-name=web
# Compare the Service selector with the Pod labels
kubectl get svc web -n shop -o jsonpath='{.spec.selector}'
kubectl get pods -n shop --show-labels

Reach the Service from your machine, and test it from inside the cluster.

bash
kubectl port-forward -n shop svc/web 8080:80
# In another terminal
curl -i http://localhost:8080/healthz
# In-cluster test, which also exercises DNS and the Service VIP
kubectl run tmp-curl -n shop --rm -it --restart=Never --image=curlimages/curl:8.8.0 -- curl -sS -m 3 http://web.shop.svc.cluster.local/healthz

Create and inspect an Ingress.

bash
kubectl create ingress web -n shop --class=nginx --rule="shop.example.com/*=web:80"
kubectl create ingress web-tls -n shop --class=nginx --rule="shop.example.com/*=web:80,tls=shop-tls" --dry-run=client -o yaml
kubectl get ingress -n shop
# Look at Backends and Events: a "<error: endpoints not found>" backend means no Endpoints
kubectl describe ingress web -n shop
kubectl get ingressclass

In practice

  • Ingress / Gateway API — Ingress is a frozen, stable API (and the community ingress-nginx controller was retired in March 2026); newer features such as header routing and traffic splitting are being added to the Gateway API instead.
  • EndpointSlice — Splits backends into slices (up to 100 endpoints each by default) so a change to one Pod does not resend the whole list to every node.
  • In practice — A Service that works through `port-forward` but not by name is usually a DNS, NetworkPolicy or `targetPort` mismatch problem, not an application bug.

Trade-offs

Pros

  • `expose` and `create ingress` skip hand-written YAML boilerplate
  • Endpoints and slices give an exact view of what the data path will use

Cons

  • Generated objects have minimal options and need editing for real use
  • `port-forward` hides Service-level problems by bypassing the Service

Create and update ConfigMaps and Secrets from literals, files and env files, mount them, and know when changes reach the Pod.

The problem

Configuration and credentials live in ConfigMaps and Secrets, and teams keep getting surprised by two things: Secrets are only base64-encoded, and a changed ConfigMap does not always reach the running Pod. Knowing the creation flags and the update rules prevents both.

How it works

  • `kubectl create configmap app --from-literal=LOG_LEVEL=info --from-file=app.properties --from-env-file=app.env`: literals give single keys, `--from-file` uses the file name as the key (or `key=path`), and `--from-env-file` turns each `KEY=VALUE` line into a key.
  • `kubectl create secret generic db --from-literal=password=...` takes the same sources; values are stored base64-encoded, which is an encoding, not encryption. Anyone who can `get secret` can decode it with `base64 -d`.
  • Real protection needs RBAC that limits `get`/`list` on Secrets, encryption at rest in the API server (an `EncryptionConfiguration`, ideally with a KMS provider) or an external secret store.
  • `kubectl create secret docker-registry regcred --docker-server=... --docker-username=... --docker-password=...` makes a `kubernetes.io/dockerconfigjson` Secret; reference it in `imagePullSecrets` on the Pod or ServiceAccount.
  • Update idempotently with `kubectl create ... --dry-run=client -o yaml | kubectl apply -f -`; plain `create` fails when the object exists.
  • As env vars (`env.valueFrom.configMapKeyRef`, `secretKeyRef`, `envFrom`) values are read once at container start, so a change needs `kubectl rollout restart` (or a new Pod).
  • As volumes, the kubelet refreshes projected files eventually (on its sync period plus cache delay, commonly up to about a minute or more), and the application must re-read the file.
  • Mounts with `subPath` never receive updates, and neither do `immutable: true` objects (they cannot be edited at all); a common pattern is to put a hash in the name so a change creates a new object and a new rollout.

Examples

Create from literals, files and env files; update idempotently.

bash
kubectl create configmap app-config -n shop --from-literal=LOG_LEVEL=info --from-file=nginx.conf --from-env-file=app.env
kubectl create secret generic db-creds -n shop --from-literal=username=app --from-file=password=./db-password.txt
# Re-run safely: render, then apply
kubectl create configmap app-config -n shop --from-file=nginx.conf --from-env-file=app.env --dry-run=client -o yaml | kubectl apply -f -
# Base64 is only an encoding
kubectl get secret db-creds -n shop -o jsonpath='{.data.username}' | base64 -d

Consume the same ConfigMap as env vars, a mounted volume and a subPath file (the last never updates).

yaml
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
    - name: web
      image: registry.example.com/web:1.4.3
      envFrom:
        - configMapRef:
            name: app-config
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password
      volumeMounts:
        - name: config
          mountPath: /etc/app
        - name: config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
  volumes:
    - name: config
      configMap:
        name: app-config

Registry credentials for private images, and reloading after a change.

bash
kubectl create secret docker-registry regcred -n shop --docker-server=registry.example.com --docker-username=ci-bot --docker-password="$REGISTRY_TOKEN"
kubectl patch serviceaccount default -n shop -p '{"imagePullSecrets":[{"name":"regcred"}]}'
# Env-var consumers only see the change after a restart
kubectl rollout restart deploy/web -n shop
# Volume consumers: check the file the Pod actually sees
kubectl exec -n shop deploy/web -- cat /etc/app/LOG_LEVEL

In practice

  • External Secrets Operator / Secrets Store CSI Driver — Sync or mount secrets from a cloud secret manager or Vault so plaintext values are not kept in Git.
  • Sealed Secrets / SOPS — Encrypt Secret manifests so they can be committed to Git and decrypted only in the cluster or by authorised users.
  • Reloader (stakater) — A controller that triggers a rolling restart when a referenced ConfigMap or Secret changes, covering the env-var case.
  • In practice — Never pass secret values on the command line in shared shells: they land in history and process lists; prefer `--from-file` or stdin-fed manifests.

Trade-offs

Pros

  • Config is decoupled from images, so one image runs in every environment
  • Volume-mounted config can be reloaded without a restart if the app supports it

Cons

  • Env-var and subPath consumption need a restart to see changes
  • Secrets are not encrypted by default and need extra layers

Namespaces, label and field selectors, jsonpath, custom-columns, sorting and context switching to get exactly the data you want.

The problem

Clusters hold thousands of objects across many namespaces. `kubectl get` with the right selector, sort order and output format answers a question in one line, and switching context carelessly is how people run a command against production.

How it works

  • `-n <ns>` scopes to a namespace and `-A` (`--all-namespaces`) covers all; `kubectl config set-context --current --namespace=<ns>` changes the default, which is convenient and easy to forget.
  • `-l app=web,tier!=db` uses equality; set-based forms `-l 'env in (prod,staging)'`, `-l 'env notin (dev)'`, `-l canary` and `-l '!canary'` need quotes; comma means AND and there is no OR.
  • `--field-selector status.phase=Running,spec.nodeName=<node>` filters on server-side fields; only `=`, `==` and `!=` work and each resource supports a limited set of fields.
  • `-o wide` adds columns, `-o yaml|json` the full object, `-o name` prints `kind/name` for scripting, and `--show-labels` appends labels.
  • `-o jsonpath='{.items[*].metadata.name}'` extracts fields; use `{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}` for rows. For anything complex pipe `-o json` into `jq`.
  • `-o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount` builds your own table.
  • `--sort-by=.metadata.creationTimestamp` (any JSONPath that yields a sortable value) orders the list; `kubectl get events --sort-by=.lastTimestamp` is the standard way to put events in order.
  • `kubectl config get-contexts`, `use-context <name>` and `current-context` manage clusters; put the context and namespace in your shell prompt, or use `--context` per command for risky actions.

Examples

Label and field selectors, sorting and labels.

bash
kubectl get pods -A -l 'env in (prod,staging),tier!=db' --show-labels
kubectl get pods -A --field-selector status.phase!=Running
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-1-23.ec2.internal
kubectl get pods -n shop --sort-by=.metadata.creationTimestamp
kubectl get pods -n shop --sort-by='.status.containerStatuses[0].restartCount'

Shape the output: names, jsonpath, custom columns.

bash
kubectl get deploy -n shop -o name
kubectl get pods -n shop -o jsonpath='{.items[*].metadata.name}'
kubectl get pods -n shop -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'
kubectl get pods -n shop -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[0].image
# Distinct images running in the cluster
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}' | tr -s ' ' '\n' | sort -u

Contexts and namespaces: check before you act.

bash
kubectl config get-contexts
kubectl config current-context
kubectl config use-context staging-eu
kubectl config set-context --current --namespace=shop
# One-off against another cluster without switching
kubectl --context prod-eu -n shop get deploy

In practice

  • kubectx / kubens — Small tools that make context and namespace switching one command, with fuzzy selection through fzf.
  • jq — For nested or conditional extraction, `-o json | jq` is usually clearer than jsonpath, which has no real conditionals beyond filters.
  • In practice — Aliases such as `k` for `kubectl` plus shell completion remove most typing; the habit worth building is reading the current context before every write.

Trade-offs

Pros

  • Server-side selectors reduce data transferred and make scripts fast
  • Output formats make kubectl scriptable without extra tools

Cons

  • jsonpath syntax is awkward and its errors are terse
  • Field selectors are limited per resource, so some filters must be done client-side

describe, events, top, exec, ephemeral containers, node debugging, auth can-i, api-resources and explain for working out what is wrong.

The problem

A workload misbehaves and you have a minute to find out why. The right order is cheap and read-only first (describe, events, top), then a look inside (exec, ephemeral containers), then the node, and checks for permissions and API shape along the way.

How it works

  • `kubectl describe pod <pod>`: read State, Last State (exit code, reason such as `OOMKilled`), Conditions and the Events at the bottom; most Pending and image-pull problems are explained there.
  • `kubectl get events --sort-by=.lastTimestamp -n <ns>` gives a timeline; add `--field-selector type=Warning`. Events are retained only briefly (about an hour by default).
  • `kubectl top pod --containers` and `kubectl top node` show current CPU and memory; they need metrics-server and report short-window usage, not history.
  • `kubectl exec -it <pod> -- sh` works only if the image has a shell. Otherwise use `kubectl debug -it <pod> --image=busybox:1.36 --target=<container>` to attach an ephemeral container that shares the Pod's network and, with `--target`, its process namespace.
  • `kubectl debug <pod> -it --copy-to=<pod>-debug --container=<container> -- sh` clones the Pod to experiment (different command or image) without touching the original.
  • `kubectl debug node/<node> -it --image=busybox:1.36` starts a Pod on that node with the host filesystem mounted at `/host`; use it for kubelet logs and disk checks, and delete the debug Pod afterward.
  • `kubectl auth can-i <verb> <resource> -n <ns>` answers 403 questions; `--as system:serviceaccount:<ns>:<sa>` tests a ServiceAccount and `--list` shows everything allowed.
  • `kubectl api-resources` lists kinds, short names, scope and verbs; `kubectl explain deploy.spec.strategy` (add `--recursive`) documents fields for the version your cluster serves, so you can write correct YAML without leaving the terminal.

Examples

The first three read-only commands for a sick Pod.

bash
kubectl describe pod checkout-6c9f7d8b5-x2k4p -n shop
kubectl get events -n shop --sort-by=.lastTimestamp --field-selector type=Warning
# Needs metrics-server
kubectl top pod -n shop --containers
kubectl top node

Ephemeral container, Pod copy and node debugging.

bash
# Attach a toolbox to a running Pod (shares network; --target shares processes)
kubectl debug -it checkout-6c9f7d8b5-x2k4p -n shop --image=busybox:1.36 --target=app
# Experiment on a copy with a different command
kubectl debug checkout-6c9f7d8b5-x2k4p -n shop -it --copy-to=checkout-debug --container=app -- sh
# Node: host filesystem is at /host
kubectl debug node/ip-10-0-1-23 -it --image=busybox:1.36
kubectl delete pod checkout-debug -n shop

Permissions and API discovery.

bash
kubectl auth can-i create deployments -n shop
kubectl auth can-i get secrets -n shop --as system:serviceaccount:shop:web
kubectl auth can-i --list -n shop
kubectl api-resources --namespaced=true -o wide
kubectl explain deployment.spec.strategy.rollingUpdate
kubectl explain pod.spec.containers.resources --recursive

In practice

  • Ephemeral containers — A stable feature since Kubernetes 1.25; they cannot be removed or restarted once added, and the Pod must be recreated to drop them.
  • metrics-server — Provides the Metrics API that `kubectl top` and the HPA read; it is not a monitoring system and keeps no history, so use Prometheus for trends.
  • netshoot — A widely used community debug image with curl, dig, tcpdump and similar tools, commonly passed to `--image` when `busybox` is too small.

Trade-offs

Pros

  • Debug tooling is brought in on demand, so production images stay minimal
  • Read-only commands are safe to run first on any cluster

Cons

  • `kubectl debug` needs RBAC for `pods/ephemeralcontainers` and, for nodes, rights to create Pods that use host namespaces and the host filesystem
  • Debug sessions bypass your image's hardening and should be audited

Resources & probes

Requests drive scheduling and limits drive runtime enforcement: CPU over the limit is throttled, memory over the limit is OOM-killed.

The problem

Every container tells Kubernetes two different things about its resources, and mixing them up causes most capacity incidents. Requests are a scheduling reservation; limits are a runtime ceiling enforced by the kernel. Set them wrong and you get Pods that will not schedule, throttled latency, or surprise OOM kills.

How it works

  • `requests` drive scheduling: the scheduler sums the requests of Pods already on a node and only places a new Pod where its requests fit in the node's allocatable. Live usage is not consulted. CPU requests also become the container's relative CPU weight when the node is contended.
  • `limits` drive enforcement at runtime. CPU is compressible: over the limit, the container is throttled by the CFS quota (a budget per 100 ms period by default) and just runs slower. Memory is not compressible: over the limit, the kernel OOM-kills the process (`OOMKilled`, exit code 137).
  • Memory sizing: set `requests.memory` equal or very close to `limits.memory`. The scheduler then reserves what the container can really use, so the node is not overcommitted on memory and you avoid node-pressure evictions of neighbours.
  • CPU limits deserve suspicion: a limit can throttle a container that is below node capacity, which shows up as p99 latency spikes with low average CPU. A common stance is to set a CPU request from measurement and omit the CPU limit, or set it generously, unless you need hard isolation or a Guaranteed class.
  • Measure before you set numbers: `kubectl top pod --containers` (needs metrics-server) gives current usage; Prometheus and cAdvisor series such as `container_cpu_cfs_throttled_periods_total` and `container_memory_working_set_bytes` give history. Size requests near typical usage and memory limits above the observed peak plus headroom.
  • A `LimitRange` in a namespace supplies `defaultRequest` and `default` (limit) values for containers that omit them, and can enforce `min`/`max`. It saves you from BestEffort Pods, but defaults are one-size-fits-all, so treat them as a safety net, not sizing.
  • Vertical Pod Autoscaler (an add-on, not built in) in `updateMode: "Off"` only publishes recommendations in its status; you read them and edit your manifests. That is the low-risk way to right-size, and avoids VPA evicting Pods on its own. Do not run it in an auto-update mode against the same metric an HPA scales on.
  • Units: CPU is in cores, `500m` is half a core; memory uses `Mi`/`Gi` (powers of 1024), and `M`/`G` are powers of 1000, which is a classic typo source. Requests above the largest node's allocatable can never schedule.

Examples

Container resources: CPU request only (no CPU limit), memory request equal to limit.

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              memory: 512Mi

Namespace defaults via LimitRange, plus a VPA that only recommends.

yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: <namespace>
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 256Mi
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api
  namespace: <namespace>
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"

Measure usage, read VPA recommendations, compare against node capacity.

bash
kubectl top pod -n "<namespace>" --containers
kubectl describe vpa api -n "<namespace>"   # Recommendation: Lower Bound / Target / Upper Bound
kubectl describe node "<node>" | sed -n '/Allocatable:/,/Allocated resources:/p'
kubectl describe node "<node>" | grep -A8 'Allocated resources'

In practice

  • metrics-server — Provides the live numbers behind `kubectl top` and HPA; it is not a history store, so pair it with Prometheus for sizing decisions.
  • cAdvisor / Prometheus — The throttling counters (`container_cpu_cfs_throttled_periods_total` against `container_cpu_cfs_periods_total`) are the standard way to prove CPU limits are hurting latency.
  • Vertical Pod Autoscaler — Maintained in the kubernetes/autoscaler project; its recommender is useful even when you never let it update Pods.
  • In practice — Copy-pasted requests from a template are a frequent cause of both stranded capacity (requests far above usage) and noisy-neighbour trouble (requests far below usage).

Trade-offs

Pros

  • Requests make bin-packing and autoscaling predictable, because the scheduler reasons about reservations rather than noisy usage.
  • Memory limits contain a leaking container to its own cgroup instead of taking the node down.
  • Right-sized requests cut cost: the cluster autoscaler adds nodes only for what Pods claim.

Cons

  • CPU limits can throttle healthy containers and are hard to spot without cAdvisor metrics.
  • Memory limit equal to request leaves no burst room; a spike kills the container instead of borrowing.
  • Requests go stale as traffic and code change; they need periodic review.

Guaranteed, Burstable and BestEffort are derived from requests and limits and shape who gets evicted or OOM-killed first.

The problem

When a node runs short of memory, something has to die, and Kubernetes does not choose at random. It derives a QoS class from each Pod's requests and limits, and that class feeds the kernel OOM killer, while the requests behind it drive the kubelet's eviction ranking. If you never set requests, you have silently chosen BestEffort for your workload.

How it works

  • `Guaranteed`: every container, init containers included, has CPU and memory limits, and requests equal to the limits (requests default to limits when omitted). Such Pods are the last candidates for eviction.
  • `Burstable`: at least one container has a request or limit, but the Pod does not qualify as Guaranteed. Memory request = limit with only a CPU request is the most common shape.
  • `BestEffort`: no container sets any request or limit. First to go under pressure, and invisible to the scheduler's capacity math.
  • The class is computed by the API server and stored in `status.qosClass`; it cannot be set directly, and it is immutable for the Pod's life. Changing resources means a new Pod.
  • Kernel OOM: the kubelet sets `oom_score_adj` per container, `-997` for Guaranteed, `1000` for BestEffort, and for Burstable a value from about 2 to 999 that shrinks as the memory request grows relative to node memory. A higher score is killed first.
  • Node-pressure eviction (memory, disk, PID thresholds) is the kubelet's own decision, normally made before the kernel steps in (if memory rises faster than the kubelet notices, the kernel OOM killer acts first): it ranks Pods by whether usage exceeds requests, then by Pod priority, then by how far above requests they run. The QoS class itself is not used, so a Burstable Pod using less than its request is ranked with Guaranteed Pods, by priority, but in practice BestEffort and over-request Burstable Pods go first.
  • Evicted Pods end up `Failed` with reason `Evicted` and a message naming the resource; the controller recreates them elsewhere. Check `kubectl get pods --field-selector=status.phase=Failed` and node conditions such as `MemoryPressure`.

Examples

A Guaranteed Pod: requests equal limits for CPU and memory on every container.

yaml
apiVersion: v1
kind: Pod
metadata:
  name: db-guaranteed
spec:
  containers:
    - name: db
      image: registry.example.com/db:2.1
      resources:
        requests:
          cpu: "2"
          memory: 4Gi
        limits:
          cpu: "2"
          memory: 4Gi

Check classes, find BestEffort Pods, and look for evictions and node pressure.

bash
kubectl get pod "<pod>" -o jsonpath='{.status.qosClass}{"\n"}'
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass' | grep BestEffort
kubectl get pods -A --field-selector=status.phase=Failed | grep Evicted
kubectl describe node "<node>" | grep -E 'MemoryPressure|DiskPressure|PIDPressure'

In practice

  • kubelet — Owns node-pressure eviction thresholds (`evictionHard`, `evictionSoft`) and sets `oom_score_adj`; system daemons reserved via `kube-reserved` and `system-reserved` protect the node itself.
  • System-critical Pods — Pods in `kube-system` such as CoreDNS or CNI agents typically carry a high `priorityClassName` (`system-cluster-critical`, `system-node-critical`), which matters for eviction ranking alongside usage over requests.
  • In practice — An `Evicted` Pod flood after a deploy is a frequent sign of missing memory requests: Pods look fine until the node fills and a pile of BestEffort or low-request Pods goes together.

Trade-offs

Pros

  • Gives a predictable order of sacrifice, so critical services can be shielded without dedicated nodes.
  • Derived from fields you already set, with no extra object to maintain.

Cons

  • Guaranteed requires CPU limits, which brings the throttling risk; you pay isolation with latency.
  • The class is only a guide: the kubelet ranks eviction by usage over requests and priority, so it is not a strict guarantee.
  • Immutable: promoting a Pod to Guaranteed means a rollout.

Liveness restarts a stuck container, readiness gates Service traffic, startup protects slow boots; most outages come from mis-tuned probes.

The problem

Kubernetes cannot see inside your process. Probes are how you tell it when to restart a container, when to send it traffic, and how long to wait for a slow boot. Done well they give self-healing and zero-downtime rollouts; done badly they cause restart loops and outages that look like the application's fault.

How it works

  • `livenessProbe` answers 'is this process stuck beyond recovery?'. On failure the kubelet kills the container and restarts it according to `restartPolicy`. It does not touch Service endpoints directly.
  • `readinessProbe` answers 'can this Pod take traffic right now?'. On failure the Pod becomes NotReady and is removed from the Service's EndpointSlices; the container keeps running. It gates rollouts too, since a Deployment waits for new Pods to be Ready.
  • `startupProbe` answers 'has the app finished booting?'. Liveness and readiness are held off until it succeeds, so slow starts (JVM warm-up, migrations, large caches) do not get killed. Budget is `failureThreshold` x `periodSeconds`; if it never succeeds in time the container is restarted.
  • Mechanisms: `httpGet` (2xx/3xx is success), `tcpSocket` (the port accepts a connection), `exec` (command exits 0; costs a process per run), and `grpc` (gRPC health checking protocol, needs a `port`; plaintext only by default, TLS is alpha behind a feature gate). Prefer `httpGet` or `grpc` over `exec`.
  • Timing fields and defaults: `initialDelaySeconds` 0, `periodSeconds` 10, `timeoutSeconds` 1, `failureThreshold` 3, `successThreshold` 1. `successThreshold` must be 1 for liveness and startup. A 1 s timeout is tight for an endpoint doing real work under load.
  • Classic mistake 1: a liveness probe that checks dependencies (database, cache, other services). When the dependency blips, every replica fails liveness together and restarts, turning a partial outage into a total one. Liveness should check only the process's own health; keep dependency checks, if any, in readiness.
  • Classic mistake 2: aggressive timing. `periodSeconds: 1` with `failureThreshold: 1` restarts on a single GC pause or CPU throttle. Give liveness a few failures of slack, and a `timeoutSeconds` that survives load.
  • Classic mistake 3: no `startupProbe` for slow starts, patched with a huge `initialDelaySeconds` on liveness. That delays detection of real hangs on every restart. A startup probe lets liveness stay tight afterwards.
  • Do not point readiness and liveness at the same expensive endpoint, and do not forget graceful shutdown: on termination the Pod is removed from endpoints asynchronously, so handle SIGTERM and keep serving briefly (a short `preStop` sleep is a common trick).

Examples

A Deployment container with resources and all three probes; startup budget is 30 x 5 s = 150 s.

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          ports:
            - containerPort: 8080
          resources:
            requests: { cpu: 250m, memory: 512Mi }
            limits: { memory: 512Mi }
          startupProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 5
            failureThreshold: 30
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            timeoutSeconds: 2
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5

See probe config, probe failure events, restarts, and whether the Pod is in the endpoints.

bash
kubectl describe pod "<pod>" | grep -E 'Liveness|Readiness|Startup'
kubectl get events --field-selector involvedObject.name="<pod>",reason=Unhealthy
kubectl get pod "<pod>" -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'
kubectl get endpointslices -l kubernetes.io/service-name="<service>"
kubectl logs "<pod>" --previous   # why did the last container die?

In practice

  • Spring Boot Actuator — Exposes separate liveness and readiness health groups (`/actuator/health/liveness`, `/actuator/health/readiness`) precisely so dependency health stays out of liveness.
  • gRPC health checking — The standard `grpc.health.v1.Health` service is what a Kubernetes `grpc` probe calls; many servers ship it.
  • kubelet — Runs all probes from the node, so they test the Pod IP and port as seen from the node; a service mesh sidecar intercepting traffic can break them, and on some setups so can network policy.
  • In practice — A restart storm that begins right when a shared dependency slows down is a common signature of a liveness probe that checks that dependency.

Trade-offs

Pros

  • Liveness gives automatic recovery from deadlocks and wedged processes without a human.
  • Readiness makes rollouts and scale-up safe: traffic arrives only when the Pod can serve it.
  • Startup probes let you keep liveness tight without punishing slow boots.

Cons

  • A wrong liveness probe is worse than none: it can restart a healthy fleet.
  • Every probe is load on the app and needs tuning per service as it evolves.
  • Probes only see what the endpoint reports; a healthy `/healthz` can hide a broken code path.

A PDB caps how many replicas voluntary disruptions such as drains and upgrades may take down at once, and can deadlock them if set too tight.

The problem

Nodes get drained for upgrades, the autoscaler removes underused nodes, and operators cordon machines, all while your service should stay up. A PodDisruptionBudget tells Kubernetes how many replicas may be down during those voluntary evictions. Too loose and a drain takes out your service; too tight and the drain never finishes.

How it works

  • A PDB selects Pods by label and sets exactly one of `minAvailable` or `maxUnavailable` (integer or percentage). Prefer `maxUnavailable` for Deployments: it scales with replica count and is harder to set to a deadlocking value.
  • Voluntary disruptions go through the Eviction API: `kubectl drain`, cluster autoscaler scale-down, Karpenter consolidation and drift, managed node-pool upgrades. These generally respect the PDB (Karpenter can be configured to proceed after a termination grace period). When an eviction would violate it, the API returns 429 and the drain retries.
  • Involuntary disruptions (node crash, kernel panic, hardware failure, OOM kill) cannot be blocked by a PDB, but they do count against the budget: a Pod already down reduces `disruptionsAllowed`.
  • PDBs do not govern Deployment rollouts (those use `maxSurge` and `maxUnavailable` on the Deployment) or direct `kubectl delete pod`, which bypasses the Eviction API.
  • The deadlock: `minAvailable` equal to replicas, `maxUnavailable: 0` or `minAvailable: 100%` leaves `disruptionsAllowed` at 0, so a drain waits forever on that Pod. A single-replica Deployment with `minAvailable: 1` does the same. Fix by allowing at least one disruption, or run more replicas.
  • Allowed disruptions are computed from healthy Pods. If Pods are NotReady, the budget is already spent, so a drain also blocks while the service is degraded. Percentages round up for `maxUnavailable` and `minAvailable`, so small replica counts behave surprisingly.
  • `unhealthyPodEvictionPolicy` (stable since 1.31): the default `IfHealthyBudget` only evicts a running-but-not-ready Pod if the budget is healthy, so a crash-looping app can block a drain. `AlwaysAllow` lets unhealthy Pods be evicted regardless, which is usually what you want to unstick drains.
  • Operations view: managed node upgrades and autoscalers wait on PDBs for a limited time and may then proceed or fail, depending on the platform, so a tight PDB shows up as a stuck upgrade. Pair PDBs with topology spread so replicas are not all on the node being drained.

Examples

A PDB that lets one replica be evicted at a time and unsticks drains blocked by unhealthy Pods.

yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
  namespace: <namespace>
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: api

Inspect budgets before a drain, then drain with a timeout.

bash
kubectl get pdb -A   # ALLOWED DISRUPTIONS of 0 will block a drain
kubectl get pdb api -n "<namespace>" -o jsonpath='{.status.disruptionsAllowed}{"\n"}'
kubectl cordon "<node>"
kubectl drain "<node>" --ignore-daemonsets --delete-emptydir-data --timeout=5m
# stuck? look for: Cannot evict pod as it would violate the pod's disruption budget.
kubectl uncordon "<node>"

In practice

  • kubectl drain — Cordons the node and evicts Pods through the Eviction API, retrying while a PDB blocks; `--timeout` is what turns a silent hang into an error.
  • Cluster Autoscaler / Karpenter — Both honour PDBs by default when removing or replacing nodes, so a PDB with zero allowed disruptions keeps a node alive and costing money.
  • Managed Kubernetes upgrades — Node-pool upgrades use the same eviction path; check your provider's documented PDB wait limit before relying on a tight budget.
  • In practice — A stuck upgrade is often one forgotten PDB with `minAvailable` equal to a Deployment's replicas, or a single-replica service nobody noticed.

Trade-offs

Pros

  • Turns 'keep N replicas up during maintenance' into declarative policy that every drain and autoscaler obeys.
  • Cheap insurance against taking down all replicas of a service in one upgrade wave.

Cons

  • A too-tight budget stalls node upgrades and scale-down, and the failure is a hang rather than an error.
  • Does nothing for involuntary failures or direct Pod deletion.
  • Needs at least two replicas to be meaningful, which costs capacity.

Troubleshooting

A fixed triage order (get, describe, logs, events, node and network) that maps the STATUS column to the right troubleshooting entry.

The problem

A Pod is not doing what you expect and the cluster offers a dozen places to look. Guessing wastes time; a fixed order does not. Each step narrows the problem to a layer (scheduling, image, container start, process, node, network), and the STATUS column already tells you which entry to open next.

How it works

  • Step 1, `kubectl get pod <pod> -o wide`: read STATUS, READY, RESTARTS, AGE, NODE. No NODE means scheduling. A high RESTARTS with a young AGE of the last restart means the container is dying. READY `0/1` with `Running` means probes.
  • Step 2, `kubectl describe pod <pod>`: go straight to the Events at the bottom, then `State`, `Last State` (reason, exit code) and `Conditions` (PodScheduled, Initialized, ContainersReady, Ready). Most causes are written there in plain words.
  • Step 3, `kubectl logs <pod> -c <container>` and `--previous`: the current container may have just restarted and have empty logs; the crash is in the previous one. Init containers have their own logs (`-c <init-container>`).
  • Step 4, events across the namespace: `kubectl get events -n <ns> --sort-by=.lastTimestamp`. The cause is often on a different object: the ReplicaSet (`FailedCreate`, quota), the PVC, the Node, or an HPA. Events expire after about an hour by default, so look early.
  • Step 5, the node and the network: `kubectl describe node <node>` (Conditions such as MemoryPressure, DiskPressure, PIDPressure, Ready; Allocated resources), `kubectl top node`, then the kubelet and container runtime logs on the node. For connectivity, test from inside the cluster with an ephemeral debug container.
  • STATUS to entry: `Pending` goes to `pod-pending`; `CrashLoopBackOff`, `Error`, `Init:CrashLoopBackOff` go to `crashloopbackoff`; `OOMKilled` or exit code 137 goes to `oomkilled`; `ErrImagePull` and `ImagePullBackOff` go to `imagepullbackoff`; `Running` but `0/1` or no traffic goes to `service-unreachable`.
  • Other values: `ContainerCreating` that never ends means volume mount or sandbox/CNI trouble (read Events for `FailedMount`, `FailedCreatePodSandBox`); `CreateContainerConfigError` means a missing Secret, ConfigMap or key; `Terminating` forever usually means a finalizer or an unreachable node; `Evicted` means node pressure. The `pod-status` table lists the next command for each.
  • Work outside-in: confirm what Kubernetes thinks (status, events, conditions) before opening a shell in the container, and change one thing at a time. If you must change a live workload, prefer `kubectl debug` on a copy over editing the running Deployment.

Examples

The first three steps, in order, for one Pod.

bash
kubectl get pod "<pod>" -n "<ns>" -o wide
kubectl describe pod "<pod>" -n "<ns>" | sed -n '/^Conditions:/,$p'
kubectl logs "<pod>" -n "<ns>" --all-containers --tail=100
# Restarted container? The crash is in the previous run:
kubectl logs "<pod>" -n "<ns>" -c "<container>" --previous

Steps 4 and 5: namespace events newest last, then the node.

bash
kubectl get events -n "<ns>" --sort-by=.lastTimestamp
kubectl get events -n "<ns>" --field-selector type=Warning
kubectl describe node "<node>" | grep -E -A8 'Conditions:|Allocated resources:'
kubectl top node
# On the node itself (SSH or a node debug session):
# journalctl -u kubelet --since "30 min ago"

In practice

  • kubectl — `describe` aggregates the object, its conditions and the related Events in one call, which is why it is step 2. `kubectl get events` shows the same Events cluster-wide or per namespace.
  • kube-state-metrics / Prometheus — Expose Pod phase, container waiting reasons and restart counts as metrics (for example `kube_pod_container_status_waiting_reason`), so alerts can fire on CrashLoopBackOff or ImagePullBackOff instead of someone noticing.
  • In practice — Events are short-lived by default; if you want history of what failed last night, ship them to your logging system with an event exporter.

Trade-offs

Pros

  • A fixed order is fast to teach, hard to skip steps in, and works the same at 3am as at noon.
  • Status and Events point to the right entry with no cluster-specific knowledge.

Cons

  • Symptoms can come from a different object than the one you are looking at (quota on the ReplicaSet, volume on the PVC).
  • Events expire quickly and logs vanish with the Pod, so a late start loses the evidence.

Read the FailedScheduling event to find which scheduler filter rejected every node: resources, selectors, taints, volumes, spread or quota.

The problem

The Pod exists but has no node, or has one but cannot start. Pending is a phase, not a cause: it covers unschedulable Pods, Pods waiting for volumes and Pods waiting for an image. The scheduler writes exactly why it rejected each node as an event, so the fix starts with reading one line.

How it works

  • Symptom: STATUS `Pending` with an empty NODE column, and a `FailedScheduling` event such as `0/3 nodes are available: 3 Insufficient cpu.` With a NODE assigned, Pending means the kubelet is still pulling or mounting (`ContainerCreating`), which is a different problem.
  • First look: `kubectl describe pod <pod>`, last lines of Events. The message is `0/N nodes are available:` followed by one count per rejection reason; the counts add up to N. No `FailedScheduling` event at all points to the scheduler not running or not watching this Pod (wrong `schedulerName`, scheduler down) or to the Pod being held back by a scheduling gate (`spec.schedulingGates`).
  • Common causes (resources and placement): `Insufficient cpu` / `Insufficient memory` (requests exceed unreserved capacity; live usage is irrelevant), `node(s) didn't match Pod's node affinity/selector` (label typo), `node(s) had untolerated taint` (dedicated or control-plane nodes), `node(s) didn't match pod topology spread constraints`, `Too many pods` (per-node Pod limit).
  • Common causes (storage and policy): `pod has unbound immediate PersistentVolumeClaims` (no matching PV or StorageClass, or provisioner failing), `node(s) had volume node affinity conflict` (the volume lives in a zone with no free node; newer releases word it `node(s) didn't match PersistentVolume's node affinity`), and `exceeded quota` (ResourceQuota; the Pod is not even created, so look at the ReplicaSet's `FailedCreate` event, not the Pod).
  • Fix: change the constraint the event names. Lower inflated requests or add capacity (the autoscaler only adds nodes for Pods that the scheduler marks unschedulable, and only if a node group could fit them); fix selectors and tolerations; make storage provisionable (`WaitForFirstConsumer` for zonal volumes); raise or free quota. Do not delete the Pod hoping it reschedules: the controller recreates the same Pod.
  • Verify: `kubectl get pod <pod> -o wide` shows a NODE and then `Running`. Quick checks while diagnosing: `kubectl get nodes -o wide` (Ready? cordoned?), `kubectl top nodes` (live usage, needs metrics-server; scheduling ignores it), `kubectl get pvc` (Bound?), `kubectl describe node | grep -A8 Allocated` (requested vs allocatable, which is what the scheduler counts).
  • Prevent: set requests from measured usage, keep a pool with headroom or an autoscaler, give zonal storage `WaitForFirstConsumer`, use `preferred` rules for soft preferences, alert on Pods Pending for more than a few minutes (`kube_pod_status_phase{phase="Pending"} == 1`), and set quotas with enough room for a rollout surge.

Examples

Find the rejection reason: Pod events, then all scheduling failures in the namespace.

bash
kubectl describe pod "<pod>" -n "<ns>" | sed -n '/^Events:/,$p'
kubectl get events -n "<ns>" --field-selector reason=FailedScheduling
# Pod not even created? Look one level up:
kubectl describe replicaset -n "<ns>" -l app="<app>" | grep -A3 FailedCreate

Capacity, volumes and taints: the four quick checks.

bash
kubectl get nodes -o wide
kubectl top nodes
kubectl get pvc -n "<ns>"
kubectl describe node "<node>" | grep -A8 Allocated
kubectl describe node "<node>" | grep -i taints
# Namespace quota usage:
kubectl describe quota -n "<ns>"

Typical fixes in one Pod: toleration for a tainted pool, a real node label, honest requests.

yaml
apiVersion: v1
kind: Pod
metadata:
  name: batch-worker
spec:
  nodeSelector:
    pool: batch
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: worker
      image: registry.example.com/worker:2.3.0
      resources:
        requests:
          cpu: 250m
          memory: 256Mi

In practice

  • kube-scheduler — Emits the `FailedScheduling` event with per-filter node counts; the same text appears in `kubectl describe pod`.
  • Cluster Autoscaler / Karpenter — React to unschedulable Pending Pods by adding nodes; they cannot help if the Pod's selector, taint or zone constraint matches no node group, or if one Pod is bigger than the largest node.
  • metrics-server — Provides `kubectl top`. Useful to see real load, but the scheduler decides on requests, so `top` can show an idle node that is still full.
  • In practice — A Pod that stays Pending right after a node pool change is very often a missing toleration or a label that exists only on the old pool.

Trade-offs

Pros

  • The scheduler states the exact failing filter and node count, so most cases are solved by reading one event.
  • Fixing the named constraint (instead of adding nodes blindly) keeps cost and bin-packing sane.

Cons

  • Events expire after about an hour, so an old Pending Pod may show no message; recreate or check the scheduler logs.
  • Message wording varies between versions, and some causes (quota, admission) appear on a different object than the Pod.

The container keeps exiting and the kubelet restarts it with growing delay; read the previous logs and the exit code to find out why.

The problem

The container starts, exits, and the kubelet restarts it again and again with a growing pause. CrashLoopBackOff is not an error in itself; it is the kubelet telling you the container keeps terminating. The real cause is in the last run's logs and exit code, which are gone after the next restart unless you ask for the previous instance.

How it works

  • Symptom: STATUS `CrashLoopBackOff` (or `Error` between attempts), RESTARTS climbing, event `Back-off restarting failed container`. The kubelet restarts with an exponential delay of 10s, 20s, 40s and so on, capped at 5 minutes, and resets the delay after the container has run cleanly for 10 minutes. Applies with `restartPolicy: Always` or `OnFailure`.
  • First look: `kubectl logs <pod> -c <container> --previous` for the crashed run, then `kubectl describe pod <pod>` and read `Last State: Terminated` (reason, exit code, started/finished times). A container that dies within a second usually fails at startup; one that runs for exactly the probe timeout was likely killed by a liveness probe.
  • Common causes (the process fails): application crash or unhandled exception (exit 1), bad config or missing env var, unreachable database or dependency at startup, wrong `command`/`args` or entrypoint (exit 127 not found, 126 not executable, `exec format error` for a wrong CPU architecture), permission denied from `securityContext` (non-root user writing to a root-owned path, read-only root filesystem).
  • Common causes (Kubernetes kills it): a liveness probe that fails on a healthy but slow-starting app (the log ends normally and the exit code is 137 or 143; the event says `Liveness probe failed`), OOM kill (exit 137, `Reason: OOMKilled`), or a process that exits 0 because it daemonised or finished its work, which `restartPolicy: Always` still restarts. A missing Secret or ConfigMap key is not this: it shows as `CreateContainerConfigError`.
  • Fix: act on the exit code. 1 or 2: fix the app or its config using the previous logs. 126/127: correct `command`, `args`, image or file permissions. 137: see `oomkilled` or loosen the probe. 0: make the process stay in the foreground, or use a Job. For slow starters add a `startupProbe` instead of a huge `initialDelaySeconds` on liveness.
  • Debugging tricks: override the command to keep the container alive (`command: ["sleep", "3600"]`), then `kubectl exec` in and run the real entrypoint by hand; or `kubectl debug <pod> -it --copy-to=<pod>-debug --container=<container> -- sh` to get a copy with a changed command; or attach an ephemeral container (`kubectl debug -it <pod> --image=busybox:1.36 --target=<container>`) to inspect a Pod that is up between crashes.
  • Verify: RESTARTS stops increasing and stays flat across several minutes, READY becomes `1/1`, and `kubectl logs <pod> -f` shows normal operation. Note the restart count does not reset by itself until the Pod is recreated, so compare it over time rather than expecting zero.
  • Prevent: validate config at CI time, make the app fail fast with a clear message, log to stdout/stderr, use `startupProbe` for slow boots, keep liveness checks cheap and independent of dependencies (a database outage should not restart every app Pod), and alert on `rate(kube_pod_container_status_restarts_total[15m]) > 0`.

Examples

Previous logs and the exit code of the last termination.

bash
kubectl logs "<pod>" -n "<ns>" -c "<container>" --previous --tail=200
kubectl get pod "<pod>" -n "<ns>" \
  -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
# Look for: reason, exitCode, startedAt, finishedAt
kubectl describe pod "<pod>" -n "<ns>" | grep -E -A6 'Last State|Liveness'

Debug without touching the live Pod: a copy with a shell, or an ephemeral container.

bash
# Copy of the Pod whose container just sleeps, so you can exec and try things:
kubectl debug "<pod>" -n "<ns>" -it --copy-to="<pod>-debug" \
  --container="<container>" -- sleep 3600
kubectl exec -it "<pod>-debug" -n "<ns>" -c "<container>" -- sh
# Or attach an ephemeral container to a Pod that is currently up:
kubectl debug -it "<pod>" -n "<ns>" --image=busybox:1.36 --target="<container>"
kubectl delete pod "<pod>-debug" -n "<ns>"

A slow-starting app: startupProbe holds off liveness; command override shown as a comment for debugging.

yaml
apiVersion: v1
kind: Pod
metadata:
  name: slow-api
spec:
  containers:
    - name: api
      image: registry.example.com/api:1.4.2
      # Debug only: keep the container alive, then exec in.
      # command: ["sleep", "3600"]
      startupProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 5
        failureThreshold: 36
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 3

In practice

  • kubelet — Owns the restart back-off (10s doubling to a 5 minute cap, reset after 10 minutes of successful running). The delay is why a fix can seem slow to take effect; delete the Pod to restart immediately.
  • kubectl debug — Creates ephemeral containers or Pod copies for debugging. Ephemeral containers are stable in current Kubernetes; they share namespaces with `--target` but cannot be removed from the Pod.
  • kube-state-metrics / Prometheus — `kube_pod_container_status_restarts_total` and the waiting-reason metric are the standard inputs for CrashLoopBackOff alerts.
  • In practice — A frequent cause is a liveness probe that checks a dependency: when the database blips, every replica restarts together and the outage gets longer.

Trade-offs

Pros

  • The back-off protects the node and registry from a tight crash loop.
  • `--previous` plus the exit code usually identifies the cause in minutes.

Cons

  • The growing delay (up to 5 minutes) slows the feedback loop while you iterate on a fix.
  • Previous logs only cover the last instance, and are lost if the Pod is deleted or evicted; central logging is needed for history.

Tell a container-limit OOM kill from a node-pressure eviction, confirm it, and fix it by right-sizing memory and runtime settings.

The problem

A container disappears with exit code 137 and the Pod restarts, often under load and without any error in the application logs, because the kernel killed the process. Two different mechanisms look alike: the container hit its own memory limit, or the node ran out of memory and the kubelet evicted Pods. The fix differs, so tell them apart first.

How it works

  • Symptom: `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137` (128+9, SIGKILL); STATUS shows `OOMKilled` or `CrashLoopBackOff` after repeats. The app log just stops mid-line, with no shutdown message.
  • First look: `kubectl describe pod <pod>` for the reason and exit code, `kubectl top pod <pod> --containers` for current memory against `limits.memory`, and your metrics (`container_memory_working_set_bytes`, the closest metric to what the kubelet and OOM killer act on) for the shape before death: a ramp is a leak or unbounded cache, a cliff is a spike (large request, batch load, burst of concurrency).
  • Common causes (container limit): the memory limit is simply lower than the app needs; the runtime does not know about the limit (a JVM heap or Go heap or Node old-space sized larger than the cgroup limit); native memory, thread stacks, buffers and metaspace add to the heap; a leak; unbounded in-memory queues or caches; `emptyDir` with `medium: Memory` and tmpfs files, which count against the container memory.
  • Common causes (node level): when the node runs short, the kubelet evicts Pods and they show STATUS `Evicted` (`The node was low on resource: memory`), not OOMKilled; the kernel OOM killer on the node can still kill a container first if memory grows faster than eviction reacts. Confirm with `kubectl describe node` (`MemoryPressure`) and kernel logs (`dmesg | grep -i 'killed process'`). The kubelet does not rank by QoS class: it evicts Pods whose usage exceeds their request first (BestEffort always does), ordered by Pod priority and then by how far over the request they are; Guaranteed Pods and Burstable Pods under their request go last. QoS does set the kernel `oom_score_adj` (BestEffort killed first, Guaranteed last).
  • Page cache is counted in the cgroup, so heavy file I/O raises memory use, but clean cache is reclaimed before the kernel kills anything. A rising `container_memory_usage_bytes` alone is not an OOM risk; watch the working set, and note that dirty or unreclaimable cache and tmpfs can still push you over.
  • Fix: right-size from measurement. Set `limits.memory` to peak working set plus headroom, and `requests.memory` near typical use (equal request and limit gives Guaranteed QoS when CPU also matches). Cap the runtime below the limit: Go `GOMEMLIMIT` (about 80-90% of the limit; Go 1.19+), JVM `-XX:MaxRAMPercentage=75` (or a fixed `-Xmx`) leaving room for off-heap, Node `--max-old-space-size` below the limit. Fix leaks and bound caches and queues.
  • Verify: RESTARTS stops climbing under the same load test, the working-set graph stays below the limit with a margin, and `Last State` no longer shows OOMKilled. Raising the limit without a cause is a bandage: if memory still grows without bound, you have a leak, so profile (heap dump, pprof) rather than keep raising.
  • Prevent: alert at about 80-90% of the limit on `container_memory_working_set_bytes / kube_pod_container_resource_limits{resource="memory"}`, load-test with realistic peaks, set memory limits on every container, give batch jobs bounded memory, and consider VPA in recommendation mode to see what real usage looks like.

Examples

Confirm an OOM kill and see which container and how much memory it was using.

bash
kubectl get pod "<pod>" -n "<ns>" \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'
kubectl top pod "<pod>" -n "<ns>" --containers
kubectl get pod "<pod>" -n "<ns>" \
  -o jsonpath='{.spec.containers[*].resources}'

Rule out node-level pressure: node conditions, evicted Pods, kernel log.

bash
kubectl describe node "<node>" | grep -E 'MemoryPressure|Allocated|memory'
kubectl get pods -A --field-selector=status.phase=Failed | grep Evicted
kubectl get events -A --field-selector reason=Evicted
# On the node (needs shell access):
# dmesg -T | grep -i -E 'out of memory|killed process'

Keep the runtime heap below the container limit (Go shown; JVM and Node equivalents in comments).

yaml
apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  containers:
    - name: api
      image: registry.example.com/api:1.4.2
      env:
        - name: GOMEMLIMIT
          value: 450MiB
        # JVM:  JAVA_TOOL_OPTIONS=-XX:MaxRAMPercentage=75.0
        # Node: NODE_OPTIONS=--max-old-space-size=384
      resources:
        requests:
          memory: 400Mi
        limits:
          memory: 512Mi

In practice

  • Linux cgroups / kernel OOM killer — The container memory limit is enforced by the cgroup; going over it with nothing reclaimable invokes the OOM killer inside that cgroup, which is why the app gets no chance to log.
  • Go and the JVM — Go 1.19 added the soft limit `GOMEMLIMIT`; modern JVMs read the cgroup limit and size the heap from `-XX:MaxRAMPercentage`. Both still use memory outside the heap, so leave headroom.
  • Vertical Pod Autoscaler — In recommendation mode it suggests requests from observed usage; useful input for right-sizing, even if you never let it resize Pods.
  • In practice — Sidecars (service mesh proxy, log shipper) have their own limits and are OOM-killed independently; check every container, not just the app.

Trade-offs

Pros

  • A hard memory limit contains a leaking container so it cannot starve its neighbours or the node.
  • Setting runtime caps below the limit turns a sudden kill into GC pressure or a catchable error.

Cons

  • Memory is not compressible like CPU: exceeding the limit kills the process instead of slowing it.
  • Large headroom wastes capacity, and a tight limit makes any spike fatal; right-sizing needs real load data.

Find out why the node cannot pull the image: wrong name or tag, missing registry credentials, rate limits, or no network path to the registry.

The problem

The Pod is scheduled but the node cannot get the image, so no container ever starts. The kubelet retries with a growing delay, which is the ImagePullBackOff state. The reason is always in the pull error in the Pod's events: name or tag, credentials, rate limit, or network.

How it works

  • Symptom: STATUS `ErrImagePull` (the attempt just failed) turning into `ImagePullBackOff` (waiting before the next try). Events show `Failed to pull image "<image>": ...` then `Error: ErrImagePull`, then `Back-off pulling image`. Both states mean the same problem; the container never started, so there are no logs.
  • First look: `kubectl describe pod <pod>` and read the text after `Failed to pull image`. Then check the exact image string with `kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}'`. The error text separates the cases: `not found` or `manifest unknown` (name or tag), `pull access denied, repository does not exist or may require authorization` (credentials or wrong name), `toomanyrequests` or a 429 (rate limit), `no such host` or `i/o timeout` (DNS or network). Exact wording varies by runtime and registry.
  • Common causes (image reference): typo in the repository or tag, a tag that was never pushed or was deleted by a retention policy, a missing registry host (so it resolves to Docker Hub), using a tag that exists only for another architecture, or a digest that no longer exists. Pushing from CI after the deploy started is a classic race.
  • Common causes (access and path): a private registry without `imagePullSecrets`, a Secret in the wrong namespace (it must be in the Pod's namespace), a Secret of the wrong type (normally `kubernetes.io/dockerconfigjson`), expired or revoked credentials (cloud registry tokens can be short-lived), Docker Hub anonymous rate limits, and nodes without a route or DNS to the registry (egress policy, proxy, private subnet without NAT or VPC endpoint).
  • Fix: correct the image reference; create the pull secret in the Pod's namespace and reference it from the Pod spec or attach it to the ServiceAccount (`kubectl patch serviceaccount default -p ...`); authenticate to avoid rate limits or use a mirror or pull-through cache; fix node egress and DNS. Test the pull from a node (`crictl pull <image>`) to separate cluster config from node network. Note that `imagePullPolicy` defaults to `Always` for `:latest` or untagged images and `IfNotPresent` otherwise.
  • Not this: `exec format error` is a different symptom. The image pulled fine, but it was built for another CPU architecture (for example arm64 image on amd64 nodes), so the container starts and dies at once with `exec format error` in its logs and CrashLoopBackOff. Build multi-arch images or pin Pods to matching nodes.
  • Verify: a new pull event `Successfully pulled image` followed by `Started container`, and the Pod reaches `Running`. Existing back-off can take up to 5 minutes to retry, so after fixing a Secret, delete the Pod to retry immediately.
  • Prevent: pin images by immutable tag or digest, push the image before rolling out, keep pull credentials managed (external secrets operator, workload identity, or ServiceAccount-level `imagePullSecrets`), mirror base images into your own registry, and keep retention rules from deleting tags still in use.

Examples

Read the pull error and the exact image string; list pull events in the namespace.

bash
kubectl describe pod "<pod>" -n "<ns>" | grep -E -A3 'Failed|Back-off'
kubectl get pod "<pod>" -n "<ns>" -o jsonpath='{.spec.containers[*].image}'
kubectl get events -n "<ns>" --field-selector reason=Failed
# Which pull secrets does the Pod actually reference?
kubectl get pod "<pod>" -n "<ns>" -o jsonpath='{.spec.imagePullSecrets}'

Create the pull Secret in the Pod's namespace and attach it to the default ServiceAccount.

bash
kubectl create secret docker-registry regcred -n "<ns>" \
  --docker-server="<registry-host>" \
  --docker-username="<user>" \
  --docker-password="<token>"
kubectl patch serviceaccount default -n "<ns>" \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'
kubectl get secret regcred -n "<ns>" -o jsonpath='{.type}'
# Expect: kubernetes.io/dockerconfigjson

Reference the pull Secret directly in the Pod spec (it must live in the same namespace).

yaml
apiVersion: v1
kind: Pod
metadata:
  name: api
  namespace: prod
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: api
      image: registry.example.com/team/api:1.4.2
      imagePullPolicy: IfNotPresent

In practice

  • Docker Hub — Applies pull rate limits to anonymous and free-tier users; hitting them typically shows as HTTP 429 / `toomanyrequests` in the pull error (wording varies by runtime). Authenticated pulls or a mirror avoid it.
  • Cloud registries (ECR, GCR/Artifact Registry, ACR) — Usually authenticate nodes through the node or workload identity rather than a static Secret; ECR login tokens expire after 12 hours, which breaks static Secrets silently.
  • crictl — Run on the node, `crictl pull <image>` reproduces the pull with the node's runtime, network and credentials, separating cluster config from node problems.
  • In practice — A Secret created in `default` but a workload in another namespace is a very common miss; pull Secrets are namespaced.

Trade-offs

Pros

  • The pull error is explicit and in the Pod's events, so the category (name, auth, limit, network) is usually clear in one read.
  • ServiceAccount-level `imagePullSecrets` fix a whole namespace at once without editing every manifest.

Cons

  • The back-off delays retries after the fix, so you often have to delete the Pod to see it work.
  • Static pull credentials expire or leak; managed identity is safer but is cloud-specific.

Walk the path layer by layer (DNS, Service selector, EndpointSlices, readiness, targetPort, NetworkPolicy, kube-proxy/CNI, Ingress) to find where traffic stops.

The problem

A client cannot reach a Service: timeouts, connection refused, or a 502/503 from the Ingress. The path has many hops (DNS, Service, EndpointSlices, Pods, NetworkPolicy, kube-proxy and CNI, Ingress), and each fails differently. Checking them in order, from the backend outwards, finds the broken hop quickly.

How it works

  • Symptom: from inside the cluster `curl http://<svc>.<ns>` times out or says `Connection refused`, `nslookup` fails, or an Ingress returns 502/503/504. Decide first: does the name resolve, does the ClusterIP answer, does a Pod IP answer? Each answer points at one layer.
  • First look: `kubectl get svc,endpointslices -n <ns>` (or `kubectl get endpoints <svc>`). Empty endpoints are the single most common finding, and they mean the Service has no ready Pods behind it. Then `kubectl get pods --show-labels` against the Service's `selector`, and READY on the Pods.
  • Common causes (selection): the Service `selector` does not match the Pod labels (typo, label changed in the Deployment template), the Pods are not Ready (failing readiness probe, so they are removed from endpoints; this is by design), the Service is in a different namespace than you queried, or the Pods are not running at all.
  • Common causes (ports): `targetPort` does not match the port the container actually listens on (or a named port that does not exist), the app listens on `127.0.0.1` instead of `0.0.0.0` so only traffic from inside the Pod works, or you are using the Service `port` where you needed the `targetPort` (or the other way round) when testing directly on a Pod.
  • Common causes (network): a NetworkPolicy that does not allow ingress from the caller (or egress to DNS); DNS broken (CoreDNS Pods down, `ndots` search path surprises, caller not allowed to reach port 53); kube-proxy not running or stale rules on a node, or a CNI problem so Pods on different nodes cannot talk. If it fails only from one node, suspect that node's kube-proxy or CNI.
  • Common causes (Ingress layer): the `backend.service.name` or port is wrong, the backend has no endpoints, the `ingressClassName` is missing or does not match an installed controller (so no controller picks it up), TLS or host/path rules do not match, or the controller itself cannot reach the Pod network. `kubectl describe ingress` shows backends and shows `<error: endpoints ... not found>` when the Service is missing.
  • Fix and isolate: run a throwaway debug Pod and test layer by layer (`nslookup`, `curl` to the Service, `curl` to a Pod IP). Use `kubectl port-forward pod/<pod> 8080:<containerPort>` to test the app directly, bypassing Service and kube-proxy; `port-forward svc/<svc>` still picks one backing Pod from the endpoints, so it checks selector and port mapping but not kube-proxy or NetworkPolicy. Then fix the selector, port, probe, policy or DNS you found.
  • Verify and prevent: `kubectl get endpointslices` shows the Pod IPs and `curl` from a debug Pod succeeds through the Service name. Prevent with Helm/Kustomize-generated labels used for both selector and template, named ports shared between Pod and Service, a smoke test after deploy, default-deny NetworkPolicies with explicit DNS egress, and blackbox probes against Services.

Examples

Layers 1-3: does the Service select any ready Pod?

bash
kubectl get svc "<svc>" -n "<ns>" -o wide
kubectl get svc "<svc>" -n "<ns>" -o jsonpath='{.spec.selector}'
kubectl get pods -n "<ns>" -l "<key>=<value>" -o wide
kubectl get endpointslices -n "<ns>" -l kubernetes.io/service-name="<svc>"
# Empty ENDPOINTS: selector mismatch, or Pods not Ready.
kubectl get pod "<pod>" -n "<ns>" -o jsonpath='{.spec.containers[*].ports}'

Test from inside the cluster, then isolate Service from Pod with port-forward.

bash
kubectl run netcheck -n "<ns>" --rm -it --restart=Never \
  --image=nicolaka/netshoot -- bash
# Inside the debug Pod:
#   nslookup "<svc>.<ns>.svc.cluster.local"
#   curl -sv "http://<svc>.<ns>:<port>/healthz"
#   curl -sv "http://<pod-ip>:<containerPort>/healthz"
# From your machine: Pod directly, bypassing Service and kube-proxy
kubectl port-forward -n "<ns>" "pod/<pod>" 8080:"<containerPort>"

Selector, targetPort and the container port must agree; a named port keeps them in one place.

yaml
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: v1
kind: Pod
metadata:
  name: api-0
  labels:
    app: api
spec:
  containers:
    - name: api
      image: registry.example.com/api:1.4.2
      ports:
        - name: http
          containerPort: 8080
      readinessProbe:
        httpGet:
          path: /healthz
          port: http

In practice

  • CoreDNS — Cluster DNS; Services resolve as `<svc>.<ns>.svc.cluster.local`. If names fail but IPs work, check CoreDNS Pods and the caller's `/etc/resolv.conf`.
  • kube-proxy / CNI — kube-proxy programs the Service virtual IP rules (iptables, IPVS or nftables mode); the CNI plugin provides Pod-to-Pod connectivity and usually enforces NetworkPolicy. Most CNIs only enforce policy if the plugin supports it.
  • EndpointSlices — The current API behind Service endpoints; the older `Endpoints` object still exists but has been deprecated since v1.33, so prefer `kubectl get endpointslices` for scripts.
  • Ingress and Gateway API controllers — Take effect only for the class they own (`ingressClassName`); an Ingress with no matching class is silently ignored.

Trade-offs

Pros

  • Layered checks turn a vague 'it is down' into one failing hop, and each hop has a one-line test.
  • Readiness-gated endpoints mean a Service only sends traffic to Pods that say they are ready, so empty endpoints are a clear signal.

Cons

  • Failures look the same from outside (timeout) for DNS, policy and routing problems, so you must test from several vantage points.
  • Debug images and ephemeral Pods may themselves be blocked by NetworkPolicy or admission rules, giving a misleading result.

Production ops

Tune maxSurge, maxUnavailable, readiness and graceful shutdown so a Deployment rollout never drops traffic, and know how to roll back.

The problem

You ship a new image and a few seconds later users see 502s, or the rollout hangs half-way. A Deployment replaces Pods gradually, but "gradually" is only safe if new Pods are truly ready before old ones go away and old Pods drain before they die.

How it works

  • `strategy.type: RollingUpdate` is the default. `maxSurge` (default 25%) is how many extra Pods may exist above `replicas`; `maxUnavailable` (default 25%) is how many may be unavailable. They cannot both be 0.
  • Percentages are rounded differently: `maxSurge` rounds up, `maxUnavailable` rounds down. With 4 replicas and 25%/25%: 1 surge, 1 unavailable. With 2 replicas, 25% unavailable rounds down to 0.
  • Safest shape for small, capacity-sensitive services: `maxUnavailable: 0` and `maxSurge: 1`. New Pod must become Ready before an old one is removed (costs one spare Pod of capacity during the rollout).
  • Readiness is the real gate: a Pod counts as available only when its `readinessProbe` passes. No probe means Ready as soon as the container starts, so traffic arrives before the app listens. `minReadySeconds` (default 0) adds a soak time before a Pod counts as available.
  • `progressDeadlineSeconds` (default 600) marks the rollout as failed (`ProgressDeadlineExceeded`) when no progress is made. It does not roll back by itself; `kubectl rollout status` exits non-zero so CI can react.
  • Connection draining: on termination the Pod is removed from Endpoints while SIGTERM is sent, and these race. Add a `preStop` sleep (a few seconds) so load balancers stop sending traffic, handle SIGTERM by finishing in-flight requests, and keep `terminationGracePeriodSeconds` (default 30) larger than preStop plus drain time; the countdown includes preStop.
  • `strategy.type: Recreate` kills all old Pods first, then starts new ones: downtime by design. Use it only when two versions must never run together (exclusive volume, schema lock).
  • Rollback: `kubectl rollout undo deployment/<name>` returns to the previous ReplicaSet (`--to-revision=N` for a specific one). `revisionHistoryLimit` (default 10) is how many old ReplicaSets are kept; setting it to 0 removes the ability to roll back.
  • Zero-downtime checklist: 2+ replicas; readiness probe that checks real dependencies lightly; `maxUnavailable: 0` or a PodDisruptionBudget for voluntary disruptions; preStop sleep + SIGTERM handling; grace period > drain time; backward-compatible schema and API between adjacent versions.

Examples

A conservative rollout: surge one Pod, never drop below full capacity, drain before exit.

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  revisionHistoryLimit: 5
  minReadySeconds: 10
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          image: registry.example.com/api:1.5.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
          lifecycle:
            preStop:
              sleep:
                seconds: 10

Roll out, wait for the result (non-zero exit on failure), inspect history and roll back.

bash
kubectl set image deployment/<name> api=registry.example.com/api:1.5.0
kubectl rollout status deployment/"<name>" --timeout=300s
kubectl rollout history deployment/"<name>"
kubectl rollout undo deployment/"<name>"
# or: kubectl rollout undo deployment/<name> --to-revision=<n>
kubectl rollout pause deployment/"<name>"   # batch several edits
kubectl rollout resume deployment/"<name>"

In practice

  • kubectl rollout status — Blocks until the rollout completes or fails; the usual CI gate after `kubectl apply` or `helm upgrade`.
  • preStop sleep action — The built-in `sleep` lifecycle handler is beta and on by default since Kubernetes 1.30 and stable (GA) since 1.34; on older clusters, or where the feature gate is off, use an `exec` sleep, which needs a `sleep` binary in the image.
  • Argo Rollouts / Flagger — Add canary and blue-green analysis on top when plain rolling updates give too little control over blast radius.
  • In practice — Most "rollout dropped requests" incidents come from a missing readiness probe, an app that ignores SIGTERM, or a grace period shorter than the longest request. A rollout is only as safe as the Pod spec; run through a pre-deploy checklist (probes, PDB, requests, shutdown handling) and test one rollout under load in staging, because the settings above reduce risk but do not guarantee zero errors.

Trade-offs

Pros

  • Built into Deployments; no extra tooling
  • Incremental, with one-command rollback while old ReplicaSets remain
  • Tunable between speed (higher surge) and safety (zero unavailable)

Cons

  • `maxUnavailable: 0` needs spare capacity for the surge Pods
  • Two versions serve traffic at once, so changes must be backward-compatible
  • No traffic-percentage canary or automatic metric-based rollback

HPA scales replicas from metrics with desired = ceil(current x metric/target); it needs requests, metrics-server and sane scale-down behavior.

The problem

Traffic changes through the day and a fixed replica count is either wasteful or too small. HPA adjusts a workload's replicas from metrics, but it only behaves if requests are set, metrics are available and scale-down is damped.

How it works

  • `autoscaling/v2` metrics: `Resource` (CPU/memory of Pods), `ContainerResource` (one container), `Pods` (custom per-Pod metric), `Object` and `External` (queue depth, requests per second from outside the cluster).
  • Utilization targets are a percentage of the Pod's *requests*, not of limits or node size. CPU `averageUtilization: 70` means 70% of the requested CPU on average across Pods.
  • Algorithm: `desired = ceil(current replicas x currentMetric / targetMetric)`. With 4 replicas at 90% and a target of 60%, desired is ceil(4 x 1.5) = 6.
  • Tolerance: if the ratio is within about 10% of 1.0 (default `--horizontal-pod-autoscaler-tolerance=0.1` on kube-controller-manager), no change is made. The control loop runs about every 15 seconds (`--horizontal-pod-autoscaler-sync-period`). Newer releases can also set a per-HPA tolerance under `behavior`, behind a feature gate.
  • `behavior` controls speed. Each of `scaleUp` and `scaleDown` takes `policies` (type `Pods` or `Percent`, per `periodSeconds`), `selectPolicy` (`Max`, `Min`, `Disabled`) and `stabilizationWindowSeconds`. Scale-down defaults to a 300s window (the highest recommendation in the window wins); scale-up has no window and by default may add up to 100% of current replicas or 4 Pods (whichever is larger) every 15s.
  • `minReplicas` and `maxReplicas` bound the result; `maxReplicas` is your cost ceiling and your downstream-protection ceiling. Scaling to 0 needs the `HPAScaleToZero` feature gate (off by default in releases before 1.37) and only works with Object or External metrics.
  • Needs a metrics source: `metrics-server` for CPU/memory (`metrics.k8s.io`); an adapter such as Prometheus Adapter for custom and external metrics. `kubectl top pods` failing usually means HPA shows `<unknown>`.
  • Pitfalls: Pods without requests cannot be measured, so no utilization HPA; HPA and VPA on the same CPU/memory metric fight each other; new Pods take time to start, so cold start causes lag (use `minReplicas` headroom and a quick readiness probe); Pods not yet ready are treated conservatively.
  • With GitOps, delete `spec.replicas` from the Deployment manifest when an HPA owns it; otherwise each sync resets replicas to the manifest value. For event-driven or scale-to-zero needs (queues, cron, Kafka lag) KEDA creates an HPA from external event sources for 1-to-N scaling and handles the 0-to-1 activation itself.

Examples

CPU target at 70% of requests, fast scale-up, slow and rate-limited scale-down.

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60

Check what the HPA sees and why it did (or did not) scale.

bash
kubectl get hpa api --watch
kubectl describe hpa api        # Conditions: AbleToScale, ScalingActive, ScalingLimited
kubectl top pods -l app=api     # needs metrics-server
kubectl get apiservices v1beta1.metrics.k8s.io
# TARGETS shows <unknown>/70%: metrics missing or Pods have no cpu request

In practice

  • metrics-server — Provides CPU and memory for `kubectl top` and resource-metric HPAs; it is a short-window metrics source, not a monitoring system.
  • KEDA — CNCF project that scales on event sources (queues, streams, cron, Prometheus queries) and can scale a workload to and from zero; it creates an HPA for 1-to-N scaling and handles 0-to-1 itself.
  • Prometheus Adapter — Exposes Prometheus series as custom and external metrics so HPA can scale on requests per second or queue depth.
  • Karpenter / Cluster Autoscaler — HPA adds Pods; a node autoscaler must add nodes for them. Without one, new replicas sit Pending.

Trade-offs

Pros

  • Matches capacity to load without manual work
  • `behavior` lets you scale up fast and down slowly
  • Works with custom and external metrics, not only CPU

Cons

  • Reactive: cold start and the 15s loop mean it lags a sharp spike
  • CPU utilization is a weak proxy for load on I/O-bound services
  • Depends on correct requests and a healthy metrics pipeline

Cap what a namespace can consume with ResourceQuota and give Pods defaults and bounds with LimitRange; quota on compute requires requests.

The problem

On a shared cluster one team's runaway Deployment or forgotten PVCs can starve everyone else. ResourceQuota caps a namespace's total consumption, and LimitRange keeps individual Pods inside sane defaults and bounds.

How it works

  • ResourceQuota is per namespace and enforced at admission time when an object is created or updated. It does not evict existing Pods, and lowering a quota below current usage only blocks new objects.
  • Object counts use `count/<resource>[.<group>]`, for example `pods`, `services`, `persistentvolumeclaims`, `count/deployments.apps`, `count/jobs.batch`.
  • Compute: `requests.cpu`, `requests.memory`, `limits.cpu`, `limits.memory`. Storage: `requests.storage`, and per class `<class>.storageclass.storage.k8s.io/requests.storage`.
  • Scopes narrow what a quota counts: `Terminating`, `NotTerminating`, `BestEffort`, `NotBestEffort`, `PriorityClass` (with `scopeSelector`), and a few others. `Terminating`/`NotTerminating` key off `activeDeadlineSeconds`, so they can give deadline-bound batch Pods a separate budget from long-running services.
  • With a quota on `cpu` or `memory`, every new Pod must set requests or limits for that resource (per the docs, either one); Pods that do not may be rejected. Pair the quota with a LimitRange that supplies defaults so developers do not have to think about it.
  • LimitRange (per namespace, applied at admission): `default` (limit), `defaultRequest`, `min`, `max`, and `maxLimitRequestRatio` for `Container` and `Pod` types (`PersistentVolumeClaim` supports only storage `min`/`max`). It is checked at admission, so existing Pods are unchanged.
  • Typical multi-tenant layout: one namespace per team or per environment, each with a ResourceQuota, a LimitRange, a default-deny NetworkPolicy and RBAC bound to that team's group.
  • When a request exceeds the budget, creation fails with a message of the form `exceeded quota: <name>, requested: ..., used: ..., limited: ...`. For a Deployment, the error shows up on the ReplicaSet's events, not on `kubectl apply`.
  • Inspect with `kubectl describe quota -n <ns>` (Used vs Hard). Alert before the ceiling: a quota hit during an incident-time scale-up is a painful way to find the limit.

Examples

A team namespace budget plus defaults so Pods without resources still pass the quota.

yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-budget
  namespace: team-a
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.memory: 60Gi
    persistentvolumeclaims: "10"
    count/deployments.apps: "30"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: team-defaults
  namespace: team-a
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 256Mi
      max:
        cpu: "4"
        memory: 8Gi

See usage against the budget and find the failing object.

bash
kubectl describe quota -n "<namespace>"
kubectl describe limitrange -n "<namespace>"
kubectl get events -n "<namespace>" --field-selector reason=FailedCreate
kubectl describe replicaset "<replicaset-name>" -n "<namespace>"
# forbidden: exceeded quota: team-budget, requested: requests.cpu=500m, used: ..., limited: ...

In practice

  • Namespace-per-team — The common unit for quota, RBAC and NetworkPolicy; quotas are the cheapest guard against a noisy neighbour.
  • Hierarchical Namespace Controller / Capsule — Community projects for multi-tenancy: HNC (a Kubernetes SIG project, archived in April 2025) adds namespace hierarchies with inherited policy, and Capsule groups namespaces into tenants with shared policy; plain ResourceQuota is per single namespace.
  • kube-state-metrics — Exports `kube_resourcequota` series, so you can alert when used/hard crosses a threshold.
  • Pitfall — Quota on `limits.*` with no LimitRange default can reject every Pod that omits limits, which often surfaces as a mysterious Deployment stuck at 0 replicas.

Trade-offs

Pros

  • Hard, cheap protection against runaway consumption
  • LimitRange gives sane defaults and keeps requests in a believable range
  • Makes team budgets explicit and visible

Cons

  • Per namespace only; no cluster-wide or cross-namespace pool
  • Static numbers need periodic review and can block legitimate scale-ups
  • Failures surface late, on the ReplicaSet or Job rather than at apply time

Design Roles, bindings and per-workload ServiceAccounts with the fewest verbs and resources, and check them with kubectl auth can-i.

The problem

A leaked token or a compromised Pod is only as dangerous as the permissions behind it. Too many clusters run workloads under the default ServiceAccount or hand out cluster-admin to make an error go away. RBAC lets you grant exactly the verbs on exactly the resources each identity needs.

How it works

  • A Role lists rules (`apiGroups`, `resources`, `verbs`, optional `resourceNames`) inside one namespace. A ClusterRole is the same but cluster-scoped, and is also how you describe non-namespaced resources (nodes) and non-resource URLs.
  • A RoleBinding grants a Role or ClusterRole within one namespace; a ClusterRoleBinding grants a ClusterRole everywhere. Binding a ClusterRole with a RoleBinding is the idiom for reusable roles scoped to one namespace. `roleRef` cannot be changed after creation; delete and recreate.
  • Start from nothing and add. List the specific verbs (see `rbac-verbs`) and resources; avoid `*` in verbs, resources or apiGroups and avoid binding `cluster-admin` to anything other than break-glass humans.
  • Use `resourceNames` to limit a rule to named objects (one ConfigMap, one Deployment). It cannot restrict top-level `create` (the name may not be known at authorization time) or `deletecollection`, and for `list`/`watch` the client must filter by `metadata.name` (field selector) to be authorized.
  • One ServiceAccount per workload, never the namespace `default`. Set `automountServiceAccountToken: false` on the ServiceAccount or Pod unless the app talks to the API server; modern tokens are short-lived, audience-bound projected tokens.
  • Verify instead of guessing: `kubectl auth can-i <verb> <resource> -n <ns> --as=system:serviceaccount:<ns>:<name>`, and `kubectl auth can-i --list` to dump what an identity can do. Run these in CI against your manifests.
  • Aggregated ClusterRoles combine other ClusterRoles through `aggregationRule.clusterRoleSelectors`; the built-in `admin`, `edit` and `view` aggregate roles labelled `rbac.authorization.k8s.io/aggregate-to-*`, which is how CRD authors extend them.
  • Escalation prevention: you cannot create or update a Role with permissions you do not hold, nor bind one, unless you also have the `escalate` or `bind` verb. `impersonate` lets an identity act as another. Treat these three, plus `create` on `pods` / `pods/exec` and `get`/`list` on `secrets`, as privileged.
  • Audit: enable API server audit logging and review who uses broad roles. Remove bindings that nobody used; stale grants outlive the people and CI jobs they were made for.

Examples

A minimal read-only Role for one workload's ServiceAccount, with no token mounted by default.

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: reporter
  namespace: app
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: app
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: reporter-pod-reader
  namespace: app
subjects:
  - kind: ServiceAccount
    name: reporter
    namespace: app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader

A CI deployer that can only roll one Deployment and read its Pods; it cannot touch Secrets.

yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deployer
  namespace: app
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["api"]
    verbs: ["get", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

Prove what an identity can and cannot do.

bash
kubectl auth can-i get pods -n app --as=system:serviceaccount:app:reporter   # yes
kubectl auth can-i get secrets -n app --as=system:serviceaccount:app:reporter # no
kubectl auth can-i --list -n app --as=system:serviceaccount:app:reporter
kubectl auth can-i patch deployments/api -n app --as=system:serviceaccount:app:ci-deployer
kubectl get rolebindings,clusterrolebindings -A -o wide | grep cluster-admin

In practice

  • Built-in ClusterRoles — `view`, `edit`, `admin` and `cluster-admin` are the starting vocabulary; bind `view` or `edit` per namespace before writing custom roles.
  • kubectl auth can-i — Asks the API server's authorizer, so it reflects real RBAC; impersonating with `--as` requires the `impersonate` verb yourself.
  • rbac-lookup / kubectl who-can plugins — Community tools that read RBAC objects for access reviews: `kubectl who-can` answers "who can do X", while `rbac-lookup` shows which roles are bound to a given user, group or ServiceAccount.
  • Pitfall — Cluster-admin handed to a CI system makes the CI token equal to full cluster compromise; scope the deployer to a namespace and the few kinds it applies.

Trade-offs

Pros

  • Limits blast radius of a leaked token or compromised Pod
  • Declarative, reviewable in Git, testable with `can-i`
  • Additive model: nothing is allowed unless a rule says so

Cons

  • Many small Roles and ServiceAccounts to maintain
  • Missing verbs surface as `forbidden` errors at runtime, so iterating takes effort
  • Subresources, CRDs and `resourceNames` have sharp edges that are easy to get wrong

Helm

The Helm 3 commands you type daily: install or upgrade a release safely, preview and diff changes, inspect, roll back and pull charts from OCI registries.

The problem

You need to ship a chart to a cluster repeatedly and safely: the same command for first install and every upgrade, a way to see what will change before it changes, and a way back when it goes wrong. Helm 3 gives you releases with numbered revisions, but only if you use the right flags.

How it works

  • Charts: `helm repo add <name> <url>` then `helm repo update` refreshes the index; `helm search repo <term> --versions` lists chart versions. Always pin `--version`, otherwise you get the latest chart on every run.
  • Idempotent deploy: `helm upgrade --install <release> <chart> -n <ns> --create-namespace` installs if absent and upgrades otherwise, so CI needs a single command.
  • Safety flags: `--wait` waits for Pods, Services and PVCs to be ready; `--timeout <dur>` bounds it (default 5m0s); `--atomic` implies `--wait` and automatically rolls back an upgrade (removes a failed install) when it fails; Helm 4 renames it `--rollback-on-failure` and deprecates `--atomic`. Size the timeout to your slowest rollout, hooks included.
  • Value precedence, lowest to highest: chart `values.yaml`, then each `-f file` in order (later wins), then the `--set` family, which always overrides files (Helm applies `--set-json`, `--set`, `--set-string`, then `--set-file`). Keep environment differences in files and use `--set` only for one-off values such as the image tag.
  • Preview before applying: `helm template` renders locally with no cluster; `helm lint` checks the chart; `helm upgrade --dry-run=server` (Helm 3.13+) renders and validates against the API server, while a plain `--dry-run` does not contact it. The `helm diff upgrade` plugin shows a live-vs-new diff.
  • Inspect: `helm list -A` (all namespaces), `helm status <rel>`, `helm get values <rel>` (user-supplied only), `helm get manifest <rel>`, `helm history <rel>`.
  • Undo: `helm rollback <rel> <revision>` creates a new revision from an old one; `helm uninstall <rel> --keep-history` removes resources but keeps revisions so you can still list or roll back.
  • State lives in the cluster: by default each revision is a Secret named `sh.helm.release.v1.<release>.v<N>` (type `helm.sh/release.v1`) in the release namespace. It holds the chart, values and manifest, so treat read access to it as read access to your values. `--history-max` caps stored revisions (default 10).
  • OCI registries (GA in Helm 3.8): `helm registry login`, `helm push <chart>.tgz oci://<registry>/<path>`, `helm pull oci://<registry>/<path>/<chart> --version <v>`, and `helm install` or `upgrade` can take the `oci://` reference directly. There is no `helm repo add` for OCI.

Examples

The one deploy command for CI: idempotent, waits, rolls back on failure.

bash
helm repo add "<repo>" https://charts.example.com
helm repo update
helm upgrade --install "<release>" "<repo>/<chart>" \
  --namespace "<ns>" --create-namespace \
  --version "<chart-version>" \
  -f values.yaml -f values-prod.yaml \
  --set-string "image.tag=<tag>" \
  --atomic --timeout 10m
helm list -n "<ns>"

Preview, inspect and undo. Output hints are comments.

bash
helm lint "./<chart-dir>" -f values-prod.yaml
helm template "<release>" "./<chart-dir>" -f values-prod.yaml | kubectl apply --dry-run=server -f -
helm plugin install https://github.com/databus23/helm-diff
helm diff upgrade "<release>" "./<chart-dir>" -n "<ns>" -f values-prod.yaml
helm history "<release>" -n "<ns>"
# REVISION  STATUS      DESCRIPTION
helm rollback "<release>" "<revision>" -n "<ns>" --wait
helm uninstall "<release>" -n "<ns>" --keep-history

Release state in Secrets, and OCI push/pull.

bash
kubectl get secrets -n "<ns>" -l owner=helm
# sh.helm.release.v1.<release>.v3   helm.sh/release.v1
helm package "./<chart-dir>"
helm registry login "<registry>" -u "<user>"
helm push "<chart>-<version>.tgz" "oci://<registry>/<path>"
helm pull "oci://<registry>/<path>/<chart>" --version "<version>"
helm upgrade --install "<release>" "oci://<registry>/<path>/<chart>" --version "<version>"

In practice

  • helm-diff plugin — A widely used community plugin (databus23/helm-diff) that adds `helm diff upgrade` and `helm diff rollback`; teams run it in CI to review changes before applying.
  • Argo CD / Flux — GitOps controllers typically render charts themselves rather than running `helm install`, so `helm list` may not show their releases; check how your controller handles Helm before relying on release history.
  • Pitfall: --reuse-values — It reuses the previous release's values and ignores new chart defaults, which hides newly added keys. Prefer passing the full set of `-f` files on every upgrade.
  • Pitfall: stuck release — An interrupted upgrade can leave a release in a pending-upgrade state that blocks the next one; inspect `helm history` and roll back to the last deployed revision before retrying.

Trade-offs

Pros

  • One command covers install and upgrade, with revisions and rollback built in
  • `--atomic` plus `--timeout` turns a half-applied release into a clean failure
  • OCI support lets charts live in the same registry as images

Cons

  • Rollback restores manifests only; data, migrations and external state are not undone
  • Release state in Secrets adds cluster objects and can hit size limits for huge charts
  • A client-side tool: two people running different Helm or chart versions can diverge without a CI gate

Keep values.yaml small and documented, validate it with a schema, use per-environment files, and know that maps merge, lists replace and null deletes.

The problem

A chart is only reusable if its configuration is predictable. Values files grow into undocumented sprawl, environments drift because of a dozen --set flags, secrets end up in Git, and nobody remembers whether a list in the override replaces or extends the default.

How it works

  • Shape of `values.yaml`: flat-ish, documented keys with a comment per key, and defaults that work for a small install without any overrides. Deeply nested trees make `--set a.b.c.d=x` painful and error-prone.
  • Validate with `values.schema.json` in the chart root (JSON Schema). Helm checks it on `install`, `upgrade`, `lint` and `template`, so a typo or wrong type fails before anything is applied.
  • One override file per environment (`values-staging.yaml`, `values-prod.yaml`) passed with `-f`, later files winning. Keep `--set` for a single moving value like the image tag; many `--set` flags are unreviewable and lost from Git.
  • No secrets in values: they end up in Git and in the release Secret. Reference an existing Secret by name (`existingSecret`) and fill it with External Secrets Operator, Sealed Secrets or your cloud secret store.
  • Merging semantics: maps are merged key by key, lists are replaced entirely (repeat every item you want), scalars are overwritten, and setting a key to `null` (`key: null` or `--set key=null`) deletes the chart default.
  • Template helpers in chart code (shown in text): `required "message" .Values.x` fails rendering with a clear error, `default "x" .Values.y` supplies a fallback, and `toYaml .Values.resources | nindent 12` embeds a block with correct indentation.
  • See the result, not your intent: `helm get values <rel>` shows only your overrides, `helm get values <rel> --all` includes computed defaults, and `helm template` shows the rendered manifests.
  • Umbrella and library charts: an umbrella chart lists `dependencies` in `Chart.yaml` (then `helm dependency update`) and configures each subchart under a top-level key named after it. The reserved `global` key is visible to the parent and every subchart. A chart with `type: library` only provides named templates and cannot be installed.

Examples

A production override file: only differences, one source of truth in Git. The null deletes a default key; the list replaces the default list.

yaml
# values-prod.yaml
replicaCount: 4
image:
  tag: "1.8.2"
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    memory: 1Gi
existingSecret: api-credentials
ingress:
  hosts:
    - api.example.com
    - api.example.net
podAnnotations: null
global:
  environment: prod

Umbrella chart: Chart.yaml with a pinned subchart. The subchart is configured under its own name in values.

yaml
# Chart.yaml
apiVersion: v2
name: platform
version: 0.4.0
dependencies:
  - name: redis
    version: 1.2.3 # illustrative placeholder
    repository: oci://registry.example.com/charts # illustrative placeholder
    condition: redis.enabled
---
# values.yaml (top-level key = subchart name)
redis:
  enabled: true
  auth:
    existingSecret: redis-credentials
global:
  imageRegistry: registry.example.com

Chart code is shown as text here because it is Go-template syntax, not YAML. Inspect the merge result.

bash
echo 'image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"'
echo 'host: {{ required "ingress.host is required" .Values.ingress.host | quote }}'
echo 'resources: {{- toYaml .Values.resources | nindent 12 }}'
helm lint "./<chart-dir>" -f values-prod.yaml
helm get values "<release>" -n "<ns>"
helm get values "<release>" -n "<ns>" --all
helm dependency update "./<chart-dir>"

In practice

  • Bitnami charts — A well-known example of documented values with a README table per key and `existingSecret`-style options; check a chart's `values.yaml` and `values.schema.json` for the conventions in practice.
  • External Secrets Operator / Sealed Secrets — Common ways to keep secret material out of values files while still deploying declaratively.
  • Pitfall: list replacement — Overriding `extraEnv` or `ingress.hosts` with one item silently drops the chart's other defaults; confirm with `helm get values --all` or `helm template`.
  • Pitfall: subchart values — Values for a subchart must sit under its name (or `global`); a key at the top level is ignored by the subchart and often fails silently.

Trade-offs

Pros

  • Per-environment files are reviewable, diffable and reproducible from Git
  • A schema catches typos and wrong types before the cluster does
  • A small, documented values surface keeps the chart upgradable

Cons

  • Maps merge but lists replace, which surprises people and needs discipline
  • Every exposed value is a promise to users; too many knobs make a chart hard to evolve
  • Umbrella charts couple subchart versions and can make `global` a dumping ground

Annotate a Job or Pod with helm.sh/hook to run it at a release lifecycle event, such as a pre-upgrade DB migration, and know how hooks differ from normal resources.

The problem

Some steps do not belong in a Deployment: a schema migration before the new Pods start, a smoke test after install, a cleanup before uninstall. Helm hooks run a Job or Pod at a release lifecycle event, but hooks behave differently from ordinary chart resources, and that difference causes most hook incidents.

How it works

  • A hook is any manifest in the chart with the annotation `helm.sh/hook: <event>`. Events: `pre-install`, `post-install`, `pre-upgrade`, `post-upgrade`, `pre-delete`, `post-delete`, `pre-rollback`, `post-rollback`, `test`. Comma-separate to use several.
  • Ordering: `helm.sh/hook-weight` is a string holding an integer (default "0"); hooks run in ascending order, and the docs say ties are sorted by kind and then name (equal-weight order is not strongly guaranteed, so set explicit weights). Use it to create a ConfigMap hook (weight -5) before the Job that mounts it (weight 0).
  • Cleanup: `helm.sh/hook-delete-policy` accepts `before-hook-creation` (default: delete the old one before a new run, so a re-run with the same name does not clash), `hook-succeeded` and `hook-failed`. Keep failed Jobs around with `before-hook-creation` alone so you can read their logs.
  • Typical use 1: a DB migration Job as `pre-upgrade,pre-install`. Helm waits for the Job to complete before touching the rest of the release; set `backoffLimit` and `activeDeadlineSeconds` so a bad migration fails fast.
  • Typical use 2: `helm.sh/hook: test` Pods that check the Service; they run only on `helm test <release>`, which is a good post-deploy step for CI.
  • Pitfall: hook resources are not part of the release manifest. They are not tracked as release objects, `helm uninstall` does not delete them, and `helm rollback` does not undo what they did (a migration stays applied). Make migrations backward compatible.
  • Pitfall: a failed `pre-upgrade` hook fails the release, and the chart's regular resources are not applied; a failed `post-*` hook marks the release failed although the new Pods are already running. With `--atomic`, a failed upgrade is rolled back (a failed install is removed).
  • Pitfall: Helm waits for hook Jobs within `--timeout` (default 5m0s). A migration longer than the timeout fails the release even if the Job later succeeds, so raise `--timeout` or run long jobs outside Helm.
  • GitOps caveat: tools like Argo CD render charts with `helm template` and map Helm hook annotations onto their own hook phases (for example pre-install and pre-upgrade onto a pre-sync phase), so install and upgrade are not distinguished, and `test` and rollback hooks are not supported. Verify in your tool's docs.

Examples

A rendered pre-upgrade migration Job (what Helm applies after templating).

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: api-migrate-v8
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/api:1.8.2
          command: ["/app/migrate", "up"]
          envFrom:
            - secretRef:
                name: api-credentials

A test hook Pod; it runs only on helm test.

yaml
apiVersion: v1
kind: Pod
metadata:
  name: api-smoke-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  restartPolicy: Never
  containers:
    - name: curl
      image: curlimages/curl:8.8.0
      command: ["curl", "--fail", "--silent", "http://api:8080/healthz"]

Run the lifecycle with enough time, then test; the Go-template annotation line is shown as text.

bash
echo '"helm.sh/hook-weight": {{ .Values.migrate.weight | default 0 | quote }}'
helm upgrade --install "<release>" "./<chart-dir>" -n "<ns>" --atomic --timeout 15m
kubectl get jobs -n "<ns>" -l "app.kubernetes.io/instance=<release>"
kubectl logs -n "<ns>" "job/<job-name>"
helm test "<release>" -n "<ns>" --logs

In practice

  • Helm chart best practices — The Helm docs describe hooks and the annotation set above; they also state that hook resources are not managed as part of the release.
  • Argo CD — Supports Helm hook annotations by mapping them to its resource hooks; because it does not run `helm install`, behaviour differs from the CLI, so test your charts under both.
  • Pitfall: non-idempotent migration — A pre-upgrade migration that is not safe to run twice or against the old code will cause an outage on retry or rollback; use a migration tool with locking and make steps expand-then-contract.
  • Pitfall: hook before its dependencies — A pre-install Job that needs a chart ServiceAccount or Secret fails because normal resources are not created yet; mark those prerequisites as hooks with a lower weight.

Trade-offs

Pros

  • Ties one-off steps like migrations and smoke tests to the release lifecycle
  • Weights and delete policies give ordering and cleanup without extra tooling
  • `pre-upgrade` can stop a bad release before any workload changes

Cons

  • Hooks are outside the release manifest, so rollback and uninstall do not manage them
  • Long jobs collide with `--timeout`, and failures can block deploys
  • Behaviour differs between Helm CLI and GitOps controllers

Scheduling

kube-scheduler filters nodes that can run the Pod, scores the rest, picks the best and binds it; requests, not usage, drive the decision.

The problem

A new Pod has no node. Something must pick one that has the resources and satisfies every constraint, quickly and without two Pods fighting over the same capacity. kube-scheduler does this as a pipeline of pluggable stages.

How it works

  • kube-scheduler watches for Pods with no `spec.nodeName`, takes one from a priority queue, and finds the best node for it.
  • Filtering finds the feasible nodes (enough CPU/memory requests, ports, volumes, node selectors, affinity, taints). The result can be empty — the Pod then stays Pending with an event explaining why.
  • Scoring ranks the feasible nodes (spreading, image locality, preferred affinity, balanced resource use). The highest score wins; ties are broken randomly.
  • Binding writes the decision to the API server (the Pod's node). The scheduling cycle is serial, one Pod at a time; the binding cycle runs asynchronously so the scheduler can move on.
  • Everything is a plugin on extension points (QueueSort, PreFilter, Filter, PostFilter, Score, Reserve, Permit, PreBind, Bind, ...) configured with scheduling profiles; you can run several profiles or whole extra schedulers via `schedulerName`.
  • Scheduling uses requests, not live usage: a node with idle CPU but fully requested CPU is full. Set requests honestly, or bin-packing and the autoscaler both go wrong.
  • PostFilter is where preemption happens when nothing fits; the autoscaler (Cluster Autoscaler / Karpenter) reacts to Pods that are unschedulable and adds nodes.

Examples

Requests drive scheduling; nodeSelector is a hard filter; schedulerName picks a profile or scheduler.

yaml
apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  schedulerName: default-scheduler
  nodeSelector:
    disktype: ssd
  containers:
    - name: api
      image: registry.example.com/api:1.4.2
      resources:
        requests:
          cpu: 500m
          memory: 256Mi
        limits:
          memory: 512Mi

Why is a Pod Pending? The scheduler reports it as an event.

bash
kubectl get pods --field-selector=status.phase=Pending
kubectl describe pod api | sed -n '/Events:/,$p'
kubectl get events --field-selector reason=FailedScheduling

In practice

  • Default scheduler — Ships with the control plane; a feasible-node search stops early on very large clusters to bound latency.
  • Cluster Autoscaler / Karpenter — Watch for unschedulable Pods and add nodes sized for their requests.
  • Volcano / Kueue — Batch and gang scheduling for ML and HPC jobs on top of (or beside) the default scheduler.
  • Pitfall — No requests means the Pod is treated as tiny: it schedules anywhere and gets throttled or OOM-killed under load.

Trade-offs

Pros

  • Declarative constraints, no manual placement
  • Pluggable: custom filters, scores and whole schedulers
  • Asynchronous binding keeps throughput high

Cons

  • Decisions use requests, which are only as good as what teams declare
  • Serial scheduling cycle bounds throughput for large bursts
  • Heavy constraints (pod affinity) get expensive on big clusters

Steer Pods with node affinity, anti-affinity, taints and tolerations, topology spread constraints and priority classes.

The problem

Defaults treat nodes as interchangeable, but real clusters have GPU nodes, zones that can fail, noisy neighbours and workloads that matter more than others. You need ways to attract Pods, repel them, spread them, and rank them.

How it works

  • nodeSelector / nodeAffinity: attract Pods to nodes by label. `requiredDuringScheduling…` is a hard filter; `preferredDuringScheduling…` only adds score. Both are ignored once the Pod runs.
  • Taints and tolerations: a taint on a node repels Pods that do not tolerate it. Effects: NoSchedule (no new Pods), PreferNoSchedule (soft), NoExecute (also evicts running Pods, optionally after `tolerationSeconds`). A toleration permits, it does not attract.
  • podAffinity / podAntiAffinity: place relative to other Pods within a `topologyKey` (node, zone). Powerful but expensive to evaluate on large clusters.
  • topologySpreadConstraints: keep the Pod count per domain within `maxSkew` of the minimum. `whenUnsatisfiable: DoNotSchedule` is hard, `ScheduleAnyway` is soft. Use `topology.kubernetes.io/zone` for AZ spread and `kubernetes.io/hostname` for node spread.
  • PriorityClass: higher priority Pods are scheduled first and may preempt lower ones when nothing fits (victims are terminated gracefully; the preemptor gets a `nominatedNodeName`, which is not a guarantee). `preemptionPolicy: Never` makes a Pod queue ahead without evicting.
  • Disruptions are separate: PodDisruptionBudgets limit voluntary evictions (drains, upgrades), they do not stop involuntary ones or preemption by priority.

Examples

Spread replicas over zones, tolerate a dedicated-node taint, and prefer SSD nodes.

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      priorityClassName: high-priority
      tolerations:
        - key: dedicated
          operator: Equal
          value: api
          effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 50
              preference:
                matchExpressions:
                  - key: disktype
                    operator: In
                    values: ["ssd"]
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
      containers:
        - name: api
          image: registry.example.com/api:1.4.2

A priority class that queues ahead of others but never evicts running Pods.

yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
preemptionPolicy: Never
globalDefault: false
description: "Schedule ahead of normal Pods without preempting them."

In practice

  • Dedicated GPU pools — Taint GPU nodes (`nvidia.com/gpu=present:NoSchedule`) and tolerate it only in GPU workloads, so general Pods do not squat on them.
  • Kafka / Redis / ClickHouse on K8s — Spread brokers, shards or replicas across zones and nodes with topology spread so one failure domain cannot take a quorum.
  • System-critical Pods — `system-cluster-critical` and `system-node-critical` priority classes protect control-plane add-ons from preemption.
  • Pitfall — Required affinity or DoNotSchedule with too few matching nodes leaves Pods Pending forever — check `kubectl describe pod` events.

Trade-offs

Pros

  • Expresses failure-domain and hardware needs declaratively
  • Topology spread gives even, zone-aware placement without per-Pod rules
  • Priority lets critical work win under pressure

Cons

  • Hard rules reduce schedulability and can strand Pods in Pending
  • Pod (anti-)affinity is costly to evaluate at scale
  • Preemption is disruptive and best-effort; it cannot replace capacity planning

Networking

A Service is a stable virtual IP; kube-proxy turns it into node-local rules that spread traffic over ready endpoints.

The problem

Pods are ephemeral and their IPs change. Clients need a stable name and address that always reaches healthy backends, without every client implementing discovery and load balancing.

How it works

  • A Service selects Pods by label and gets a stable virtual IP (ClusterIP) and a DNS name (`svc.namespace.svc.cluster.local`, served by CoreDNS).
  • EndpointSlices list the ready Pod IPs behind a Service; the control plane keeps them current as Pods start, fail readiness and terminate. Only ready endpoints receive traffic.
  • kube-proxy runs on each node, watches Services and EndpointSlices, and programs the node to redirect the ClusterIP to a backend. Modes on Linux: iptables (the default today, random backend choice), ipvs, and nftables (documented as the future default).
  • There is no real proxy process on the data path in iptables mode: it is destination NAT in the kernel, so there is no per-connection health checking beyond readiness and no least-connections balancing.
  • Service types: ClusterIP (internal), NodePort (a port on every node), LoadBalancer (cloud load balancer on top of NodePort), ExternalName (DNS alias). A headless Service (`clusterIP: None`) skips the virtual IP and returns Pod IPs in DNS — what StatefulSets use for stable identities.
  • sessionAffinity: ClientIP pins a client to one backend; `externalTrafficPolicy: Local` preserves the client IP and avoids a second hop, at the cost of uneven balancing.
  • Set the kube-proxy mode explicitly in configuration so an upgrade cannot change it under you.

Examples

A normal Service and a headless Service for stable per-Pod DNS names.

yaml
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: redis-headless
spec:
  clusterIP: None
  selector:
    app: redis
  ports:
    - name: redis
      port: 6379

Inspect what a Service actually routes to.

bash
kubectl get svc api -o wide
kubectl get endpointslices -l kubernetes.io/service-name=api
kubectl -n kube-system get cm kube-proxy -o yaml | grep -i mode

In practice

  • StatefulSets (Kafka, Redis, ClickHouse) — A headless Service gives each Pod a stable DNS name such as `kafka-0.kafka-headless`, which brokers advertise and peers use to find each other.
  • Cilium — eBPF datapath can replace kube-proxy entirely, avoiding iptables rule scaling issues.
  • Cloud load balancers — `type: LoadBalancer` provisions an external L4 load balancer; Ingress / Gateway API add L7 routing.
  • Pitfall — A Pod that is Running but not Ready receives no Service traffic; failing readiness probes look like “the Service is broken”.

Trade-offs

Pros

  • Stable address and DNS name regardless of Pod churn
  • No client-side discovery code; works for any TCP/UDP protocol
  • Readiness-aware: unhealthy Pods drop out of rotation

Cons

  • L4 only, random-ish balancing: no retries, outlier detection or per-request routing
  • iptables rules scale linearly with Services and endpoints; very large clusters need ipvs, nftables or an eBPF datapath
  • Long-lived connections (gRPC, HTTP/2) stay pinned to the backend chosen at connect time

A proxy layer for mTLS, retries, timeouts, traffic splitting and telemetry — as sidecars or, in Istio ambient mode, per-node ztunnels.

The problem

Every service needs mutual TLS, timeouts, retries, circuit breaking, canary routing and consistent telemetry. Implementing that in every language and library is slow and inconsistent. A mesh moves it into a shared proxy layer driven by central policy.

How it works

  • Data plane: Envoy proxies that mediate all traffic between services — service discovery, load balancing, TLS termination, circuit breaking, fault injection and metrics.
  • Control plane (Istio's istiod): service discovery, configuration management and certificate issuance; it translates high-level routing rules into Envoy configuration pushed to every proxy.
  • Sidecar mode puts an Envoy next to each Pod (injected, no app changes). Ambient mode replaces per-Pod sidecars with a node-level ztunnel for L4 security (mTLS, identity) and optional waypoint proxies for L7 features, reducing per-Pod overhead.
  • Traffic management: weighted routing (90/10 canary), header-based routing, retries with budgets, timeouts, outlier detection, mirroring.
  • Security: workload identity from certificates, automatic mTLS, and authorization policy by identity rather than by IP.
  • The mesh does not remove the app's duties: it still has to forward trace headers between inbound and outbound requests, and retries need idempotent endpoints.
  • Adopt incrementally: start with telemetry and mTLS in permissive mode, then add traffic policy per service.

Examples

Istio canary: 90% to v1, 10% to v2, with a timeout and retries.

yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
    - api
  http:
    - route:
        - destination:
            host: api
            subset: v1
          weight: 90
        - destination:
            host: api
            subset: v2
          weight: 10
      timeout: 2s
      retries:
        attempts: 2
        perTryTimeout: 500ms
        retryOn: connect-failure,refused-stream,503
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: api
spec:
  host: api
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

In practice

  • Istio — Envoy-based; sidecar or ambient data plane, istiod control plane; the most feature-rich and the heaviest to operate.
  • Linkerd — Purpose-built Rust micro-proxy; smaller feature set, lower overhead and simpler operations.
  • Cilium service mesh — eBPF and per-node Envoy; sidecar-less by design.
  • Pitfall — Retries multiply load during an outage (retry storm); cap retries with budgets and make endpoints idempotent.

Trade-offs

Pros

  • Uniform mTLS, retries, timeouts and telemetry across languages, with no code changes
  • Central, auditable traffic and security policy
  • Progressive delivery (canary, mirroring) without app involvement

Cons

  • Extra hop and CPU/memory per Pod (less with ambient), plus tail-latency cost
  • Large operational surface: upgrades, debugging through proxies, certificate lifecycle
  • Misconfigured retries and timeouts can cause outages the apps never had

Extensibility

A custom resource plus a controller that reconciles actual state toward desired state — encoding operational knowledge as code.

The problem

Running a stateful system like Kafka, Redis or ClickHouse takes operational knowledge: bootstrap, scaling, failover, upgrades, backups. A runbook cannot react at 3 a.m. The operator pattern turns that knowledge into a controller that runs inside the cluster.

How it works

  • A CustomResourceDefinition (CRD) adds a new API type (e.g. `ClickHouseInstallation`); users declare desired state in a custom resource (CR).
  • A controller runs a control loop: watch the API server, compare desired state (`spec`) with observed state, and act to move the world toward it. That loop is the whole pattern.
  • Reconciliation is level-triggered and must be idempotent: it is called repeatedly, for any reason, and must converge from any starting state; never assume you saw every event.
  • Report progress in `status` (conditions, observed generation) and use owner references so child objects (StatefulSets, Services, ConfigMaps) are garbage-collected with the CR.
  • Finalizers let the controller clean up external resources (volumes, cloud resources, cluster membership) before the object is deleted.
  • Run several replicas and elect a leader with a Lease so only one reconciles at a time; rate-limit and back off on errors with a work queue.
  • Build with controller-runtime / Kubebuilder or Operator SDK; the Operator Lifecycle Manager and Helm cover packaging and install.
  • Operators typically automate deploying on demand, backups and restores, upgrades including schema or config changes, publishing a Service, and failure testing.

Examples

Go
import (
    "context"
    "fmt"
    "time"
)

// Desired state declared by the user (the custom resource's spec).
type Spec struct {
    Replicas int
    Version  string
}

// Observed state read from the cluster.
type Observed struct {
    Replicas int
    Version  string
    Ready    int
}

type Cluster interface {
    Get(ctx context.Context, name string) (Spec, Observed, error)
    ScaleTo(ctx context.Context, name string, replicas int) error
    Upgrade(ctx context.Context, name, version string) error
    SetStatus(ctx context.Context, name, message string) error
}

// Reconcile is level-triggered and idempotent: it looks only at the
// current spec and observed state, takes at most one step toward the goal,
// and asks to be called again instead of blocking.
func Reconcile(ctx context.Context, c Cluster, name string) (time.Duration, error) {
    spec, got, err := c.Get(ctx, name)
    if err != nil {
        return 0, err
    }
    switch {
    case got.Replicas != spec.Replicas:
        return time.Second, c.ScaleTo(ctx, name, spec.Replicas)
    case got.Version != spec.Version && got.Ready == got.Replicas:
        // Upgrade only a fully healthy cluster, one step at a time.
        return time.Second, c.Upgrade(ctx, name, spec.Version)
    case got.Ready != got.Replicas:
        msg := fmt.Sprintf("%d/%d replicas ready", got.Ready, got.Replicas)
        return 5 * time.Second, c.SetStatus(ctx, name, msg)
    }
    return 0, c.SetStatus(ctx, name, "Ready")
}

A custom resource: the user states what, the operator works out how.

yaml
apiVersion: example.com/v1
kind: CacheCluster
metadata:
  name: sessions
spec:
  replicas: 3
  version: "7.4"
  persistence:
    mode: aof
    storage: 20Gi
status:
  conditions:
    - type: Ready
      status: "True"

In practice

  • Strimzi — Runs Kafka (KRaft or ZooKeeper) on Kubernetes: topics and users are CRs, rolling restarts respect ISR.
  • Altinity clickhouse-operator — Declares shards and replicas in a ClickHouseInstallation and wires Keeper, macros and cluster config.
  • Prometheus Operator / CloudNativePG — Monitoring stacks and PostgreSQL clusters managed through CRs, including failover and backups.
  • kube-controller-manager — The built-in controllers (Deployment, StatefulSet, Job, ...) are the same pattern applied to core types.

Trade-offs

Pros

  • Declarative, self-healing operations for stateful systems
  • Reuses Kubernetes API, RBAC, watch and GitOps tooling
  • Operational knowledge is versioned, tested and shared

Cons

  • A controller is software you must write, test and operate; bugs can delete data
  • CRD versions and conversions need care; schema changes are API changes
  • Cluster-scoped controllers are a powerful, high-blast-radius component

Control plane

etcd (Raft) stores all cluster state behind the API server; resourceVersion gives optimistic concurrency and Leases give leader election.

The problem

Controllers, schedulers and kubelets all need one consistent view of the cluster and a safe way to avoid acting on stale data or running two active copies of a singleton. Kubernetes answers with a consistent store, versioned objects and leases.

How it works

  • etcd is a Raft-replicated key-value store holding all cluster state. Only the API server talks to it; everything else goes through the API, which gives validation, auth, admission and watch.
  • Every object has a `resourceVersion` taken from etcd's revision. Updates are compare-and-swap: writing with a stale resourceVersion fails with 409 Conflict, so clients re-read and retry — optimistic concurrency.
  • Watch streams changes from a resourceVersion, so controllers react to events instead of polling; if a watch falls behind compaction (410 Gone), they re-list.
  • Leases (`coordination.k8s.io/v1`) are small objects with a holder and a renew time. kube-controller-manager and kube-scheduler use one for leader election: the holder renews it, standbys take over after it expires.
  • Each node has a Lease in `kube-node-lease` that kubelet renews as a cheap heartbeat; API servers publish identity Leases too.
  • Leader election gives at-most-one active controller only while clocks and renewals behave: a paused leader can overlap with its successor, so controllers must stay idempotent (and use fencing where side effects are external).
  • Run 3 or 5 etcd members on fast, low-latency disks; writes cost a fsync and a quorum round trip. Watch DB size against the quota (2 GiB default, raise with care) and compact and defragment.

Examples

Look at the Leases that hold leadership, and the node heartbeats.

bash
kubectl -n kube-system get lease kube-scheduler kube-controller-manager
kubectl -n kube-node-lease get lease | head
kubectl get --raw '/readyz?verbose' | head -20

A Lease used for leader election in a custom controller.

yaml
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-controller
  namespace: my-system
spec:
  holderIdentity: my-controller-7d9f_2b1c
  leaseDurationSeconds: 15
  renewTime: "2026-10-07T04:00:00.000000Z"

In practice

  • etcd — Defaults to a 100 ms heartbeat and 1000 ms election timeout; cross-region stretched clusters need those tuned and still pay the latency.
  • client-go leaderelection — The library controllers use to acquire, renew and release a Lease with configurable durations.
  • Operators — Run two or three replicas with leader election so a node failure does not stop reconciliation.
  • Pitfall — An unbounded list of objects or an over-large etcd database makes API server and etcd slow for the whole cluster.

Trade-offs

Pros

  • A single strongly consistent source of truth with change notification
  • Optimistic concurrency avoids distributed locks for ordinary updates
  • Leases give cheap leader election and failure detection

Cons

  • Writes go through one Raft leader; etcd is not a high-volume database
  • Leases are time-based, so a stalled leader can overlap its successor
  • Control-plane health depends on etcd disk latency and size