Kubernetes Pods That Won't Delete: Follow ownerReferences, Not the Force Flag

A pod that reappears after kubectl delete isn’t a bug in Kubernetes. Something owns it, and that owner is doing exactly what it was designed to do: recreating a missing replica. The fix is almost never --force. It’s finding the owner (Job, ReplicaSet, StatefulSet or Helm release) and deleting that instead.

The incident

I ran into this while deploying GitLab through Helm in 2018, but nothing about the mechanics has changed: three migration pods kept refusing to die.

# kubectl get pods -n kube-system
NAME                            READY    STATUS     RESTARTS   AGE
gitlab-migrations.1-2km-4rwnm   1/1      Running    158        21h
gitlab-migrations.1-30d-gx9nz   1/1      Running    164        21h
gitlab-migrations.1-dsw-m5qbc   1/1      Running    167        21h

Look at the restart counts. 158, 164, 167 restarts in 21 hours. These pods weren’t stuck in the “won’t terminate” sense — they were crash-looping, and every time I deleted one, it came back within seconds. Deleting the pod was never going to work, because the pod wasn’t the problem.

If you see the same pattern today (three-digit restart counts, identical ages, a suspicious symmetry), stop deleting pods. You’re fighting a controller, and the controller wins every round.

Why deleting the pod does nothing

Kubernetes objects form an ownership tree. You delete a leaf and the branch grows a new one:

Deployment ──► ReplicaSet ──► Pod
CronJob    ──► Job        ──► Pod
StatefulSet ─────────────────► Pod
Helm release ──► (any of the above, via labels)

kubectl delete pod sends a termination request to that single pod. The owning controller notices its replica count is below the desired value and schedules a replacement. This is not a malfunction: it’s the reconcile loop doing its job, the same loop that will recreate your pod at 3 a.m. when a node dies.

The only reliable way out: delete the owner. The pods go with it.

Step 1 — Read the description before touching anything

kubectl describe on one of the ghosts was the turning point:

# kubectl describe pod -n kube-system gitlab-migrations.1-30d-gx9nz
...
Controlled By:  Job/gitlab-migrations.1-30d
...
Labels:         app=migrations
                controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209
                job-name=gitlab-migrations.1-30d
                release=gitlab
...
Events:
  Warning  FailedMount  6m (x655 over 21h)   kubelet  MountVolume.SetUp failed for volume
           "migrations-config": configmaps "gitlab-migrations" not found
  Warning  BackOff      3m (x3516 over 21h)  kubelet  Back-off restarting failed container

Two findings in one screen:

Controlled By: Job/gitlab-migrations.1-30d — the pods belong to Jobs, not to a Deployment. That’s the owner I need to remove.
configmaps "gitlab-migrations" not found ×655. The containers kept crashing because a previous (failed) redeploy had removed the ConfigMap these pods mount, so every start attempt died with FailedMount, then the kubelet backed off and restarted. The pods were working as intended. The environment wasn’t.

That second finding matters for a practical reason: even after deleting the Job, you should understand what got deleted, because the recreate button (Helm, GitOps, a human with a YAML file) is still live. If the ConfigMap is genuinely gone, the next deploy fails the same way.

Tip: kubectl describe prints environment variables in full. The output above contained a live GITLAB_ROOT_PASSWORD in plain text. Scrub secrets before pasting describe output into tickets, chats or blog posts — I nearly didn’t.

Step 2 — Walk the ownership chain

You don’t have to grep labels by hand. The owner is recorded in metadata.ownerReferences:

kubectl get pod -n kube-system gitlab-migrations.1-30d-gx9nz 
  -o jsonpath='{range .metadata.ownerReferences[*]}{.kind}/{.name}{"n"}{end}'

For a Deployment-managed pod this prints ReplicaSet/something-7d9f8c; for our case it’s Job/gitlab-migrations.1-30d. Follow the chain one level at a time until the top level is a Deployment, StatefulSet, DaemonSet, CronJob — or nothing at all.

Then confirm what actually exists in the namespace:

kubectl get jobs -n kube-system

NAME                      DESIRED   SUCCESSFUL   AGE
gitlab-migrations.1-2km   1         0            23h
gitlab-migrations.1-30d   1         0            23h
gitlab-migrations.1-dsw   1         0            23h

Three Jobs, zero successful, 23 hours old. kubectl get all would have shown them, but get all has a dirty secret: it skips plenty of resource types (CRDs, for one), and plenty of people don’t read its output carefully. Checking the specific kind you found in ownerReferences is more reliable.

Step 3 — Delete the owner, then confirm

kubectl delete job gitlab-migrations.1-2km -n kube-system
kubectl delete job gitlab-migrations.1-30d -n kube-system
kubectl delete job gitlab-migrations.1-dsw  -n kube-system

Deleting a Job cascades to its pods by default. One get pods later, the infinite loop was over:

NAME                               READY   STATUS        RESTARTS   AGE
gitlab-migrations.1-30d-gx9nz      1/1     Terminating   173        23h
gitlab-migrations.1-dsw-m5qbc      0/1     Terminating   176        23h

A few seconds after that, both were gone. A day of kubectl delete pod had accomplished nothing; deleting the owner took one command.

Same principle for the other owners: kubectl delete deployment for Deployments, kubectl delete statefulset for StatefulSets. And if the workload came from Helm, resist hand-deleting the leaves first. helm uninstall <release> -n <ns> removes the whole release as a unit, which is what you actually want.

When the pod is really stuck (Terminating forever)

Everything above concerns pods that keep coming back. A different failure mode is a pod that stays in Terminating — it has a deletionTimestamp but never finishes. The usual suspects:

Symptom	Likely cause	First check
Terminating for minutes	Kubelet unreachable / node NotReady	`kubectl get nodes`
Terminating with a volume	Attached volume that won’t detach	`kubectl describe pod`, cloud provider console
Terminating, custom resource	A finalizer never removed	`kubectl get pod X -o jsonpath='{.metadata.finalizers}'`
Terminating after node deletion	Node object gone, kubelet gone	Node still in API? `kubectl get node`
Long terminations, no error	Grace period + SIGTERM-ignoring process	`terminationGracePeriodSeconds`, app logs

The reality check every time: is the node still alive? If it is, figure out why the kubelet can’t kill the container (runtime stuck, containerd problem, PLEG issues) and fix it there. If the node is genuinely dead and gone from the cluster, force-deletion is the acceptable last resort:

kubectl delete pod <name> -n <ns> --grace-period=0 --force

Understand the trade-off before typing it: force-deletion doesn’t wait for the kubelet to confirm the container died. If the node actually comes back, you now have a pod the API server forgot but the runtime never stopped — two copies writing to the same volume, which is how data corruption stories start. A removed append-only finalizer (kubectl patch pod X --type=merge -p '{"metadata":{"finalizers":null}}') is the same category of move: fine as a last resort on an unreachable node, dangerous anywhere else.

The 2026 version of the same lesson

Kubernetes itself got better at this. The mechanics stayed identical, so here’s what the modern setup looks like:

Helm 3 dropped Tiller entirely; helm uninstall release -n ns is the clean teardown. Leftover migration Jobs from a broken upgrade are still common — that part of 2018 never left.
ttlSecondsAfterFinished (GA since 1.23) garbage-collects finished Jobs automatically. If you generate Jobs from CronJobs or CI, set it. Old completed Jobs in a namespace are exactly the kind of clutter that makes “why won’t this pod delete” investigations longer than they need to be.
GitOps makes hand-deletion temporary. Argo CD or Flux will happily recreate anything you kubectl delete if the manifest still declares it. Under GitOps, the fix for a runaway workload is a commit, not a terminal command. Delete via the source of truth or you’ll be repeating yourself in five minutes.
kubectl events --for pod/X (1.23+) and kubectl debug (1.18+) replaced a lot of the manual poking. kubectl debug -it pod/X --image=nicolaka/netshoot gets you a shell in a copy of the pod when the original has no tools (or no working structure) left.

Tip: One habit that saves hours: before any kubectl delete pod, run kubectl get pod X -o jsonpath=...ownerReferences first. Ten seconds of checking beats an afternoon of whack-a-mole. Aliasing it (k 'get pod' ...) pays for itself within a month.

If you want a throwaway cluster to practice this on, break a Job deliberately, watch it resurrect pods, then trace the owner. A $6 VPS running k3s is enough. Any provider will do (DigitalOcean, Vultr, Hetzner); one node is all this needs.

Checklist

Pod won’t delete → check ownerReferences before retrying the delete
Pod keeps coming back → owner is a Job / ReplicaSet / StatefulSet / Helm release; delete the owner
Read the Events section — the crash reason (missing ConfigMap, image pull error) is usually printed there
Terminating forever → check node health first, force-delete only when the node is confirmed dead
Helm-managed workload → helm uninstall instead of hand-deleting objects
Under GitOps → fix the manifest, don’t fight the reconciler

The takeaway

The delete button doesn’t delete. It asks, and the controller answers. Once you internalize that every stubborn pod has a parent that’s proud of its work, “How do I force this pod to die?” turns into the more useful question: “Who keeps bringing it back, and why does it keep failing?”

Related reading on this site: the Alpine DNS resolution problem — another case where the pod wasn’t the problem, its environment was. 中文原文：kubernetes 无法删除 pod 问题的解决。

Kubernetes Pods That Won’t Delete: Follow ownerReferences, Not the Force Flag