kubernetes 无法删除 pod 问题的解决

[摘要] kubernetes 可能会产生垃圾或者僵尸pod，在删除rc的时候，相应的pod没有被删除，手动删除pod后会自动重新创建，这时一般需要先删除掉相关联的resources，实际中还要具体情况具体分析。

在使用Helm部署gitlab的时候发现有垃圾的pod存在，一直删除不掉

# kubectl get pods -n kube-system
NAME                            READY    STATUS    RESTARTS   AGE
gitlab-migrations.1-2km-4rwnm      1/1   Running       158    21h
gitlab-migrations.1-30d-gx9nz      1/1   Running       164    21h
gitlab-migrations.1-dsw-m5qbc      1/1   Running       167    21h

一般来说是要删除pod关联的rc，rs以及deployment等等，可以执行如下命令查询

kubectl get ClusterRole -n kube-system
kubectl get ClusterRoleBinding -n kube-system
kubectl get Deployments -n kube-system
kubectl get Pods -n kube-system
kubectl get ServiceAccounts -n kube-system
kubectl get CustomResourceDefinition -n kube-system
kubectl get replicasets -n kube-system
kubectl get replicationcontrollers -n kube-system
kubectl get services -n kube-system

找到对应的标签进行删除，对应的pod一般就会被删除掉，如果pod还在的话再手动删除pod。
实际中我这三个pod还是无法删除掉，删除之后仍然会重新启动，所以仔细看了下pod的具体描述内容。

# kubectl describe pod -n kube-system gitlab-migrations.1-30d-gx9nz
Name:           gitlab-migrations.1-30d-gx9nz
Namespace:      kube-system
Node:           172.30.0.169/172.30.0.169
Start Time:     Mon, 26 Mar 2018 11:53:45 +0800
Labels:         app=migrations
                controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209
                job-name=gitlab-migrations.1-30d
                release=gitlab
Annotations:    <none>
Status:         Running
IP:             172.100.14.7
Controlled By:  Job/gitlab-migrations.1-30d
Init Containers:
  configure:
    Container ID:  docker://a3051d24e65267489aa97ae368bcbe5819e4ae4f3b5ddc4d277b0b25ee585716
    Image:         busybox
    Image ID:      docker-pullable://docker.io/busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21
    Port:          <none>
    Command:
      sh
      /config/configure
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 26 Mar 2018 11:54:24 +0800
      Finished:     Mon, 26 Mar 2018 11:54:24 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /config from migrations-config (ro)
      /init-config from init-migrations-secrets (ro)
      /init-secrets from migrations-secrets (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)
Containers:
  migrations:
    Container ID:  docker://c33e809de76ba44da330063d9e3c6f58631b30e3c805ea561dc68af38ffe3926
    Image:         registry.gitlab.com/gitlab-org/build/cng/gitlab-rails:latest
    Image ID:      docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-rails@sha256:7dca5eb3b686342a9cff798f8349ad07459a1ef1e9d47a23b787b284d874cc17
    Port:          <none>
    Args:
      /scripts/wait-for-deps
      /scripts/db-migrate
    State:          Running
      Started:      Tue, 27 Mar 2018 09:50:09 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 27 Mar 2018 09:41:52 +0800
      Finished:     Tue, 27 Mar 2018 09:45:02 +0800
    Ready:          True
    Restart Count:  164
    Environment:
      CONFIG_TEMPLATE_DIRECTORY:  /var/opt/gitlab/templates
      CONFIG_DIRECTORY:           /var/opt/gitlab/config/gitlab/
      GITLAB_ROOT_PASSWORD:       Vi4ree2R
    Mounts:
      /etc/gitlab from migrations-secrets (ro)
      /home/git/gitlab/config/secrets.yml from migrations-secrets (rw)
      /var/opt/gitlab/templates from migrations-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  migrations-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gitlab-migrations
    Optional:  false
  init-migrations-secrets:
  <unknown>
  migrations-secrets:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory
  default-token-6rdp2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6rdp2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason       Age                  From                   Message
  ----     ------       ----                 ----                   -------
  Warning  FailedMount  6m (x655 over 21h)   kubelet, 172.30.0.169  MountVolume.SetUp failed for volume "migrations-config" : configmaps "gitlab-migrations" not found
  Warning  BackOff      3m (x3516 over 21h)  kubelet, 172.30.0.169  Back-off restarting failed container

需要关注的地方是Labels标签

Labels:         app=migrations
                controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209
                job-name=gitlab-migrations.1-30d
                release=gitlab

发现有个job-name=gitlab-migrations.xxx
于是继续查看kubernetes是否有job存在

# kubectl get jobs -n kube-system
NAME                      DESIRED   SUCCESSFUL   AGE
gitlab-migrations.1-2km   1         0            23h
gitlab-migrations.1-30d   1         0            23h
gitlab-migrations.1-dsw   1         0            23h

先执行删除job看下：

# kubectl delete job gitlab-migrations.1-2km -n kube-system
job "gitlab-migrations.1-2km" deleted
# kubectl delete job gitlab-migrations.1-30d -n kube-system
job "gitlab-migrations.1-30d" deleted
# kubectl delete job gitlab-migrations.1-dsw -n kube-system
job "gitlab-migrations.1-dsw" deleted

再看pod情况：

# kubectl get pods -n kube-system
NAME                               READY STATUS     RESTARTS  AGE 
gitlab-migrations.1-30d-gx9nz      1/1   Terminating   173    23h
gitlab-migrations.1-dsw-m5qbc      0/1   Terminating   176    23h

附图为kubernets 的一些对象资源：

再看pod已经被删除了，还有处于Terminating状态的，一会就会消失掉。至此pod删除成功，所以删除与pods相关联的依赖资源是解决问题的最终方法！

1 Comment