kubernetes - Pod staying in Pending state

Question

I have a kubernetes pod that is staying in Pending state. When I describe the pod, I am not seeing why it fails to start, I can just see Back-off restarting failed container.

This is what I can see when I describe the pod.

kubectl describe po jenkins-68d5474964-slpkj -n infrastructure

Name:               jenkins-68d5474964-slpkj
Namespace:          infrastructure
Priority:           0
PriorityClassName:  <none>
Node:               ip-172-20-120-29.eu-west-1.compute.internal/172.20.120.29
Start Time:         Fri, 05 Feb 2021 17:10:34 +0100
Labels:             app=jenkins
                    chart=jenkins-0.35.0
                    component=jenkins-jenkins-master
                    heritage=Tiller
                    pod-template-hash=2481030520
                    release=jenkins
Annotations:        checksum/config=fc546aa316b7bb9bd6a7cbeb69562ca9f224dbfe53973411f97fea27e90cd4d7
Status:             Pending
IP:                 100.125.247.153
Controlled By:      ReplicaSet/jenkins-68d5474964
Init Containers:
  copy-default-config:
    Container ID:  docker://a6ce91864c181d4fc851afdd4a6dc2258c23e75bbed6981fe1cafad74a764ff2
    Image:         jenkins/jenkins:2.248
    Image ID:      docker-pullable://jenkins/jenkins@sha256:352f10079331b1e63c170b6f4b5dc5e2367728f0da00b6ad34424b2b2476426a
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      /var/jenkins_config/apply_config.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 05 Feb 2021 17:15:16 +0100
      Finished:     Fri, 05 Feb 2021 17:15:36 +0100
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     2560m
      memory:  2Gi
    Requests:
      cpu:     50m
      memory:  256Mi
    Environment:
      ADMIN_PASSWORD:  <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: false
      ADMIN_USER:      <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: false
    Mounts:
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (rw)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_plugins from plugin-dir (rw)
      /var/run/docker.sock from docker-sock (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5tbbb (rw)
Containers:
  jenkins:
    Container ID:  
    Image:         jenkins/jenkins:2.248
    Image ID:      
    Ports:         8080/TCP, 50000/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
      --argumentsRealm.roles.$(ADMIN_USER)=admin
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2560m
      memory:  2Gi
    Requests:
      cpu:     50m
      memory:  256Mi
    Environment:
      JAVA_OPTS:                 
      JENKINS_OPTS:              
      JENKINS_SLAVE_AGENT_PORT:  50000
      ADMIN_PASSWORD:            <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: false
      ADMIN_USER:                <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: false
    Mounts:
      /usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (ro)
      /var/jenkins_home from jenkins-home (rw)
      /var/run/docker.sock from docker-sock (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5tbbb (rw)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  jenkins-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      jenkins
    Optional:  false
  plugin-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  secrets-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins
    ReadOnly:   false
  default-token-5tbbb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5tbbb
    Optional:    false
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  
QoS Class:         Burstable
Node-Selectors:    nodePool=ci
Tolerations:       node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                                                  Message
  ----     ------     ----              ----                                                  -------
  Normal   Scheduled  7m                default-scheduler                                     Successfully assigned infrastructure/jenkins-68d5474964-slpkj to ip-172-20-120-29.eu-west-1.compute.internal
  Normal   Started    5m (x4 over 7m)   kubelet, ip-172-20-120-29.eu-west-1.compute.internal  Started container
  Normal   Pulling    4m (x5 over 7m)   kubelet, ip-172-20-120-29.eu-west-1.compute.internal  pulling image "jenkins/jenkins:2.248"
  Normal   Pulled     4m (x5 over 7m)   kubelet, ip-172-20-120-29.eu-west-1.compute.internal  Successfully pulled image "jenkins/jenkins:2.248"
  Normal   Created    4m (x5 over 7m)   kubelet, ip-172-20-120-29.eu-west-1.compute.internal  Created container
  Warning  BackOff    2m (x14 over 6m)  kubelet, ip-172-20-120-29.eu-west-1.compute.internal  Back-off restarting failed container

Once I run helm upgrade for that container, I can see:

RESOURCES:
==> v1/ConfigMap
NAME             DATA  AGE
jenkins          5     441d
jenkins-configs  1     441d
jenkins-tests    1     441d

==> v1/Deployment
NAME     READY  UP-TO-DATE  AVAILABLE  AGE
jenkins  0/1    1           0          441d

==> v1/PersistentVolumeClaim
NAME     STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
jenkins  Bound   pvc-8813319f-0d37-11ea-9864-0a7b1d347c8a  4Gi       RWO           aws-efs       441d

==> v1/Pod(related)
NAME                      READY  STATUS    RESTARTS  AGE
jenkins-7b85495f65-2w5mv  0/1    Init:0/1  3         2m9s

==> v1/Secret
NAME             TYPE    DATA  AGE
jenkins          Opaque  2     441d
jenkins-secrets  Opaque  3     441d

==> v1/Service
NAME           TYPE          CLUSTER-IP     EXTERNAL-IP       PORT(S)         AGE
jenkins        LoadBalancer  100.65.2.235   a881a20a40d37...  8080:31962/TCP  441d
jenkins-agent  ClusterIP     100.64.69.113  <none>            50000/TCP       441d

==> v1/ServiceAccount
NAME     SECRETS  AGE
jenkins  1        441d

==> v1beta1/ClusterRoleBinding
NAME                  AGE
jenkins-role-binding  441d

Can someone advice?

question from:https://stackoverflow.com/questions/66066843/pod-staying-in-pending-state

Answer 1

Now you cannot get any logs by kubectl logs pod_name because the pod status is initializing. When you use kubectl logs command;

• If the pod has multiple containers, you have to specify the container name explicitly.
• If you have only one container, then no need to specify the container name.
• If you want to get logs of initContainers, you need to specify the initContainer name.

For your case, the pod has one init container and seems it stuck now.

Init Containers:
  copy-default-config:
    Command:
      sh
      /var/jenkins_config/apply_config.sh

You can check the log of this container.

kubectl logs jenkins-68d5474964-slpkj copy-default-config