kubernetes - Cert-Manager: renewing dns01 certificate not working

Question

We are using Cert-manager to manage the tls certifications for a website. The website's certificate expired yesterday, I tried to investigate why cert-manager was not doing its job.

I have checked the details of certificate fakename-io-cert, looks like cert-manager tried to renew the cert one month ago?:

$ kubectl describe cert/fakename-io-cert

Name:         fakename-io-cert
Namespace:    stage
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-10-28T17:10:08Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:commonName:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:privateKey:
          .:
          f:rotationPolicy:
        f:secretName:
    Manager:      kubectl
    Operation:    Update
    Time:         2020-10-28T17:10:08Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:nextPrivateKeySecretName:
        f:notAfter:
        f:notBefore:
        f:renewalTime:
        f:revision:
    Manager:         controller
    Operation:       Update
    Time:            2021-01-27T20:26:11Z
  Resource Version:  28153132
  Self Link:         /apis/cert-manager.io/v1/namespaces/stage/certificates/fakename-io-cert
  UID:               193717dd-0c00-43c5-8bde-5b7f981a5558
Spec:
  Common Name:  *.fakename.io
  Dns Names:
    fakename.io
    *.fakename.io
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-prod
  Private Key:
    Rotation Policy:  Always
  Secret Name:        cert-stage-wildcard
Status:
  Conditions:
    Last Transition Time:        2021-01-27T20:26:11Z
    Message:                     Certificate expired on Tue, 26 Jan 2021 16:10:10 UTC
    Reason:                      Expired
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2020-12-27T16:10:10Z
    Message:                     Renewing certificate as renewal was scheduled at 2020-12-27 16:10:10 +0000 UTC
    Reason:                      Renewing
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  fakename-io-cert-cjmpk
  Not After:                     2021-01-26T16:10:10Z
  Not Before:                    2020-10-28T16:10:10Z
  Renewal Time:                  2020-12-27T16:10:10Z
  Revision:                      1
Events:                          <none>

The following are the information of other related resources, such as certificaterequests, certificates, secrets:

$ kubectl get certificaterequests  
NAME                     READY   AGE
fakename-io-cert-8nxb6   False   31d
fakename-io-cert-k79kq   True    91d

$ kubectl get certificates
NAME               READY   SECRET                AGE
fakename-io-cert   False   cert-stage-wildcard   91d

$ kubectl get secrets
NAME                              TYPE                                  DATA   AGE
cert-stage-wildcard               kubernetes.io/tls                     2      91d
fakename-io-cert-cjmpk            Opaque                                1      31d

$ kubectl describe secrets/cert-stage-wildcard
Name:         cert-stage-wildcard
Namespace:    stage
Labels:       <none>
Annotations:  cert-manager.io/alt-names: *.fakename.io,fakename.io
              cert-manager.io/certificate-name: fakename-io-cert
              cert-manager.io/common-name: *.fakename.io
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: 
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-prod
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
tls.crt:  3570 bytes
tls.key:  1675 bytes

$ kubectl describe secrets/fakename-io-cert-cjmpk
Name:         fakename-io-cert-cjmpk
Namespace:    stage
Labels:       cert-manager.io/next-private-key=true
Annotations:  <none>

Type:  Opaque

Data
====
tls.key:  1700 bytes

And then the ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: issuer-account-key
    solvers:
      - selector:
          dnsNames:
          - fakename.io
        dns01:
          digitalocean:
            tokenSecretRef:
              name: digitalocean-dns
              key: do-access-token

Anyone has any idea of why this is not working?

UPDATE: The following logs are found in cert-manager's logs:

I0122 20:40:15.494843       1 reflector.go:207] Starting reflector *v1.Secret (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
I0122 20:40:15.495228       1 reflector.go:207] Starting reflector *v1.Pod (30s) from external/io_k8s_client_go/tools/cache/reflector.go:156
E0122 20:40:15.527258       1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io "fakename-io-cert-8nxb6" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="fakename-io-cert-8nxb6" "related_resource_namespace"="stage" "resource_kind"="Order" "resource_name"="fakename-io-cert-8nxb6-31268985" "resource_namespace"="stage" "resource_version"="v1" 
E0122 20:40:15.540473       1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io "fakename-io-cert-k79kq" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="fakename-io-cert-k79kq" "related_resource_namespace"="stage" "resource_kind"="Order" "resource_name"="fakename-io-cert-k79kq-31268985" "resource_namespace"="stage" "resource_version"="v1" 
E0122 20:40:17.801204       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="ACME client for issuer not initialised/available" "key"="stage/fakename-io-cert-k79kq-31268985" 
E0122 20:40:17.850290       1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="ACME client for issuer not initialised/available" "key"="stage/fakename-io-cert-8nxb6-31268985" 
I0122 20:40:17.917857       1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 
I0122 20:40:17.919532       1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1" 
E0122 20:40:23.180388       1 sync.go:110] cert-manager/controller/orders "msg"="Failed to determine the list of Challenge resources needed for the Order" "error"="no configured challenge solvers can be used for this challenge" "resource_kind"="Order" "resource_name"="fakename-io-cert-8nxb6-31268985" "resource_namespace"="stage" "resource_version"="v1" 
I0122 20:40:23.490123       1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1" 
I0122 20:40:23.502923       1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 

question from:https://stackoverflow.com/questions/65927833/cert-manager-renewing-dns01-certificate-not-working

Answer 1

Waitting for answers