How to renew certificates on Kubernetes?

Question

First of all, this is not a duplicated question, as I will describe below I have tried almost every solution that I have found and it didn't work.

My environment:

• Single master K8s cluster running on bare metal
• Host OS: Ubuntu 18.04
• K8s version 1.16

Yesterday I found out that kubectl commands are not working on our k8s cluster and after some digging, I found out that the reason is our certificates have been expired after 1 year. Based on K8s documentation (https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/) you can renew your certificates manually by running kubeadm alpha certs renew which I did but still kubectl was not able to connect to the cluster but the cluster was up and pods were working.

Then I found this answer https://stackoverflow.com/a/56334732/5553963 which creates new certificates and config files. After following that kubectl started to work but kubelet on the master node went down and it said that it cannot authenticate and all nodes became Not Ready.

After digging even more in the kubernetes github issues I found a solution that deleting the /var/lib/kubelet/pki certificates might help (in addition to editing the /etc/kubernetes/kubelet.conf) and after deleting those certificates and when they got recreated automatically again kubelet was able to start and master node became ready.

Then I had to find out how to renew the certificates for the workers but I couldn't find anything that directly was a solution for workers so I created a new join token with below command on the master and I ran it on the worker nodes but I got below error:

# on Master
kubeadm token create --print-join-command  --ttl=0

# on worker
$ sudo kubeadm join 10.203.20.160:6443 --token fecjk7.ni262bvxx8zwmn5k     --discovery-token-ca-cert-hash sha256:02c76ff08c29a6d65ec580f8988a21ac75f77cc3ea655615e02d7331cf5ffb34 --v=5

I0127 14:44:57.968121   13747 token.go:199] [discovery] Trying to connect to API Server "10.203.20.160:6443"
I0127 14:44:57.970371   13747 token.go:74] [discovery] Created cluster-info discovery client, requesting info from "https://10.203.20.160:6443"
I0127 14:44:57.973778   13747 token.go:202] [discovery] Failed to connect to API Server "10.203.20.160:6443": cluster CA found in cluster-info configmap is invalid: none of the public keys "sha256:1e202253d5bc97848fdd3a3d3a00961bcce7f29b86d494a248ec953e7bfab73e" are pinned
I0127 14:45:02.973962   13747 token.go:199] [discovery] Trying to connect to API Server "10.203.20.160:6443"
I0127 14:45:02.974873   13747 token.go:74] [discovery] Created cluster-info discovery client, requesting info from "https://10.203.20.160:6443"
I0127 14:45:02.978465   13747 token.go:202] [discovery] Failed to connect to API Server "10.203.20.160:6443": cluster CA found in cluster-info configmap is invalid: none of the public keys "sha256:1e202253d5bc97848fdd3a3d3a00961bcce7f29b86d494a248ec953e7bfab73e" are pinned

# on master checking the public key again
$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
02c76ff08c29a6d65ec580f8988a21ac75f77cc3ea655615e02d7331cf5ffb34

I double-checked the published key of that token on the master again, rebooted the worker, deleted the kubectl and kubeadm and everything else that I could think of but with no success. Now I just gave up and we need the cluster so I want to reinstall the K8s again but since we have another cluster that its certificates will be expired in 22 days I have some questions:

1. Can someone please write all the steps on how to renew the certificates?
2. We just created a cluster on AWS EKS to test it, how EKS manages the certificates and how we shall renew the certificates on EKS?

question from:https://stackoverflow.com/questions/65921084/how-to-renew-certificates-on-kubernetes

Answer 1

Waitting for answers