kubeadm alpha phase certs renew all should also update certs in KubeConfig files

@MalloZup
of course, but please note that the join phases are a high priority.

sounds good! thanks a lot.

Hi,

there is one more thing regarding this topic.

kubeadm alpha phase kubeconfig all shows this message if conf files are in place when issuing the command:

[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"

It does not check if certs are expired, so in my opinion up-to-date is missleading.

To get the updated certs into the files one MUST remove the files upfront, than the log looks like:

[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"   

In my case I though I'm fine, but a few days later to static pods couldn't communicate due to outdated certificates.

Best Regards
Andreas

Assigned to @MalloZup

@MalloZup: GitHub didn't allow me to assign the following users: MalloZup.

Note that only kubernetes members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

hi @adoerler thx for issue. Regarding the misleading info i have sent a PR https://github.com/kubernetes/kubernetes/pull/73798.

I will have a look on the rest of the issue once i have time. Thx for the time and precision of the issue

@adoerler i have sent a DOC pr for your suggestion. Feel free to have a look tia :rocket:
(https://github.com/kubernetes/website/pull/12579)

Hi @MalloZup,

thanks for PR!

I'm missing a sentence about the kubeconfig files, because certs renew is only one part of the game.
Something like:

After certs have been renewed, don't forget to recreate the KubeConfig files using kubeadm alpha phase kubeconfig ...

Thx. I didnt add the doc because I was thinking that actually we could renew Also kubeconfig files. The rest restarting pods we can delegate to user and write minimal doc. @fabriziopandini @lubomir @ereslibre I m missing something on this implementation ? Tia

@MalloZup I have not a deep knowledge of how certs renewal works.

Personally, I would like to clarify a little bit the overall history before taking actions - included what proposed above - :

• what should be managed by kubeadm alpha phase certs renew
• what should be managed automatically during kubeadm upgrade
• what should be documented (and mananged by the users)
• how this apply to HA clusters
• how this is impacted by cluster variants (like e.g. External etcd, External CA)
• etc.

but I leave final word to people more skilled than me in this area

i think we should reserve time on a meeting to discuss what our recommended certs renewal policy should be. the page about certs management might need some extra detail:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs

and we need to write a small guide, for single control plane clusters as a start at least.

what users have been doing is figuring things out on their own:
https://github.com/kubernetes/kubeadm/issues/581#issuecomment-421477139
^ this comment and one above contain user made guides.

this is a sign that we need to add an official guide.
cc @timothysc @liztio

/assign @ereslibre

Our cluster with a couple hundreds users is stuck at the moment. Could I have a very quick guide what to do with expired cert?

@dimm0

what users have been doing is figuring things out on their own:
#581 (comment)
^ this comment and one above contain user made guides.

these are the only guides we have ATM.

[root@controller0 ~]# kubeadm alpha phase certs apiserver --apiserver-advertise-address 1.2.3.4
Error: unknown flag: --apiserver-advertise-address
Usage:

Flags:
  -h, --help   help for phase

Global Flags:
      --log-file string   If non-empty, use this log file
      --rootfs string     [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers      If true, avoid header prefixes in the log messages
  -v, --v Level           log level for V logs

error: unknown flag: --apiserver-advertise-address
[root@controller0 ~]# kubeadm alpha phase certs apiserver
This command is not meant to be run on its own. See list of available subcommands.

in 1.13 init phases have graduated to to parent init command:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-certs

in 1.12 the flag should be there:
https://v1-12.docs.kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-certs

1.11 is soon going out of support.

removing the lifecycle/active label.
moving to 1.15.

possible docs update ideas here:
https://github.com/kubernetes/kubeadm/issues/1361#issuecomment-463192631

@neolit123
Question: in 1.14 with master HA, is it enough to follow the https://github.com/kubernetes/kubeadm/issues/581#issuecomment-421477139 on a single master, or we have to rejoin the secondary masters again in order to refetch the certs?

rejoining the secondary control plane nodes, seems like a quick and viable option in 1,14.
we don't have any docs yet in terms of HA cert rotations.
(not to mention we are yet to add proper steps like https://github.com/kubernetes/kubeadm/issues/581#issuecomment-421477139 did).

Doesn't --experimental-upload-certs provide the basis for an easier solution to certificate rotation in HA?

one way to do HA cert rotation is:

• on a single control-plane node follow the above mentioned steps to update it's certs
• on the same CP node call:

kubeadm init phase upload-certs --experimental-upload-certs

store the certs key.

kubeadm token create --print-join-command

store the join command with the token.

rejoin the rest of the control plane nodes using the token and certs key, one by one using --certs-key .... --experimental-control-plane-join

for the workers: drain, rejoin using the new token, uncordon, one by one.

optionally delete the resulted tokens.

@neolit123
In an 3 masters clusters, the moment we change the certs on the "primary" master, the etcd will stop working, as the certs are changed (and quorum has to be min 51%)? If so, maybe we have to somehow cordon the 2 secondary masters and only then change certs? Is "cordon master" possible?

I am not an expert here, but I don't think automatic cert copy should get into this picture

Automatic copy certificates handles CA, front-proxy-CA, etcd-CA (with 10 years TTL) and SA key (without TTL)

Cert renew command touch all the other certificates (with 1 year TTL), that are different across masters.
AFAIK, currently there is nothing that handles certificates renewal for kubeconfig files

ok, i did not consider what "certs copy" actually does here.
we need to write proper cert rotate docs, either way.

/assign
/lifecycle active

I'm starting to work on this issue.
There are different points to be addressed (_updated 14 May 2019_)

• Even with Certificate Rotation enabled, kubelet.conf points to outdated certs (already tracked by https://github.com/kubernetes/kubeadm/issues/1317)
• Certificate Rotation does not update apiserver/etcd/front-proxy-client certs (fixed by https://github.com/kubernetes/kubernetes/pull/76862)
• The Command kubeadm alpha phase certs renew all does not update KubeConfig files (fixed by https://github.com/kubernetes/kubernetes/pull/77180)
• Documentation about certs renewal (with more detail about where the command should be run, when, kubeconfig, HA)

And I will tackle all of them in separate PRs

@neolit123 @fabriziopandini
are the steps you mentioned also for rotating the CA cert? Can this be documented as well? What about rotating the private keys including the one for the CA?

@tushar00jain rotation of CA cert is tracked in another issue https://github.com/kubernetes/kubeadm/issues/1350
This issue focus on signed certs only

@fabriziopandini i was looking at closing this ticket today as you were able to send PRs for the renewal parts. should the ticket be closed?

Even with Certificate Rotation enabled, kubelet.conf points to outdated certs (already tracked by #1317)

yes this is tracked in a separate issue, possibly needs discussion/docs in terms of what workarounds we should provide.

Certificate Rotation does not update apiserver/etcd/front-proxy-client certs (fixed by kubernetes/kubernetes#76862)

The Command kubeadm alpha phase certs renew all does not update KubeConfig files (fixed by kubernetes/kubernetes#77180)

Documentation about certs renewal (with more detail about where the command should be run, when, kubeconfig, HA)

the 3 above should be done.

/close
As per comment above most of the work is already completed; the missing bit is tracked in a separate/dedicated issue

@fabriziopandini: Closing this issue.

Can someone please explain to me how was the "Even with Certificate Rotation enabled, kubelet.conf points to outdated certs" part addressed? The only issue linked that mentions this explicitly closed in favour of another issues which is closed with "I'm not sure if this an issue so open a new ticket if it is".
I'm on 1.16 do not see any renewal happening at kubelet.conf with sudo kubeadm alpha certs renew all. What am missing? @neolit123

a quick recap of a very very long discussion.

1. certificate rotation for all the certs but kubelet.conf are now managed by kubeadm alpha cert renew.
2. certificate rotation for kubelet.conf will be managed by kubelet itself (unless the user opts out from automatic certificate rotation)

This second point already as of today works for all the nodes except the one where you run kubeadm init; https://github.com/kubernetes/kubernetes/pull/84118 is going to fix that

@fabriziopandini Thank you for this, it makes sense.

For anyone else facing the issue of the certs in kubelte.conf being out of date between now and when the above is fixed I found this article helpful:

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration

On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

@AndrewSav Thank you for this. I have used the promethes operator to monitor the cluster. I recently received an alert "Kubernetes API certificate is expiring in less than 7 days", I think it is related to this issue. I have updated the content of kubelet.conf on the master nodes. But I still get the alert. Do you have any suggestions? Tks.

@tannh if you installed the cluster with kubeadm, use kubeadm to check the certs experation. Otherwise your issue is probably not related.

On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:

this will also be in the release notes for 1.17.

@adoerler I am still running old version of kubeadm, how can I update the kubelet.conf, admin.con, ... etc, after certificate renewal?

I ran "kubeadm alpha certs renew all", which generated new certificates, then I need to edit all .conf under /etc/kubernetes, how? where exactly they should point?
and in case of multi master nodes, should I run the command in all masters?

Hi @SuleimanWA ,

I cannot tell you what to do on a multi master env, I've had only single master in my setup.

This is what I've done:

First of all make sure to move existing conf files out of the way, because existing files will not get overwritten!

mv /etc/kubernetes/admin.conf /backup
mv /etc/kubernetes/kubelet.conf /backup
mv /etc/kubernetes/controller-manager.conf /backup
mv /etc/kubernetes/scheduler.conf /backup

then update these files:

user@master:~$ sudo kubeadm alpha phase kubeconfig all --apiserver-advertise-address=<INSERT-YOUR-APISERVER-IP-HERE>
I0124 21:56:14.253641   15040 version.go:236] remote version is much newer: v1.13.2; falling back to: stable-1.12
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"    

To apply the new certificates in the static system pods the easiest way for me was to simply reboot the master server.

Don't forget copy client-certificate-data and client-key-data from /etc/kubernetes/admin.conf to your local .kube/config.

Hope this helps

Andreas

Any idea how to run this command on 1.14.10? All I get is:

kubeadm alpha phase kubeconfig all --apiserver-advertise-address=192.168.102.170 Error: unknown flag: --apiserver-advertise-address

Then the docs say:
kubeadm alpha phase kubeconfig all
and I get:
This command is not meant to be run on its own. See list of available subcommands.

Thanks

Hi @provgregoryabdo,

whats your kubeadm version output?

BR Andreas

@provgregoryabdo the phase commands moved out of alpha and to init in later versions so you can use something like

kubeadm init phase kubeconfig all --apiserver-advertise-address=<your_address>

@adoerler thanks for the help!