Kubeadm: use signed kubelet serving certificates

Created on 9 Nov 2018  ·  38Comments  ·  Source: kubernetes/kubeadm

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Opening the kubeadm side for this issue on the metrics-server

Versions

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Kubernetes version (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    Any
  • OS (e.g. from /etc/os-release):
    Any
  • Kernel (e.g. uname -a):
$ uname -a
Linux ip-172-31-1-118 4.15.0-1023-aws #23-Ubuntu SMP Mon Sep 24 16:31:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Others:

What happened?

kubeadm creates certs under /var/lib/kubelet/pki/kubelet.* signed with a different CA from the one under /etc/kubernetes/pki/ca.pem

What you expected to happen?

As a result some apps like the metrics-server cannot collect stats from a secured kubelet because the kubelet has certs signed by a different ca from the K8s master(s)

Error sample:

E1108 23:49:32.090084       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-x-x-x-x: unable to fetch metrics from Kubelet ip-x-x-x-x (ip-x-x-x-x): Get https://ip-x-x-x-x:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-x-x-x-x: unable to fetch metrics from Kubelet ip-x-x-x-x (ip-x-x-x-x): Get https://ip-x-x-x-x:10250/stats/summary/: x509: certificate is valid for x.x.x.x not ip-x-x-x-x]

How to reproduce it (as minimally and precisely as possible)?

Install the metrics-server on run:

$ kubectl -n kube-system logs

Anything else we need to know?

Some more background here

There also steps in there that I followed to fix the issue.


edit: neolit123

the problem here is that the serving cert is self-signed by default:
see https://github.com/kubernetes/website/pull/27071 for documentation update.

aresecurity help wanted kinbug kinfeature lifecyclfrozen prioritimportant-longterm

Most helpful comment

lets summarize the issue:

as outlined by @anitgandhi:
https://github.com/kubernetes/kubeadm/issues/1223#issuecomment-454572577

the problem with kubeadm here is that we are not passing a couple of flags to the kubelet:

--tls-cert-file=<some-path>/kubelet.crt
--tls-private-key-file=<some-path>/kubelet.key

without these flags the kubelet defaults to self-signing it's serving certificate when it first runs, which can be verified with:

sudo openssl verify -verbose -CAfile /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt

with a self-signed certificate instead of certificate signed by the cluster CA (/etc/kubernetes/ca.crt), deployments like the metrics server cannot scrape the kubelet, because the self-signed cert SAN will only include DNS:hostname.

possible solutions:
A) implement singing of a new kubelet.crt/key pair, ideally under /var/lib/kubelet/pki and setting the extra kubelet flags --tls-cert-file, --tls-private-key-file.

B) document means to enable this on demand similarly to how @raravena80 did it here: https://stackoverflow.com/questions/53212149/x509-certificate-signed-by-unknown-authority-kubeadm/53218524#53218524
except possibly using Kubernetes CSRs / kubeadm commands.

C) as commented by @alexbrand

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

D) ?

@kubernetes/sig-cluster-lifecycle
to me this seems in the space between bug/feature.

also see:
https://github.com/kubernetes/community/pull/602/files

All 38 comments

@raravena80 I'm not aware of any certificate created by kubeadm under /var/lib/kubelet/pki/.. could you kindly provide more info? e.g kubeadm config files, steps to create the cluster

@fabriziopandini I'm not entirely sure if the certs are created by the kubeadm, but the general procedure is described here.

This is what the content of the directory looks like:

root@ip-172-31-1-118:/var/lib/kubelet/pki# pwd
/var/lib/kubelet/pki
root@ip-172-31-1-118:/var/lib/kubelet/pki# ls -al
total 24
drwxr-xr-x 2 root root 4096 Jul 23 21:10 .
drwxr-xr-x 7 root root 4096 Nov 12 04:52 ..
-rw------- 1 root root 2810 Jul 23 21:09 kubelet-client-2018-07-23-21-09-53.pem
-rw------- 1 root root 1159 Jul 23 21:10 kubelet-client-2018-07-23-21-10-43.pem
lrwxrwxrwx 1 root root   59 Jul 23 21:10 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2018-07-23-21-10-43.pem
-rw-r--r-- 1 root root 1501 Nov  8 23:53 kubelet.crt
-rw------- 1 root root 1679 Nov  8 23:53 kubelet.key
root@ip-172-31-1-118:/var/lib/kubelet/pki#

Are they kubelet.crt and kubelet.key files created by the kubelet the first time it loads?

@raravena80 thanks for the clarification
Probably I don't have the full context here so I leave room to others for answering.

Only one side note (might be it can help)
Kubeadm already creates a certificate named apiserver-kubelet-client for letting the api server talking securely with the kubelets; it is signed by ca and bound to necessary RBAC rules.

/assign @liztio

I think this is to pre-generate kubelet's server certs. I tinkered with trying to use the Kubelet flags for TLS server bootstrap and rotate server certs, unfortunately I could not get Kubelet to request a server cert for itself using the bootstrap token. Kubelet ends up falling back to its default behavior for server certs, which is to generate a self-signed one.

To the best of my knowledge, right now the only way around that is to generate Kubelet's server certs out-of-band and place them at a deterministic path and kubelet (configured by kubeadm) will pick it up, and set some kubelet flags accordingly; reference: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates

The apiserver-kubelet-client is the client cert that API server will present to a kubelet, but kubelet is configured to trust clients that are signed by the k8s CA:

# cat /var/lib/kubelet/config.yaml 
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt

It's the kubelet's identity as a server being presented that needs to be signed by the k8s CA, which comes around to the original question.

There's also some relevant discussion at the end of this thread: https://github.com/kubernetes/kubeadm/issues/118

I think kubeadm may have to add a CSR approver for server cert requests with a valid bootstrap token, just like it does for client cert requests?

What about having the kubelet upload its self signed ca to a configmap somewhere? the nodeadmission plugin could restrict it to just its own configmap. metrics-server can use it to contact the node.

Any ideas about that?

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

@alexbrand I agree on that

kubelet TLS bootstrapping only generates client certificates for whatever reason:
--bootstrap-kubeconfig string
Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.

And kubeadm already does this. Perhaps this is a kubelet feature request?

lets summarize the issue:

as outlined by @anitgandhi:
https://github.com/kubernetes/kubeadm/issues/1223#issuecomment-454572577

the problem with kubeadm here is that we are not passing a couple of flags to the kubelet:

--tls-cert-file=<some-path>/kubelet.crt
--tls-private-key-file=<some-path>/kubelet.key

without these flags the kubelet defaults to self-signing it's serving certificate when it first runs, which can be verified with:

sudo openssl verify -verbose -CAfile /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt

with a self-signed certificate instead of certificate signed by the cluster CA (/etc/kubernetes/ca.crt), deployments like the metrics server cannot scrape the kubelet, because the self-signed cert SAN will only include DNS:hostname.

possible solutions:
A) implement singing of a new kubelet.crt/key pair, ideally under /var/lib/kubelet/pki and setting the extra kubelet flags --tls-cert-file, --tls-private-key-file.

B) document means to enable this on demand similarly to how @raravena80 did it here: https://stackoverflow.com/questions/53212149/x509-certificate-signed-by-unknown-authority-kubeadm/53218524#53218524
except possibly using Kubernetes CSRs / kubeadm commands.

C) as commented by @alexbrand

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

D) ?

@kubernetes/sig-cluster-lifecycle
to me this seems in the space between bug/feature.

also see:
https://github.com/kubernetes/community/pull/602/files

I think something between options B+C should be done since a lot of the bootstrap token client cert/CSR logic kubelet+kubeadm would have common logic to this.

lets summarize the issue:

as outlined by @anitgandhi:
#1223 (comment)

the problem with kubeadm here is that we are not passing a couple of flags to the kubelet:

--tls-cert-file=<some-path>/kubelet.crt
--tls-private-key-file=<some-path>/kubelet.key

without these flags the kubelet defaults to self-signing it's serving certificate when it first runs, which can be verified with:

sudo openssl verify -verbose -CAfile /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt

with a self-signed certificate instead of certificate signed by the cluster CA (/etc/kubernetes/ca.crt), deployments like the metrics server cannot scrape the kubelet, because the self-signed cert SAN will only include DNS:hostname.

possible solutions:
A) implement singing of a new kubelet.crt/key pair, ideally under /var/lib/kubelet/pki and setting the extra kubelet flags --tls-cert-file, --tls-private-key-file.

B) document means to enable this on demand similarly to how @raravena80 did it here: https://stackoverflow.com/questions/53212149/x509-certificate-signed-by-unknown-authority-kubeadm/53218524#53218524
except possibly using Kubernetes CSRs / kubeadm commands.

C) as commented by @alexbrand

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

D) ?

@kubernetes/sig-cluster-lifecycle
to me this seems in the space between bug/feature.

also see:
https://github.com/kubernetes/community/pull/602/files

great summary @neolit123 . Do you know if this will slip onto next cycle or work in progress as we speak ? Asking mainly because of the metrics-server which imo every deployment wants to have it ;)

@randomvariable mentioned that there is another workaround for that.
from the discussions thus far, we are hesitant about signing the kubelet-serving cert with the cluster CA. this topic needs further discussion.

/remove-help

because the solution to implement has not been chosen yet.

Any movement on this? I'm running up against it to support autoscaling features within a kubeadm deployed cluster.

Current workaround is to turn off CA checking of the kubelet certificate.

helm install --set 'args={--kubelet-insecure-tls}' --namespace kube-system metrics stable/metrics-serve

not really, it's blocked on design proposals.
there are a number of workarounds but the work of documenting those stalled:
https://github.com/kubernetes/kubeadm/issues/1602

--kubelet-insecure-tls

this might not be ideal for all users.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/lifecycle frozen

Running into this exact issue creating v1.18.2 cluster with kubeadm.

When setting up metrics-server it doesn't work without either setting it's kubelet-insecure-tls flag OR issue certificates for kublet "out of band", signing it with the kubernetes CA.

I thought about re-using the kubelet client certificate but that's of course issued to CN = system:node:nodename and no SANs. And I did test it though wich ofcourse changes the error to indicate just that. The same certificate could be used as both server/client if it had the nodename as a subject alternate name? But I'm guessing it'd be more proper to use separate certs for server/client?

/remove-lifecycle frozen

/lifecycle frozen

it's frozen so that the bots don't close the issue.

The same certificate could be used as both server/client if it had the nodename as a subject alternate name?

in theory and unless the kubelet validates them - e.g. "client cert must not have SANs".

But I'm guessing it'd be more proper to use separate certs for server/client?

it is the common practice to use them separate even in cases where it seems avoidable. it's unlikely that the kubelet/auth{z|n} maintainers would change this detail.

Hey. Did a little more diggin. Kubelet configuration option serverTLSBootstrap: true can actually create a CSR for the serving certificate. But it leaves it unapproved. Which may be ok?

Setting both rotateCertificates: true and serverTLSBootsrap: true and then approving the CSR for the serving certificate seem like the easiest way to go here. The serving certificate that is reqested/issued is for O = system:nodes, CN = system:node:<nodename> with Subject Alternate Names for DNS: <nodename>, IP Address: <node IP address>

Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?

Hey. Did a little more diggin. Kubelet configuration option serverTLSBootstrap: true can actually create a CSR for the serving certificate. But it leaves it unapproved. Which may be ok?

Setting both rotateCertificates: true and serverTLSBootsrap: true and then approving the CSR for the serving certificate seem like the easiest way to go here. The serving certificate that is reqested/issued is for O = system:nodes, CN = system:node:<nodename> with Subject Alternate Names for DNS: <nodename>, IP Address: <node IP address>

Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?

Not sure the security implementations but you can combine serverTLSBootstrap with this operator to auto-approve the CSRs https://github.com/kontena/kubelet-rubber-stamp

Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?

kubeadm cannot do the approval because kubeadm is not a daemon. it has to deploy a controller / operator that manages that for the user. maybe in the future.

the certificates API should go GA soon and hopefully we are going to have a better way to manage this in k8s. please watch:
https://github.com/kubernetes/enhancements/issues/267
(yet unclear to me what we are going to end up with...)

we also have alternatives ideas. but if all this is trying to solve the metric-server issue, you might as well just use https://github.com/brancz/kube-rbac-proxy that can perform SAR on the MS requests to the kubelet . sadly this is not yet documented on our side:
https://github.com/kubernetes/kubeadm/issues/1602

@neolit123 I at least started looking into it when trying to stand up metrics server on kubeadm and "the hard way" clusters for learning experience. Easiest way there was of course to flag MS with --kubelet-insecure-tls, but I really wanted to see how to fix it in a secure way and then just got interested in the issue. 🙂

For now it's easy enough for me to add the serverTLSbootstrap flag to the kubelet config and manually approve the certificates. I have noticed a downside of it though, which is that you can't fully interact with pods on the node until you approve the cert. (kubectl exec fails to run command on pods running on a node before approval for example)

I'll follow along the enhancements issue too. Thanks.

That's really sad that with kubeadm which seems mature enough the out of the box result for kubeletet cert is to be self signed & many people choose kubelet-insecure-tls for metrics server instead of doing things properly & etc :(

it's a complicated problem.

please try:
https://github.com/kontena/kubelet-rubber-stamp
or
https://github.com/brancz/kube-rbac-proxy
as workarounds

it's a complicated problem.

please try:
https://github.com/kontena/kubelet-rubber-stamp
or
https://github.com/brancz/kube-rbac-proxy
as workarounds

Actually https://github.com/kontena/kubelet-rubber-stamp works pretty good & imo seems to be a more correct solution instead of proxy.

Step 1:
Add
serverTLSBootstrap: true to the end of every /var/lib/kubelet/config.yaml for kubelets reconfiguration & don't forget to apply config (or just reboot them)

Step 2:
Deploy kubelet-rubber-stamp

service_account.yaml
role.yaml
role_binding.yaml
operator.yaml

Step 3:
Edit metrics-server deployment & remove --kubelet-insecure-tls

Result:

kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR          CONDITION
csr-7dvsx   31m   kubernetes.io/kubelet-serving   system:node:u-02   Approved,Issued
csr-d6rvm   31m   kubernetes.io/kubelet-serving   system:node:u-03   Approved,Issued
csr-szblz   31m   kubernetes.io/kubelet-serving   system:node:u-01   Approved,Issued
csr-zjfgj   31m   kubernetes.io/kubelet-serving   system:node:u-04   Approved,Issued

Hey, just to add to that @vainkop
During your initial kubeadm init to create the cluster you should also be able to pass in a KubeletConfiguration API object file to set the serverTLSBootstrap

```kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration

...

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true

`kubeadm init --config=kubeadm-config.yaml`

Then all kubelet's will automatically be set up using the `serverTLSBootstrap` flag.

To get the CSRs

kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-2qkdw 2m1s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:fcufbo Approved,Issued
csr-9wvgt 114s kubernetes.io/kubelet-serving system:node:worker-1 Pending
csr-lz97v 4m58s kubernetes.io/kubelet-serving system:node:master-1 Pending
csr-rsdsp 4m59s kubernetes.io/kube-apiserver-client-kubelet system:node:master-1 Approved,Issued
csr-wgxqs 4m49s kubernetes.io/kubelet-serving system:node:master-1 Pending

Then either approve them manually or deploy https://github.com/kontena/kubelet-rubber-stamp which approves them automatically. I just tried it with kubelet-rubber-stamp and it works great.

Also I did not seem to need to restart the kubelet's this way, they picked up their certificates as soon as I approvde the CSR, but a caveat is that the kublet's have NO cert until the CSR is approved, it does not get a self signed certificate first.

kubectl certificate approve csr-ab123 # OR deploy rubber-stamp!

kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-9wvgt 3m kubernetes.io/kubelet-serving system:node:worker-1 Approved,Issued
...
```

Another strange thing seems to happen here btw, which is that the master node seems to create it’s CSR twice. (At least the two times I tried this)

But as @nijave says in a comment above, I'm not sure what the security implications of using the rubber-stamper are.

@allir, @vainkop as far is i can see the kubelet-rubber-stamp does only verify if the common name of the CSR matches the requester name but does not verify whether the additional hostnames and IP Addresses requested by the kubelet are valid. This means an attacker that has access to the kubelet client certificate can create certificates for basically any domain name or IP address. All clients that are configured to trust the root CA will then accept this certificate.
Of course validating the hostname and IP addresses that are valid for a given kubelet is hard since there is currently no authority that can confirm what a kubelet is allowed to request. For example using the node object on the API server is not sufficient because kubelets can update the object without limits.

Hey, just to add to that @vainkop
During your initial kubeadm init to create the cluster you should also be able to pass in a KubeletConfiguration API object file to set the serverTLSBootstrap
kubeadm init --config=kubeadm-config.yaml
Then all kubelet's will automatically be set up using the serverTLSBootstrap flag.

Or for an existing K8s setup using Ansible it can be:

  tasks:
    - name: Insert a line at the end of /var/lib/kubelet/config.yaml
      lineinfile:
        path: /var/lib/kubelet/config.yaml
        line: 'serverTLSBootstrap: true'

+restart kubelets

Wow I'm so glad that I found this issue, and I'm not alone who wants to make this proper way. :)

Now let me share my thoughts on this issue (please correct me if I'm somewhere wrong):

First my vision of the original problem:
Currently kubeadm enables webhook authentication for all kubelets by default, so kubelet is validating client certificates for incoming connections with no problem even if --kubelet-insecure-tls option is specified.
From the other side metrics-server have no oportunity to verify specific kubelet certificate because it is self-signed on the node.

Possible risks of using --kubelet-insecure-tls for metrics-server:
While the kubelet data is somewhat secured and will never be provided to the metrics-server without successful webhook authentification.
In theory someone can compromise the server IP or hostname and provide wrong statistics. But for establishing the connections metricserver is using an IP address and hostnames specified for node via kube-apiserver, so the attacker needs to hack the API-server, DNS or the node IP-address first.

Little observation:
The metrics-server is not single service which is accessing the kubelets directly. Kube-apiserver also doing this for read the container logs or execute the shell on them. The good question is how kube-apiserver makes sure that it establishing connection with the specific kubelet while it has no any information about CA that issued it's certificate?
Doesn't it behave the same as the metrics server with the --kubelet-insecure-tls option in this case?

Possible solution:
Nowdays the webhooks and API aggregation are quite popular in Kubernetes. All of them are behave the similar way, by generating it's own CA and crt/key pair. The CA hash is also stored in a specific resource to provide kube-apiserver with information about which certificate it can trust.

For example:

  • APIServices are storing related CA hash in their apiservices.apiregistration.k8s.io resource:

    spec:
    caBundle: <ca-hash>
    
  • Webhooks are storing related CA hash in their validatingwebhookconfigurations.admissionregistration.k8s.io and mutatingwebhookconfigurations.admissionregistration.k8s.io resources:

    webhooks:
    - clientConfig:
      caBundle: <ca-hash>
    

To me it is pretty obvious that each node resource should have similar caBundle in their spec, where kubelets can register their own CA for serving using their client certificate:

spec:
  caBundle: <ca-hash>

Both metris-server and kube-apiserver should use these certificates to verify and trust the connection to the kubelets.

thanks to @kfox1111 who expressed a similar idea previously https://github.com/kubernetes/kubeadm/issues/1223#issuecomment-460854312

The good question is how kube-apiserver makes sure that it establishing connection with the specific kubelet while it has no any information about CA that issued it's certificate?
Doesn't it behave the same as the metrics server with the --kubelet-insecure-tls option in this case?

To answer this question I can quote @luxas here:

Right, we can't make the connections from the api server to the kubelet servers verified, as each kubelet has its own self-signed cert. We might consider a manual approving flow wrt kubelet serving certs in the future but that's not secured by default at the moment.

from https://github.com/kubernetes/kubeadm/issues/118#issuecomment-407498529

hope it can be resolved some day

[root@jenkins metrics-server]# kubectl -n kube-system logs -f metrics-server-6955d88db9-lftlz
I1120 08:23:09.094132       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1120 08:23:09.094234       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1120 08:23:09.094270       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:09.094279       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:09.094307       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:09.094315       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:09.095064       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1120 08:23:09.095207       1 secure_serving.go:197] Serving securely on [::]:4443
I1120 08:23:09.095259       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1120 08:23:09.194453       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I1120 08:23:09.194660       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I1120 08:23:09.194455       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
E1120 08:23:10.420643       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node k8s-master3: unable to fetch metrics from node k8s-master3: Get "https://10.39.140.250:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.250 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master1: unable to fetch metrics from node k8s-master1: Get "https://10.39.140.248:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.248 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master2: unable to fetch metrics from node k8s-master2: Get "https://10.39.140.249:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.249 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-node1: unable to fetch metrics from node k8s-node1: Get "https://10.39.140.251:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.251 because it doesn't contain any IP SANs]
I1120 08:23:33.874949       1 requestheader_controller.go:183] Shutting down RequestHeaderAuthRequestController
I1120 08:23:33.874978       1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:33.874993       1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:33.875019       1 tlsconfig.go:255] Shutting down DynamicServingCertificateController
I1120 08:23:33.875026       1 dynamic_serving_content.go:145] Shutting down serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1120 08:23:33.875041       1 secure_serving.go:241] Stopped listening on [::]:4443

No news from this issue? I would be interested to have a solution for this as well.

we are documenting workarounds here:
https://github.com/kubernetes/website/pull/27071
https://github.com/kubernetes/kubeadm/issues/1602

we can keep this issue open, but due to the complexities of requiring to deploy a signer with kubeadm by default, it's unlikely that we are going to make this change anytime soon.

Was this page helpful?
0 / 5 - 0 ratings