<p>kubeadm init tidak mengkonfigurasi RBAC untuk configmaps dengan benar</p>

Dibuat pada 9 Jun 2018 · 17Komentar · Sumber: kubernetes/kubeadm

Apakah ini LAPORAN BUG atau PERMINTAAN FITUR?

LAPORAN BUG

Versi

versi kubeadm (gunakan kubeadm version ): "v1.12.0-alpha.0.957 + 1235adac3802fd-dirty"

Apa yang terjadi?

Saya membuat node bidang kontrol dengan kubeadm init . Saya menjalankan kubeadm join pada node terpisah dan mendapatkan pesan kesalahan ini:

[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace                                                                        
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:4ipkdk" cannot get configmaps in the namespace "kube-system"

Apa yang Anda harapkan terjadi?

Saya berharap kubeadm join berhasil menyelesaikannya

Bagaimana cara memperbanyaknya (seminimal dan setepat mungkin)?

Sejauh yang saya tahu, jalankan kubeadm init dan kubeadm join pada node lain. Saya memiliki banyak kode tambahan / yaml yang seharusnya tidak memengaruhi peta konfigurasi (diperlukan untuk penerapan yang menyenangkan). Tetapi jika ini ternyata tidak dapat direproduksi, saya akan memberikan instruksi yang lebih mendalam.

Ada hal lain yang perlu kami ketahui?

Saya pikir juga kubeadm join dan kubeadm init menamai peta konfigurasi secara tidak konsisten. Perintah init menggunakan kubernetesVersion ditentukan dalam file konfigurasi dan perintah join menggunakan versi kubelet untuk nama peta konfigurasi (misalnya kubelet-config-1.1). Ini baik-baik saja kecuali Anda memiliki versi yang tidak cocok maka tidak masalah.

Perintah init membuat aturan RBAC untuk akses anonim ke peta konfigurasi di namespace kube-public tetapi tampaknya tidak meletakkan konfigurasi kubelet di namespace publik dan dengan demikian node yang bergabung tidak memiliki akses ke saya t.

kinbug prioritimportant-longterm

Sumber

chuckha

Komentar yang paling membantu

Ini baik-baik saja kecuali Anda memiliki versi yang tidak cocok maka tidak masalah.

text

neolit123 pada 9 Jun 2018

😄22 👍7

Semua 17 komentar

Ini baik-baik saja kecuali Anda memiliki versi yang tidak cocok maka tidak masalah.

text

neolit123 pada 9 Jun 2018

😄22 👍7

@chuckha Jika saya ingat dengan baik kubelet-* config maps harus kube-system dan kubeadm membuat aturan untuk mengizinkan akses ke token dan node bootstrap. Tapi saya akan memeriksa ini lagi setelah putaran terakhir perubahan

fabriziopandini pada 11 Jun 2018

@chuckha Saya berhasil bergabung dengan cluster dengan semua komponen yang dibangun dari master + nomor rilis dipaksa ke v1.11.0

kubelet-config-1.11 dibuat di kube-system
peran kubeadm:kubelet-config-1.11 dibuat di kube-system dengan mendapatkan izin di peta konfigurasi
rolebinding kubeadm:kubelet-config-1.11 dibuat di kube-system untuk system:nodes dan system:bootstrappers:kubeadm:default-node-token

Jadi IMO:

RBAC dibuat
jika kita berada dalam skenario "versi konsisten", semuanya bekerja

Bagian yang masih harus diselidiki adalah

Perintah init menggunakan kubernetesVersion yang ditentukan dalam file config dan perintah join menggunakan versi kubelet untuk nama config map (misalnya kubelet-config-1.1)

fabriziopandini pada 11 Jun 2018

👍2

Ya, ini pasti masalah versi.

Saya tidak mengatur versinya ketika saya membangun sehingga semua binari mengira mereka 1.12.0+ tetapi saya menginstal dan memaksa kubeadm untuk menggunakan v1.11.

Ini menghasilkan

root@ip-10-0-0-7:~# k get cm -n kube-system
NAME                                 DATA      AGE
...
kubelet-config-1.11                  1         32m

dan kemudian saat bergabung:

[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace                                                                        
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:u3ns6m" cannot get configmaps in the namespace "kube-system"

Saya akan membangun kembali binari dan memaksanya ke versi yang benar dan mencoba lagi.

chuckha pada 11 Jun 2018

Semuanya baik-baik saja jika versi kubelet dan kubeadm cocok. Menutup demi perbaikan yang tidak terlalu mendesak terkait inkonsistensi (mungkin disengaja) antara menulis config map dan mengambil configmap.

chuckha pada 11 Jun 2018

Saya tidak menggunakan kubeadm init, tetapi memanggil fase secara individual. Tidak ada configmap di kube-system atau izin untuk mengaturnya dengan benar. Fase apa ini?

drewwells pada 9 Agu 2018

@drewwells Saya mengalami masalah yang sama seperti yang Anda alami. Saya menjalankan fase secara individual, dan tidak ada konfigurasi mpas.

sudo kubectl get cm -n kube-system --kubeconfig=/etc/kubernetes/admin.conf
NAME                                 DATA   AGE
calico-config                        2      11m
coredns                              1      15m
extension-apiserver-authentication   6      15m
kube-proxy                           2      15m

Sudahkah Anda menemukan solusinya?

Saya juga harus menyebutkan, semua komponen adalah 1.11.4

Ini lebih jauh. Saya mem-bootstrap cluster dengan kubeadm init, dan saya memiliki peta konfigurasi yang benar sekarang:

ubuntu@master-1-test2:~$ sudo kubectl get cm -n kube-system --kubeconfig=/etc/kubernetes/admin.conf
NAME                                 DATA      AGE
coredns                              1         41m
extension-apiserver-authentication   6         41m
kube-proxy                           2         41m
kubeadm-config                       1         41m
kubelet-config-1.11                  1         41m
ubuntu@master-1-test2:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.4", GitCommit:"bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b", GitTreeState:"clean", BuildDate:"2018-10-25T19:13:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Di node:

$ sudo kubeadm -v=10 join k8s.oz.noris.de:6443 --token 36etul.nv5lz6hjfifdd4c9 --discovery-token-unsafe-skip-ca-verification                                                                                             I1107 12:57:01.340638    3631 join.go:226] [join] found NodeName empty
I1107 12:57:01.340816    3631 join.go:227] [join] considered OS hostname as NodeName
[preflight] running pre-flight checks
I1107 12:57:01.341152    3631 join.go:238] [preflight] running various checks on all nodes
I1107 12:57:01.341265    3631 checks.go:253] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1107 12:57:01.341677    3631 checks.go:291] validating the existence of file /etc/kubernetes/pki/ca.crt
I1107 12:57:01.341774    3631 checks.go:291] validating the existence of file /etc/kubernetes/kubelet.conf
I1107 12:57:01.341857    3631 checks.go:291] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1107 12:57:01.341947    3631 kernelcheck_linux.go:45] validating the kernel module IPVS required exists in machine or not
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs:{} ip_
vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I1107 12:57:01.349448    3631 checks.go:138] validating if the service is enabled and active
I1107 12:57:01.361957    3631 checks.go:340] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1107 12:57:01.362034    3631 checks.go:340] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1107 12:57:01.362076    3631 checks.go:653] validating whether swap is enabled or not
I1107 12:57:01.362134    3631 checks.go:381] validating the presence of executable crictl
I1107 12:57:01.362204    3631 checks.go:381] validating the presence of executable ip
I1107 12:57:01.362244    3631 checks.go:381] validating the presence of executable iptables
I1107 12:57:01.362281    3631 checks.go:381] validating the presence of executable mount
I1107 12:57:01.362320    3631 checks.go:381] validating the presence of executable nsenter
...
[discovery] Trying to connect to API Server "mycluster.example.com:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://mycluster.example.com:6443"
I1107 12:57:01.487256    3631 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.4 (linux/amd64) kubernetes/bf9a868" 'https://mycluster.example.com:6443/api/v1/namespaces/kube-public/config
maps/cluster-info'
I1107 12:57:01.504539    3631 round_trippers.go:405] GET https://mycluster.example.com:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 200 OK in 17 milliseconds
I1107 12:57:01.504720    3631 round_trippers.go:411] Response Headers:
I1107 12:57:01.504818    3631 round_trippers.go:414]     Content-Type: application/json
I1107 12:57:01.504914    3631 round_trippers.go:414]     Content-Length: 2217
I1107 12:57:01.505003    3631 round_trippers.go:414]     Date: Wed, 07 Nov 2018 12:57:01 GMT
I1107 12:57:01.505174    3631 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"kube-public","selfLink":"/api/v1/namespaces/kube-public/configmaps/cluster-info","uid":"97e9
a3d1-e286-11e8-9272-fa163ef9e3af","resourceVersion":"425","creationTimestamp":"2018-11-07T12:13:53Z"},"data":{"jws-kubeconfig-36etul":"eyJhbGciOiJIUzI1NiIsImtpZCI6IjM2ZXR1bCJ9..zRgexonkjOpLJS0q3IignURwTcpBuQy7gv35Qhhsl_k","jws-kubeconfig-
eth6o8":"eyJhbGciOiJIUzI1NiIsImtpZCI6ImV0aDZvOCJ9..kWj4cI2j1WgKfNG07IGiIij4CSb9kWUbaM2mixlYThY","jws-kubeconfig-rbxd02":"eyJhbGciOiJIUzI1NiIsImtpZCI6InJieGQwMiJ9..HwIWDwfIbAjNM1EGbWdXYOhC8z1MxgwuzhjlJRaZ_pc","kubeconfig":"apiVersion: v1\n
clusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNE1URXdNekEwTlRrME4xb1hEVEk0TVRBek1UQTB
OVGswTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTU9BCkNHQU5jUjVRQWV3MlljY2V0eWVyYktiODd4RWRPVlp2aUdneElrbkpKTTZwZFVBbzMwSWVxckRqSnlFaTFVeDcKU0c5NS9sRlBqU1htdHhhNHMvc1g1KzNTVW4zZ
EtFRWw5TFhXa0lzeTRJYzRFUTMwWE9WcnNuYTYwN1UzNmQyaAp3NHdTK1dveE5QR3dqZDM2bXQzMFR4bUluYk54ZVl5d2NnVU1tMlZFZXM4dGhVaVhZMXB1N1Y2SUNCY243cE9NCkdoT2xlRXg4SmlEVnhuSGlpSm9oYytCbGNIdHdLU1pzK2cvZUhwdGdlSDdaQlZNRC8zZVFvZXVsUGVvTEkwamEKc09jTENMTkpEVVB
LUWJqRnRNbkFZSXVvOENHSXpFTzBDaDZNeW5vb1pTL0E0bEs1MXJmTVdkTkZ4N0dVdnQxYQo5KzZzMHo2NEpHeVFBdmtBcWhVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEbWs1NEYzZ1BqOS91NzlRbTg2V1Mzc
k5YaFoKZG16Wmt3TXRDajRuTXdsSndGQy9iZUU4ZUdsWnFxWDcrdEpYUDVaY0xLNE1pSnM1U2JTMjd5NDF3WTRRTFFWaQpVWmRocEFHUTBOSlpHSGhWMTVDczlVQTA1ZTFNajNCaHZ6SG5VV2t1ZUhYbW84VmI4SkI5RGloeGdiUW5GY2FQCjRWcVhWY0pBemxVQ0V5aXhreVRGendZTklJbzJHdGtCdlI1YkxCM0doT2R
sQURmQzEwdzgvTmQveFFmRnRWdmYKL3lHaktpbW8rT2xERkV5YittcHVKMVdiN3Y3bnJJSzlSSy9WbVhUWENiOWZLQ3BmQ0hMU0hpa0lEWklZK0wxTQpwbWpXYXZFcjFLSlE5UEJIYmdZSHkxK1F0bkpXRDNjNnJrOUtoNU1zMFhTVmpBc2Z1RWdXaG9CYnlVdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=\n   
 server: https://k8s.oz.noris.de:6443\n  name: \"\"\ncontexts: []\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: []\n"}}
[discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "k8s.oz.noris.de:6443"
[discovery] Successfully established connection with API Server "mycluster.example.com:6443"
I1107 12:57:01.509945    3631 join.go:260] [join] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1107 12:57:01.617006    3631 loader.go:359] Config loaded from file /etc/kubernetes/bootstrap-kubelet.conf
I1107 12:57:01.617871    3631 join.go:283] Stopping the kubelet
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
I1107 12:57:01.627838    3631 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.4 (linux/amd64) kubernetes/bf9a868" -H "Authorization: Bearer 36etul.nv5lz6hjfifdd4c9" 'https://mycluster.example.com:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.12'
I1107 12:57:01.639396    3631 round_trippers.go:405] GET https://mycluster.example.com:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.12 403 Forbidden in 11 milliseconds
I1107 12:57:01.639427    3631 round_trippers.go:411] Response Headers:
I1107 12:57:01.639443    3631 round_trippers.go:414]     Content-Length: 311
I1107 12:57:01.639464    3631 round_trippers.go:414]     Date: Wed, 07 Nov 2018 12:57:01 GMT
I1107 12:57:01.639477    3631 round_trippers.go:414]     Content-Type: application/json
I1107 12:57:01.639492    3631 round_trippers.go:414]     X-Content-Type-Options: nosniff
I1107 12:57:01.639525    3631 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubelet-config-1.12\" is forbidden: User \"system:bootstrap:36etul\" cannot get confi
gmaps in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kubelet-config-1.12","kind":"configmaps"},"code":403}
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:36etul" cannot get configmaps in the namespace "kube-system"

perhatikan hal-hal yang melelahkan?

Rupanya, itu semua akan terjadi, jika Anda memeriksa versi kubelet! Saya telah menggunakan kubelet versi 1.12.2.
Saya mendapat petunjuk dari kode

oz123 pada 7 Nov 2018

Saya mengalami masalah yang sama, kecuali satu versi yang lebih baru.

Di master:

$ lsb_release -d
Description:    Ubuntu 16.04.5 LTS
$ dpkg -l | grep kub
ii  kubeadm                          1.12.1-00                                  amd64        Kubernetes Cluster Bootstrapping Tool
ii  kubectl                          1.12.1-00                                  amd64        Kubernetes Command Line Tool
ii  kubelet                          1.12.1-00                                  amd64        Kubernetes Node Agent
ii  kubernetes-cni                   0.6.0-00                                   amd64        Kubernetes CNI
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Apa, dari mana server versi 1.13.0 itu berasal? Saya tidak menginstal itu.

Bagaimanapun, sebelum saya menjalankan kubeadm init di VM ini, saya mengkloningnya, jadi saya memiliki satu lagi yang siap untuk menjadi node lain di cluster. Karena itu klon, ia juga memiliki 1.12.1. Ketika saya mencoba untuk bergabung:

$ kubeadm join --token blahblah 10.138.0.3:6443 --discovery-token-ca-cert-hash sha256:deadbeefdeadbeefetc 
[preflight] running pre-flight checks
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
[discovery] Trying to connect to API Server "10.138.0.3:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.138.0.3:6443"
[discovery] Requesting info from "https://10.138.0.3:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will 
use API Server "10.138.0.3:6443"
[discovery] Successfully established connection with API Server "10.138.0.3:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:3ai26q" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Dan mengapa itu tidak bisa mendapatkan configmap kubelet-config-1.12? Karena tidak ada. Kembali pada master:

$ sudo kubectl get cm -n kube-system --kubeconfig=/etc/kubernetes/admin.conf
NAME                                 DATA   AGE
calico-config                        4      26m
coredns                              1      29m
extension-apiserver-authentication   6      29m
kube-proxy                           2      29m
kubeadm-config                       2      29m
kubelet-config-1.13                  1      29m

brianriceca pada 7 Des 2018

@brianriceca : menghadapi masalah yang sama persis seperti Anda ... solusi apa pun untuk ini ..

Master: ram @ k8master1 : ~ $ versi kubeadm
versi kubeadm: & version.Info {Mayor: "1", Minor: "12", GitVersion: "v1.12.1", GitCommit: "4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState: "clean", BuildDate: "2018-10-05T16: 43: 08Z ", GoVersion:" go1.10.4 ", Penyusun:" gc ", Platform:" linux / amd64 "}
ram @ k8master1 : ~ $ dpkg -l | grep kub
ii kubeadm 1.12.1-00 amd64 Alat Bootstrap Cluster Kubernetes
ii kubectl 1.12.1-00 amd64 Alat Baris Perintah Kubernetes
ii kubelet 1.12.1-00 amd64 Agen Node Kubernetes
ii kubernetes-cni 0.6.0-00 amd64 Kubernetes CNI
ram @ k8master1 : ~ versi $ kubectl
Versi Klien: version.Info {Major: "1", Minor: "12", GitVersion: "v1.12.1", GitCommit: "4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState: "clean", BuildDate: "2018-10-05T16: 46: 06Z ", GoVersion:" go1.10.4 ", Penyusun:" gc ", Platform:" linux / amd64 "}
Versi Server: version.Info {Mayor: "1", Minor: "13", GitVersion: "v1.13.0", GitCommit: "ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState: "clean", BuildDate: "2018-12-03T20: 56: 12Z ", GoVersion:" go1.11.2 ", Penyusun:" gc ", Platform:" linux / amd64 "}

ram @ k8master1 : ~ $ kubectl dapatkan node
NAMA STATUS PERAN VERSI USIA
k8master1.example.com Siap master 101m v1.12.1
ram @ k8master1 : ~ $ kubectl dapatkan pods --all-namespaces
NAMESPACE NAMA STATUS SIAP DIMULAI USIA
kube-system calico-node-r248v 2/2 Berjalan 0 99m
kube-system coredns-869f847d58-72lqd 1/1 Berjalan 0101m
kube-system coredns-869f847d58-p2zzs 1/1 Berjalan 0101m
kube-system etcd-k8master1.example.com 1/1 Berjalan 0 100m
kube-system kube-apiserver-k8master1.example.com 1/1 Berjalan 0 100m
kube-system kube-controller-manager-k8master1.example.com 1/1 Berjalan 0 100m
kube-system kube-proxy-77qbx 1/1 Berjalan 0101m
kube-system kube-scheduler-k8master1.example.com 1/1 Berjalan 0 100m

Node Pekerja:
root @ k8worker1 : ~ # dpkg -l | grep -i kub
ii kubeadm 1.12.1-00 amd64 Alat Bootstrap Cluster Kubernetes
ii kubectl 1.12.1-00 amd64 Alat Baris Perintah Kubernetes
ii kubelet 1.12.1-00 amd64 Agen Node Kubernetes
ii kubernetes-cni 0.6.0-00 amd64 Kubernetes CNI
root @ k8worker1 : ~ # versi kubectl
Versi Klien: version.Info {Major: "1", Minor: "12", GitVersion: "v1.12.1", GitCommit: "4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState: "clean", BuildDate: "2018-10-05T16: 46: 06Z ", GoVersion:" go1.10.4 ", Penyusun:" gc ", Platform:" linux / amd64 "}
Koneksi ke server localhost: 8080 ditolak - apakah Anda menentukan host atau port yang benar?

root @ k8worker1 : ~ # kubeadm gabung 10.0.0.61:6443 --token xjxgqa.h2vnld3x9ztgf3pr --discovery-token-ca-cert-hash sha256: 7c18b654b623ee84164bb0dadfa79409c821398fe096882543444462a096882543
[preflight] menjalankan pemeriksaan pra-penerbangan
[PERINGATAN DiperlukanIPVSKernelModulesAvailable]: proxy IPVS tidak akan digunakan, karena modul kernel yang diperlukan berikut tidak dimuat: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] atau tidak ada dukungan ipvs kernel bawaan: mapr [nf_conntrack_ipv4: {} ip_vs: {} ip_vs: {} ip_vs: {} ip_vs ip_vs_wrr: {} ip_vs_sh: {}]
Anda dapat mengatasi masalah ini dengan metode berikut:

Jalankan 'modprobe -' untuk memuat modul kernel yang hilang;
1. Sediakan dukungan ipv kernel bawaan yang hilang

[penemuan] Mencoba untuk terhubung ke Server API "10.0.0.61:6443"
[penemuan] Membuat klien penemuan info cluster, meminta info dari " https://10.0.0.61 : 6443"
[penemuan] Meminta info dari " https://10.0.0.61 : 6443" lagi untuk memvalidasi TLS terhadap kunci publik yang disematkan
[penemuan] Tanda tangan info cluster dan kontennya valid dan sertifikat TLS divalidasi terhadap root yang disematkan, akan menggunakan API Server "10.0.0.61:6443"
[penemuan] Berhasil membuat koneksi dengan API Server "10.0.0.61:6443"
[kubelet] Mendownload konfigurasi untuk kubelet dari ConfigMap "kubelet-config-1.12" di namespace sistem kube
configmaps "kubelet-config-1.12" dilarang: Pengguna " system: bootstrap : xjxgqa" tidak bisa mendapatkan sumber daya "configmaps" di grup API "" di namespace "kube-system"
root @ k8worker1 : ~ # kubeadm gabung 10.0.0.61:6443 --token xjxgqa.h2vnld3x9ztgf3pr --discovery-token-ca-cert-hash sha256: 7c18b654b623ee84164bb0dadfa79409c821398fe096882543444462a096882543
[preflight] menjalankan pemeriksaan pra-penerbangan
[PERINGATAN DiperlukanIPVSKernelModulesAvailable]: proxy IPVS tidak akan digunakan, karena modul kernel yang diperlukan berikut tidak dimuat: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] atau tidak ada dukungan ipv kernel bawaan: map [ip_vs_rr: {} ip_vs_ nf_conntrack_ipv4: {} ip_vs: {}]
Anda dapat mengatasi masalah ini dengan metode berikut:

Jalankan 'modprobe -' untuk memuat modul kernel yang hilang;
1. Sediakan dukungan ipv kernel bawaan yang hilang

[preflight] Beberapa kesalahan fatal terjadi:
[ERROR FileAvailable - etc-kubernetes-bootstrap-kubelet.conf]: /etc/kubernetes/bootstrap-kubelet.conf sudah ada
[ERROR FileAvailable - etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt sudah ada
[preflight] Jika Anda tahu apa yang Anda lakukan, Anda dapat membuat cek non-fatal dengan --ignore-preflight-errors=...

ramanjk pada 10 Des 2018

sama seperti @brianriceca dan @ramanjk. Versi kubadm dan kubelet adalah 1.12.1 pada kedua node, kubelet-config-1.13 configmap ada di sana, dan melihat configmaps "kubelet-config-1.12" is forbidden juga

timwebster9 pada 17 Des 2018

D'oh! Yang belum saya ketahui adalah bahwa kubeadm selalu mendownload versi terbaru dari control plane Kubernetes dari gcr.io kecuali Anda memberi tahu sebaliknya. Jadi jika saya ingin menginstal 1.12.1 meskipun 1.13 tersedia, saya perlu melakukannya
kubeadm init --kubernetes-version 1.12.1 --pod-network-cidr _ terserah / terserah_

brianriceca pada 17 Des 2018

👍3

Saya menghapus kedua node dan mencoba lagi menggunakan versi 1.12.2 semuanya, dan kali ini masalahnya tidak terjadi. Ada beberapa keluaran tentang adanya versi yang lebih baru tetapi itu akan kembali ke 1.12 (Saya sudah kehilangannya sekarang)

timwebster9 pada 17 Des 2018

Saya menderita bug yang sama, sementara (tampaknya) menggunakan versi yang sama di seeder seperti di node.

Di seeder (setelah inisialisasi):

$ kubeadm version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "13",
    "gitVersion": "v1.13.0",
    "gitCommit": "ddf47ac13c1a9483ea035a79cd7c10005ff21a6d",
    "gitTreeState": "clean",
    "buildDate": "2018-12-11T17:03:40Z",
    "goVersion": "go1.11.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
$
$ kubectl get cm --all-namespaces
NAMESPACE     NAME                                 DATA   AGE
kube-public   cluster-info                         2      174m
kube-system   coredns                              1      174m
kube-system   extension-apiserver-authentication   6      174m
kube-system   flannel-plugin-config-map            2      174m
kube-system   kube-proxy                           2      174m
kube-system   kubeadm-config                       2      174m
kube-system   kubelet-config-1.13                  1      174m
kube-system   kubic-init-config-seeder             1      174m
$
$
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"archive", BuildDate:"2018-12-07T12:00:00Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Di node:

$ cat config.txt 
apiVersion: kubeadm.k8s.io/v1beta1
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: 192.168.100.1:6443
    token: 94dcda.c271f4ff502789ca
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: 94dcda.c271f4ff502789ca
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/crio/crio.sock
  kubeletExtraArgs:
    cni-bin-dir: /var/lib/kubelet/cni/bin
    cni-conf-dir: /etc/cni/net.d
    container-runtime-endpoint: unix:///var/run/crio/crio.sock
    network-plugin: cni

$
$ kubeadm join --v=8 --config=config.txt 
I1220 11:55:56.879023       7 join.go:299] [join] found NodeName empty; using OS hostname as NodeName
I1220 11:55:56.880357       7 joinconfiguration.go:72] loading configuration from the given file
[preflight] Running pre-flight checks
I1220 11:55:56.890498       7 join.go:328] [preflight] Running general checks
I1220 11:55:56.891937       7 checks.go:245] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1220 11:55:56.893051       7 checks.go:283] validating the existence of file /etc/kubernetes/kubelet.conf
I1220 11:55:56.894239       7 checks.go:283] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:55:56.895384       7 checks.go:104] validating the container runtime
I1220 11:55:57.072517       7 checks.go:373] validating the presence of executable crictl
I1220 11:55:57.073553       7 checks.go:332] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1220 11:55:57.074479       7 checks.go:332] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1220 11:55:57.075518       7 checks.go:644] validating whether swap is enabled or not
I1220 11:55:57.076499       7 checks.go:373] validating the presence of executable ip
I1220 11:55:57.077424       7 checks.go:373] validating the presence of executable iptables
I1220 11:55:57.078594       7 checks.go:373] validating the presence of executable mount
I1220 11:55:57.079564       7 checks.go:373] validating the presence of executable nsenter
I1220 11:55:57.080425       7 checks.go:373] validating the presence of executable ebtables
I1220 11:55:57.081391       7 checks.go:373] validating the presence of executable ethtool
I1220 11:55:57.082170       7 checks.go:373] validating the presence of executable socat
I1220 11:55:57.084207       7 checks.go:373] validating the presence of executable tc
I1220 11:55:57.085250       7 checks.go:373] validating the presence of executable touch
I1220 11:55:57.086132       7 checks.go:515] running all checks
I1220 11:55:57.137681       7 checks.go:403] checking whether the given node name is reachable using net.LookupHost
I1220 11:55:57.150619       7 checks.go:613] validating kubelet version
I1220 11:55:57.450319       7 checks.go:130] validating if the service is enabled and active
I1220 11:55:57.554984       7 checks.go:208] validating availability of port 10250
I1220 11:55:57.556700       7 checks.go:283] validating the existence of file /etc/kubernetes/pki/ca.crt
I1220 11:55:57.557579       7 checks.go:430] validating if the connectivity type is via proxy or direct
[preflight] Some fatal errors occurred:
    [ERROR FileAvailable--etc-kubernetes-bootstrap-kubelet.conf]: /etc/kubernetes/bootstrap-kubelet.conf already exists
    [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
sh-4.4# rm -f /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/pki/ca.crt
sh-4.4# kubeadm join --v=8 --config=config.txt 
I1220 11:56:10.073469      30 join.go:299] [join] found NodeName empty; using OS hostname as NodeName
I1220 11:56:10.074575      30 joinconfiguration.go:72] loading configuration from the given file
[preflight] Running pre-flight checks
I1220 11:56:10.085937      30 join.go:328] [preflight] Running general checks
I1220 11:56:10.086871      30 checks.go:245] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1220 11:56:10.087809      30 checks.go:283] validating the existence of file /etc/kubernetes/kubelet.conf
I1220 11:56:10.088573      30 checks.go:283] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:56:10.089370      30 checks.go:104] validating the container runtime
I1220 11:56:10.126939      30 checks.go:373] validating the presence of executable crictl
I1220 11:56:10.128075      30 checks.go:332] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1220 11:56:10.129096      30 checks.go:332] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1220 11:56:10.129993      30 checks.go:644] validating whether swap is enabled or not
I1220 11:56:10.131006      30 checks.go:373] validating the presence of executable ip
I1220 11:56:10.132983      30 checks.go:373] validating the presence of executable iptables
I1220 11:56:10.139740      30 checks.go:373] validating the presence of executable mount
I1220 11:56:10.140267      30 checks.go:373] validating the presence of executable nsenter
I1220 11:56:10.140738      30 checks.go:373] validating the presence of executable ebtables
I1220 11:56:10.141092      30 checks.go:373] validating the presence of executable ethtool
I1220 11:56:10.141459      30 checks.go:373] validating the presence of executable socat
I1220 11:56:10.142799      30 checks.go:373] validating the presence of executable tc
I1220 11:56:10.145062      30 checks.go:373] validating the presence of executable touch
I1220 11:56:10.145954      30 checks.go:515] running all checks
I1220 11:56:10.189173      30 checks.go:403] checking whether the given node name is reachable using net.LookupHost
I1220 11:56:10.204103      30 checks.go:613] validating kubelet version
I1220 11:56:10.529594      30 checks.go:130] validating if the service is enabled and active
I1220 11:56:10.556043      30 checks.go:208] validating availability of port 10250
I1220 11:56:10.557915      30 checks.go:283] validating the existence of file /etc/kubernetes/pki/ca.crt
I1220 11:56:10.559380      30 checks.go:430] validating if the connectivity type is via proxy or direct
I1220 11:56:10.560242      30 join.go:334] [preflight] Fetching init configuration
I1220 11:56:10.561013      30 join.go:601] [join] Discovering cluster-info
[discovery] Trying to connect to API Server "192.168.100.1:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.1:6443"
I1220 11:56:10.567171      30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-public/configmaps/cluster-info
I1220 11:56:10.568131      30 round_trippers.go:390] Request Headers:
I1220 11:56:10.568891      30 round_trippers.go:393]     Accept: application/json, */*
I1220 11:56:10.569609      30 round_trippers.go:393]     User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.586461      30 round_trippers.go:408] Response Status: 200 OK in 16 milliseconds
I1220 11:56:10.587241      30 round_trippers.go:411] Response Headers:
I1220 11:56:10.588006      30 round_trippers.go:414]     Content-Type: application/json
I1220 11:56:10.588757      30 round_trippers.go:414]     Content-Length: 1991
I1220 11:56:10.589497      30 round_trippers.go:414]     Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.590141      30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"kube-public","selfLink":"/api/v1/namespaces/kube-public/configmaps/cluster-info","uid":"c8b93b6b-0436-11e9-b4e4-4845202d6379","resourceVersion":"368","creationTimestamp":"2018-12-20T09:08:15Z"},"data":{"jws-kubeconfig-94dcda":"eyJhbGciOiJIUzI1NiIsImtpZCI6Ijk0ZGNkYSJ9..qJePAaUQp5APwTC-dSSzvL3MEVE8PQxgbvipbsC1faA","kubeconfig":"apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNE1USXlNREE1TURjMU0xb1hEVEk0TVRJeE56QTVNRGMxTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTkF3CmRET2tVSXI4M3BDMkY2bFE2UFZMYzlMbGhtenNnc3NPMWRQZWhVWTZ0azJwNUZIdmRlNEwwdkVNWHpaZU5oSGUKNFNnd1A1cTMxd1F0Wkx3aEFKWDRDR1dzNGVFZG9LNnZqMDVJYzQ1SDZhMDNvN3RqSWhNcUsvTHVrdDAzR2Q1Lwp0OENXUllHYjV5TWtBNldIMkxDZkwvSXlYQWU1NTV0STR1RTRuTTFSUXNSMFNMdy8rR2FK [truncated 967 chars]
[discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "192.168.100.1:6443"
[discovery] Successfully established connection with API Server "192.168.100.1:6443"
I1220 11:56:10.596836      30 join.go:608] [join] Retrieving KubeConfig objects
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
I1220 11:56:10.600835      30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config
I1220 11:56:10.601554      30 round_trippers.go:390] Request Headers:
I1220 11:56:10.602287      30 round_trippers.go:393]     User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.603124      30 round_trippers.go:393]     Accept: application/json, */*
I1220 11:56:10.603831      30 round_trippers.go:393]     Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.633321      30 round_trippers.go:408] Response Status: 200 OK in 28 milliseconds
I1220 11:56:10.634283      30 round_trippers.go:411] Response Headers:
I1220 11:56:10.635127      30 round_trippers.go:414]     Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.635912      30 round_trippers.go:414]     Content-Type: application/json
I1220 11:56:10.636635      30 round_trippers.go:414]     Content-Length: 1316
I1220 11:56:10.637413      30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kubeadm-config","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/kubeadm-config","uid":"c8069fb3-0436-11e9-b4e4-4845202d6379","resourceVersion":"173","creationTimestamp":"2018-12-20T09:08:14Z"},"data":{"ClusterConfiguration":"apiServer:\n  certSANs:\n  - 192.168.100.1\n  extraArgs:\n    authorization-mode: Node,RBAC\n    oidc-ca-file: /etc/kubernetes/pki/ca.crt\n    oidc-client-id: kubernetes\n    oidc-groups-claim: group\n    oidc-issuer-url: https://192.168.0.154:32000\n    oidc-username-claim: email\n  timeoutForControlPlane: 4m0s\napiVersion: kubeadm.k8s.io/v1beta1\ncertificatesDir: /etc/kubernetes/pki\nclusterName: kubernetes\ncontrolPlaneEndpoint: \"\"\ncontrollerManager: {}\ndns:\n  type: CoreDNS\netcd:\n  local:\n    dataDir: /var/lib/etcd\n    imageRepository: registry.opensuse.org/devel/kubic/containers/container/kubic\n    imageTag: \"3.3\"\nimageRepository: k8s.gcr.io\nkind: ClusterConfiguration\nkubernetesVer [truncated 292 chars]
I1220 11:56:10.643565      30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy
I1220 11:56:10.644430      30 round_trippers.go:390] Request Headers:
I1220 11:56:10.645126      30 round_trippers.go:393]     Accept: application/json, */*
I1220 11:56:10.645791      30 round_trippers.go:393]     User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.646455      30 round_trippers.go:393]     Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.654053      30 round_trippers.go:408] Response Status: 200 OK in 6 milliseconds
I1220 11:56:10.655099      30 round_trippers.go:411] Response Headers:
I1220 11:56:10.655921      30 round_trippers.go:414]     Content-Type: application/json
I1220 11:56:10.656796      30 round_trippers.go:414]     Content-Length: 1655
I1220 11:56:10.657597      30 round_trippers.go:414]     Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.658883      30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kube-proxy","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/kube-proxy","uid":"c8de0370-0436-11e9-b4e4-4845202d6379","resourceVersion":"229","creationTimestamp":"2018-12-20T09:08:15Z","labels":{"app":"kube-proxy"}},"data":{"config.conf":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nbindAddress: 0.0.0.0\nclientConnection:\n  acceptContentTypes: \"\"\n  burst: 10\n  contentType: application/vnd.kubernetes.protobuf\n  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\n  qps: 5\nclusterCIDR: 172.16.0.0/13\nconfigSyncPeriod: 15m0s\nconntrack:\n  max: null\n  maxPerCore: 32768\n  min: 131072\n  tcpCloseWaitTimeout: 1h0m0s\n  tcpEstablishedTimeout: 24h0m0s\nenableProfiling: false\nhealthzBindAddress: 0.0.0.0:10256\nhostnameOverride: \"\"\niptables:\n  masqueradeAll: false\n  masqueradeBit: 14\n  minSyncPeriod: 0s\n  syncPeriod: 30s\nipvs:\n  excludeCIDRs: null\n  minSyncPeriod: 0s\n  scheduler: \"\"\n  syncPeriod: 30s\nkind: Kub [truncated 631 chars]
I1220 11:56:10.664746      30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.13
I1220 11:56:10.665883      30 round_trippers.go:390] Request Headers:
I1220 11:56:10.666731      30 round_trippers.go:393]     User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.667616      30 round_trippers.go:393]     Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.668451      30 round_trippers.go:393]     Accept: application/json, */*
I1220 11:56:10.676896      30 round_trippers.go:408] Response Status: 200 OK in 7 milliseconds
I1220 11:56:10.677820      30 round_trippers.go:411] Response Headers:
I1220 11:56:10.680010      30 round_trippers.go:414]     Content-Type: application/json
I1220 11:56:10.681115      30 round_trippers.go:414]     Content-Length: 2134
I1220 11:56:10.682015      30 round_trippers.go:414]     Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.683204      30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kubelet-config-1.13","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.13","uid":"c80c1601-0436-11e9-b4e4-4845202d6379","resourceVersion":"176","creationTimestamp":"2018-12-20T09:08:14Z"},"data":{"kubelet":"address: 0.0.0.0\napiVersion: kubelet.config.k8s.io/v1beta1\nauthentication:\n  anonymous:\n    enabled: false\n  webhook:\n    cacheTTL: 2m0s\n    enabled: true\n  x509:\n    clientCAFile: /etc/kubernetes/pki/ca.crt\nauthorization:\n  mode: Webhook\n  webhook:\n    cacheAuthorizedTTL: 5m0s\n    cacheUnauthorizedTTL: 30s\ncgroupDriver: cgroupfs\ncgroupsPerQOS: true\nclusterDNS:\n- 172.24.0.10\nclusterDomain: cluster.local\nconfigMapAndSecretChangeDetectionStrategy: Watch\ncontainerLogMaxFiles: 5\ncontainerLogMaxSize: 10Mi\ncontentType: application/vnd.kubernetes.protobuf\ncpuCFSQuota: true\ncpuCFSQuotaPeriod: 100ms\ncpuManagerPolicy: none\ncpuManagerReconcilePeriod: 10s\nenableControllerAttachDetach: tr [truncated 1110 chars]
I1220 11:56:10.688139      30 interface.go:384] Looking for default routes with IPv4 addresses
I1220 11:56:10.688797      30 interface.go:389] Default route transits interface "eth0"
I1220 11:56:10.689612      30 interface.go:196] Interface eth0 is up
I1220 11:56:10.690375      30 interface.go:244] Interface "eth0" has 2 addresses :[192.168.100.220/24 fe80::d0a8:62ff:fe54:b6e9/64].
I1220 11:56:10.690995      30 interface.go:211] Checking addr  192.168.100.220/24.
I1220 11:56:10.691796      30 interface.go:218] IP found 192.168.100.220
I1220 11:56:10.692489      30 interface.go:250] Found valid IPv4 address 192.168.100.220 for interface "eth0".
I1220 11:56:10.693168      30 interface.go:395] Found active IP 192.168.100.220 
I1220 11:56:10.694393      30 join.go:341] [preflight] Running configuration dependant checks
I1220 11:56:10.695211      30 join.go:478] [join] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:56:10.942159      30 loader.go:359] Config loaded from file /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:56:10.943961      30 join.go:503] Stopping the kubelet
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
I1220 11:56:10.977300      30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.12
I1220 11:56:10.978035      30 round_trippers.go:390] Request Headers:
I1220 11:56:10.978844      30 round_trippers.go:393]     User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.979502      30 round_trippers.go:393]     Accept: application/json, */*
I1220 11:56:10.980081      30 round_trippers.go:393]     Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.983223      30 round_trippers.go:408] Response Status: 403 Forbidden in 2 milliseconds
I1220 11:56:10.984240      30 round_trippers.go:411] Response Headers:
I1220 11:56:10.985065      30 round_trippers.go:414]     Content-Type: application/json
I1220 11:56:10.985883      30 round_trippers.go:414]     X-Content-Type-Options: nosniff
I1220 11:56:10.987515      30 round_trippers.go:414]     Content-Length: 342
I1220 11:56:10.989207      30 round_trippers.go:414]     Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.990506      30 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubelet-config-1.12\" is forbidden: User \"system:bootstrap:94dcda\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kubelet-config-1.12","kind":"configmaps"},"code":403}
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:94dcda" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
$
$
$ kubeadm version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "13",
    "gitVersion": "v1.13.0",
    "gitCommit": "ddf47ac13c1a9483ea035a79cd7c10005ff21a6d",
    "gitTreeState": "archive",
    "buildDate": "2018-12-07T12:00:00Z",
    "goVersion": "go1.11.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

Untuk beberapa alasan tampaknya mencari kubelet-config-1.12 sedangkan ConfigMap yang tepat harus kubelet-config-1.13 .

Setelah memeriksa kubelet --version saya melihat bahwa:

$ kubelet --version
Kubernetes v1.12.0

Apakah nama configmap berasal dari ini?

inercia pada 20 Des 2018

@inercia nama configmap diturunkan dari versi kubelet. Lihat link saya di atas.

oz123 pada 20 Des 2018

Terima kasih atas klarifikasi @ oz123.

Saya bertanya-tanya apa yang akan terjadi pada pembaruan. Sebagai contoh,

1) seeder diinstal dengan my-distribution-1.13 yang mencakup kubeadm-1.13
2) seeder adalah init ialized
3) beberapa waktu setelah itu, node diinstal dengan distro yang sama, menginstal kubeadm-1.13 dan kubelet-1.13
4) tetapi beberapa pembaruan diterapkan di akhir penginstalan, dan kubelet-1.14 diinstal
5) kubeadm join mencoba mencari configmap untuk 1.14 tetapi tidak ada ...

inercia pada 20 Des 2018

Saya melihat masalah ini menginstal k8s 1.13.1 dengan versi kubeadm yang cocok, tetapi diisolasi ke kube-proxy:

kubeadm join --config /etc/kubernetes/kubeadm-client.conf --ignore-preflight-errors=all
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "xxx.xxx.xxx.xxx:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://xxx.xxx.xxx.xxx:6443"
[discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "xxx.xxx.xxx.xxx:6443"
[discovery] Successfully established connection with API Server "xxx.xxx.xxx.xxx:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
unable to fetch the kubeadm-config ConfigMap: failed to get component configs: configmaps "kube-proxy" is forbidden: User "system:bootstrap:3tw24k" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Jika saya membuat RoleBinding solusi secara manual, saya dapat bergabung dengan node.

# on controlplane node
kubectl create rolebinding -n kube-system --role kube-proxy --group system:bootstrappers:kubeadm:default-node-token kubeadm:kube-proxy-bootstrap

# on joining node
...
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ci-pdk1-debug4144-k8sne-1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

Apakah kube-proxy RoleBinding untuk system:bootstrappers:kubeadm:default-node-token Grup seharusnya terisi secara otomatis? Tahu apa yang terjadi di sini?

EDIT: Juga menemukan bahwa menjalankan kubeadm init phase addon kube-proxy secara eksplisit setelah kubeadm init pertama saya membuat RoleBinding kube-proxy untuk system:bootstrappers:kubeadm:default-node-token . Masih tidak yakin mengapa kubeadm init awal saya melewatkan pembuatan RoleBinding ini.

chadswen pada 9 Jan 2019

Ini berhasil untuk saya:

Periksa dengan pasti versi kubeadm dan kubelet Anda, pastikan versi yang sama dari paket-paket ini digunakan di semua node Anda. Sebelum memasang, Anda harus "menandai dan menahan" versi Anda ini di host Anda:

periksa versi Anda saat ini dari masing-masing:
kubelet --version

periksa kubeadm
versi kubeadm

jika mereka berbeda, Anda punya masalah. Anda harus menginstal ulang versi yang sama di antara semua node Anda dan mengizinkan penurunan. Versi saya pada perintah di bawah ini mungkin lebih lama dari yang saat ini keluar, Anda dapat mengganti nomor versi dengan yang lebih baru, tetapi ini akan berhasil:
sudo apt-get install -y docker-ce = 18.06.1 ~ ce ~ 3-0 ~ ubuntu kubelet = 1.12.2-00 kubeadm = 1.12.2-00 kubectl = 1.12.2-00 --allow-downgrades

kemudian, setelah dipasang, tandai dan tahan sehingga tidak dapat ditingkatkan secara otomatis, dan hancurkan sistem Anda
sudo apt-mark hold docker-ce kubelet kubeadm kubectl