๋ฒ๊ทธ ๋ณด๊ณ ์
kubeadm ๋ฒ์ ( kubeadm version
) : "v1.12.0-alpha.0.957 + 1235adac3802fd-dirty"
kubeadm init
๋ก ์ ์ด ํ๋ ์ธ ๋
ธ๋๋ฅผ ์์ฑํ์ต๋๋ค. ๋ณ๋์ ๋
ธ๋์์ kubeadm join
๋ฅผ ์คํํ๋๋ฐ ๋ค์ ์ค๋ฅ ๋ฉ์์ง๊ฐ ๋ํ๋ฉ๋๋ค.
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:4ipkdk" cannot get configmaps in the namespace "kube-system"
kubeadm join
๊ฐ) ์ฑ๊ณต์ ์ผ๋ก ์๋ฃ ๋ ๊ฒ์ผ๋ก ์์ํ์ต๋๋ค.
๋ด๊ฐ ์ ์์๋ ํ, ๋ค๋ฅธ ๋
ธ๋์์ kubeadm init
๋ฐ kubeadm join
๋ฅผ ์คํํฉ๋๋ค. ๊ตฌ์ฑ ๋งต์ ์ํฅ์์ฃผ์ง ์์์ผํ๋ ์ถ๊ฐ ์ฝ๋ / yaml์ด ๋ง์ด ์์ต๋๋ค (ํ๋ณตํ AWS ๋ฐฐํฌ์ ํ์). ๊ทธ๋ฌ๋ ์ด๊ฒ์ด ์ฌํ ํ ์์๋ ๊ฒ์ผ๋ก ํ๋ช
๋๋ฉด ๋ ์์ธํ ์ง์นจ์ ์ ๊ณต ํ ๊ฒ์
๋๋ค.
kubeadm join
๋ฐ kubeadm init
๋ ๊ตฌ์ฑ ๋งต์ ์ด๋ฆ์ ์ผ๊ด์ฑ์์ด ์ง์ ํ๋ค๊ณ ์๊ฐํฉ๋๋ค. init
๋ช
๋ น์ ๊ตฌ์ฑ ํ์ผ์ ์ง์ ๋ kubernetesVersion
ํ๊ณ join
๋ช
๋ น์ ๊ตฌ์ฑ ๋งต์ ์ด๋ฆ์ผ๋ก kubelet ๋ฒ์ ์ ์ฌ์ฉํฉ๋๋ค (์ : kubelet-config-1.1). ์ผ์นํ์ง ์๋ ๋ฒ์ ์ด ์์ผ๋ฉด ๊ด์ฐฎ์ต๋๋ค.
init
๋ช
๋ น์ kube-public
๋ค์ ์คํ์ด์ค์ ๊ตฌ์ฑ ๋งต์ ๋ํ ์ต๋ช
์ก์ธ์ค์ ๋ํ RBAC ๊ท์น์ ์์ฑํ์ง๋ง kubelet ๊ตฌ์ฑ์ ๊ณต์ฉ ๋ค์ ์คํ์ด์ค์ ๋ฃ์ง ์๋ ๊ฒ์ฒ๋ผ ๋ณด์ด๋ฏ๋ก ๋
ธ๋ ๊ฐ์
์ ์ก์ธ์ค ํ ์ ์์ต๋๋ค. ๊ทธ๊ฒ.
์ผ์นํ์ง ์๋ ๋ฒ์ ์ด ์์ผ๋ฉด ๊ด์ฐฎ์ต๋๋ค.
@chuckha ์ ๊ธฐ์ตํ๋ค๋ฉด kubelet-*
๊ตฌ์ฑ ๋งต์ kube-system
์ด์ด์ผํ๋ฉฐ kubeadm์ ๋ถํธ ์คํธ๋ฉ ํ ํฐ ๋ฐ ๋
ธ๋์ ๋ํ ์ก์ธ์ค๋ฅผ ํ์ฉํ๋ ๊ท์น์ ๋ง๋ญ๋๋ค. ํ์ง๋ง ๋ง์ง๋ง ๋ณ๊ฒฝ ํ ๋ค์ ํ์ธํ๊ฒ ์ต๋๋ค.
@chuckha ๋ง์คํฐ + ๋ฆด๋ฆฌ์ค ๋ฒํธ์์ v1.11.0์ผ๋ก ๊ฐ์ ๋น๋ ๋ ๋ชจ๋ ๊ตฌ์ฑ ์์๋ฅผ ์ฌ์ฉํ์ฌ ํด๋ฌ์คํฐ์์ ์ฑ๊ณต์ ์ผ๋ก ์กฐ์ธ์ ์๋ฃํ์ต๋๋ค.
kubelet-config-1.11
์ด (๊ฐ) kube-system
์์ ์์ฑ๋์์ต๋๋ค.kubeadm:kubelet-config-1.11
์ญํ ์ด ๊ตฌ์ฑ ๋งต์ ๋ํ ๊ถํ ๊ฐ์ ธ ์ค๊ธฐ๋ก kube-system
์์ ์์ฑ๋์์ต๋๋ค.system:nodes
๋ฐ system:bootstrappers:kubeadm:default-node-token
๋ํ ์ญํ ๋ฐ์ธ๋ฉ kubeadm:kubelet-config-1.11
์ด kube-system
์์ ์์ฑ๋์์ต๋๋ค.๊ทธ๋์ IMO :
์์ง ์กฐ์ฌํด์ผ ํ ๋ถ๋ถ์
init ๋ช ๋ น์ ๊ตฌ์ฑ ํ์ผ์ ์ง์ ๋ kubernetesVersion์ ์ฌ์ฉํ๊ณ join ๋ช ๋ น์ ๊ตฌ์ฑ ๋งต์ ์ด๋ฆ์ผ๋ก kubelet ๋ฒ์ ์ ์ฌ์ฉํฉ๋๋ค (์ : kubelet-config-1.1).
์, ๋ฒ์ ๋ฌธ์ ์ฌ์ผํฉ๋๋ค.
๋น๋ ํ ๋ ๋ฒ์ ์ ์ค์ ํ์ง ์์์ผ๋ฏ๋ก ๋ฐ์ด๋๋ฆฌ๊ฐ ๋ชจ๋ 1.12.0 ์ด์์ด๋ผ๊ณ ์๊ฐํ์ง๋ง kubeadm์ ์ค์นํ๊ณ ๊ฐ์ ๋ก v1.11์ ์ฌ์ฉํ๋๋กํ์ต๋๋ค.
์ด๋ก ์ธํด
root@ip-10-0-0-7:~# k get cm -n kube-system
NAME DATA AGE
...
kubelet-config-1.11 1 32m
๊ทธ๋ฆฌ๊ณ ๊ฐ์ ํ ๋ :
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:u3ns6m" cannot get configmaps in the namespace "kube-system"
๋ฐ์ด๋๋ฆฌ๋ฅผ ๋ค์ ๋น๋ํ๊ณ ์ฌ๋ฐ๋ฅธ ๋ฒ์ ์ผ๋ก ๊ฐ์ ์ค์ ํ๊ณ ๋ค์ ์๋ํฉ๋๋ค.
kubelet ๋ฐ kubeadm ๋ฒ์ ์ด ์ผ์นํ๋ฉด ๋ชจ๋ ๊ฒ์ด ์ ์์ ๋๋ค. ๊ตฌ์ฑ ๋งต ์์ฑ๊ณผ ๊ตฌ์ฑ ๋งต ๊ฐ์ ธ ์ค๊ธฐ ์ฌ์ด์ ๋ถ์ผ์น (์๋์ ์ผ ์ ์์)์ ๊ด๋ จํ์ฌ ๋ ๊ธด๊ธํ ์์ ์ ์ํด ์ข ๋ฃํฉ๋๋ค.
kubeadm init๋ฅผ ์ฌ์ฉํ์ง ์๊ณ ๋จ๊ณ๋ฅผ ๊ฐ๋ณ์ ์ผ๋ก ํธ์ถํฉ๋๋ค. kube-system์๋ configmap์ด ์์ผ๋ฉฐ ์ฌ๋ฐ๋ฅด๊ฒ ์ค์ ํ ์์๋ ๊ถํ๋ ์์ต๋๋ค. ์ด๊ฒ์ ์ด๋ค ๋จ๊ณ์ ๋๊น?
@drewwells ๋น์ ๊ณผ ๊ฐ์ ๋ฌธ์ ๊ฐ ๋ฐ์ํ์ต๋๋ค. ๋จ๊ณ๋ฅผ ๊ฐ๋ณ์ ์ผ๋ก ์คํ ์ค์ด๋ฉฐ ๊ตฌ์ฑ mpas๊ฐ ์์ต๋๋ค.
sudo kubectl get cm -n kube-system --kubeconfig=/etc/kubernetes/admin.conf
NAME DATA AGE
calico-config 2 11m
coredns 1 15m
extension-apiserver-authentication 6 15m
kube-proxy 2 15m
ํด๊ฒฐ์ฑ ์ ์ฐพ์์ต๋๊น?
๋ํ ๋ชจ๋ ๊ตฌ์ฑ ์์๋ 1.11.4์ ๋๋ค.
์ด๊ฒ์ ๋ ๋์๊ฐ๋๋ค. kubeadm init์ ์ฌ์ฉํ์ฌ ํด๋ฌ์คํฐ๋ฅผ ๋ถํธ ์คํธ๋ฉํ๊ณ ์ด์ ์ฌ๋ฐ๋ฅธ ๊ตฌ์ฑ ๋งต์ด ์ ์๋ฆฌ์ ์์ต๋๋ค.
ubuntu@master-1-test2:~$ sudo kubectl get cm -n kube-system --kubeconfig=/etc/kubernetes/admin.conf
NAME DATA AGE
coredns 1 41m
extension-apiserver-authentication 6 41m
kube-proxy 2 41m
kubeadm-config 1 41m
kubelet-config-1.11 1 41m
ubuntu@master-1-test2:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.4", GitCommit:"bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b", GitTreeState:"clean", BuildDate:"2018-10-25T19:13:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
๋ ธ๋์์ :
$ sudo kubeadm -v=10 join k8s.oz.noris.de:6443 --token 36etul.nv5lz6hjfifdd4c9 --discovery-token-unsafe-skip-ca-verification I1107 12:57:01.340638 3631 join.go:226] [join] found NodeName empty
I1107 12:57:01.340816 3631 join.go:227] [join] considered OS hostname as NodeName
[preflight] running pre-flight checks
I1107 12:57:01.341152 3631 join.go:238] [preflight] running various checks on all nodes
I1107 12:57:01.341265 3631 checks.go:253] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1107 12:57:01.341677 3631 checks.go:291] validating the existence of file /etc/kubernetes/pki/ca.crt
I1107 12:57:01.341774 3631 checks.go:291] validating the existence of file /etc/kubernetes/kubelet.conf
I1107 12:57:01.341857 3631 checks.go:291] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1107 12:57:01.341947 3631 kernelcheck_linux.go:45] validating the kernel module IPVS required exists in machine or not
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs:{} ip_
vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
I1107 12:57:01.349448 3631 checks.go:138] validating if the service is enabled and active
I1107 12:57:01.361957 3631 checks.go:340] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1107 12:57:01.362034 3631 checks.go:340] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1107 12:57:01.362076 3631 checks.go:653] validating whether swap is enabled or not
I1107 12:57:01.362134 3631 checks.go:381] validating the presence of executable crictl
I1107 12:57:01.362204 3631 checks.go:381] validating the presence of executable ip
I1107 12:57:01.362244 3631 checks.go:381] validating the presence of executable iptables
I1107 12:57:01.362281 3631 checks.go:381] validating the presence of executable mount
I1107 12:57:01.362320 3631 checks.go:381] validating the presence of executable nsenter
...
[discovery] Trying to connect to API Server "mycluster.example.com:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://mycluster.example.com:6443"
I1107 12:57:01.487256 3631 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.4 (linux/amd64) kubernetes/bf9a868" 'https://mycluster.example.com:6443/api/v1/namespaces/kube-public/config
maps/cluster-info'
I1107 12:57:01.504539 3631 round_trippers.go:405] GET https://mycluster.example.com:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 200 OK in 17 milliseconds
I1107 12:57:01.504720 3631 round_trippers.go:411] Response Headers:
I1107 12:57:01.504818 3631 round_trippers.go:414] Content-Type: application/json
I1107 12:57:01.504914 3631 round_trippers.go:414] Content-Length: 2217
I1107 12:57:01.505003 3631 round_trippers.go:414] Date: Wed, 07 Nov 2018 12:57:01 GMT
I1107 12:57:01.505174 3631 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"kube-public","selfLink":"/api/v1/namespaces/kube-public/configmaps/cluster-info","uid":"97e9
a3d1-e286-11e8-9272-fa163ef9e3af","resourceVersion":"425","creationTimestamp":"2018-11-07T12:13:53Z"},"data":{"jws-kubeconfig-36etul":"eyJhbGciOiJIUzI1NiIsImtpZCI6IjM2ZXR1bCJ9..zRgexonkjOpLJS0q3IignURwTcpBuQy7gv35Qhhsl_k","jws-kubeconfig-
eth6o8":"eyJhbGciOiJIUzI1NiIsImtpZCI6ImV0aDZvOCJ9..kWj4cI2j1WgKfNG07IGiIij4CSb9kWUbaM2mixlYThY","jws-kubeconfig-rbxd02":"eyJhbGciOiJIUzI1NiIsImtpZCI6InJieGQwMiJ9..HwIWDwfIbAjNM1EGbWdXYOhC8z1MxgwuzhjlJRaZ_pc","kubeconfig":"apiVersion: v1\n
clusters:\n- cluster:\n certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNE1URXdNekEwTlRrME4xb1hEVEk0TVRBek1UQTB
OVGswTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTU9BCkNHQU5jUjVRQWV3MlljY2V0eWVyYktiODd4RWRPVlp2aUdneElrbkpKTTZwZFVBbzMwSWVxckRqSnlFaTFVeDcKU0c5NS9sRlBqU1htdHhhNHMvc1g1KzNTVW4zZ
EtFRWw5TFhXa0lzeTRJYzRFUTMwWE9WcnNuYTYwN1UzNmQyaAp3NHdTK1dveE5QR3dqZDM2bXQzMFR4bUluYk54ZVl5d2NnVU1tMlZFZXM4dGhVaVhZMXB1N1Y2SUNCY243cE9NCkdoT2xlRXg4SmlEVnhuSGlpSm9oYytCbGNIdHdLU1pzK2cvZUhwdGdlSDdaQlZNRC8zZVFvZXVsUGVvTEkwamEKc09jTENMTkpEVVB
LUWJqRnRNbkFZSXVvOENHSXpFTzBDaDZNeW5vb1pTL0E0bEs1MXJmTVdkTkZ4N0dVdnQxYQo5KzZzMHo2NEpHeVFBdmtBcWhVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEbWs1NEYzZ1BqOS91NzlRbTg2V1Mzc
k5YaFoKZG16Wmt3TXRDajRuTXdsSndGQy9iZUU4ZUdsWnFxWDcrdEpYUDVaY0xLNE1pSnM1U2JTMjd5NDF3WTRRTFFWaQpVWmRocEFHUTBOSlpHSGhWMTVDczlVQTA1ZTFNajNCaHZ6SG5VV2t1ZUhYbW84VmI4SkI5RGloeGdiUW5GY2FQCjRWcVhWY0pBemxVQ0V5aXhreVRGendZTklJbzJHdGtCdlI1YkxCM0doT2R
sQURmQzEwdzgvTmQveFFmRnRWdmYKL3lHaktpbW8rT2xERkV5YittcHVKMVdiN3Y3bnJJSzlSSy9WbVhUWENiOWZLQ3BmQ0hMU0hpa0lEWklZK0wxTQpwbWpXYXZFcjFLSlE5UEJIYmdZSHkxK1F0bkpXRDNjNnJrOUtoNU1zMFhTVmpBc2Z1RWdXaG9CYnlVdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=\n
server: https://k8s.oz.noris.de:6443\n name: \"\"\ncontexts: []\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: []\n"}}
[discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "k8s.oz.noris.de:6443"
[discovery] Successfully established connection with API Server "mycluster.example.com:6443"
I1107 12:57:01.509945 3631 join.go:260] [join] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1107 12:57:01.617006 3631 loader.go:359] Config loaded from file /etc/kubernetes/bootstrap-kubelet.conf
I1107 12:57:01.617871 3631 join.go:283] Stopping the kubelet
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
I1107 12:57:01.627838 3631 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.4 (linux/amd64) kubernetes/bf9a868" -H "Authorization: Bearer 36etul.nv5lz6hjfifdd4c9" 'https://mycluster.example.com:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.12'
I1107 12:57:01.639396 3631 round_trippers.go:405] GET https://mycluster.example.com:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.12 403 Forbidden in 11 milliseconds
I1107 12:57:01.639427 3631 round_trippers.go:411] Response Headers:
I1107 12:57:01.639443 3631 round_trippers.go:414] Content-Length: 311
I1107 12:57:01.639464 3631 round_trippers.go:414] Date: Wed, 07 Nov 2018 12:57:01 GMT
I1107 12:57:01.639477 3631 round_trippers.go:414] Content-Type: application/json
I1107 12:57:01.639492 3631 round_trippers.go:414] X-Content-Type-Options: nosniff
I1107 12:57:01.639525 3631 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubelet-config-1.12\" is forbidden: User \"system:bootstrap:36etul\" cannot get confi
gmaps in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kubelet-config-1.12","kind":"configmaps"},"code":403}
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:36etul" cannot get configmaps in the namespace "kube-system"
์ด์ํ ๋ฌผ๊ฑด์ ๋ณด์ จ๋์?
๋ถ๋ช
ํ kubelet์ ๋ฒ์ ์ ํ์ธํ๋ฉด ๋ชจ๋ ๊ฒ์ด ๋ง๋ค์ด์ง๋๋ค! kubelet ๋ฒ์ 1.12.2๊ฐ ์์ต๋๋ค.
์ฝ๋์์ ํํธ๋ฅผ ์ป์์ต๋๋ค.
ํ๋์ ์ต์ ๋ฒ์ ์ ์ ์ธํ๊ณ ๋์ผํ ๋ฌธ์ ๊ฐ ๋ฐ์ํฉ๋๋ค.
๋ง์คํฐ์์ :
$ lsb_release -d
Description: Ubuntu 16.04.5 LTS
$ dpkg -l | grep kub
ii kubeadm 1.12.1-00 amd64 Kubernetes Cluster Bootstrapping Tool
ii kubectl 1.12.1-00 amd64 Kubernetes Command Line Tool
ii kubelet 1.12.1-00 amd64 Kubernetes Node Agent
ii kubernetes-cni 0.6.0-00 amd64 Kubernetes CNI
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
๊ทธ ์๋ฒ ๋ฒ์ 1.13.0์ ์ด๋์์ ์์ต๋๊น? ๋๋ ๊ทธ๊ฒ์ ์ค์นํ์ง ์์๋ค.
์ด์จ๋ ์ด VM์์ kubeadm init๋ฅผ ์คํํ๊ธฐ ์ ์ ๋ณต์ ํ์ผ๋ฏ๋ก ํด๋ฌ์คํฐ์ ๋ค๋ฅธ ๋ ธ๋๊ฐ ๋ ์ค๋น๊ฐ ๋ ๋ค๋ฅธ VM์ด ์์ต๋๋ค. ํด๋ก ์ด๊ธฐ ๋๋ฌธ์ 1.12.1๋ ์์ต๋๋ค. ๊ฐ์ ํ๋ ค๊ณ ํ ๋ :
$ kubeadm join --token blahblah 10.138.0.3:6443 --discovery-token-ca-cert-hash sha256:deadbeefdeadbeefetc
[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
[discovery] Trying to connect to API Server "10.138.0.3:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.138.0.3:6443"
[discovery] Requesting info from "https://10.138.0.3:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will
use API Server "10.138.0.3:6443"
[discovery] Successfully established connection with API Server "10.138.0.3:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:3ai26q" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
๊ทธ๋ฆฌ๊ณ ์ kubelet-config-1.12 configmap์ ์ป์ ์ ์์ต๋๊น? ํ๋๋ ์์ผ๋๊น. ๋ง์คํฐ๋ก ๋์ ๊ฐ๊ธฐ :
$ sudo kubectl get cm -n kube-system --kubeconfig=/etc/kubernetes/admin.conf
NAME DATA AGE
calico-config 4 26m
coredns 1 29m
extension-apiserver-authentication 6 29m
kube-proxy 2 29m
kubeadm-config 2 29m
kubelet-config-1.13 1 29m
@brianriceca : ๋น์ ๊ณผ ๋๊ฐ์ ๋ฌธ์ ์ ์ง๋ฉด ... ์ด์ ๋ํ ๋ชจ๋ ํด๊ฒฐ์ฑ ..
Master : ram @ k8master1 : ~ $ kubeadm ๋ฒ์
kubeadm ๋ฒ์ : & version.Info {Major : "1", Minor : "12", GitVersion : "v1.12.1", GitCommit : "4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState : "clean", BuildDate : "2018-10-05T16 : 43 : 08Z ", GoVersion :"go1.10.4 ", ์ปดํ์ผ๋ฌ :"gc ", ํ๋ซํผ :"linux / amd64 "}
ram @ k8master1 : ~ $ dpkg -l | grep kub
ii kubeadm 1.12.1-00 amd64 Kubernetes ํด๋ฌ์คํฐ ๋ถํธ ์คํธ๋ฉ ๋๊ตฌ
ii kubectl 1.12.1-00 amd64 Kubernetes ๋ช
๋ น ์ค ๋๊ตฌ
ii kubelet 1.12.1-00 amd64 Kubernetes ๋
ธ๋ ์์ด์ ํธ
ii kubernetes-cni 0.6.0-00 amd64 Kubernetes CNI
ram @ k8master1 : ~ $ kubectl ๋ฒ์
ํด๋ผ์ด์ธํธ ๋ฒ์ : version.Info {Major : "1", Minor : "12", GitVersion : "v1.12.1", GitCommit : "4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState : "clean", BuildDate : "2018-10-05T16 : 46 : 06Z ", GoVersion :"go1.10.4 ", ์ปดํ์ผ๋ฌ :"gc ", ํ๋ซํผ :"linux / amd64 "}
์๋ฒ ๋ฒ์ : version.Info {Major : "1", Minor : "13", GitVersion : "v1.13.0", GitCommit : "ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState : "clean", BuildDate : "2018-12-03T20 : 56 : 12Z ", GoVersion :"go1.11.2 ", ์ปดํ์ผ๋ฌ :"gc ", ํ๋ซํผ :"linux / amd64 "}
ram @ k8master1 : ~ $ kubectl ๋
ธ๋ ๊ฐ์ ธ ์ค๊ธฐ
์ด๋ฆ ์ํ ์ญํ ์ฐ๋ น ๋ฒ์
k8master1.example.com ์ค๋น ๋ง์คํฐ 101m v1.12.1
ram @ k8master1 : ~ $ kubectl get pods --all-namespaces
๋ค์ ์คํ์ด์ค ์ด๋ฆ ์ค๋น ์ํ ๋ค์ ์์ ๋์ด
kube-system calico-node-r248v 2/2 Running 0 99m
kube-system coredns-869f847d58-72lqd 1/1 Running 0 101m
kube-system coredns-869f847d58-p2zzs 1/1 Running 0 101m
kube-system etcd-k8master1.example.com 1/1 ๋ฌ๋ฆฌ๊ธฐ 0 100m
kube-system kube-apiserver-k8master1.example.com 1/1 ๋ฌ๋ฆฌ๊ธฐ 0 100m
kube-system kube-controller-manager-k8master1.example.com 1/1 ๋ฌ๋ฆฌ๊ธฐ 0 100m
kube-system kube-proxy-77qbx 1/1 Running 0 101m
kube-system kube-scheduler-k8master1.example.com 1/1 ๋ฌ๋ฆฌ๊ธฐ 0 100m
์์
์ ๋
ธ๋ :
root @ k8worker1 : ~ # dpkg -l | grep -i kub
ii kubeadm 1.12.1-00 amd64 Kubernetes ํด๋ฌ์คํฐ ๋ถํธ ์คํธ๋ฉ ๋๊ตฌ
ii kubectl 1.12.1-00 amd64 Kubernetes ๋ช
๋ น ์ค ๋๊ตฌ
ii kubelet 1.12.1-00 amd64 Kubernetes ๋
ธ๋ ์์ด์ ํธ
ii kubernetes-cni 0.6.0-00 amd64 Kubernetes CNI
root @ k8worker1 : ~ # kubectl ๋ฒ์
ํด๋ผ์ด์ธํธ ๋ฒ์ : version.Info {Major : "1", Minor : "12", GitVersion : "v1.12.1", GitCommit : "4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState : "clean", BuildDate : "2018-10-05T16 : 46 : 06Z ", GoVersion :"go1.10.4 ", ์ปดํ์ผ๋ฌ :"gc ", ํ๋ซํผ :"linux / amd64 "}
์๋ฒ localhost : 8080 ์ ๋ํ ์ฐ๊ฒฐ์ด ๊ฑฐ๋ถ๋์์ต๋๋ค. ์ฌ๋ฐ๋ฅธ ํธ์คํธ ๋๋ ํฌํธ๋ฅผ ์ง์ ํ์ต๋๊น?
root @ k8worker1 : ~ # kubeadm join 10.0.0.61:6443 --token xjxgqa.h2vnld3x9ztgf3pr --discovery-token-ca-cert-hash sha256 : 7c18b654b623ee84164bb0dfa79409c821398f1a968843446af525ec72e0fdad
[ํ๋ฆฌ ํ๋ผ์ดํธ] ํ๋ฆฌ ํ๋ผ์ดํธ ๊ฒ์ฌ ์คํ
[๊ฒฝ๊ณ ํ์ IPVSKernelModulesAvailable] : ๋ค์ ํ์ ์ปค๋ ๋ชจ๋์ด๋ก๋๋์ง ์์์ผ๋ฏ๋ก IPVS ํ๋ก์๊ฐ ์ฌ์ฉ๋์ง ์์ต๋๋ค. [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] ๋๋ ๋ด์ฅ ์ปค๋ ipvs ์ง์ ์์ : map [nf_conntrack_iprrv4 : {} ip_vs : {} ip_vs : {} ip_vs_wrr : {} ip_vs_sh : {}]
๋ค์ ๋ฐฉ๋ฒ์ผ๋ก์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ ์ ์์ต๋๋ค.
[๋ฐ๊ฒฌ] API ์๋ฒ "10.0.0.61:6443"์ ์ ์ ์๋ ์ค
[๊ฒ์] ํด๋ฌ์คํฐ ์ ๋ณด ๊ฒ์ ํด๋ผ์ด์ธํธ ์์ฑ, " https://10.0.0.61 : 6443"์์ ์ ๋ณด ์์ฒญ
[๋ฐ๊ฒฌ] ๊ณ ์ ๋ ๊ณต๊ฐ ํค์ ๋ํด TLS์ ์ ํจ์ฑ์ ๊ฒ์ฌํ๊ธฐ ์ํด " https://10.0.0.61 : 6443"์์ ๋ค์ ์ ๋ณด ์์ฒญ
[๋ฐ๊ฒฌ] ํด๋ฌ์คํฐ ์ ๋ณด ์๋ช
๋ฐ ์ฝํ
์ธ ๊ฐ ์ ํจํ๋ฉฐ TLS ์ธ์ฆ์๊ฐ ๊ณ ์ ๋ ๋ฃจํธ์ ๋ํด ์ ํจ์ฑ์ ๊ฒ์ฌํ๊ณ API ์๋ฒ "10.0.0.61:6443"์ ์ฌ์ฉํฉ๋๋ค.
[๋ฐ๊ฒฌ] API ์๋ฒ "10.0.0.61:6443"๊ณผ ์ฑ๊ณต์ ์ผ๋ก ์ฐ๊ฒฐ
[kubelet] kube-system ๋ค์ ์คํ์ด์ค์ "kubelet-config-1.12"ConfigMap์์ kubelet ๊ตฌ์ฑ ๋ค์ด๋ก๋
configmaps "kubelet-config-1.12"๋ ๊ธ์ง๋จ : ์ฌ์ฉ์ " system : bootstrap : xjxgqa"๋ ๋ค์ ์คํ์ด์ค "kube-system"์ API ๊ทธ๋ฃน ""์์ "configmaps"๋ฆฌ์์ค๋ฅผ ๊ฐ์ ธ์ฌ ์ ์์ต๋๋ค.
root @ k8worker1 : ~ # kubeadm join 10.0.0.61:6443 --token xjxgqa.h2vnld3x9ztgf3pr --discovery-token-ca-cert-hash sha256 : 7c18b654b623ee84164bb0dfa79409c821398f1a968843446af525ec72e0fdad
[ํ๋ฆฌ ํ๋ผ์ดํธ] ํ๋ฆฌ ํ๋ผ์ดํธ ๊ฒ์ฌ ์คํ
[๊ฒฝ๊ณ ํ์ IPVSKernelModulesAvailable] : ๋ค์ ํ์ ์ปค๋ ๋ชจ๋์ด๋ก๋๋์ง ์์๊ธฐ ๋๋ฌธ์ IPVS ํ๋ก์๊ฐ ์ฌ์ฉ๋์ง ์์ต๋๋ค. [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] ๋๋ ๋ด์ฅ ์ปค๋ ipvs ์ง์ ์์ : map [ip_vs_rr : {} ip_vs_rr : {} ip_vs_rr : {} ip_vs_rr : {} nf_conntrack_ipv4 : {} ip_vs : {}]
๋ค์ ๋ฐฉ๋ฒ์ผ๋ก์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ ์ ์์ต๋๋ค.
[ํ๋ฆฌ ํ๋ผ์ดํธ] ๋ช ๊ฐ์ง ์น๋ช
์ ์ธ ์ค๋ฅ๊ฐ ๋ฐ์ํ์ต๋๋ค.
[์ค๋ฅ FileAvailable--etc-kubernetes-bootstrap-kubelet.conf] : /etc/kubernetes/bootstrap-kubelet.conf๊ฐ ์ด๋ฏธ ์์ต๋๋ค.
[์ค๋ฅ FileAvailable--etc-kubernetes-pki-ca.crt] : /etc/kubernetes/pki/ca.crt๊ฐ ์ด๋ฏธ ์์ต๋๋ค.
[ํ๋ฆฌ ํ๋ผ์ดํธ] ์ํ์ค์ธ ์์
์ ์๊ณ ์๋ค๋ฉด --ignore-preflight-errors=...
ํ์ฌ ์น๋ช
์ ์ด์ง ์์ ์ํ๋ฅผ ๋ง๋ค ์ ์์ต๋๋ค.
@brianriceca ๋ฐ kubelet-config-1.13
๋งต์ด ์์ผ๋ฉฐ configmaps "kubelet-config-1.12" is forbidden
๋ฉ๋๋ค.
์ค! ๋ด๊ฐ ์์ง ๋ชฐ๋๋ ๊ฒ์ kubeadm์ด ๋ฌ๋ฆฌ ์ง์ ํ์ง ์๋ ํ ํญ์ gcr.io์์ Kubernetes ์ ์ด ํ๋ ์ธ์ ์ต์ ๋ฒ์ ์ ๋ค์ด๋ก๋ํ๋ค๋ ๊ฒ์
๋๋ค. ๋ฐ๋ผ์ 1.13์ ์ฌ์ฉํ ์ ์๋๋ฐ๋ 1.12.1์ ์ค์นํ๋ ค๋ฉด ๋ค์์ ์ํํด์ผํฉ๋๋ค.
kubeadm init --kubernetes-version 1.12.1 --pod-network-cidr
_ ๋ฌด์์ด๋ / ๋ฌด์์ด๋ _
๋ ๋
ธ๋๋ฅผ ๋ชจ๋ ์ญ์ ํ๊ณ ๋ชจ๋ ๋ฒ์ 1.12.2
๋ฅผ ์ฌ์ฉํ์ฌ ๋ค์ ์๋ํ์ง๋ง ์ด๋ฒ์๋ ๋ฌธ์ ๊ฐ ๋ฐ์ํ์ง ์์์ต๋๋ค. ์ต์ ๋ฒ์ ์ด ์๋ค๋ ์ถ๋ ฅ์ด ์์์ง๋ง 1.12๋ก ๋๋๋ฆด ๊ฒ์
๋๋ค (์ง๊ธ ์์ด ๋ฒ๋ ธ์ต๋๋ค)
๋๋ (๋ถ๋ช ํ) ์๋์์ ๋ ธ๋์ ๋์ผํ ๋ฒ์ ์ ์ฌ์ฉํ๋ ๋์ ๋์ผํ ๋ฒ๊ทธ๋ฅผ ๊ฒช๊ณ ์์ต๋๋ค.
์๋์์ (์ด๊ธฐํ ํ) :
$ kubeadm version -o json
{
"clientVersion": {
"major": "1",
"minor": "13",
"gitVersion": "v1.13.0",
"gitCommit": "ddf47ac13c1a9483ea035a79cd7c10005ff21a6d",
"gitTreeState": "clean",
"buildDate": "2018-12-11T17:03:40Z",
"goVersion": "go1.11.2",
"compiler": "gc",
"platform": "linux/amd64"
}
}
$
$ kubectl get cm --all-namespaces
NAMESPACE NAME DATA AGE
kube-public cluster-info 2 174m
kube-system coredns 1 174m
kube-system extension-apiserver-authentication 6 174m
kube-system flannel-plugin-config-map 2 174m
kube-system kube-proxy 2 174m
kube-system kubeadm-config 2 174m
kube-system kubelet-config-1.13 1 174m
kube-system kubic-init-config-seeder 1 174m
$
$
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"archive", BuildDate:"2018-12-07T12:00:00Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
๋ ธ๋์์ :
$ cat config.txt
apiVersion: kubeadm.k8s.io/v1beta1
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
bootstrapToken:
apiServerEndpoint: 192.168.100.1:6443
token: 94dcda.c271f4ff502789ca
unsafeSkipCAVerification: true
timeout: 5m0s
tlsBootstrapToken: 94dcda.c271f4ff502789ca
kind: JoinConfiguration
nodeRegistration:
criSocket: /var/run/crio/crio.sock
kubeletExtraArgs:
cni-bin-dir: /var/lib/kubelet/cni/bin
cni-conf-dir: /etc/cni/net.d
container-runtime-endpoint: unix:///var/run/crio/crio.sock
network-plugin: cni
$
$ kubeadm join --v=8 --config=config.txt
I1220 11:55:56.879023 7 join.go:299] [join] found NodeName empty; using OS hostname as NodeName
I1220 11:55:56.880357 7 joinconfiguration.go:72] loading configuration from the given file
[preflight] Running pre-flight checks
I1220 11:55:56.890498 7 join.go:328] [preflight] Running general checks
I1220 11:55:56.891937 7 checks.go:245] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1220 11:55:56.893051 7 checks.go:283] validating the existence of file /etc/kubernetes/kubelet.conf
I1220 11:55:56.894239 7 checks.go:283] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:55:56.895384 7 checks.go:104] validating the container runtime
I1220 11:55:57.072517 7 checks.go:373] validating the presence of executable crictl
I1220 11:55:57.073553 7 checks.go:332] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1220 11:55:57.074479 7 checks.go:332] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1220 11:55:57.075518 7 checks.go:644] validating whether swap is enabled or not
I1220 11:55:57.076499 7 checks.go:373] validating the presence of executable ip
I1220 11:55:57.077424 7 checks.go:373] validating the presence of executable iptables
I1220 11:55:57.078594 7 checks.go:373] validating the presence of executable mount
I1220 11:55:57.079564 7 checks.go:373] validating the presence of executable nsenter
I1220 11:55:57.080425 7 checks.go:373] validating the presence of executable ebtables
I1220 11:55:57.081391 7 checks.go:373] validating the presence of executable ethtool
I1220 11:55:57.082170 7 checks.go:373] validating the presence of executable socat
I1220 11:55:57.084207 7 checks.go:373] validating the presence of executable tc
I1220 11:55:57.085250 7 checks.go:373] validating the presence of executable touch
I1220 11:55:57.086132 7 checks.go:515] running all checks
I1220 11:55:57.137681 7 checks.go:403] checking whether the given node name is reachable using net.LookupHost
I1220 11:55:57.150619 7 checks.go:613] validating kubelet version
I1220 11:55:57.450319 7 checks.go:130] validating if the service is enabled and active
I1220 11:55:57.554984 7 checks.go:208] validating availability of port 10250
I1220 11:55:57.556700 7 checks.go:283] validating the existence of file /etc/kubernetes/pki/ca.crt
I1220 11:55:57.557579 7 checks.go:430] validating if the connectivity type is via proxy or direct
[preflight] Some fatal errors occurred:
[ERROR FileAvailable--etc-kubernetes-bootstrap-kubelet.conf]: /etc/kubernetes/bootstrap-kubelet.conf already exists
[ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
sh-4.4# rm -f /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/pki/ca.crt
sh-4.4# kubeadm join --v=8 --config=config.txt
I1220 11:56:10.073469 30 join.go:299] [join] found NodeName empty; using OS hostname as NodeName
I1220 11:56:10.074575 30 joinconfiguration.go:72] loading configuration from the given file
[preflight] Running pre-flight checks
I1220 11:56:10.085937 30 join.go:328] [preflight] Running general checks
I1220 11:56:10.086871 30 checks.go:245] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1220 11:56:10.087809 30 checks.go:283] validating the existence of file /etc/kubernetes/kubelet.conf
I1220 11:56:10.088573 30 checks.go:283] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:56:10.089370 30 checks.go:104] validating the container runtime
I1220 11:56:10.126939 30 checks.go:373] validating the presence of executable crictl
I1220 11:56:10.128075 30 checks.go:332] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1220 11:56:10.129096 30 checks.go:332] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1220 11:56:10.129993 30 checks.go:644] validating whether swap is enabled or not
I1220 11:56:10.131006 30 checks.go:373] validating the presence of executable ip
I1220 11:56:10.132983 30 checks.go:373] validating the presence of executable iptables
I1220 11:56:10.139740 30 checks.go:373] validating the presence of executable mount
I1220 11:56:10.140267 30 checks.go:373] validating the presence of executable nsenter
I1220 11:56:10.140738 30 checks.go:373] validating the presence of executable ebtables
I1220 11:56:10.141092 30 checks.go:373] validating the presence of executable ethtool
I1220 11:56:10.141459 30 checks.go:373] validating the presence of executable socat
I1220 11:56:10.142799 30 checks.go:373] validating the presence of executable tc
I1220 11:56:10.145062 30 checks.go:373] validating the presence of executable touch
I1220 11:56:10.145954 30 checks.go:515] running all checks
I1220 11:56:10.189173 30 checks.go:403] checking whether the given node name is reachable using net.LookupHost
I1220 11:56:10.204103 30 checks.go:613] validating kubelet version
I1220 11:56:10.529594 30 checks.go:130] validating if the service is enabled and active
I1220 11:56:10.556043 30 checks.go:208] validating availability of port 10250
I1220 11:56:10.557915 30 checks.go:283] validating the existence of file /etc/kubernetes/pki/ca.crt
I1220 11:56:10.559380 30 checks.go:430] validating if the connectivity type is via proxy or direct
I1220 11:56:10.560242 30 join.go:334] [preflight] Fetching init configuration
I1220 11:56:10.561013 30 join.go:601] [join] Discovering cluster-info
[discovery] Trying to connect to API Server "192.168.100.1:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.1:6443"
I1220 11:56:10.567171 30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-public/configmaps/cluster-info
I1220 11:56:10.568131 30 round_trippers.go:390] Request Headers:
I1220 11:56:10.568891 30 round_trippers.go:393] Accept: application/json, */*
I1220 11:56:10.569609 30 round_trippers.go:393] User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.586461 30 round_trippers.go:408] Response Status: 200 OK in 16 milliseconds
I1220 11:56:10.587241 30 round_trippers.go:411] Response Headers:
I1220 11:56:10.588006 30 round_trippers.go:414] Content-Type: application/json
I1220 11:56:10.588757 30 round_trippers.go:414] Content-Length: 1991
I1220 11:56:10.589497 30 round_trippers.go:414] Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.590141 30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"kube-public","selfLink":"/api/v1/namespaces/kube-public/configmaps/cluster-info","uid":"c8b93b6b-0436-11e9-b4e4-4845202d6379","resourceVersion":"368","creationTimestamp":"2018-12-20T09:08:15Z"},"data":{"jws-kubeconfig-94dcda":"eyJhbGciOiJIUzI1NiIsImtpZCI6Ijk0ZGNkYSJ9..qJePAaUQp5APwTC-dSSzvL3MEVE8PQxgbvipbsC1faA","kubeconfig":"apiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNE1USXlNREE1TURjMU0xb1hEVEk0TVRJeE56QTVNRGMxTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTkF3CmRET2tVSXI4M3BDMkY2bFE2UFZMYzlMbGhtenNnc3NPMWRQZWhVWTZ0azJwNUZIdmRlNEwwdkVNWHpaZU5oSGUKNFNnd1A1cTMxd1F0Wkx3aEFKWDRDR1dzNGVFZG9LNnZqMDVJYzQ1SDZhMDNvN3RqSWhNcUsvTHVrdDAzR2Q1Lwp0OENXUllHYjV5TWtBNldIMkxDZkwvSXlYQWU1NTV0STR1RTRuTTFSUXNSMFNMdy8rR2FK [truncated 967 chars]
[discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "192.168.100.1:6443"
[discovery] Successfully established connection with API Server "192.168.100.1:6443"
I1220 11:56:10.596836 30 join.go:608] [join] Retrieving KubeConfig objects
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
I1220 11:56:10.600835 30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config
I1220 11:56:10.601554 30 round_trippers.go:390] Request Headers:
I1220 11:56:10.602287 30 round_trippers.go:393] User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.603124 30 round_trippers.go:393] Accept: application/json, */*
I1220 11:56:10.603831 30 round_trippers.go:393] Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.633321 30 round_trippers.go:408] Response Status: 200 OK in 28 milliseconds
I1220 11:56:10.634283 30 round_trippers.go:411] Response Headers:
I1220 11:56:10.635127 30 round_trippers.go:414] Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.635912 30 round_trippers.go:414] Content-Type: application/json
I1220 11:56:10.636635 30 round_trippers.go:414] Content-Length: 1316
I1220 11:56:10.637413 30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kubeadm-config","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/kubeadm-config","uid":"c8069fb3-0436-11e9-b4e4-4845202d6379","resourceVersion":"173","creationTimestamp":"2018-12-20T09:08:14Z"},"data":{"ClusterConfiguration":"apiServer:\n certSANs:\n - 192.168.100.1\n extraArgs:\n authorization-mode: Node,RBAC\n oidc-ca-file: /etc/kubernetes/pki/ca.crt\n oidc-client-id: kubernetes\n oidc-groups-claim: group\n oidc-issuer-url: https://192.168.0.154:32000\n oidc-username-claim: email\n timeoutForControlPlane: 4m0s\napiVersion: kubeadm.k8s.io/v1beta1\ncertificatesDir: /etc/kubernetes/pki\nclusterName: kubernetes\ncontrolPlaneEndpoint: \"\"\ncontrollerManager: {}\ndns:\n type: CoreDNS\netcd:\n local:\n dataDir: /var/lib/etcd\n imageRepository: registry.opensuse.org/devel/kubic/containers/container/kubic\n imageTag: \"3.3\"\nimageRepository: k8s.gcr.io\nkind: ClusterConfiguration\nkubernetesVer [truncated 292 chars]
I1220 11:56:10.643565 30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy
I1220 11:56:10.644430 30 round_trippers.go:390] Request Headers:
I1220 11:56:10.645126 30 round_trippers.go:393] Accept: application/json, */*
I1220 11:56:10.645791 30 round_trippers.go:393] User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.646455 30 round_trippers.go:393] Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.654053 30 round_trippers.go:408] Response Status: 200 OK in 6 milliseconds
I1220 11:56:10.655099 30 round_trippers.go:411] Response Headers:
I1220 11:56:10.655921 30 round_trippers.go:414] Content-Type: application/json
I1220 11:56:10.656796 30 round_trippers.go:414] Content-Length: 1655
I1220 11:56:10.657597 30 round_trippers.go:414] Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.658883 30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kube-proxy","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/kube-proxy","uid":"c8de0370-0436-11e9-b4e4-4845202d6379","resourceVersion":"229","creationTimestamp":"2018-12-20T09:08:15Z","labels":{"app":"kube-proxy"}},"data":{"config.conf":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nbindAddress: 0.0.0.0\nclientConnection:\n acceptContentTypes: \"\"\n burst: 10\n contentType: application/vnd.kubernetes.protobuf\n kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\n qps: 5\nclusterCIDR: 172.16.0.0/13\nconfigSyncPeriod: 15m0s\nconntrack:\n max: null\n maxPerCore: 32768\n min: 131072\n tcpCloseWaitTimeout: 1h0m0s\n tcpEstablishedTimeout: 24h0m0s\nenableProfiling: false\nhealthzBindAddress: 0.0.0.0:10256\nhostnameOverride: \"\"\niptables:\n masqueradeAll: false\n masqueradeBit: 14\n minSyncPeriod: 0s\n syncPeriod: 30s\nipvs:\n excludeCIDRs: null\n minSyncPeriod: 0s\n scheduler: \"\"\n syncPeriod: 30s\nkind: Kub [truncated 631 chars]
I1220 11:56:10.664746 30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.13
I1220 11:56:10.665883 30 round_trippers.go:390] Request Headers:
I1220 11:56:10.666731 30 round_trippers.go:393] User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.667616 30 round_trippers.go:393] Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.668451 30 round_trippers.go:393] Accept: application/json, */*
I1220 11:56:10.676896 30 round_trippers.go:408] Response Status: 200 OK in 7 milliseconds
I1220 11:56:10.677820 30 round_trippers.go:411] Response Headers:
I1220 11:56:10.680010 30 round_trippers.go:414] Content-Type: application/json
I1220 11:56:10.681115 30 round_trippers.go:414] Content-Length: 2134
I1220 11:56:10.682015 30 round_trippers.go:414] Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.683204 30 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kubelet-config-1.13","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.13","uid":"c80c1601-0436-11e9-b4e4-4845202d6379","resourceVersion":"176","creationTimestamp":"2018-12-20T09:08:14Z"},"data":{"kubelet":"address: 0.0.0.0\napiVersion: kubelet.config.k8s.io/v1beta1\nauthentication:\n anonymous:\n enabled: false\n webhook:\n cacheTTL: 2m0s\n enabled: true\n x509:\n clientCAFile: /etc/kubernetes/pki/ca.crt\nauthorization:\n mode: Webhook\n webhook:\n cacheAuthorizedTTL: 5m0s\n cacheUnauthorizedTTL: 30s\ncgroupDriver: cgroupfs\ncgroupsPerQOS: true\nclusterDNS:\n- 172.24.0.10\nclusterDomain: cluster.local\nconfigMapAndSecretChangeDetectionStrategy: Watch\ncontainerLogMaxFiles: 5\ncontainerLogMaxSize: 10Mi\ncontentType: application/vnd.kubernetes.protobuf\ncpuCFSQuota: true\ncpuCFSQuotaPeriod: 100ms\ncpuManagerPolicy: none\ncpuManagerReconcilePeriod: 10s\nenableControllerAttachDetach: tr [truncated 1110 chars]
I1220 11:56:10.688139 30 interface.go:384] Looking for default routes with IPv4 addresses
I1220 11:56:10.688797 30 interface.go:389] Default route transits interface "eth0"
I1220 11:56:10.689612 30 interface.go:196] Interface eth0 is up
I1220 11:56:10.690375 30 interface.go:244] Interface "eth0" has 2 addresses :[192.168.100.220/24 fe80::d0a8:62ff:fe54:b6e9/64].
I1220 11:56:10.690995 30 interface.go:211] Checking addr 192.168.100.220/24.
I1220 11:56:10.691796 30 interface.go:218] IP found 192.168.100.220
I1220 11:56:10.692489 30 interface.go:250] Found valid IPv4 address 192.168.100.220 for interface "eth0".
I1220 11:56:10.693168 30 interface.go:395] Found active IP 192.168.100.220
I1220 11:56:10.694393 30 join.go:341] [preflight] Running configuration dependant checks
I1220 11:56:10.695211 30 join.go:478] [join] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:56:10.942159 30 loader.go:359] Config loaded from file /etc/kubernetes/bootstrap-kubelet.conf
I1220 11:56:10.943961 30 join.go:503] Stopping the kubelet
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
I1220 11:56:10.977300 30 round_trippers.go:383] GET https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.12
I1220 11:56:10.978035 30 round_trippers.go:390] Request Headers:
I1220 11:56:10.978844 30 round_trippers.go:393] User-Agent: kubeadm/v1.13.0 (linux/amd64) kubernetes/ddf47ac
I1220 11:56:10.979502 30 round_trippers.go:393] Accept: application/json, */*
I1220 11:56:10.980081 30 round_trippers.go:393] Authorization: Bearer 94dcda.c271f4ff502789ca
I1220 11:56:10.983223 30 round_trippers.go:408] Response Status: 403 Forbidden in 2 milliseconds
I1220 11:56:10.984240 30 round_trippers.go:411] Response Headers:
I1220 11:56:10.985065 30 round_trippers.go:414] Content-Type: application/json
I1220 11:56:10.985883 30 round_trippers.go:414] X-Content-Type-Options: nosniff
I1220 11:56:10.987515 30 round_trippers.go:414] Content-Length: 342
I1220 11:56:10.989207 30 round_trippers.go:414] Date: Thu, 20 Dec 2018 11:56:11 GMT
I1220 11:56:10.990506 30 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubelet-config-1.12\" is forbidden: User \"system:bootstrap:94dcda\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kubelet-config-1.12","kind":"configmaps"},"code":403}
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:94dcda" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
$
$
$ kubeadm version -o json
{
"clientVersion": {
"major": "1",
"minor": "13",
"gitVersion": "v1.13.0",
"gitCommit": "ddf47ac13c1a9483ea035a79cd7c10005ff21a6d",
"gitTreeState": "archive",
"buildDate": "2018-12-07T12:00:00Z",
"goVersion": "go1.11.2",
"compiler": "gc",
"platform": "linux/amd64"
}
}
์ด๋ค ์ด์ ๋ก kubelet-config-1.12
๋ฅผ ์ฐพ๊ณ ์๋ ๋ฐ๋ฉด ์ค๋ฅธ์ชฝ ConfigMap์ kubelet-config-1.13
์ด์ด์ผํฉ๋๋ค.
kubelet --version
ํ์ธ ํ ๋ค์์ ํ์ธํฉ๋๋ค.
$ kubelet --version
Kubernetes v1.12.0
configmap ์ด๋ฆ์ด ์ด๊ฒ์์ ํ์ ๋์์ต๋๊น?
@inercia configmap ์ด๋ฆ์ kubelet ๋ฒ์ ์์ ํ์๋ฉ๋๋ค. ์์ ๋ด ๋งํฌ๋ฅผ ์ฐธ์กฐํ์ญ์์ค.
@ oz123์ ์ค๋ช ํด ์ฃผ์ ์ ๊ฐ์ฌํฉ๋๋ค.
์ ๋ฐ์ดํธ์ ์ด๋ค ์ผ์ด ์ผ์ด๋ ์ง ๊ถ๊ธํฉ๋๋ค. ์๋ฅผ ๋ค๋ฉด
1) my-distribution-1.13
๊ฐ ํฌํจ ๋ kubeadm-1.13
my-distribution-1.13
์ ํจ๊ป ์๋๊ฐ ์ค์น๋ฉ๋๋ค.
2) ์๋๊ฐ init
ialized
3) ๊ทธ ํ ์ผ๋ง ํ ๋์ผํ ๋ฐฐํฌํ์ผ๋ก ๋
ธ๋๊ฐ ์ค์น๋๊ณ kubeadm-1.13
๋ฐ kubelet-1.13
4) ๊ทธ๋ฌ๋ ์ผ๋ถ ์
๋ฐ์ดํธ๋ ์ค์น๊ฐ ๋๋ ๋ ์ ์ฉ๋๊ณ ์๋ก์ด kubelet-1.14
๊ฐ ์ค์น๋ฉ๋๋ค.
5) kubeadm join
๋ 1.14์ ๋ํ configmap์ ์ฐพ์ผ๋ ค๊ณ ํ์ง๋ง ๊ฑฐ๊ธฐ์ ์์ต๋๋ค ...
์ผ์นํ๋ kubeadm ๋ฒ์ ์ผ๋ก k8s 1.13.1์ ์ค์นํ์ง๋ง kube-proxy์ ๊ฒฉ๋ฆฌ ๋์ด ๋ฌธ์ ๊ฐ ํ์๋ฉ๋๋ค.
kubeadm join --config /etc/kubernetes/kubeadm-client.conf --ignore-preflight-errors=all
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "xxx.xxx.xxx.xxx:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://xxx.xxx.xxx.xxx:6443"
[discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "xxx.xxx.xxx.xxx:6443"
[discovery] Successfully established connection with API Server "xxx.xxx.xxx.xxx:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
unable to fetch the kubeadm-config ConfigMap: failed to get component configs: configmaps "kube-proxy" is forbidden: User "system:bootstrap:3tw24k" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
RoleBinding ํด๊ฒฐ ๋ฐฉ๋ฒ์ ์๋์ผ๋ก ์์ฑํ๋ฉด ๋ ธ๋์ ๊ฐ์ ํ ์ ์์ต๋๋ค.
# on controlplane node
kubectl create rolebinding -n kube-system --role kube-proxy --group system:bootstrappers:kubeadm:default-node-token kubeadm:kube-proxy-bootstrap
# on joining node
...
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ci-pdk1-debug4144-k8sne-1" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
system:bootstrappers:kubeadm:default-node-token
๊ทธ๋ฃน์ ๋ํ kube-proxy
RoleBinding์ด ์๋์ผ๋ก ์ฑ์ ์ ธ์ผํฉ๋๊น? ์ฌ๊ธฐ์ ๋ฌด์จ ์ผ์ด ์ผ์ด๋๊ณ ์๋์ง ์์ญ๋๊น?
ํธ์ง : ๋ํ ์ฒซ ๋ฒ์งธ kubeadm init
์ดํ์ ๋ช
์ ์ ์ผ๋ก kubeadm init phase addon kube-proxy
์คํํ๋ฉด system:bootstrappers:kubeadm:default-node-token
๋ํ kube-proxy RoleBinding์ด ์์ฑ๋๋ค๋ ์ฌ์ค์ ๋ฐ๊ฒฌํ์ต๋๋ค. ์ ๋ด ์ด๊ธฐ kubeadm init
์์์ด RoleBinding ์์ฑ์ ๊ฑด๋ ๋ฐ๋ ์ง ๋ชจ๋ฅด๊ฒ ์ต๋๋ค.
์ด๊ฒ์ ๋๋ฅผ ์ํด ์ผํ์ต๋๋ค.
kubeadm ๋ฐ kubelet์ ๋ฒ์ ์ ํ์คํ ํ์ธํ๊ณ ์ด๋ฌํ ํจํค์ง์ ๋์ผํ ๋ฒ์ ์ด ๋ชจ๋ ๋ ธ๋์์ ์ฌ์ฉ๋๋์ง ํ์ธํ์ญ์์ค. ์ค์นํ๊ธฐ ์ ์ ํธ์คํธ์์ ๋ค์ ๋ฒ์ ์ "ํ์ํ๊ณ ์ ์ง"ํด์ผํฉ๋๋ค.
๊ฐ๊ฐ์ ํ์ฌ ๋ฒ์ ์ ํ์ธํ์ญ์์ค.
kubelet-๋ฒ์
kubeadm ํ์ธ
kubeadm ๋ฒ์
๊ทธ๋ค์ด ๋ค๋ฅด๋ฉด ๋ฌธ์ ๊ฐ ์์ต๋๋ค. ๋ชจ๋ ๋
ธ๋์ ๋์ผํ ๋ฒ์ ์ ๋ค์ ์ค์นํ๊ณ ๋ค์ด ๊ทธ๋ ์ด๋๋ฅผ ํ์ฉํด์ผํฉ๋๋ค. ์๋ ๋ช
๋ น์ ๋ด ๋ฒ์ ์ด ํ์ฌ ์ถ์ ๋ ๊ฒ๋ณด๋ค ์ด์ ๋ฒ์ ์ผ ์ ์์ต๋๋ค. ๋ฒ์ ๋ฒํธ๋ฅผ ์ต์ ๋ฒ์ ์ผ๋ก ๋ฐ๊ฟ ์ ์์ง๋ง์ด ๋ฐฉ๋ฒ์ ์๋ํฉ๋๋ค.
sudo apt-get install -y docker-ce = 18.06.1 ~ ce ~ 3-0 ~ ubuntu kubelet = 1.12.2-00 kubeadm = 1.12.2-00 kubectl = 1.12.2-00 --allow-downgrades
๊ทธ๋ฐ ๋ค์ ์ค์น๊ฐ ์๋ฃ๋๋ฉด ์๋์ผ๋ก ์
๊ทธ๋ ์ด๋๋์ง ์๋๋ก ํ์ํ๊ณ ๋๋ฅด๊ณ ์์ผ๋ฉด ์์คํ
์ด ์์๋ฉ๋๋ค.
sudo apt-mark hold docker-ce kubelet kubeadm kubectl
๊ฐ์ฅ ์ ์ฉํ ๋๊ธ