dns, resolv.conf, coredns
kubeadm ๋ฒ์ ( kubeadm version
) :
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
ํ๊ฒฝ :
kubectl version
) : Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
uname -a
) :kubeadm์ ํตํด kubernetes๋ฅผ ์ค์น ํ ํ coredns ํฌ๋๊ฐ ํ์๋์ง ์์ต๋๋ค. kubectl get pods --all-namespaces
๋ ๋ค์์ ์ธ์ํฉ๋๋ค.
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcdf6894-kgg8d 0/1 Pending 0 2h
kube-system coredns-78fcdf6894-vl9jf 0/1 Pending 0 2h
kube-system etcd-beetlejuice 1/1 Running 0 2h
kube-system kube-apiserver-beetlejuice 1/1 Running 0 2h
kube-system kube-controller-manager-beetlejuice 1/1 Running 0 2h
kube-system kube-proxy-bjdqd 1/1 Running 0 2h
kube-system kube-scheduler-beetlejuice 1/1 Running 0 2h
coredns๊ฐ '์คํ ์ค'์ํ๋ก ๋ณ๊ฒฝ๋๊ณ kubernetes๊ฐ ๋ฌธ์ ์์ด ์คํ ์ค์ ๋๋ค.
๋ค์์ ์ค์น์ ์ฌ์ฉํ ์คํฌ๋ฆฝํธ์ ๋๋ค.
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y docker.io
apt-get install -y kubeadm
kubeadm init --pod-network-cidr=10.27.0.0/16
๊ทธ๋ ๋ค๊ณ ์๊ฐํ์ง๋ง ๋ญ์ง ๋ชจ๋ฅด๊ฒ ์ด์ .. ์ด๋ค ์ข ๋ฅ์ ๋ก๊ทธ๊ฐ ํ์ํ๋ฉด ๋งํด์ฃผ์ธ์.
๋ฟก๋ฟก
์ถ๋ ฅ :
kubectl describe pod <coredns-pod-ids>
๋ฐ ๊ด๋ จ ์ค๋ฅ :
'systemctl status kubelet'
'journalctl -xeu kubelet'
๋ ๋ง์ ๋์์ด ๋ ๊ฒ์
๋๋ค.
๊ฐ์ฌ.
kubectl describe pod coredns-78fcdf6894-kgg8d -n kube-system
:
Name: coredns-78fcdf6894-kgg8d
Namespace: kube-system
Node: <none>
Labels: k8s-app=kube-dns
pod-template-hash=3497892450
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/coredns-78fcdf6894
Containers:
coredns:
Image: k8s.gcr.io/coredns:1.1.3
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-4fqm7 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-4fqm7:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-4fqm7
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4m (x1436 over 4h) default-scheduler 0/1 nodes are available: 1 node(s) were not ready.
kubectl describe pod coredns-78fcdf6894-vl9jf -n kube-system
:
Name: coredns-78fcdf6894-vl9jf
Namespace: kube-system
Node: <none>
Labels: k8s-app=kube-dns
pod-template-hash=3497892450
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/coredns-78fcdf6894
Containers:
coredns:
Image: k8s.gcr.io/coredns:1.1.3
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-4fqm7 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-4fqm7:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-4fqm7
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 1m (x1467 over 4h) default-scheduler 0/1 nodes are available: 1 node(s) were not ready.
systemctl status kubelet
์ค๋ฅ๊ฐ ํ์๋์ง ์์ต๋๋ค. ๋ค์์ ์ ์ฒด ์ถ๋ ฅ์
๋๋ค.
โ kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
โโ10-kubeadm.conf, override.conf
Active: active (running) since Mon 2018-07-09 17:43:53 CEST; 4h 7min ago
Docs: http://kubernetes.io/docs/
Main PID: 26710 (kubelet)
Tasks: 32 (limit: 4915)
CGroup: /system.slice/kubelet.service
โโ26710 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-co
journalctl -xeu kubelet
๋ค์ ์ค์ ์ฌ๋ฌ ๋ฒ ๋ด
๋๋ค.
Jul 09 21:54:48 beetlejuice kubelet[26710]: E0709 21:54:48.883071 26710 kubelet.go:2112] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninit
Jul 09 21:54:49 beetlejuice kubelet[26710]: E0709 21:54:49.566069 26710 dns.go:131] Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 213.133.99.99 213.133.98.98 213.133.100.100
Jul 09 21:54:53 beetlejuice kubelet[26710]: W0709 21:54:53.884846 26710 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
์ฐธ๊ณ ๋ก /etc/resolv.conf
:
### Hetzner Online GmbH installimage
# nameserver config
nameserver 213.133.99.99
nameserver 213.133.98.98
nameserver 213.133.100.100
nameserver 2a01:4f8:0:1::add:1010
nameserver 2a01:4f8:0:1::add:9999
nameserver 2a01:4f8:0:1::add:9898
์ฌ๊ธฐ์ Hetzner๋ ๋ฐ์ดํฐ ์ผํฐ ์ด์์์ ์ด๋ฆ์ ๋๋ค.
๋ฟก๋ฟก
์ค์ผ์ค๋ฌ๊ฐ ์คํจํ์ง๋ง ์ด์ ๊ฐ ๋ช
ํํ์ง ์์ต๋๋ค.
kube-scheduler ํฌ๋์ ID๋ฅผ ์ฐพ์ ์ ์์ต๋๋ค.
kubectl get pods --all-namespace
๊ทธ๋ฐ ๋ค์ ํด๋น ํฌ๋์์ ํฐ๋ฏธ๋์ ์์ํ๊ธฐ ์ํด ์ด๊ฒ์ ์๋ ํ ์ ์์ต๋๋ค.
kubectl exec -ti <POD-ID-HERE> bash -n kube-system
๊ฑฐ๊ธฐ์์ ๋ก๊ทธ๋ฅผ ์ดํด๋ณผ ์ ์์ต๋๋ค.
cat /var/log
๊ด๋ จ์ฑ์ด ์๋ค๋ ๋ณด์ฅ์ ์์ต๋๋ค.
๋ค๋ฅธ ๋ช ๊ฐ์ง :
init
์ดํ์ pod-network๋ฅผ ์ค์นํ๊ณ ์์ต๋๊น (๊ฒฝ์ฐ์ ๋ฐ๋ผ ์ง๋ฌธ)?init
๋น์ ์ด ์ป๋ ๋ฌด์จ ๋ด์ฉ /var/lib/kubelet/kubeadm-flags.env
?kubectl exec -ti kube-scheduler-beetlejuice bash -n kube-system
๋ผ๊ณ
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown command terminated with exit code 126
๊ทธ๋์ kubectl exec -ti kube-scheduler-beetlejuice sh -n kube-system
์๋ ํ์ผ๋ฏ๋ก sh
๋์ bash
...ํ์ง๋ง ํ์ผ์ด ์์ต๋๋ค /var/log
ls -lAh /var
:
drwxr-xr-x 3 root root 4.0K May 22 17:00 spool
drwxr-xr-x 2 root root 4.0K May 22 17:00 www
์ด์ ๋ ๋จ์ผ ๋ ธ๋ kube๊ฐ ๋ ๊ฒ์ ๋๋ค.
/var/lib/kubelet/kubeadm-flags.env
์ ๋ด์ฉ์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni --resolv-conf=/run/systemd/resolve/resolv.conf
์ค์ผ์ค๋ฌ์ ๋ก๊ทธ๊ฐ ์๋ค๋ ๊ฒ์ด ๋๋ฌด ๋์ฉ๋๋ค.
/var/lib/kubelet/kubeadm-flags.env์ ๋ด์ฉ์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
--resolv-conf
ํ๋๊ทธ๊ฐ ์ถ๊ฐ ๋ ๊ฒ ๊ฐ์ผ๋ฏ๋ก ๊ด์ฐฎ์ต๋๋ค.
์ด์ ๋ ๋จ์ผ ๋ ธ๋ kube๊ฐ ๋ ๊ฒ์ ๋๋ค.
kubeadm reset
์ ํ ํ ๋ค์ kubeadm init ...
๋ค์ ์ ํํ์ธ์.
๊ทธ๋ฐ ๋ค์ ๊ตฌ์ฑ์ ์ฌ์ฉ์ ๋๋ ํ ๋ฆฌ์ ๋ณต์ฌํ๊ณ ํฌ๋ ๋คํธ์ํฌ ํ๋ฌ๊ทธ์ธ (weave)์ ์ค์นํด๋ณด์ญ์์ค.
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
DNS ํฌ๋๊ฐ ์ค๋น๋์๋์ง ํ์ธํฉ๋๋ค. ๋ง์ด ์ ๋ ์๋ ์์ง๋ง ์๋ํด๋ณด์ธ์.
์๋ก์ด ์ด๊ธฐํ ํฌ๋๊ฐ (๋ค์) ๋ค์๊ณผ ๊ฐ์ด ๋ณด์ ๋๋ค.
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcdf6894-lcmg6 0/1 Pending 0 1m
kube-system coredns-78fcdf6894-wd9nt 0/1 Pending 0 1m
kube-system etcd-beetlejuice 1/1 Running 0 18s
kube-system kube-apiserver-beetlejuice 1/1 Running 0 36s
kube-system kube-controller-manager-beetlejuice 1/1 Running 0 12s
kube-system kube-proxy-zrhgj 1/1 Running 0 1m
kube-system kube-scheduler-beetlejuice 1/1 Running 0 24s
weave ํ๋ฌ๊ทธ์ธ์ ์ค์นํ๋ฉด ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcdf6894-lcmg6 1/1 Running 0 2m
kube-system coredns-78fcdf6894-wd9nt 1/1 Running 0 2m
kube-system etcd-beetlejuice 1/1 Running 0 1m
kube-system kube-apiserver-beetlejuice 1/1 Running 0 1m
kube-system kube-controller-manager-beetlejuice 1/1 Running 0 58s
kube-system kube-proxy-zrhgj 1/1 Running 0 2m
kube-system kube-scheduler-beetlejuice 1/1 Running 0 1m
kube-system weave-net-ldxg5 2/2 Running 0 24s
์ด์ ์๋ํฉ๋๋ค ๐
์ง๊ธ์ ๋ฌธ์ ์์ด kubernetes-dashboard
์ค์นํ๋๋ฐ ์ด์ ์๋ ์๋ํ์ง ์์์ต๋๋ค. ๊ทธ๋ฌ๋ ์ด๊ฒ์ด์ด ๋ฌธ์ ์ ๋ํ ํด๊ฒฐ์ฑ
์
๋๊น?
ํ ์คํธ ํด์ฃผ์ ์ ๊ฐ์ฌํฉ๋๋ค.
๊ทธ๋ฌ๋ ์ด๊ฒ์ด์ด ๋ฌธ์ ์ ๋ํ ํด๊ฒฐ์ฑ ์ ๋๊น?
๋๋ ๋งํ ๊ฒ์ด๋ค-์, ์ง๊ธ์.
CLI์ ์ค๋ช
์ ๋ชจ๋ ์ฌ์ฉ์์๊ฒ init
๋ฐ๋ก ๋ค์ pod-network ํ๋ฌ๊ทธ์ธ์ ์ค์นํ๋๋ก ์ง์ํฉ๋๋ค.
์ด ๋จ๊ณ๋ฅผ ๊ฑด๋ ๋ฐ๋ ๊ฒฝ์ฐ ๋ฐ์ํ๋ ์ํฉ์ ์ ํํ ๋ฌธ์ํํ์ง ์์์ง๋ง ํด๋ฌ์คํฐ๊ฐ ์ ๋๋ก ์๋ํ์ง ์์ ๊ฒ์ผ๋ก ์์๋ฉ๋๋ค.
๋๊ตฐ๊ฐ์ด ๋ฌธ์ ๋ฅผ ์ข ๊ฒฐํด์๋ ์๋๋ค๊ณ ์๊ฐํ๋ฉด ๋ค์ ์ด์ด์ฃผ์ธ์.
์ง์กฐ ํ๋ฌ๊ทธ์ธ์ด ํธ๋ฆญ์ ์ํํ์ง ์์์ง๋ง ์ ํํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค. ContainerCreating ์ํ์์ coredns ํฌ๋๊ฐ ๊ณ์ ํ์๋ฉ๋๋ค. ์ด์ ๊ฑฐ์ ํ ์๊ฐ์ด ์ง๋ฌ์ผ๋ฏ๋ก ...
linux-uwkw:~ # kubectl cluster-info
Kubernetes master is running at https://192.168.178.163:6443
KubeDNS is running at https://192.168.178.163:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
linux-uwkw:~ # cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
--resolv-conf
ํ๋๊ทธ๊ฐ ์์ต๋๋ค. ๊ทธ๋์ ์ด๊ฒ์ด ๋ฌธ์ ๋ผ๊ณ ์๊ฐํฉ๋๋ค. kubeqdm์ ๊ตฌ์ฑํ์ฌ ์ถ๊ฐํ๋ ค๋ฉด ์ด๋ป๊ฒํด์ผํฉ๋๊น?--cni-bin-dir=/opt/cni/bin
์ด ์๋ชป๋์์ต๋๋ค. ๋ด ์์คํ
์๋ ์ด๊ฒ์ด ์์ต๋๋ค.linux-uwkw:~ # rpm -ql cni
/etc/cni
/etc/cni/net.d
/etc/cni/net.d/99-loopback.conf.sample
/usr/lib/cni
/usr/lib/cni/noop
/usr/sbin/cnitool
/usr/share/doc/packages/cni
/usr/share/doc/packages/cni/CONTRIBUTING.md
/usr/share/doc/packages/cni/DCO
/usr/share/doc/packages/cni/README.md
/usr/share/licenses/cni
/usr/share/licenses/cni/LICENSE
๋๋ ๊ฑฐ๊ธฐ์ /usr/sbin
๋ฅผ ๋ฃ์ด์ผํ๋ค๊ณ ์๊ฐํ๋ค, ๊ทธ๋ ์ง?
๋ํ ์ค์ผ์ค๋ฌ์ ๋ก๊ทธ๋ ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
linux-uwkw:~ # docker logs k8s_kube-scheduler_kube-scheduler-linux-uwkw_kube-system_a00c35e56ebd0bdfcd77d53674a5d2a1_0
I0813 21:18:19.816990 1 server.go:126] Version: v1.11.2
W0813 21:18:19.821719 1 authorization.go:47] Authorization is disabled
W0813 21:18:19.821744 1 authentication.go:55] Authentication is disabled
I0813 21:18:19.821787 1 insecure_serving.go:47] Serving healthz insecurely on 127.0.0.1:10251
E0813 21:18:25.603025 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope
E0813 21:18:25.603122 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope
E0813 21:18:25.603161 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list services at the cluster scope
E0813 21:18:25.603253 1 reflector.go:205] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list pods at the cluster scope
E0813 21:18:25.603286 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0813 21:18:25.603335 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope
E0813 21:18:25.603364 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope
E0813 21:18:25.603437 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list poddisruptionbudgets.policy at the cluster scope
E0813 21:18:25.603491 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: replicasets.extensions is forbidden: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope
E0813 21:18:25.605642 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list nodes at the cluster scope
E0813 21:18:26.603723 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list persistentvolumes at the cluster scope
E0813 21:18:26.606225 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list services at the cluster scope
E0813 21:18:26.606295 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list replicationcontrollers at the cluster scope
E0813 21:18:26.607860 1 reflector.go:205] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list pods at the cluster scope
E0813 21:18:26.611457 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0813 21:18:26.612777 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list persistentvolumeclaims at the cluster scope
E0813 21:18:26.616076 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list statefulsets.apps at the cluster scope
E0813 21:18:26.616779 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list poddisruptionbudgets.policy at the cluster scope
E0813 21:18:26.619308 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: replicasets.extensions is forbidden: User "system:kube-scheduler" cannot list replicasets.extensions at the cluster scope
E0813 21:18:26.620048 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list nodes at the cluster scope
I0813 21:18:28.429769 1 controller_utils.go:1025] Waiting for caches to sync for scheduler controller
I0813 21:18:28.533687 1 controller_utils.go:1032] Caches are synced for scheduler controller
I0813 21:18:28.533868 1 leaderelection.go:185] attempting to acquire leader lease kube-system/kube-scheduler...
I0813 21:18:28.539621 1 leaderelection.go:194] successfully acquired lease kube-system/kube-scheduler
์ ํํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค
๋๋ ๊ทธ๊ฒ์ด ๊ฐ์ ๋ฌธ์ ๋ผ๊ณ ์๊ฐํ์ง ์๋๋ค.
KubeDNS๋
CoreDNS๋ 1.11.x์ ๊ธฐ๋ณธ DNS ์๋ฒ์ ๋๋ค. ์๋์ ์ผ๋ก ํ์ฑํ ํ์ต๋๊น?
๊ณ ์์ด /var/lib/kubelet/kubeadm-flags.env
/var/lib/kubelet/kubeadm-flags.env
์ kubeadm ๋ฐํ์์์ ์๋ ์์ฑ๋๋ฉฐ ํธ์ง ํ ์ ์์ต๋๋ค.
/etc/default/kubelet
์ ํ๋๊ทธ๋ฅผ ์ถ๊ฐ ํ ์ ์์ต๋๋ค.
์ฌ๊ธฐ์์ ์ ๋ณด๋ฅผ ์ฐธ์กฐํ์ญ์์ค.
https://github.com/kubernetes/kubernetes/blob/master/build/debs/10-kubeadm.conf
--resolv-conf ํ๋๊ทธ๊ฐ ์์ต๋๋ค.
๋ฐฐํฌํ์ด ๋ค์์ ์ฌ์ฉํ๋ ๊ฒฝ์ฐ์๋ง ์ถ๊ฐ๋ฉ๋๋ค.
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html
--cni-bin-dir = / opt / cni / bin --cni-conf-dir = / etc / cni / net.d
์ด๊ฒ์ด ๊ธฐ๋ณธ๊ฐ์ด์ง๋ง AFAIK๋ ๋ฐํ์์ ์๋ ์ถ๊ฐ๋์ง ์์์ผํฉ๋๋ค.
๊ทํ์ ๊ฒฝ์ฐ์ ๋ฌธ์ ๊ฐ ๋ฌด์์ธ์ง ๋งํ๊ธฐ๊ฐ ์ด๋ ต์ต๋๋ค.
๋ ๋์ ์๋ก์ด ๋ฌธ์ ๋ฅผ ์ด๊ณ ๋ฌธ์ ๋ณด๊ณ ์ ํ
ํ๋ฆฟ์ ๋ฐ๋ฅด์ญ์์ค.
๊ฒฐ๊ตญ ๋ด ๋ฌธ์ ๋ฅผ ๋ฐ๊ฒฌํ์ต๋๋ค. /opt/cni/bin
์์ CNI์ loopback
๋ฐ์ด๋๋ฆฌ๊ฐ ๋๋ฝ๋์์ต๋๋ค.
cd /opt/cni/bin
curl -L -O https://github.com/containernetworking/cni/releases/download/v0.4.0/cni-amd64-v0.4.0.tgz
tar -xf cni-amd64-v0.4.0.tgz
systemctl restart kubelet
๋๋ ๊ฐ์ ๋ฌธ์ ๋ฅผ ๋ง๋ฌ๊ณ ํ๋๋ฌ์ ์ค์นํ์ฌ ํด๊ฒฐํ์ต๋๋ค. ํ๋๋ฌ์ ์ค์น ํ ํ coredns ๋ฐ ๊ธฐํ ํฌ๋๋ฅผ ์ฌ๋ฐ๋ฅด๊ฒ ์์ฑ ํ ์ ์์ต๋๋ค.
ํ๋๋ฌ์ ์ค์นํ๋ฉด ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋๋ ์ด์ ๋ ๋ฌด์์ ๋๊น?
๋ํ kubeadm์ ์ฌ์ฉํ์ฌ K8s ํด๋ฌ์คํฐ๋ฅผ ์ด๊ธฐํํ๊ณ --pod-network-cidr
์ธ์๋ฅผ ์ถ๊ฐํฉ๋๋ค. ๊ทธ๋์ ํ๋๋ฌ์ด๋ ๋ค๋ฅธ ํฌ๋ ๋คํธ์ํฌ ์ ๋์จ์ด ์ค์น๋์ด ์์ง ์์ผ๋ฉด kubelet์ด ํฌ๋์ ip๋ฅผ ํ ๋นํ๋ ๋ฐฉ๋ฒ์ ์ ์์๊ณ ์ค๋ฅ๊ฐ ๋ฐ์ํ๋ค๊ณ ์๊ฐํฉ๋๋ค.
journal -xeu kubelet
์ฌ์ฉํ์ฌ ๋ค์ ๋ฉ์์ง๊ฐ ํ์๋๋ค๊ณ ๋งํ์ต๋๋ค.
26710 kubelet.go:2112] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninit
Jul 09 21:54:49 beetlejuice kubelet[26710]: E0709 21:54:49.566069 26710 dns.go:131] Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 213.133.99.99 213.133.98.98 213.133.100.100
Jul 09 21:54:53 beetlejuice kubelet[26710]: W0709 21:54:53.884846 26710 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
๋๋ ๋ํ ๋ด ์ปดํจํฐ์์ ๋น์ทํ ๋ก๊ทธ๋ฅผ ๋ณด์์ผ๋ฏ๋ก ์ด๊ฒ์ด ๋ฌธ์ ์ ์์ธ์ด๋ผ๊ณ ์๊ฐํฉ๋๋ค.
ํ๋๋ฌ์ ์ค์นํ๋ฉด ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋๋ ์ด์ ๋ ๋ฌด์์ ๋๊น?
CNI ํ๋ฌ๊ทธ์ธ์ ์ค์นํด์ผํฉ๋๋ค.
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm
Installing a pod network add-on
You must install a pod network add-on so that your pods can communicate with each other.
CNI ํ๋ฌ๊ทธ์ธ์ ์ค์นํด์ผํฉ๋๋ค.
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm
kubernetes๋ฅผ ์ด๊ธฐํ ํ ํ ํฌ๋ ๋คํธ์ํฌ ์ ๋์จ์ด ์ค์น๋์ง ์์๊ธฐ ๋๋ฌธ์ coredns๋ฅผ ์์ํ ์ ์์ต๋๋ค. ์ด์ ์๊ณ ์์ต๋๋ค. ๊ฐ์ฌํฉ๋๋ค.
์ค์ผ์ค๋ฌ์ ๋ก๊ทธ๊ฐ ์๋ค๋ ๊ฒ์ด ๋๋ฌด ๋์ฉ๋๋ค.
/var/lib/kubelet/kubeadm-flags.env์ ๋ด์ฉ์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
--resolv-conf
ํ๋๊ทธ๊ฐ ์ถ๊ฐ ๋ ๊ฒ ๊ฐ์ผ๋ฏ๋ก ๊ด์ฐฎ์ต๋๋ค.์ด์ ๋ ๋จ์ผ ๋ ธ๋ kube๊ฐ ๋ ๊ฒ์ ๋๋ค.
kubeadm reset
์ ํ ํ ๋ค์kubeadm init ...
๋ค์ ์ ํํ์ธ์.
๊ทธ๋ฐ ๋ค์ ๊ตฌ์ฑ์ ์ฌ์ฉ์ ๋๋ ํ ๋ฆฌ์ ๋ณต์ฌํ๊ณ ํฌ๋ ๋คํธ์ํฌ ํ๋ฌ๊ทธ์ธ (weave)์ ์ค์นํด๋ณด์ญ์์ค.kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
DNS ํฌ๋๊ฐ ์ค๋น๋์๋์ง ํ์ธํฉ๋๋ค. ๋ง์ด ์ ๋ ์๋ ์์ง๋ง ์๋ํด๋ณด์ธ์.
๊ทธ๊ฒ์ ๋๋ฅผ ์ํด ์๋ํฉ๋๋ค! ๊ณ ๋ง์
์ค์ผ์ค๋ฌ์ ๋ก๊ทธ๊ฐ ์๋ค๋ ๊ฒ์ด ๋๋ฌด ๋์ฉ๋๋ค.
/var/lib/kubelet/kubeadm-flags.env์ ๋ด์ฉ์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
--resolv-conf
ํ๋๊ทธ๊ฐ ์ถ๊ฐ ๋ ๊ฒ ๊ฐ์ผ๋ฏ๋ก ๊ด์ฐฎ์ต๋๋ค.์ด์ ๋ ๋จ์ผ ๋ ธ๋ kube๊ฐ ๋ ๊ฒ์ ๋๋ค.
kubeadm reset
์ ํ ํ ๋ค์kubeadm init ...
๋ค์ ์ ํํ์ธ์.
๊ทธ๋ฐ ๋ค์ ๊ตฌ์ฑ์ ์ฌ์ฉ์ ๋๋ ํ ๋ฆฌ์ ๋ณต์ฌํ๊ณ ํฌ๋ ๋คํธ์ํฌ ํ๋ฌ๊ทธ์ธ (weave)์ ์ค์นํด๋ณด์ญ์์ค.kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
DNS ํฌ๋๊ฐ ์ค๋น๋์๋์ง ํ์ธํฉ๋๋ค. ๋ง์ด ์ ๋ ์๋ ์์ง๋ง ์๋ํด๋ณด์ธ์.
serviceaccount / weave-net ๊ตฌ์ฑ
clusterrole.rbac.authorization.k8s.io/weave-net ๊ตฌ์ฑ
clusterrolebinding.rbac.authorization.k8s.io/weave-net ๊ตฌ์ฑ
role.rbac.authorization.k8s.io/weave-net ๊ตฌ์ฑ
rolebinding.rbac.authorization.k8s.io/weave-net ๊ตฌ์ฑ
"extensions / v1beta1"๋ฒ์ ์์ "DaemonSet"์ข
๋ฅ์ ์ผ์นํ๋ ํญ๋ชฉ์ด ์์์ ์ธ์ ํ ์ ์์ต๋๋ค.
๊ตฌ์ฑ ํ ์ ์์ต๋๋ค.
"extensions / v1beta1"๋ฒ์ ์์ "DaemonSet"์ข ๋ฅ์ ์ผ์นํ๋ ํญ๋ชฉ์ด ์์์ ์ธ์ ํ ์ ์์ต๋๋ค.
๊ทธ๊ฒ์ CNI ํ๋ฌ๊ทธ์ธ ์ธก์ ๋ฒ๊ทธ์
๋๋ค.
๋์ Callico CNI ํ๋ฌ๊ทธ์ธ์ ์ฌ์ฉํด๋ณด์ญ์์ค.
๊ฐ์ฅ ์ ์ฉํ ๋๊ธ
์ค์ผ์ค๋ฌ์ ๋ก๊ทธ๊ฐ ์๋ค๋ ๊ฒ์ด ๋๋ฌด ๋์ฉ๋๋ค.
--resolv-conf
ํ๋๊ทธ๊ฐ ์ถ๊ฐ ๋ ๊ฒ ๊ฐ์ผ๋ฏ๋ก ๊ด์ฐฎ์ต๋๋ค.kubeadm reset
์ ํ ํ ๋ค์kubeadm init ...
๋ค์ ์ ํํ์ธ์.๊ทธ๋ฐ ๋ค์ ๊ตฌ์ฑ์ ์ฌ์ฉ์ ๋๋ ํ ๋ฆฌ์ ๋ณต์ฌํ๊ณ ํฌ๋ ๋คํธ์ํฌ ํ๋ฌ๊ทธ์ธ (weave)์ ์ค์นํด๋ณด์ญ์์ค.
DNS ํฌ๋๊ฐ ์ค๋น๋์๋์ง ํ์ธํฉ๋๋ค. ๋ง์ด ์ ๋ ์๋ ์์ง๋ง ์๋ํด๋ณด์ธ์.