Kubeadm: CoreDNS๊ฐ€ k8s 1.11 ๋ฐ weave(CentOS 7)์—์„œ ์‹œ์ž‘๋˜์ง€ ์•Š์Œ

์— ๋งŒ๋“  2018๋…„ 07์›” 17์ผ  ยท  33์ฝ”๋ฉ˜ํŠธ  ยท  ์ถœ์ฒ˜: kubernetes/kubeadm

์ด๊ฒƒ์€ ๋ฒ„๊ทธ ๋ณด๊ณ ์„œ์ž…๋‹ˆ๊นŒ ์•„๋‹ˆ๋ฉด ๊ธฐ๋Šฅ ์š”์ฒญ์ž…๋‹ˆ๊นŒ?

๋ฒ„๊ทธ ๋ณด๊ณ 

๋ฒ„์ „

kubeadm ๋ฒ„์ „ 1.11

ํ™˜๊ฒฝ :

  • Kubernetes ๋ฒ„์ „ ( kubectl version ์‚ฌ์šฉ): 1.11
  • ํด๋ผ์šฐ๋“œ ๊ณต๊ธ‰์ž ๋˜๋Š” ํ•˜๋“œ์›จ์–ด ๊ตฌ์„ฑ : aws ec2(16vcpus 64gb RAM)
  • OS (์˜ˆ: /etc/os-release): centos 7
  • ์ปค๋„ (์˜ˆ: uname -a ): 3.10.0-693.17.1.el7.x86_64
  • ๊ธฐํƒ€ : cni ์• ๋“œ์˜จ์œผ๋กœ weave

๋ฌด์Šจ ์ผ์ด์—์š”?

kubeadm init ํ›„ coreos ํฌ๋“œ๊ฐ€ ์˜ค๋ฅ˜ ์ƒํƒœ๋กœ ์œ ์ง€๋จ

NAME                                   READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-ljdjp               0/1       Error     6          9m
coredns-78fcdf6894-p6flm               0/1       Error     6          9m
etcd-master                            1/1       Running   0          8m
heapster-5bbdfbff9f-h5h2n              1/1       Running   0          9m
kube-apiserver-master                  1/1       Running   0          8m
kube-controller-manager-master         1/1       Running   0          8m
kube-proxy-5642r                       1/1       Running   0          9m
kube-scheduler-master                  1/1       Running   0          8m
kubernetes-dashboard-6948bdb78-bwkvx   1/1       Running   0          9m
weave-net-r5jkg                        2/2       Running   0          9m

๋‘ ํฌ๋“œ์˜ ๋กœ๊ทธ์—๋Š” ๋‹ค์Œ์ด ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.
standard_init_linux.go:178: exec user process caused "operation not permitted"

kindocumentation lifecyclactive prioritimportant-soon
์ถœ์ฒ˜
picture carlosrmendes
👍3

๊ฐ€์žฅ ์œ ์šฉํ•œ ๋Œ“๊ธ€

๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค, @chrisohaver !

์ด๊ฒƒ์€ ํšจ๊ณผ๊ฐ€ ์žˆ์—ˆ๋‹ค:

kubectl -n kube-system get deployment coredns -o yaml | \
  sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
  kubectl apply -f -
picture kuznero ์— 2018๋…„ 08์›” 10์ผ
👍17 🎉5 ❤3

๋ชจ๋“  33 ๋Œ“๊ธ€

@kubernetes/sig-network-bugs

@carlosmkb , ๋„์ปค ๋ฒ„์ „์ด ๋ฌด์—‡์ž…๋‹ˆ๊นŒ?

neolit123 picture neolit123 ์— 2018๋…„ 07์›” 17์ผ

๋ฏฟ๊ธฐ โ€‹โ€‹์–ด๋ ต์Šต๋‹ˆ๋‹ค. ์šฐ๋ฆฌ ์ธก์—์„œ CentOS 7์„ ๊ฝค ๊ด‘๋ฒ”์œ„ํ•˜๊ฒŒ ํ…Œ์ŠคํŠธํ•ฉ๋‹ˆ๋‹ค.

์‹œ์Šคํ…œ ๋ฐ ํฌ๋“œ ๋กœ๊ทธ๊ฐ€ ์žˆ์Šต๋‹ˆ๊นŒ?

timothysc picture timothysc ์— 2018๋…„ 07์›” 17์ผ
👎8

์ด ํ•˜๋‚˜? https://stackoverflow.com/questions/44127247/does-anyone-know-a-workaround-for-no-new-privileges-blocking-selinux-transitions

dims picture dims ์— 2018๋…„ 07์›” 17์ผ

@dims , ์˜๋ฏธ๊ฐ€ ์žˆ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‹œ๋„ํ•ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

@neolit123 ๋ฐ @timothysc

๋„์ปค ๋ฒ„์ „: docker-1.13.1-63.git94f4240.el7.centos.x86_64

coredns ํฌ๋“œ ๋กœ๊ทธ : standard_init_linux.go:178: exec user process caused "operation not permitted"
์‹œ์Šคํ…œ ๋กœ๊ทธ journalctl -xeu kubelet :

Jul 17 23:45:17 server.raid.local kubelet[20442]: E0717 23:45:17.679867   20442 pod_workers.go:186] Error syncing pod dd030886-89f4-11e8-9786-0a92797fa29e ("cas-7d6d97c7bd-mzw5j_raidcloud(dd030886-89f4-11e8-9786-0a92797fa29e)"), skipping: failed to "StartContainer" for "cas" with ImagePullBackOff: "Back-off pulling image \"registry.raidcloud.io/raidcloud/cas:180328.pvt.01\""
Jul 17 23:45:18 server.raid.local kubelet[20442]: I0717 23:45:18.679059   20442 kuberuntime_manager.go:513] Container {Name:json2ldap Image:registry.raidcloud.io/raidcloud/json2ldap:180328.pvt.01 Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:default-token-f2cmq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:30,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Jul 17 23:45:18 server.raid.local kubelet[20442]: E0717 23:45:18.680001   20442 pod_workers.go:186] Error syncing pod dcc39ce2-89f4-11e8-9786-0a92797fa29e ("json2ldap-666fc85686-tmxrr_raidcloud(dcc39ce2-89f4-11e8-9786-0a92797fa29e)"), skipping: failed to "StartContainer" for "json2ldap" with ImagePullBackOff: "Back-off pulling image \"registry.raidcloud.io/raidcloud/json2ldap:180328.pvt.01\""
Jul 17 23:45:21 server.raid.local kubelet[20442]: I0717 23:45:21.678232   20442 kuberuntime_manager.go:513] Container {Name:coredns Image:k8s.gcr.io/coredns:1.1.3 Command:[] Args:[-conf /etc/coredns/Corefile] WorkingDir: Ports:[{Name:dns HostPort:0 ContainerPort:53 Protocol:UDP HostIP:} {Name:dns-tcp HostPort:0 ContainerPort:53 Protocol:TCP HostIP:} {Name:metrics HostPort:0 ContainerPort:9153 Protocol:TCP HostIP:}] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:178257920 scale:0} d:{Dec:<nil>} s:170Mi Format:BinarySI}] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} memory:{i:{value:73400320 scale:0} d:{Dec:<nil>} s:70Mi Format:BinarySI}]} VolumeMounts:[{Name:config-volume ReadOnly:true MountPath:/etc/coredns SubPath: MountPropagation:<nil>} {Name:coredns-token-6nhgg ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:5,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[NET_BIND_SERVICE],Drop:[all],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:*false,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Jul 17 23:45:21 server.raid.local kubelet[20442]: I0717 23:45:21.678311   20442 kuberuntime_manager.go:757] checking backoff for container "coredns" in pod "coredns-78fcdf6894-znfvw_kube-system(9b44aa92-89f7-11e8-9786-0a92797fa29e)"
Jul 17 23:45:21 server.raid.local kubelet[20442]: I0717 23:45:21.678404   20442 kuberuntime_manager.go:767] Back-off 5m0s restarting failed container=coredns pod=coredns-78fcdf6894-znfvw_kube-system(9b44aa92-89f7-11e8-9786-0a92797fa29e)
Jul 17 23:45:21 server.raid.local kubelet[20442]: E0717 23:45:21.678425   20442 pod_workers.go:186] Error syncing pod 9b44aa92-89f7-11e8-9786-0a92797fa29e ("coredns-78fcdf6894-znfvw_kube-system(9b44aa92-89f7-11e8-9786-0a92797fa29e)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=coredns pod=coredns-78fcdf6894-znfvw_kube-system(9b44aa92-89f7-11e8-9786-0a92797fa29e)"
Jul 17 23:45:22 server.raid.local kubelet[20442]: I0717 23:45:22.679145   20442 kuberuntime_manager.go:513] Container {Name:login Image:registry.raidcloud.io/raidcloud/admin:180329.pvt.05 Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:login-config ReadOnly:true MountPath:/usr/share/nginx/conf/ SubPath: MountPropagation:<nil>} {Name:default-token-f2cmq ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Jul 17 23:45:22 server.raid.local kubelet[20442]: E0717 23:45:22.679941   20442 pod_workers.go:186] Error syncing pod dc8392a9-89f4-11e8-9786-0a92797fa29e ("login-85ffb66bb8-5l9fq_raidcloud(dc8392a9-89f4-11e8-9786-0a92797fa29e)"), skipping: failed to "StartContainer" for "login" with ImagePullBackOff: "Back-off pulling image \"registry.raidcloud.io/raidcloud/admin:180329.pvt.05\""
Jul 17 23:45:23 server.raid.local kubelet[20442]: I0717 23:45:23.678172   20442 kuberuntime_manager.go:513] Container {Name:coredns Image:k8s.gcr.io/coredns:1.1.3 Command:[] Args:[-conf /etc/coredns/Corefile] WorkingDir: Ports:[{Name:dns HostPort:0 ContainerPort:53 Protocol:UDP HostIP:} {Name:dns-tcp HostPort:0 ContainerPort:53 Protocol:TCP HostIP:} {Name:metrics HostPort:0 ContainerPort:9153 Protocol:TCP HostIP:}] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:178257920 scale:0} d:{Dec:<nil>} s:170Mi Format:BinarySI}] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI} memory:{i:{value:73400320 scale:0} d:{Dec:<nil>} s:70Mi Format:BinarySI}]} VolumeMounts:[{Name:config-volume ReadOnly:true MountPath:/etc/coredns SubPath: MountPropagation:<nil>} {Name:coredns-token-6nhgg ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:5,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[NET_BIND_SERVICE],Drop:[all],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:*false,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Jul 17 23:45:23 server.raid.local kubelet[20442]: I0717 23:45:23.678412   20442 kuberuntime_manager.go:757] checking backoff for container "coredns" in pod "coredns-78fcdf6894-lcqt5_kube-system(9b45a068-89f7-11e8-9786-0a92797fa29e)"
Jul 17 23:45:23 server.raid.local kubelet[20442]: I0717 23:45:23.678532   20442 kuberuntime_manager.go:767] Back-off 5m0s restarting failed container=coredns pod=coredns-78fcdf6894-lcqt5_kube-system(9b45a068-89f7-11e8-9786-0a92797fa29e)
Jul 17 23:45:23 server.raid.local kubelet[20442]: E0717 23:45:23.678554   20442 pod_workers.go:186] Error syncing pod 9b45a068-89f7-11e8-9786-0a92797fa29e ("coredns-78fcdf6894-lcqt5_kube-system(9b45a068-89f7-11e8-9786-0a92797fa29e)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=coredns pod=coredns-78fcdf6894-lcqt5_kube-system(9b45a068-89f7-11e8-9786-0a92797fa29e)"
carlosrmendes picture carlosrmendes ์— 2018๋…„ 07์›” 18์ผ

๊ณผ๊ฑฐ์— ๋‹ค๋ฅธ ์‹œ๋‚˜๋ฆฌ์˜ค์—์„œ ๋ณด๊ณ ๋œ ๋™์ผํ•œ ์˜ค๋ฅ˜์˜ ๋ช‡ ๊ฐ€์ง€ ์ธ์Šคํ„ด์Šค๋ฅผ ์ฐพ์•˜์Šต๋‹ˆ๋‹ค.
CoreDNS ๋ฐฐํฌ์—์„œ "allowPrivilegeEscalation: false"๋ฅผ ์ œ๊ฑฐํ•˜์—ฌ ๋„์›€์ด ๋˜๋Š”์ง€ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

chrisohaver picture chrisohaver ์— 2018๋…„ 07์›” 19์ผ
❤3

์ €์—๊ฒŒ๋„ ๊ฐ™์€ ๋ฌธ์ œ์ž…๋‹ˆ๋‹ค. ์œ ์‚ฌํ•œ ์„ค์ • CentOS 7.4.1708, Docker ๋ฒ„์ „ 1.13.1, ๋นŒ๋“œ 94f4240/1.13.1(CentOS์™€ ํ•จ๊ป˜ ์ œ๊ณต):

[root@faas-A01 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                 READY     STATUS             RESTARTS   AGE
kube-system   calico-node-2vssv                                    2/2       Running            0          9m
kube-system   calico-node-4vr7t                                    2/2       Running            0          7m
kube-system   calico-node-nlfnd                                    2/2       Running            0          17m
kube-system   calico-node-rgw5w                                    2/2       Running            0          23m
kube-system   coredns-78fcdf6894-p4wbl                             0/1       CrashLoopBackOff   9          30m
kube-system   coredns-78fcdf6894-r4pwf                             0/1       CrashLoopBackOff   9          30m
kube-system   etcd-faas-a01.sl.cloud9.ibm.com                      1/1       Running            0          29m
kube-system   kube-apiserver-faas-a01.sl.cloud9.ibm.com            1/1       Running            0          29m
kube-system   kube-controller-manager-faas-a01.sl.cloud9.ibm.com   1/1       Running            0          29m
kube-system   kube-proxy-55csj                                     1/1       Running            0          17m
kube-system   kube-proxy-56r8c                                     1/1       Running            0          30m
kube-system   kube-proxy-kncql                                     1/1       Running            0          9m
kube-system   kube-proxy-mf2bp                                     1/1       Running            0          7m
kube-system   kube-scheduler-faas-a01.sl.cloud9.ibm.com            1/1       Running            0          29m
[root@faas-A01 ~]# kubectl logs --namespace=all coredns-78fcdf6894-p4wbl
Error from server (NotFound): namespaces "all" not found
[root@faas-A01 ~]# kubectl logs --namespace=kube-system coredns-78fcdf6894-p4wbl
standard_init_linux.go:178: exec user process caused "operation not permitted"
sectorsize512 picture sectorsize512 ์— 2018๋…„ 07์›” 19์ผ

๋งŒ์ผ์„ ๋Œ€๋น„ํ•˜์—ฌ selinux๋Š” ๋ชจ๋“  ๋…ธ๋“œ์—์„œ ํ—ˆ์šฉ ๋ชจ๋“œ์ž…๋‹ˆ๋‹ค.

sectorsize512 picture sectorsize512 ์— 2018๋…„ 07์›” 19์ผ

์ €๋Š” Calico๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค(@carlosmkb๋กœ ์งœ์ง€ ์•Š์Œ).

sectorsize512 picture sectorsize512 ์— 2018๋…„ 07์›” 19์ผ

[ root@faas-A01 ~]# kubectl ๋กœ๊ทธ --namespace=kube-system coredns-78fcdf6894-p4wbl
standard_init_linux.go:178: exec ์‚ฌ์šฉ์ž ํ”„๋กœ์„ธ์Šค๋กœ ์ธํ•ด "์ž‘์—…์ด ํ—ˆ์šฉ๋˜์ง€ ์•Š์Œ"์ด ๋ฐœ์ƒํ–ˆ์Šต๋‹ˆ๋‹ค.

์•„ - ์ด๊ฒƒ์€ ๋กœ๊ทธ์˜ ๋‚ด์šฉ์ด ์•„๋‹ˆ๋ผ ๋กœ๊ทธ๋ฅผ ๊ฐ€์ ธ์˜ค๋ ค๊ณ  ํ•  ๋•Œ kubectl์˜ ์˜ค๋ฅ˜์ž…๋‹ˆ๋‹ค...

chrisohaver picture chrisohaver ์— 2018๋…„ 07์›” 23์ผ

@chrisohaver kubectl logs ๋Š” ๋‹ค๋ฅธ kube ์‹œ์Šคํ…œ ํฌ๋“œ์™€ ํ•จ๊ป˜ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค.

carlosrmendes picture carlosrmendes ์— 2018๋…„ 07์›” 23์ผ

ํ™•์ธ - CoreDNS ๋ฐฐํฌ์—์„œ "allowPrivilegeEscalation: false"๋ฅผ ์ œ๊ฑฐํ•˜์—ฌ ๋„์›€์ด ๋˜๋Š”์ง€ ํ™•์ธํ•ด๋ณด์…จ์Šต๋‹ˆ๊นŒ?

chrisohaver picture chrisohaver ์— 2018๋…„ 07์›” 23์ผ
👍1

... coredns ํฌ๋“œ์˜ kubectl describe ๊ฐ€ ํฅ๋ฏธ๋กœ์šด ๊ฒƒ์„ ๋ณด์—ฌ์ค๋‹ˆ๊นŒ?

chrisohaver picture chrisohaver ์— 2018๋…„ 07์›” 23์ผ

์ €์—๊ฒŒ๋„ ๊ฐ™์€ ๋ฌธ์ œ์ž…๋‹ˆ๋‹ค.
CentOS Linux ๋ฆด๋ฆฌ์Šค 7.5.1804(์ฝ”์–ด)
Docker ๋ฒ„์ „ 1.13.1, ๋นŒ๋“œ dded712/1.13.1
cni ์• ๋“œ์˜จ์œผ๋กœ ํ”Œ๋ž€๋„ฌ

[root<strong i="9">@k8s</strong> ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY     STATUS             RESTARTS   AGE
kube-system   coredns-78fcdf6894-cfmm7             0/1       CrashLoopBackOff   12         15m
kube-system   coredns-78fcdf6894-k65js             0/1       CrashLoopBackOff   11         15m
kube-system   etcd-k8s.master                      1/1       Running            0          14m
kube-system   kube-apiserver-k8s.master            1/1       Running            0          13m
kube-system   kube-controller-manager-k8s.master   1/1       Running            0          14m
kube-system   kube-flannel-ds-fts6v                1/1       Running            0          14m
kube-system   kube-proxy-4tdb5                     1/1       Running            0          15m
kube-system   kube-scheduler-k8s.master            1/1       Running            0          14m
[root<strong i="10">@k8s</strong> ~]# kubectl logs coredns-78fcdf6894-cfmm7 -n kube-system
standard_init_linux.go:178: exec user process caused "operation not permitted"
[root<strong i="11">@k8s</strong> ~]# kubectl describe pods coredns-78fcdf6894-cfmm7 -n kube-system
Name:           coredns-78fcdf6894-cfmm7
Namespace:      kube-system
Node:           k8s.master/192.168.150.40
Start Time:     Fri, 27 Jul 2018 00:32:09 +0800
Labels:         k8s-app=kube-dns
                pod-template-hash=3497892450
Annotations:    <none>
Status:         Running
IP:             10.244.0.12
Controlled By:  ReplicaSet/coredns-78fcdf6894
Containers:
  coredns:
    Container ID:  docker://3b7670fbc07084410984d7e3f8c0fa1b6d493a41d2a4e32f5885b7db9d602417
    Image:         k8s.gcr.io/coredns:1.1.3
    Image ID:      docker-pullable://k8s.gcr.io/coredns<strong i="12">@sha256</strong>:db2bf53126ed1c761d5a41f24a1b82a461c85f736ff6e90542e9522be4757848
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 27 Jul 2018 00:46:30 +0800
      Finished:     Fri, 27 Jul 2018 00:46:30 +0800
    Ready:          False
    Restart Count:  12
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-vqslm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-vqslm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-vqslm
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From                 Message
  ----     ------            ----                ----                 -------
  Warning  FailedScheduling  16m (x6 over 16m)   default-scheduler    0/1 nodes are available: 1 node(s) were not ready.
  Normal   Scheduled         16m                 default-scheduler    Successfully assigned kube-system/coredns-78fcdf6894-cfmm7 to k8s.master
  Warning  BackOff           14m (x10 over 16m)  kubelet, k8s.master  Back-off restarting failed container
  Normal   Pulled            14m (x5 over 16m)   kubelet, k8s.master  Container image "k8s.gcr.io/coredns:1.1.3" already present on machine
  Normal   Created           14m (x5 over 16m)   kubelet, k8s.master  Created container
  Normal   Started           14m (x5 over 16m)   kubelet, k8s.master  Started container
  Normal   Pulled            11m (x4 over 12m)   kubelet, k8s.master  Container image "k8s.gcr.io/coredns:1.1.3" already present on machine
  Normal   Created           11m (x4 over 12m)   kubelet, k8s.master  Created container
  Normal   Started           11m (x4 over 12m)   kubelet, k8s.master  Started container
  Warning  BackOff           2m (x56 over 12m)   kubelet, k8s.master  Back-off restarting failed container
[root<strong i="13">@k8s</strong> ~]# uname
Linux
[root<strong i="14">@k8s</strong> ~]# uname -a
Linux k8s.master 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root<strong i="15">@k8s</strong> ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[root<strong i="16">@k8s</strong> ~]# docker --version
Docker version 1.13.1, build dded712/1.13.1
Leozki picture Leozki ์— 2018๋…„ 07์›” 26์ผ

selinux๊ฐ€ ํ—ˆ์šฉ ๋ชจ๋“œ์— ์žˆ์„ ๋•Œ๋„ ๋™์ผํ•œ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. /etc/selinux/conf SELINUX=disabled์—์„œ ๋น„ํ™œ์„ฑํ™”ํ•˜๊ณ  ์‹œ์Šคํ…œ์„ ์žฌ๋ถ€ํŒ…ํ•˜๋ฉด ํฌ๋“œ๊ฐ€ ์‹œ์ž‘๋ฉ๋‹ˆ๋‹ค.

๋ ˆ๋“œํ–‡ 7.4, ์ปค๋„ 3.10.0-693.11.6.el7.x86_64
๋„์ปค-1.13.1-68.gitdded712.el7.x86_64

zimnyjakub picture zimnyjakub ์— 2018๋…„ 07์›” 27์ผ

์ฐธ๊ณ ๋กœ SELinux๊ฐ€ ๋น„ํ™œ์„ฑํ™”๋œ ์ƒํƒœ์—์„œ๋„ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค(ํ—ˆ์šฉ๋˜์ง€๋Š” ์•Š์ง€๋งŒ _disabled_).
Docker ๋ฒ„์ „ 1.13.1, ๋นŒ๋“œ dded712/1.13.1
์„ผํŠธOS 7

[root<strong i="8">@centosk8s</strong> ~]# kubectl logs coredns-78fcdf6894-rhx9p -n kube-system
.:53
CoreDNS-1.1.3
linux/amd64, go1.10.1, b0fd575c
2018/07/27 16:37:31 [INFO] CoreDNS-1.1.3
2018/07/27 16:37:31 [INFO] linux/amd64, go1.10.1, b0fd575c
2018/07/27 16:37:31 [INFO] plugin/reload: Running configuration MD5 = 2a066f12ec80aeb2b92740dd74c17138
chrisohaver picture chrisohaver ์— 2018๋…„ 07์›” 27์ผ

์šฐ๋ฆฌ๋„ ์ด ๋ฌธ์ œ๋ฅผ ๊ฒช๊ณ  ์žˆ์œผ๋ฉฐ ์ž๋™ํ™”๋ฅผ ํ†ตํ•ด ์ธํ”„๋ผ๋ฅผ ํ”„๋กœ๋น„์ €๋‹ํ•˜๋ฏ€๋กœ selinux๋ฅผ ์™„์ „ํžˆ ๋น„ํ™œ์„ฑํ™”ํ•˜๊ธฐ ์œ„ํ•ด ๋‹ค์‹œ ์‹œ์ž‘ํ•ด์•ผ ํ•˜๋Š” ๊ฒƒ์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ด ๋ฌธ์ œ๊ฐ€ ํ•ด๊ฒฐ๋˜๊ธฐ๋ฅผ ๊ธฐ๋‹ค๋ฆฌ๋Š” ๋‹ค๋ฅธ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์ด ์žˆ์Šต๋‹ˆ๊นŒ?

lareeth picture lareeth ์— 2018๋…„ 07์›” 30์ผ

CoreDNS ๋ฐฐํฌ์—์„œ "allowPrivilegeEscalation: false"๋ฅผ ์ œ๊ฑฐํ•˜์—ฌ ๋„์›€์ด ๋˜๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.
์ตœ์‹  ๋ฒ„์ „์˜ docker(1.13 ์ด์ƒ)๋กœ ์—…๋ฐ์ดํŠธํ•˜๋Š” ๊ฒƒ๋„ ๋„์›€์ด ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

chrisohaver picture chrisohaver ์— 2018๋…„ 07์›” 30์ผ

์—ฌ๊ธฐ์„œ๋„ ๊ฐ™์€ ๋ฌธ์ œ
๋„์ปค ๋ฒ„์ „ 1.2.6
์„ผํŠธOS 7
@lareeth์™€ ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ kubeadm ์„ ์‚ฌ์šฉํ•  ๋•Œ kubernetes ์ž๋™ํ™”๋ฅผ ํ”„๋กœ๋น„์ €๋‹ํ•˜๊ณ  selinux๋ฅผ ์™„์ „ํžˆ ๋น„ํ™œ์„ฑํ™”ํ•˜๊ธฐ ์œ„ํ•ด ๋‹ค์‹œ ์‹œ์ž‘ํ•ด์•ผ ํ•˜๋Š” ๊ฒƒ๋„ ํ—ˆ์šฉ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
@chrisohaver ๋Š” selinux๋ฅผ ์™„์ „ํžˆ ๋น„ํ™œ์„ฑํ™”ํ•˜๊ธฐ ์œ„ํ•ด ๋‹ค์‹œ ์‹œ์ž‘์„ ์š”๊ตฌํ•˜๋Š” ๊ฒƒ์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์œ ์šฉํ•˜๊ฒŒ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค !
๊ทธ๋Ÿฌ๋‚˜ ๋‚ด๊ฐ€ ์•Œ๊ณ  ์žˆ๋Š” ๋ฐ”์™€ ๊ฐ™์ด coredns ์˜ต์…˜์€ kubeadm ๊ตฌ์„ฑ์—์„œ ์„ค์ •๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
๋‹ค๋ฅธ ๋ฐฉ๋ฒ•์€ ์—†๋‚˜์š”?

thdrnsdk picture thdrnsdk ์— 2018๋…„ 08์›” 01์ผ

CoreDNS ๋ฐฐํฌ์—์„œ "allowPrivilegeEscalation: false"๋ฅผ ์ œ๊ฑฐํ•˜์—ฌ ๋„์›€์ด ๋˜๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.
์ตœ์‹  ๋ฒ„์ „์˜ docker(์˜ˆ: k8s์—์„œ ๊ถŒ์žฅํ•˜๋Š” ๋ฒ„์ „)๋กœ ์—…๋ฐ์ดํŠธํ•˜๋Š” ๊ฒƒ๋„ ๋„์›€์ด ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

chrisohaver picture chrisohaver ์— 2018๋…„ 08์›” 01์ผ

coredns ๋ฐฐํฌ์—์„œ "allowPrivilegeEscalation: false"๋ฅผ ์ œ๊ฑฐํ•˜๋ฉด ๋ฌธ์ œ๊ฐ€ ํ•ด๊ฒฐ๋จ์„ ํ™•์ธํ–ˆ์Šต๋‹ˆ๋‹ค(SE linux๊ฐ€ ํ—ˆ์šฉ ๋ชจ๋“œ์—์„œ ํ™œ์„ฑํ™”๋จ).

chrisohaver picture chrisohaver ์— 2018๋…„ 08์›” 01์ผ
👍3

๋˜ํ•œ Kubernetes์—์„œ ๊ถŒ์žฅํ•˜๋Š” docker ๋ฒ„์ „(docker 17.03)์œผ๋กœ ์—…๊ทธ๋ ˆ์ด๋“œํ•˜๋ฉด "allowPrivilegeEscalation: false"๊ฐ€ coredns ๋ฐฐํฌ์— ๋‚จ์•„ ์žˆ๊ณ  SELinux๊ฐ€ ํ—ˆ์šฉ ๋ชจ๋“œ์—์„œ ํ™œ์„ฑํ™”๋œ ์ƒํƒœ์—์„œ ๋ฌธ์ œ๊ฐ€ ํ•ด๊ฒฐ๋จ์„ ํ™•์ธํ–ˆ์Šต๋‹ˆ๋‹ค.

chrisohaver picture chrisohaver ์— 2018๋…„ 08์›” 01์ผ

๋”ฐ๋ผ์„œ ์ด์ „ ๋ฒ„์ „์˜ docker์™€ SELinux ์‚ฌ์ด์— allowPrivilegeEscalation ์ง€์‹œ๋ฌธ์ด ์žˆ๋Š” ๋น„ํ˜ธํ™˜์„ฑ์ด ์žˆ๋Š” ๊ฒƒ์ฒ˜๋Ÿผ ๋ณด์ž…๋‹ˆ๋‹ค. ์ด ์ง€์‹œ๋ฌธ์€ ์ดํ›„ ๋ฒ„์ „์˜ docker์—์„œ ๋ถ„๋ช…ํžˆ ํ•ด๊ฒฐ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

3๊ฐ€์ง€ ๋‹ค๋ฅธ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์ด ์žˆ๋Š” ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  • ์ตœ์‹  ๋ฒ„์ „์˜ docker(์˜ˆ: 17.03, ํ˜„์žฌ k8s์—์„œ ๊ถŒ์žฅํ•˜๋Š” ๋ฒ„์ „)๋กœ ์—…๊ทธ๋ ˆ์ด๋“œ
  • ๋˜๋Š” ๋ฐฐํฌ์˜ ํฌ๋“œ ์‚ฌ์–‘์—์„œ allowPrivilegeEscalation=false๋ฅผ ์ œ๊ฑฐํ•˜์‹ญ์‹œ์˜ค.
  • ๋˜๋Š” SELinux ๋น„ํ™œ์„ฑํ™”
chrisohaver picture chrisohaver ์— 2018๋…„ 08์›” 01์ผ
👍12

@chrisohaver ์ตœ์‹  ๋ฒ„์ „์˜ docker 17.03์œผ๋กœ ์—…๊ทธ๋ ˆ์ด๋“œํ•˜์—ฌ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ–ˆ์Šต๋‹ˆ๋‹ค. ๊ณ ๋งˆ์›Œ

Leozki picture Leozki ์— 2018๋…„ 08์›” 01์ผ

์กฐ์‚ฌ์— ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค @chrisohaver :100:

neolit123 picture neolit123 ์— 2018๋…„ 08์›” 01์ผ

๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค, @chrisohaver !

์ด๊ฒƒ์€ ํšจ๊ณผ๊ฐ€ ์žˆ์—ˆ๋‹ค:

kubectl -n kube-system get deployment coredns -o yaml | \
  sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
  kubectl apply -f -
kuznero picture kuznero ์— 2018๋…„ 08์›” 10์ผ
👍17 🎉5 ❤3

@chrisohaver
SELinux ๋…ธ๋“œ์— ๋Œ€ํ•œ kubeadm ๋ฌธ์ œ ํ•ด๊ฒฐ ๊ฐ€์ด๋“œ ์—์„œ ์ด ๋‹จ๊ณ„๋ฅผ ๋‹ค์Œ ํ–‰์œผ๋กœ ๋ฌธ์„œํ™”ํ•ด์•ผ ํ•œ๋‹ค๊ณ  ์ƒ๊ฐํ•˜์‹ญ๋‹ˆ๊นŒ?

coredns ํฌ๋“œ์—๋Š” CrashLoopBackOff ๋˜๋Š” Error ์ƒํƒœ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

์ด์ „ ๋ฒ„์ „์˜ Docker์™€ ํ•จ๊ป˜ SELinux๋ฅผ ์‹คํ–‰ํ•˜๋Š” ๋…ธ๋“œ๊ฐ€ ์žˆ๋Š” ๊ฒฝ์šฐ coredns ํŒŸ(Pod)์ด ์‹œ์ž‘๋˜์ง€ ์•Š๋Š” ์‹œ๋‚˜๋ฆฌ์˜ค๊ฐ€ ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๋ฉด ๋‹ค์Œ ์˜ต์…˜ ์ค‘ ํ•˜๋‚˜๋ฅผ ์‹œ๋„ํ•ด ๋ณด์„ธ์š”.

  • ์ตœ์‹  ๋ฒ„์ „์˜ Docker๋กœ ์—…๊ทธ๋ ˆ์ด๋“œ - 17.03์ด ์ž‘๋™ํ•˜๋Š” ๊ฒƒ์œผ๋กœ ํ™•์ธ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
  • SELinux๋ฅผ ๋น„ํ™œ์„ฑํ™”ํ•ฉ๋‹ˆ๋‹ค.
  • allowPrivilegeEscalation ๋ฅผ true ๋กœ ์„ค์ •ํ•˜๋„๋ก coredns ๋ฐฐํฌ๋ฅผ ์ˆ˜์ •ํ•ฉ๋‹ˆ๋‹ค.
kubectl -n kube-system get deployment coredns -o yaml | \
  sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
  kubectl apply -f -

์–ด๋–ป๊ฒŒ ์ƒ๊ฐํ•˜์‹ญ๋‹ˆ๊นŒ? ๋ญ”๊ฐ€ ๊ฐœ์„ ํ•  ์ˆ˜ ์žˆ๋‹ค๊ณ  ์ƒ๊ฐ๋˜๋ฉด ํ…์ŠคํŠธ ์ˆ˜์ •์„ ์ œ์•ˆํ•˜์‹ญ์‹œ์˜ค.

neolit123 picture neolit123 ์— 2018๋…„ 08์›” 10์ผ
👍1

๊ดœ์ฐฎ์•„. SELinux๋ฅผ ๋น„ํ™œ์„ฑํ™”ํ•˜๊ฑฐ๋‚˜ allowPrivilegeEscalation ์„ค์ •์„ ๋ณ€๊ฒฝํ•  ๋•Œ ๋ณด์•ˆ์— ๋ถ€์ •์ ์ธ ์˜ํ–ฅ์ด ์žˆ์Œ์„ ์–ธ๊ธ‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

๊ฐ€์žฅ ์•ˆ์ „ํ•œ ์†”๋ฃจ์…˜์€ Docker๋ฅผ Kubernetes๊ฐ€ ๊ถŒ์žฅ ํ•˜๋Š” ๋ฒ„์ „(17.03)์œผ๋กœ ์—…๊ทธ๋ ˆ์ด๋“œํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

chrisohaver picture chrisohaver ์— 2018๋…„ 08์›” 10์ผ
👍1

@chrisohaver
์ดํ•ดํ•˜๋ฉด ์‚ฌ๋ณธ์„ ์ˆ˜์ •ํ•˜๊ณ  ์ด์— ๋Œ€ํ•œ PR์„ ์ œ์ถœํ•  ๊ฒƒ์ž…๋‹ˆ๋‹ค.

neolit123 picture neolit123 ์— 2018๋…„ 08์›” 10์ผ

stackoverflow์—๋„ ์ด์— ๋Œ€ํ•œ ๋‹ต๋ณ€์ด ์žˆ์Šต๋‹ˆ๋‹ค.
https://stackoverflow.com/questions/53075796/coredns-pods-have-crashloopbackoff-or-error-state

์ด ์˜ค๋ฅ˜

[FATAL] plugin/loop: Seen "HINFO IN 6900627972087569316.7905576541070882081." more than twice, loop detected

CoreDNS๊ฐ€ ํ™•์ธ ๊ตฌ์„ฑ์—์„œ ๋ฃจํ”„๋ฅผ ๊ฐ์ง€ํ•  ๋•Œ ๋ฐœ์ƒํ•˜๋ฉฐ ์ด๋Š” ์˜๋„๋œ ๋™์ž‘์ž…๋‹ˆ๋‹ค. ๋‹น์‹ ์€ ์ด ๋ฌธ์ œ๋ฅผ ๊ฒช๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค:

https://github.com/kubernetes/kubeadm/issues/1162

https://github.com/coredns/coredns/issues/2087

Hacky ์†”๋ฃจ์…˜: CoreDNS ๋ฃจํ”„ ๊ฐ์ง€ ๋น„ํ™œ์„ฑํ™”

CoreDNS ๊ตฌ์„ฑ ๋งต์„ ํŽธ์ง‘ํ•ฉ๋‹ˆ๋‹ค.

kubectl -n kube-system edit configmap coredns

loop ํ–‰์„ ์ œ๊ฑฐํ•˜๊ฑฐ๋‚˜ ์ฃผ์„ ์ฒ˜๋ฆฌํ•˜๊ณ  ์ €์žฅํ•˜๊ณ  ์ข…๋ฃŒํ•ฉ๋‹ˆ๋‹ค.

๊ทธ๋Ÿฐ ๋‹ค์Œ CoreDNS ํฌ๋“œ๋ฅผ ์ œ๊ฑฐํ•˜์—ฌ ์ƒˆ ๊ตฌ์„ฑ์œผ๋กœ ์ƒˆ ํฌ๋“œ๋ฅผ ์ƒ์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

kubectl -n kube-system delete pod -l k8s-app=kube-dns

๊ทธ ํ›„์—๋Š” ๋ชจ๋“  ๊ฒƒ์ด ์ข‹์•„์•ผํ•ฉ๋‹ˆ๋‹ค.

์„ ํ˜ธํ•˜๋Š” ์†”๋ฃจ์…˜: DNS ๊ตฌ์„ฑ์—์„œ ๋ฃจํ”„ ์ œ๊ฑฐ

๋จผ์ € systemd-resolved ๋ฅผ ์‚ฌ์šฉ ์ค‘์ธ์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค. Ubuntu 18.04๋ฅผ ์‹คํ–‰ ์ค‘์ด๋ผ๋ฉด ์•„๋งˆ๋„ ๊ทธ๋Ÿด ๊ฒƒ์ž…๋‹ˆ๋‹ค.

systemctl list-unit-files | grep enabled | grep systemd-resolved

๊ทธ๋ ‡๋‹ค๋ฉด ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์ฐธ์กฐ๋กœ ์‚ฌ์šฉ ์ค‘์ธ resolv.conf ํŒŒ์ผ์„ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.

ps auxww | grep kubelet

๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ค„์ด ํ‘œ์‹œ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

/usr/bin/kubelet ... --resolv-conf=/run/systemd/resolve/resolv.conf

์ค‘์š”ํ•œ ๋ถ€๋ถ„์€ --resolv-conf ์ž…๋‹ˆ๋‹ค. systemd resolv.conf๊ฐ€ ์‚ฌ์šฉ๋˜๋Š”์ง€ ์—ฌ๋ถ€๋ฅผ ์•Œ์•„๋ƒ…๋‹ˆ๋‹ค.

systemd resolv.conf ์ธ ๊ฒฝ์šฐ ๋‹ค์Œ์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

/run/systemd/resolve/resolv.conf ์˜ ๋‚ด์šฉ์„ ํ™•์ธํ•˜์—ฌ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ ˆ์ฝ”๋“œ๊ฐ€ ์žˆ๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.

nameserver 127.0.0.1

127.0.0.1 ๊ฐ€ ์žˆ์œผ๋ฉด ๋ฃจํ”„๋ฅผ ์ผ์œผํ‚ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

์ œ๊ฑฐํ•˜๋ ค๋ฉด ํ•ด๋‹น ํŒŒ์ผ์„ ํŽธ์ง‘ํ•˜์ง€ ๋ง๊ณ  ๋‹ค๋ฅธ ์œ„์น˜์—์„œ ์ œ๋Œ€๋กœ ์ƒ์„ฑ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.

/etc/systemd/network ์•„๋ž˜์˜ ๋ชจ๋“  ํŒŒ์ผ์„ ํ™•์ธํ•˜๊ณ  ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ ˆ์ฝ”๋“œ๋ฅผ ์ฐพ์œผ๋ฉด

DNS=127.0.0.1

๊ทธ ๊ธฐ๋ก์„ ์‚ญ์ œํ•ฉ๋‹ˆ๋‹ค. ๋˜ํ•œ /etc/systemd/resolved.conf ๋ฅผ ํ™•์ธํ•˜๊ณ  ํ•„์š”ํ•œ ๊ฒฝ์šฐ ๋™์ผํ•œ ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ ์–ด๋„ ํ•˜๋‚˜ ๋˜๋Š” ๋‘ ๊ฐœ์˜ DNS ์„œ๋ฒ„๊ฐ€ ๊ตฌ์„ฑ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.

DNS=1.1.1.1 1.0.0.1

๋ชจ๋“  ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•œ ํ›„ ๋ณ€๊ฒฝ ์‚ฌํ•ญ์„ ์ ์šฉํ•˜๋ ค๋ฉด systemd ์„œ๋น„์Šค๋ฅผ ๋‹ค์‹œ ์‹œ์ž‘ํ•˜์„ธ์š”.
systemctl ๋‹ค์‹œ ์‹œ์ž‘ systemd-networkd systemd-resolved

๊ทธ๋Ÿฐ ๋‹ค์Œ resolv.conf ํŒŒ์ผ์— DNS=127.0.0.1 ๊ฐ€ ๋” ์ด์ƒ ์—†๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

cat /run/systemd/resolve/resolv.conf

๋งˆ์ง€๋ง‰์œผ๋กœ DNS ํฌ๋“œ์˜ ์žฌ์ƒ์„ฑ์„ ํŠธ๋ฆฌ๊ฑฐํ•ฉ๋‹ˆ๋‹ค.

kubectl -n kube-system delete pod -l k8s-app=kube-dns

์š”์•ฝ: ์†”๋ฃจ์…˜์—๋Š” ํ˜ธ์ŠคํŠธ DNS ๊ตฌ์„ฑ์—์„œ DNS ์กฐํšŒ ๋ฃจํ”„์ฒ˜๋Ÿผ ๋ณด์ด๋Š” ๊ฒƒ์„ ์ œ๊ฑฐํ•˜๋Š” ๊ฒƒ์ด ํฌํ•จ๋ฉ๋‹ˆ๋‹ค. ๋‹จ๊ณ„๋Š” resolv.conf ๊ด€๋ฆฌ์ž/๊ตฌํ˜„๋งˆ๋‹ค ๋‹ค๋ฆ…๋‹ˆ๋‹ค.

mydockergit picture mydockergit ์— 2019๋…„ 01์›” 23์ผ
👍3

๊ฐ์‚ฌ ํ•ด์š”. CoreDNS ๋ฃจํ”„ ํ”Œ๋Ÿฌ๊ทธ์ธ ์ถ”๊ฐ€ ์ •๋ณด ์—์„œ๋„ ๋‹ค๋ฃน๋‹ˆ๋‹ค...

chrisohaver picture chrisohaver ์— 2019๋…„ 01์›” 23์ผ

๋‚˜๋Š” ๊ฐ™์€ ๋ฌธ์ œ๊ฐ€ ์žˆ๊ณ  ๋˜ ๋‹ค๋ฅธ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
1. dns๋ฅผ ์ฐพ์„ ์ˆ˜ ์—†์Œ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค. ์˜ค๋ฅ˜๋Š”
[์˜ค๋ฅ˜] ํ”Œ๋Ÿฌ๊ทธ์ธ/์˜ค๋ฅ˜: 2 2115717704248378980.1120568170924441806. ํžŒํŠธ: ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋Š” ๋ฐฑ์—”๋“œ: udp 10.224.0.3:57088->8.8.8.8:53 ์ฝ๊ธฐ: i/o ์‹œ๊ฐ„ ์ดˆ๊ณผ
[์˜ค๋ฅ˜] ํ”Œ๋Ÿฌ๊ทธ์ธ/์˜ค๋ฅ˜: 2 2115717704248378980.1120568170924441806. ํžŒํŠธ: ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋Š” ๋ฐฑ์—”๋“œ: ์ฝ๊ธฐ udp 10.224.0.3:38819->172.16.254.1:53: i/o ์‹œ๊ฐ„ ์ดˆ๊ณผ
........

๋‚ด /etc/resolv.com
๊ฐ€์ง€๊ณ  ์žˆ์ง€ ์•Š๋‹ค
๋„ค์ž„์„œ๋ฒ„ 172.16.254.1 #์ด๊ฒƒ์€ ๋‚ด DNS์ž…๋‹ˆ๋‹ค
๋„ค์ž„์„œ๋ฒ„ 8.8.8.8 #๋„ท์˜ ๋˜ ๋‹ค๋ฅธ dns
๋‚œ ๋‹ฌ๋ฆฐ๋‹ค

kubectl -n kube-system ๋ฐฐํฌ coredns -o yaml ๊ฐ€์ ธ์˜ค๊ธฐ | \
sed '/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
kubectl ์ ์šฉ -f -

๊ทธ๋Ÿฐ ๋‹ค์Œ ํฌ๋“œ ์žฌ๊ตฌ์ถ•์—๋Š” ํ•˜๋‚˜์˜ ์˜ค๋ฅ˜๋งŒ ์žˆ์Šต๋‹ˆ๋‹ค.

[์˜ค๋ฅ˜] ํ”Œ๋Ÿฌ๊ทธ์ธ/์˜ค๋ฅ˜: 2 10594135170717325.8545646296733374240. ํžŒํŠธ: ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋Š” ๋ฐฑ์—”๋“œ: ์—…์ŠคํŠธ๋ฆผ ํ˜ธ์ŠคํŠธ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

๊ทธ๊ฒŒ ์ •์ƒ์ธ์ง€ ๋ชจ๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค. ์•„๋งˆ๋„

2, coredns๊ฐ€ ๋‚ด API ์„œ๋น„์Šค๋ฅผ ์ฐพ์„ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ์˜ค๋ฅ˜๋Š”

kube-dns *v1.Endpoints getsockopt๋ฅผ ๋‚˜์—ดํ•˜์ง€ ๋ชปํ–ˆ์Šต๋‹ˆ๋‹ค: 10.96.0.1:6443 API ์—ฐ๊ฒฐ์ด ๊ฑฐ๋ถ€๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

coredns๊ฐ€ ๊ณ„์†ํ•ด์„œ ๋‹ค์‹œ ์‹œ์ž‘๋˜๊ณ  ๋งˆ์นจ๋‚ด CrashLoopBackOff๊ฐ€ ๋ฉ๋‹ˆ๋‹ค.

๊ทธ๋ž˜์„œ ๋งˆ์Šคํ„ฐ ๋…ธ๋“œ์—์„œ coredns๋ฅผ ์‹คํ–‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

kubectl ํŽธ์ง‘ ๋ฐฐํฌ/coredns --namespace=kube-system
์ŠคํŽ™.ํ…œํ”Œ๋ฆฟ.์ŠคํŽ™
๋…ธ๋“œ ์„ ํƒ๊ธฐ:
node-role.kubernetes.io/master: ""

๊ทธ๊ฒŒ ์ •์ƒ์ธ์ง€ ๋ชจ๋ฅด๊ฒ ์–ด

๋งˆ์ง€๋ง‰์œผ๋กœ ๋‚ด env๋ฅผ ์ œ๊ณต

๋ฆฌ๋ˆ…์Šค 4.20.10-1.el7.elrepo.x86_64 /// ์„ผํ† ์Šค 7

๋„์ปค ๋ฒ„์ „: 18.09.3

[ root@k8smaster00 ~]# ๋„์ปค ์ด๋ฏธ์ง€ ls -a
์ €์žฅ์†Œ ํƒœ๊ทธ ์ด๋ฏธ์ง€ ID ์ƒ์„ฑ๋œ ํฌ๊ธฐ
k8s.gcr.io/kube-controller-manager v1.13.3 0482f6400933 6์ฃผ ์ „ 146MB
k8s.gcr.io/kube-proxy v1.13.3 98db19758ad4 6์ฃผ ์ „ 80.3MB
k8s.gcr.io/kube-apiserver v1.13.3 fe242e556a99 6์ฃผ ์ „ 181MB
k8s.gcr.io/kube-scheduler v1.13.3 3a6f709e97a0 6์ฃผ ์ „ 79.6MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 7์ฃผ ์ „ 52.6MB
k8s.gcr.io/coredns 1.2.6 f59dcacceff4 4๊ฐœ์›” ์ „ 40MB
k8s.gcr.io/etcd 3.2.24 3cab8e1b9802 6 ๊ฐœ์›” ์ „ 220MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 15๊ฐœ์›” ์ „ 742kB

์ฟ ๋ฒ ๋„ท์€ 1.13.3์ž…๋‹ˆ๋‹ค.

์ด๊ฒƒ์€ ๋ฒ„๊ทธ๋ผ๊ณ  ์ƒ๊ฐํ•ฉ๋‹ˆ๋‹ค. ๊ณต์‹ ์—…๋ฐ์ดํŠธ ๋˜๋Š” ์†”๋ฃจ์…˜์„ ๊ธฐ๋Œ€ํ•˜์‹ญ์‹œ์˜ค

mengxifl picture mengxifl ์— 2019๋…„ 03์›” 20์ผ

๋‚˜๋Š” ๊ฐ™์€ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค ...

@mengxifl , ์ด๋Ÿฌํ•œ ์˜ค๋ฅ˜๋Š” ์ด๋ฒˆ ํ˜ธ์—์„œ ๋ณด๊ณ ๋˜๊ณ  ๋…ผ์˜๋œ ์˜ค๋ฅ˜์™€ ํฌ๊ฒŒ ๋‹ค๋ฆ…๋‹ˆ๋‹ค.

[์˜ค๋ฅ˜] ํ”Œ๋Ÿฌ๊ทธ์ธ/์˜ค๋ฅ˜: 2 2115717704248378980.1120568170924441806. ํžŒํŠธ: ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋Š” ๋ฐฑ์—”๋“œ: udp 10.224.0.3:57088->8.8.8.8:53 ์ฝ๊ธฐ: i/o ์‹œ๊ฐ„ ์ดˆ๊ณผ
[์˜ค๋ฅ˜] ํ”Œ๋Ÿฌ๊ทธ์ธ/์˜ค๋ฅ˜: 2 2115717704248378980.1120568170924441806. ํžŒํŠธ: ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋Š” ๋ฐฑ์—”๋“œ: ์ฝ๊ธฐ udp 10.224.0.3:38819->172.16.254.1:53: i/o ์‹œ๊ฐ„ ์ดˆ๊ณผ

์ด๋Ÿฌํ•œ ์˜ค๋ฅ˜๋Š” CoreDNS ํฌ๋“œ(๋ฐ ์•„๋งˆ๋„ ๋‹ค๋ฅธ ๋ชจ๋“  ํฌ๋“œ)๊ฐ€ ๋„ค์ž„์„œ๋ฒ„์— ๋„๋‹ฌํ•  ์ˆ˜ ์—†์Œ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค. ์ด๋Š” ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋„คํŠธ์›Œํ‚น ๋ฌธ์ œ๋ฅผ ์™ธ๋ถ€ ์„ธ๊ณ„์— ์•”์‹œํ•ฉ๋‹ˆ๋‹ค. ํ”Œ๋ž€๋„ฌ ๊ตฌ์„ฑ์ด ์ž˜๋ชป๋˜์—ˆ๊ฑฐ๋‚˜ ๋ฐฉํ™”๋ฒฝ์ด ์žˆ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

coredns์—์„œ ๋‚ด API ์„œ๋น„์Šค๋ฅผ ์ฐพ์„ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค...
๊ทธ๋ž˜์„œ ๋งˆ์Šคํ„ฐ ๋…ธ๋“œ์—์„œ coredns๋ฅผ ์‹คํ–‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

์ด๊ฒƒ๋„ ์ •์ƒ์ด ์•„๋‹™๋‹ˆ๋‹ค. ๋‚ด๊ฐ€ ๋‹น์‹ ์„ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ์ดํ•ดํ•œ๋‹ค๋ฉด CoreDNS๊ฐ€ ๋งˆ์Šคํ„ฐ ๋…ธ๋“œ์—์„œ API์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ์ง€๋งŒ ๋‹ค๋ฅธ ๋…ธ๋“œ์—์„œ๋Š” ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋‹ค๊ณ  ๋งํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ํด๋Ÿฌ์Šคํ„ฐ ๋‚ด์˜ ๋…ธ๋“œ ๊ฐ„์˜ ๋„คํŠธ์›Œํ‚น ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด ํฌ๋“œ๋ฅผ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ์•„๋งˆ๋„ ํ”Œ๋ž€๋„ฌ ๊ตฌ์„ฑ ๋˜๋Š” ๋ฐฉํ™”๋ฒฝ ๋ฌธ์ œ์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

chrisohaver picture chrisohaver ์— 2019๋…„ 03์›” 20์ผ

๋‚˜๋Š” ๊ฐ™์€ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค ...

@mengxifl , ์ด๋Ÿฌํ•œ ์˜ค๋ฅ˜๋Š” ์ด๋ฒˆ ํ˜ธ์—์„œ ๋ณด๊ณ ๋˜๊ณ  ๋…ผ์˜๋œ ์˜ค๋ฅ˜์™€ ํฌ๊ฒŒ ๋‹ค๋ฆ…๋‹ˆ๋‹ค.

[์˜ค๋ฅ˜] ํ”Œ๋Ÿฌ๊ทธ์ธ/์˜ค๋ฅ˜: 2 2115717704248378980.1120568170924441806. ํžŒํŠธ: ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋Š” ๋ฐฑ์—”๋“œ: udp 10.224.0.3:57088->8.8.8.8:53 ์ฝ๊ธฐ: i/o ์‹œ๊ฐ„ ์ดˆ๊ณผ
[์˜ค๋ฅ˜] ํ”Œ๋Ÿฌ๊ทธ์ธ/์˜ค๋ฅ˜: 2 2115717704248378980.1120568170924441806. ํžŒํŠธ: ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋Š” ๋ฐฑ์—”๋“œ: ์ฝ๊ธฐ udp 10.224.0.3:38819->172.16.254.1:53: i/o ์‹œ๊ฐ„ ์ดˆ๊ณผ

์ด๋Ÿฌํ•œ ์˜ค๋ฅ˜๋Š” CoreDNS ํฌ๋“œ(๋ฐ ์•„๋งˆ๋„ ๋‹ค๋ฅธ ๋ชจ๋“  ํฌ๋“œ)๊ฐ€ ๋„ค์ž„์„œ๋ฒ„์— ๋„๋‹ฌํ•  ์ˆ˜ ์—†์Œ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค. ์ด๋Š” ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋„คํŠธ์›Œํ‚น ๋ฌธ์ œ๋ฅผ ์™ธ๋ถ€ ์„ธ๊ณ„์— ์•”์‹œํ•ฉ๋‹ˆ๋‹ค. ํ”Œ๋ž€๋„ฌ ๊ตฌ์„ฑ์ด ์ž˜๋ชป๋˜์—ˆ๊ฑฐ๋‚˜ ๋ฐฉํ™”๋ฒฝ์ด ์žˆ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

coredns์—์„œ ๋‚ด API ์„œ๋น„์Šค๋ฅผ ์ฐพ์„ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค...
๊ทธ๋ž˜์„œ ๋งˆ์Šคํ„ฐ ๋…ธ๋“œ์—์„œ coredns๋ฅผ ์‹คํ–‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

์ด๊ฒƒ๋„ ์ •์ƒ์ด ์•„๋‹™๋‹ˆ๋‹ค. ๋‚ด๊ฐ€ ๋‹น์‹ ์„ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ์ดํ•ดํ•œ๋‹ค๋ฉด CoreDNS๊ฐ€ ๋งˆ์Šคํ„ฐ ๋…ธ๋“œ์—์„œ API์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ์ง€๋งŒ ๋‹ค๋ฅธ ๋…ธ๋“œ์—์„œ๋Š” ์—ฐ๊ฒฐํ•  ์ˆ˜ ์—†๋‹ค๊ณ  ๋งํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ํด๋Ÿฌ์Šคํ„ฐ ๋‚ด์˜ ๋…ธ๋“œ ๊ฐ„์˜ ๋„คํŠธ์›Œํ‚น ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด ํฌ๋“œ๋ฅผ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ์•„๋งˆ๋„ ํ”Œ๋ž€๋„ฌ ๊ตฌ์„ฑ ๋˜๋Š” ๋ฐฉํ™”๋ฒฝ ๋ฌธ์ œ์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹น์‹ ์˜ ๋‹ต๋ณ€์— ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค

์•„๋งˆ๋„ ๋‚ด yaml ํŒŒ์ผ์„ ์˜ฌ๋ ค์•ผ ํ•  ๊ฒƒ์ž…๋‹ˆ๋‹ค

๋‚˜๋Š” ์‚ฌ์šฉํ•œ๋‹ค
kubeadm ์ดˆ๊ธฐํ™” --config=config.yaml

๋‚ด config.yaml ๋‚ด์šฉ์€

apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
apiEndpoint:
  advertiseAddress: "172.16.254.74"
  bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: "v1.13.3"
etcd:
  external:
    endpoints:
    - "https://172.16.254.86:2379" 
    - "https://172.16.254.87:2379"
    - "https://172.16.254.88:2379"
    caFile: /etc/kubernetes/pki/etcd/ca.pem
    certFile: /etc/kubernetes/pki/etcd/client.pem
    keyFile: /etc/kubernetes/pki/etcd/client-key.pem
networking:
  podSubnet: "10.224.0.0/16"
  serviceSubnet: "10.96.0.0/12"
apiServerCertSANs:
- k8smaster00
- k8smaster01
- k8snode00
- k8snode01
- 172.16.254.74
- 172.16.254.79
- 172.16.254.80
- 172.16.254.81
- 172.16.254.85 #Vip
- 127.0.0.1
clusterName: "cluster"
controlPlaneEndpoint: "172.16.254.85:6443"

apiServerExtraArgs:
  service-node-port-range: 20-65535

๋‚ด fannel yaml์€ ๊ธฐ๋ณธ์ž…๋‹ˆ๋‹ค

https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

systemctl status firewalld
๋ชจ๋“  ๋…ธ๋“œ ๋ง
์žฅ์น˜ firewalld.service๋ฅผ ์ฐพ์„ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.

cat /etc/sysconfig/iptables
๋ชจ๋“  ๋…ธ๋“œ ๋ง
*ํ•„ํ„ฐ
:์ž…๋ ฅ ์ˆ˜๋ฝ [0:0]
:์•ž์œผ๋กœ ์ˆ˜๋ฝ [0:0]
:์ถœ๋ ฅ ์Šน์ธ [0:0]
-A ์ž…๋ ฅ -p tcp -m tcp --dport 1:65535 -j ์ˆ˜๋ฝ
-A INPUT -m ์ƒํƒœ --state RELATED,ESTABLISHED -j ACCEPT
-A ์ž…๋ ฅ -p icmp -j ์ˆ˜๋ฝ
-A ์ž…๋ ฅ -i lo -j ์ˆ˜๋ฝ
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A ์ถœ๋ ฅ -p tcp -m tcp --sport 1:65535 -j ์ˆ˜๋ฝ
-A ์•ž์œผ๋กœ -p tcp -m tcp --dport 1:65535 -j ์ˆ˜๋ฝ
-A ์•ž์œผ๋กœ -p tcp -m tcp --sport 1:65535 -j ์ˆ˜๋ฝ
COMMI

cat /etc/resolv.conf & ping bing.com
๋ชจ๋“  ๋…ธ๋“œ ๋ง
[1] 6330
๋„ค์ž„์„œ๋ฒ„ 172.16.254.1
๋„ค์ž„์„œ๋ฒ„ 8.8.8.8
PING bing.com(13.107.21.200) 56(84) ๋ฐ”์ดํŠธ ๋ฐ์ดํ„ฐ.
13.107.21.200(13.107.21.200)์˜ 64๋ฐ”์ดํŠธ: icmp_seq=2 ttl=111 ์‹œ๊ฐ„=149ms

uname -rs
๋งˆ์Šคํ„ฐ ๋…ธ๋“œ ๋ง
๋ฆฌ๋ˆ…์Šค 4.20.10-1.el7.elrepo.x86_64

uname -rs
์Šฌ๋ ˆ์ด๋ธŒ ๋…ธ๋“œ ๋ง
๋ฆฌ๋ˆ…์Šค 4.4.176-1.el7.elrepo.x86_64

๊ทธ๋ž˜์„œ ๋ฐฉํ™”๋ฒฝ์ด mybe fannel์— ๋ฌธ์ œ๊ฐ€ ์žˆ๋‹ค๊ณ  ์ƒ๊ฐํ•˜์ง€ ์•Š์Šต๋‹ˆ๊นŒ? ํ•˜์ง€๋งŒ ๊ธฐ๋ณธ ๊ตฌ์„ฑ์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ์•„๋งˆ๋„ ๋ฆฌ๋ˆ…์Šค ๋ฒ„์ „ . ๋ชจ๋ฅด๊ฒ ์–ด์š” .

์ข‹์•„, ๋‚˜๋Š” ๋‹ฌ๋ฆฐ๋‹ค
/sbin/iptables -t nat -I POSTROUTING -s 10.224.0.0/16 -j MASQUERADE

๋‚˜๋ฅผ ์œ„ํ•ด ์ž‘๋™ํ•˜๋Š” ๋ชจ๋“  ๋…ธ๋“œ์—์„œ. ๊ฐ์‚ฌ ํ•ด์š”

mengxifl picture mengxifl ์— 2019๋…„ 03์›” 21์ผ
์ด ํŽ˜์ด์ง€๊ฐ€ ๋„์›€์ด ๋˜์—ˆ๋‚˜์š”?
0 / 5 - 0 ๋“ฑ๊ธ‰

๊ด€๋ จ ๋ฌธ์ œ

mattmoyer picture mattmoyer  ยท  4์ฝ”๋ฉ˜ํŠธ
ggaaooppeenngg picture ggaaooppeenngg  ยท  4์ฝ”๋ฉ˜ํŠธ
jbrandes picture jbrandes  ยท  4์ฝ”๋ฉ˜ํŠธ
imedaouidene picture imedaouidene  ยท  3์ฝ”๋ฉ˜ํŠธ
atoato88 picture atoato88  ยท  4์ฝ”๋ฉ˜ํŠธ