<p>kubeadm init "첫 번째 λ…Έλ“œκ°€ λ“±λ‘λ˜μ—ˆμ§€λ§Œ 아직 μ€€λΉ„λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€"μ—μ„œ 멈좀</p>

에 λ§Œλ“  2017λ…„ 03μ›” 29일  Β·  52μ½”λ©˜νŠΈ  Β·  좜처: kubernetes/kubeadm

이 문제λ₯Ό μ œμΆœν•˜κΈ° 전에 Kubernetes λ¬Έμ œμ—μ„œ μ–΄λ–€ ν‚€μ›Œλ“œλ₯Ό κ²€μƒ‰ν•˜μ…¨μŠ΅λ‹ˆκΉŒ? (쀑볡 ν•­λͺ©μ„ 찾은 경우 λŒ€μ‹  거기에 νšŒμ‹ ν•΄μ•Ό ν•©λ‹ˆλ‹€.): kubeadm

이것은 버그 λ³΄κ³ μ„œμž…λ‹ˆκΉŒ μ•„λ‹ˆλ©΄ κΈ°λŠ₯ μš”μ²­μž…λ‹ˆκΉŒ? (ν•˜λ‚˜ 선택): 버그 보고

Kubernetes 버전 ( kubectl version ): 1.6.0

ν™˜κ²½ :

  • ν΄λΌμš°λ“œ 제곡자 λ˜λŠ” ν•˜λ“œμ›¨μ–΄ ꡬ성 : Raspberry Pi 3 Model B
  • OS (예: /etc/os-release): Hypriot 1.4.0(Dockerλ₯Ό 1.12.6으둜 μˆ˜λ™μœΌλ‘œ λ‹€μš΄κ·Έλ ˆμ΄λ“œν•œ 경우, Hypriot 1.4.0은 Docker 17.03.0-ce와 ν•¨κ»˜ 제곡됨)
  • 컀널 (예: uname -a ): 4.4.50-hypriotos-v7+
  • μ„€μΉ˜ 도ꡬ : kubeadm
  • 기타 :

무슨 일이 μžˆμ—ˆλŠ”μ§€ :

kubeadm μ‹œμž‘ν•˜κΈ° κ°€μ΄λ“œλ₯Ό μ •ν™•νžˆ λ”°λ₯΄μ‹­μ‹œμ˜€ .

# kubeadm init --apiserver-cert-extra-sans redacted --pod-network-cidr 10.244.0.0/16
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.0
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [kube-01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local redacted] and IPs [10.96.0.1 10.0.1.101]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 206.956919 seconds
[apiclient] Waiting for at least one node to register and become ready
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet

"첫 번째 λ…Έλ“œκ°€ λ“±λ‘λ˜μ—ˆμ§€λ§Œ 아직 μ€€λΉ„λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€"λΌλŠ” λ§ˆμ§€λ§‰ λ©”μ‹œμ§€κ°€ λ¬΄ν•œ 반볡되고 kubeadm이 μ™„λ£Œλ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. λͺ¨λ“  Docker μ»¨ν…Œμ΄λ„ˆκ°€ μ˜ˆμƒλŒ€λ‘œ μ‹€ν–‰ 쀑이고 λ‹€μŒκ³Ό 같은지 ν™•μΈν•˜κΈ° μœ„ν•΄ λ‹€λ₯Έ μ„Έμ…˜μ—μ„œ λ§ˆμŠ€ν„° μ„œλ²„μ— μ—°κ²°ν–ˆμŠ΅λ‹ˆλ‹€.

$ docker ps
CONTAINER ID        IMAGE                                                                                                                          COMMAND                  CREATED             STATUS              PORTS               NAMES
54733aa1aae3        gcr.io/google_containers/kube-controller-manager-arm<strong i="6">@sha256</strong>:22f30303212b276b6868b89c8e92c5fb2cb93641e59c312b254c6cb0fa111b2a   "kube-controller-mana"   10 minutes ago      Up 10 minutes                           k8s_kube-controller-manager_kube-controller-manager-kube-01_kube-system_d44abf63e3ab24853ab86643e0b96d81_0
55b6bf2cc09e        gcr.io/google_containers/etcd-arm<strong i="7">@sha256</strong>:0ce1dcd85968a3242995dfc168abba2c3bc03d0e3955f52a0b1e79f90039dcf2                      "etcd --listen-client"   11 minutes ago      Up 11 minutes                           k8s_etcd_etcd-kube-01_kube-system_90ab26991bf9ad676a430c7592d08bee_0
bd0dc34d5e77        gcr.io/google_containers/kube-apiserver-arm<strong i="8">@sha256</strong>:c54b8c609a6633b5397173c763aba0656c6cb2601926cce5a5b4870d58ba67bd            "kube-apiserver --ins"   12 minutes ago      Up 12 minutes                           k8s_kube-apiserver_kube-apiserver-kube-01_kube-system_4d99c225ec157dc715c26b59313aeac8_1
1c4c7b69a3eb        gcr.io/google_containers/kube-scheduler-arm<strong i="9">@sha256</strong>:827449ef1f3d8c0a54d842af9d6528217ccd2d36cc2b49815d746d41c7302050            "kube-scheduler --kub"   13 minutes ago      Up 13 minutes                           k8s_kube-scheduler_kube-scheduler-kube-01_kube-system_3ef1979df7569495bb727d12ac1a7a6f_0
4fd0635f9439        gcr.io/google_containers/pause-arm:3.0                                                                                         "/pause"                 14 minutes ago      Up 14 minutes                           k8s_POD_kube-controller-manager-kube-01_kube-system_d44abf63e3ab24853ab86643e0b96d81_0
cfb4a758ad96        gcr.io/google_containers/pause-arm:3.0                                                                                         "/pause"                 14 minutes ago      Up 14 minutes                           k8s_POD_etcd-kube-01_kube-system_90ab26991bf9ad676a430c7592d08bee_0
a631d8b6c11c        gcr.io/google_containers/pause-arm:3.0                                                                                         "/pause"                 14 minutes ago      Up 14 minutes                           k8s_POD_kube-scheduler-kube-01_kube-system_3ef1979df7569495bb727d12ac1a7a6f_0
309b62fff122        gcr.io/google_containers/pause-arm:3.0                                                                                         "/pause"                 14 minutes ago      Up 14 minutes                           k8s_POD_kube-apiserver-kube-01_kube-system_4d99c225ec157dc715c26b59313aeac8_0

admin kubeconfigλ₯Ό 둜컬 머신에 λ³΅μ‚¬ν•˜κ³  kubectl(1.6.0)을 μ‚¬μš©ν•˜μ—¬ kubeadm이 λ“±λ‘λ˜μ—ˆλ‹€κ³  μ£Όμž₯ν•˜λŠ” λ…Έλ“œμ—μ„œ 무슨 일이 μΌμ–΄λ‚˜κ³  μžˆλŠ”μ§€ ν™•μΈν–ˆμŠ΅λ‹ˆλ‹€.

$ kubectl describe node kube-01
Name:           kube-01
Role:
Labels:         beta.kubernetes.io/arch=arm
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=kube-01
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:         <none>
CreationTimestamp:  Tue, 28 Mar 2017 22:06:40 -0700
Phase:
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  OutOfDisk         False   Tue, 28 Mar 2017 22:17:24 -0700     Tue, 28 Mar 2017 22:06:40 -0700     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Tue, 28 Mar 2017 22:17:24 -0700     Tue, 28 Mar 2017 22:06:40 -0700     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Tue, 28 Mar 2017 22:17:24 -0700     Tue, 28 Mar 2017 22:06:40 -0700     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         False   Tue, 28 Mar 2017 22:17:24 -0700     Tue, 28 Mar 2017 22:06:40 -0700     KubeletNotReady         runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:      10.0.1.101,10.0.1.101,kube-01
Capacity:
 cpu:       4
 memory:    882632Ki
 pods:      110
Allocatable:
 cpu:       4
 memory:    780232Ki
 pods:      110
System Info:
 Machine ID:            9989a26f06984d6dbadc01770f018e3b
 System UUID:           9989a26f06984d6dbadc01770f018e3b
 Boot ID:           7a77e2e8-dd62-4989-b9e7-0fb52747162a
 Kernel Version:        4.4.50-hypriotos-v7+
 OS Image:          Raspbian GNU/Linux 8 (jessie)
 Operating System:      linux
 Architecture:          arm
 Container Runtime Version: docker://1.12.6
 Kubelet Version:       v1.6.0
 Kube-Proxy Version:        v1.6.0
PodCIDR:            10.244.0.0/24
ExternalID:         kube-01
Non-terminated Pods:        (4 in total)
  Namespace         Name                        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                        ------------    ----------  --------------- -------------
  kube-system           etcd-kube-01                0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-apiserver-kube-01          250m (6%)   0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-controller-manager-kube-01     200m (5%)   0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-scheduler-kube-01          100m (2%)   0 (0%)      0 (0%)      0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  550m (13%)    0 (0%)      0 (0%)      0 (0%)
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  14m       14m     1   kubelet, kube-01            Normal      Starting        Starting kubelet.
  14m       10m     55  kubelet, kube-01            Normal      NodeHasSufficientDisk   Node kube-01 status is now: NodeHasSufficientDisk
  14m       10m     55  kubelet, kube-01            Normal      NodeHasSufficientMemory Node kube-01 status is now: NodeHasSufficientMemory
  14m       10m     55  kubelet, kube-01            Normal      NodeHasNoDiskPressure   Node kube-01 status is now: NodeHasNoDiskPressure

이것은 kubelet이 μ€€λΉ„λ˜μ§€ μ•Šμ€ 이유λ₯Ό λ°ν˜€λƒˆμŠ΅λ‹ˆλ‹€:

"λŸ°νƒ€μž„ λ„€νŠΈμ›Œν¬κ°€ μ€€λΉ„λ˜μ§€ μ•ŠμŒ: NetworkReady=false 이유:NetworkPluginNotReady λ©”μ‹œμ§€: docker : λ„€νŠΈμ›Œν¬ ν”ŒλŸ¬κ·ΈμΈμ΄ μ€€λΉ„λ˜μ§€ μ•ŠμŒ: cni config"

kubeadm 1.5λ₯Ό μ‚¬μš©ν•œ μ‹€ν—˜μ—μ„œ λ§ˆμŠ€ν„° λ…Έλ“œλ₯Ό λΆˆλŸ¬μ˜€λŠ” 데 CNIκ°€ ν•„μš”ν•˜μ§€ μ•Šμ•˜κΈ° λ•Œλ¬Έμ— 이것은 λ†€λΌμš΄ μΌμž…λ‹ˆλ‹€. μ‹œμž‘ κ°€μ΄λ“œμ—μ„œλ„ CNI ν”ŒλŸ¬κ·ΈμΈ 배포둜 μ΄λ™ν•˜κΈ° 전에 kubeadm init κ°€ μ„±κ³΅μ μœΌλ‘œ μ™„λ£Œλ˜μ–΄μ•Ό ν•œλ‹€κ³  μ œμ•ˆν•©λ‹ˆλ‹€.

μ–΄μ¨Œλ“  둜컬 μ»΄ν“¨ν„°μ—μ„œ kubectl을 μ‚¬μš©ν•˜μ—¬ ν”Œλž€λ„¬μ„ λ°°ν¬ν–ˆμŠ΅λ‹ˆλ‹€.

$ kubectl apply -f kube-flannel.yml

파일 λ‚΄μš©μ΄ μžˆμ—ˆλ˜ μœ„μΉ˜:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      serviceAccountName: flannel
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

κ·ΈλŸ¬λ‚˜ 그것은 κ²°μ½” μ˜ˆμ •λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€:

$ kubectl describe ds kube-flannel-ds -n kube-system
Name:       kube-flannel-ds
Selector:   app=flannel,tier=node
Node-Selector:  beta.kubernetes.io/arch=amd64
Labels:     app=flannel
        tier=node
Annotations:    kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"DaemonSet","metadata":{"annotations":{},"labels":{"app":"flannel","tier":"node"},"name":"kube-flannel-ds","n...
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app=flannel
            tier=node
  Service Account:  flannel
  Containers:
   kube-flannel:
    Image:  quay.io/coreos/flannel:v0.7.0-amd64
    Port:
    Command:
      /opt/bin/flanneld
      --ip-masq
      --kube-subnet-mgr
    Environment:
      POD_NAME:      (v1:metadata.name)
      POD_NAMESPACE:     (v1:metadata.namespace)
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run from run (rw)
   install-cni:
    Image:  quay.io/coreos/flannel:v0.7.0-amd64
    Port:
    Command:
      /bin/sh
      -c
      set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done
    Environment:    <none>
    Mounts:
      /etc/cni/net.d from cni (rw)
      /etc/kube-flannel/ from flannel-cfg (rw)
  Volumes:
   run:
    Type:   HostPath (bare host directory volume)
    Path:   /run
   cni:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/cni/net.d
   flannel-cfg:
    Type:   ConfigMap (a volume populated by a ConfigMap)
    Name:   kube-flannel-cfg
    Optional:   false
Events:     <none>

μ–΄λ–»κ²Œ λ˜λŠ”μ§€ 보기 μœ„ν•΄ μ–΄μ¨Œλ“  λ‹€λ₯Έ μ„œλ²„ 쀑 ν•˜λ‚˜μ— κ°€μž…ν•˜λ €κ³  ν–ˆμŠ΅λ‹ˆλ‹€. kubeadm token create λ₯Ό μ‚¬μš©ν•˜μ—¬ λ‹€λ₯Έ μ»΄ν“¨ν„°μ—μ„œ μ‚¬μš©ν•  수 μžˆλŠ” 토큰을 μˆ˜λ™μœΌλ‘œ μƒμ„±ν–ˆμŠ΅λ‹ˆλ‹€. λ‹€λ₯Έ μ»΄ν“¨ν„°μ—μ„œ:

kubeadm join --token $TOKEN 10.0.1.101:6443
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "10.0.1.101:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.1.101:6443"
[discovery] Failed to request cluster info, will try again: [User "system:anonymous" cannot get configmaps in the namespace "kube-public". (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [User "system:anonymous" cannot get configmaps in the namespace "kube-public". (get configmaps cluster-info)]
[discovery] Failed to request cluster info, will try again: [User "system:anonymous" cannot get configmaps in the namespace "kube-public". (get configmaps cluster-info)]

그리고 λ§ˆμ§€λ§‰ λ©”μ‹œμ§€λŠ” μ˜μ›νžˆ λ°˜λ³΅λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

μ˜ˆμƒν–ˆλ˜ 일 :

kubeadm init λŠ” μ™„λ£Œλ˜κ³  λΆ€νŠΈμŠ€νŠΈλž© 토큰을 생성해야 ν•©λ‹ˆλ‹€.

κ°€μž₯ μœ μš©ν•œ λŒ“κΈ€

flannel이 APIμ—μ„œ 읽을 수 μžˆλ„λ‘ κΆŒν•œμ„ λΆ€μ—¬ν•˜λ €λ©΄ rbac 역할을 μΆ”κ°€ν•΄μ•Ό ν•©λ‹ˆλ‹€.

λ‹€λ₯Έ μ‚¬λžŒμ΄ 이것이 μ˜λ―Έν•˜λŠ” λ°”κ°€ κΆκΈˆν•˜λ‹€λ©΄ flannel을 λ§Œλ“€κΈ° 전에 kube-flannel-rbac.yml λ₯Ό λ§Œλ“€μ–΄μ•Ό ν•˜λŠ” 것 κ°™μŠ΅λ‹ˆλ‹€.

kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

λͺ¨λ“  52 λŒ“κΈ€

Ubuntu 16.04.02, GCE 및 둜컬 VMWare μ„€μΉ˜, Docker 버전 1.12.6, 컀널 4.8.0-44-일반 47~16.04.1-Ubuntu SMPμ—μ„œ λ˜‘κ°™μ€ 일이 λ°œμƒν•©λ‹ˆλ‹€.

kubelet λ‘œκ·ΈλŠ” jimmycuadra의 λ³΄κ³ μ„œμ—μ„œ λ³Ό 수 μžˆλŠ” 였λ₯˜ 전에 λˆ„λ½λœ /etc/cni/net.d에 λŒ€ν•œ κ²½κ³ λ₯Ό ν‘œμ‹œν•©λ‹ˆλ‹€.

Mar 29 04:43:25 instance-1 kubelet[6800]: W0329 04:43:25.763117    6800 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Mar 29 04:43:25 instance-1 kubelet[6800]: E0329 04:43:25.763515    6800 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Ubuntu AWS VMμ—μ„œλ„ λ™μΌν•œ λ¬Έμ œμž…λ‹ˆλ‹€. 도컀 1.12.5

root@ip-10-43-0-20 :~# kubeadm 버전
kubeadm 버전: version.Info{μ£Ό:"1", λΆ€:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState::"clean", BuildDate:"2241 30Z", GoVersion:"go1.7.5"

root@ip-10-43-0-20 :~#
Linux ip-10-43-0-20 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@ip-10-43-0-20 :~# kubeadm init --config cfg.yaml
[kubeadm] κ²½κ³ : kubeadm은 베타 λ²„μ „μ΄λ―€λ‘œ ν”„λ‘œλ•μ…˜ ν΄λŸ¬μŠ€ν„°μ— μ‚¬μš©ν•˜μ§€ λ§ˆμ‹­μ‹œμ˜€.
[μ΄ˆκΈ°ν™”] Kubernetes 버전 μ‚¬μš©: v1.6.0
[μ΄ˆκΈ°ν™”] κΆŒν•œ λΆ€μ—¬ λͺ¨λ“œ μ‚¬μš©: RBAC
[μ΄ˆκΈ°ν™”] κ²½κ³ : ν΄λΌμš°λ“œ κ³΅κΈ‰μž 톡합이 μž‘λ™ν•˜λ €λ©΄ ν΄λŸ¬μŠ€ν„°μ˜ λͺ¨λ“  kubelet에 λŒ€ν•΄ --cloud-providerκ°€ μ„€μ •λ˜μ–΄μ•Ό ν•©λ‹ˆλ‹€.
(/etc/systemd/system/kubelet.service.d/10-kubeadm.confλŠ” 이λ₯Ό μœ„ν•΄ νŽΈμ§‘λ˜μ–΄μ•Ό 함)
[μ‹€ν–‰ μ „] μ‹€ν–‰ μ „ 검사 μ‹€ν–‰
[ν”„λ¦¬ν”ŒλΌμ΄νŠΈ] kubelet μ„œλΉ„μŠ€ μ‹œμž‘ν•˜κΈ°
[μΈμ¦μ„œ] μƒμ„±λœ CA μΈμ¦μ„œ 및 ν‚€μž…λ‹ˆλ‹€.
[μΈμ¦μ„œ] μƒμ„±λœ API μ„œλ²„ μΈμ¦μ„œ 및 ν‚€μž…λ‹ˆλ‹€.
[μΈμ¦μ„œ] API μ„œλ²„ μ„œλΉ™ μΈμ¦μ„œλŠ” DNS 이름 [ip-10-43-0-20 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] 및 IP [10.96.0.1 10.43.0.20]에 λŒ€ν•΄ μ„œλͺ…λ˜μ—ˆμŠ΅λ‹ˆλ‹€. ]
[μΈμ¦μ„œ] API μ„œλ²„ kubelet ν΄λΌμ΄μ–ΈνŠΈ μΈμ¦μ„œ 및 ν‚€λ₯Ό μƒμ„±ν–ˆμŠ΅λ‹ˆλ‹€.
[μΈμ¦μ„œ] μƒμ„±λœ μ„œλΉ„μŠ€ 계정 토큰 μ„œλͺ… ν‚€ 및 곡개 ν‚€.
[μΈμ¦μ„œ] ν”„λ‘ νŠΈ ν”„λ‘μ‹œ CA μΈμ¦μ„œ 및 ν‚€λ₯Ό μƒμ„±ν–ˆμŠ΅λ‹ˆλ‹€.
[μΈμ¦μ„œ] ν”„λ‘ νŠΈ ν”„λ‘μ‹œ ν΄λΌμ΄μ–ΈνŠΈ μΈμ¦μ„œ 및 ν‚€λ₯Ό μƒμ„±ν–ˆμŠ΅λ‹ˆλ‹€.
[μΈμ¦μ„œ] μœ νš¨ν•œ μΈμ¦μ„œ 및 ν‚€κ°€ 이제 "/etc/kubernetes/pki"에 μžˆμŠ΅λ‹ˆλ‹€.
[kubeconfig] λ””μŠ€ν¬μ— KubeConfig 파일 μž‘μ„±: "/etc/kubernetes/admin.conf"
[kubeconfig] λ””μŠ€ν¬μ— KubeConfig 파일 μž‘μ„±: "/etc/kubernetes/kubelet.conf"
[kubeconfig] λ””μŠ€ν¬μ— KubeConfig 파일 μž‘μ„±: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] λ””μŠ€ν¬μ— KubeConfig 파일 μž‘μ„±: "/etc/kubernetes/scheduler.conf"
[apiclient] API ν΄λΌμ΄μ–ΈνŠΈ 생성, 컨트둀 ν”Œλ ˆμΈ μ€€λΉ„ λŒ€κΈ°
[apiclient] λͺ¨λ“  μ œμ–΄ 평면 ꡬ성 μš”μ†ŒλŠ” 16.531681초 후에 μ •μƒμž…λ‹ˆλ‹€.
[apiclient] ν•˜λ‚˜ μ΄μƒμ˜ λ…Έλ“œκ°€ λ“±λ‘λ˜κ³  쀀비될 λ•ŒκΉŒμ§€ λŒ€κΈ° 쀑
[apiclient] 첫 번째 λ…Έλ“œκ°€ λ“±λ‘λ˜μ—ˆμ§€λ§Œ 아직 μ€€λΉ„λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.
[apiclient] 첫 번째 λ…Έλ“œκ°€ λ“±λ‘λ˜μ—ˆμ§€λ§Œ 아직 μ€€λΉ„λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.
[apiclient] 첫 번째 λ…Έλ“œκ°€ λ“±λ‘λ˜μ—ˆμ§€λ§Œ 아직 μ€€λΉ„λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.

++ λ™μΌν•œ 문제(Ubuntu 16.04.1)

Ubuntu 16.04μ—μ„œλ„ λ§ˆμ°¬κ°€μ§€μž…λ‹ˆλ‹€.

CentOS 7μ—μ„œ kubelet을 1.5.4 λ‹€μš΄κ·Έλ ˆμ΄λ“œν–ˆμŠ΅λ‹ˆλ‹€. 그것은 λ‚˜λ₯Ό μœ„ν•΄ 그것을 ν•΄κ²°ν–ˆμŠ΅λ‹ˆλ‹€. 1.6.0 kubeletμ—μ„œ μ€€λΉ„ 확인이 λ‹€λ₯΄κ²Œ μž‘λ™ν•˜λŠ” 것 κ°™μŠ΅λ‹ˆλ‹€.

k8s 1.6.0으둜 μ—…κ·Έλ ˆμ΄λ“œν•œ 이후 λ² μ–΄λ©”νƒˆ x64 μ‹œμŠ€ν…œμ˜ CentOS 7μ—μ„œλ„ λ™μΌν•œ 문제

Ubuntu 16.04μ—μ„œ λ™μΌν•œ 문제

Ubuntu 16.04μ—μ„œ λ™μΌν•œ 문제둜 kubelet νŒ¨ν‚€μ§€λ₯Ό μˆ˜λ™μœΌλ‘œ λ‹€μš΄κ·Έλ ˆμ΄λ“œν•˜λ©΄ λ¬Έμ œκ°€ ν•΄κ²°λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

# apt install kubelet=1.5.6-00

@ctrlaltdel 그것은 λ‚˜λ₯Ό μœ„ν•΄ μž‘λ™ν•˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.

λ‚˜λŠ” 이것이 Kubelet 문제라고 μƒκ°ν•©λ‹ˆλ‹€. CNIκ°€ ꡬ성 ν•΄μ œλ˜μ–΄ μžˆμ„ λ•Œ λ…Έλ“œλ₯Ό μ€€λΉ„λ˜μ§€ μ•Šμ€ κ²ƒμœΌλ‘œ ν‘œμ‹œν•΄μ„œλŠ” μ•ˆ λ©λ‹ˆλ‹€. CNIκ°€ ν•„μš”ν•œ ν¬λ“œλ§Œ μ€€λΉ„λ˜μ§€ μ•Šμ€ κ²ƒμœΌλ‘œ ν‘œμ‹œν•΄μ•Ό ν•©λ‹ˆλ‹€.

@jbeda 이 λ¬Έμ œκ°€ μ–Έμ œ ν•΄κ²°λ˜λŠ”μ§€ μ•„μ‹­λ‹ˆκΉŒ?

@kristiandrucker -- μ•„λ‹ˆ -- μ—¬μ „νžˆ 무슨 일이 μΌμ–΄λ‚˜κ³  μžˆλŠ”μ§€ νŒŒμ•…ν•˜κ³  μžˆμŠ΅λ‹ˆλ‹€. 원인을 λ¨Όμ € νŒŒμ•…ν•΄μ•Ό ν•©λ‹ˆλ‹€.

@jbeda μ•Œκ² μŠ΅λ‹ˆλ‹€ . ν•˜μ§€λ§Œ λ¬Έμ œκ°€ ν•΄κ²°λœ ν›„μ—λŠ” μ–΄λ–»κ²Œ λ κΉŒμš”? μ†ŒμŠ€μ—μ„œ kubelet을 λ‹€μ‹œ λΉŒλ“œν•˜μ‹œκ² μŠ΅λ‹ˆκΉŒ?

@kristiandrucker 이것은 kubelet 문제인 경우 k8s의 포인트 λ¦΄λ¦¬μŠ€μ—μ„œ λ‚˜μ™€μ•Ό ν•©λ‹ˆλ‹€.

https://github.com/kubernetes/kubernetes/pull/43474 κ°€ κ·Όλ³Έ 원인이라고 μƒκ°ν•©λ‹ˆλ‹€. 버그λ₯Ό μ‹ κ³ ν•˜κ³  λ„€νŠΈμ›Œν¬ μ‚¬λžŒλ“€μ—κ²Œ 후속 쑰치λ₯Ό μ·¨ν•˜λ €κ³  ν•©λ‹ˆλ‹€.

@dcbw 주변에 μžˆλ‚˜μš”?

λ¬Έμ œλŠ” DaemonSet이 Net workReady:false 쑰건이 μžˆλŠ” λ…Έλ“œμ— μ˜ˆμ•½λ˜μ§€ μ•Šμ€ 것 stNetwork:true인 ν¬λ“œλŠ” Net workReady:false 인 λ…Έλ“œμ—μ„œ μ˜ˆμ•½ν•΄μ•Ό ν•˜μ§€λ§Œ ho stNetwork:false ν¬λ“œλŠ” μ˜ˆμ•½ν•˜μ§€ μ•Šμ•„μ•Ό ν•©λ‹ˆλ‹€.

ν•΄κ²° λ°©λ²•μœΌλ‘œ DaemonSet에 scheduler.alpha.kubernetes.io/critical-pod 주석을 μΆ”κ°€ν•˜λ©΄ λ‹€μ‹œ μž‘λ™ν•©λ‹ˆκΉŒ?

@janetkuo @lukaszo DS λ™μž‘μ„ λΆ„λ₯˜ν•  수 μžˆμŠ΅λ‹ˆκΉŒ?

#sig-network on slack, btwμ—μ„œλ„ 진행쀑인 토둠이 μžˆμŠ΅λ‹ˆλ‹€.

같은 문제 CentOS 7 x64

@prapdm 이것은 μ‹€ν–‰ 쀑인 λ°°ν¬νŒμ— λŒ€ν•΄ 무방비 μƒνƒœμΈ κ²ƒμœΌλ‘œ λ³΄μž…λ‹ˆλ‹€.

CentOS Linux 릴리슀 7.3.1611(μ½”μ–΄)

Ubuntu 16.04κ°€ μžˆλŠ” ν•œ λ…Έλ“œμ—μ„œ μ‹œλ„ν–ˆμŠ΅λ‹ˆλ‹€. "아직 μ€€λΉ„λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€"λΌλŠ” λ©”μ‹œμ§€μ™€ ν•¨κ»˜ 멈μΆ₯λ‹ˆλ‹€. λ˜ν•œ μˆ˜λ™μœΌλ‘œ ν”Œλž€λ„¬ DaemonSet을 μƒμ„±ν–ˆμ§€λ§Œ 제 κ²½μš°μ—λŠ” 문제 없이 ν•˜λ‚˜μ˜ ν¬λ“œλ₯Ό μ˜ˆμ•½ν–ˆμŠ΅λ‹ˆλ‹€. 데λͺ¬ ν¬λ“œ μžμ²΄κ°€ E0329 22:57:03.065651 1 main.go:127] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-z3xgn': the server does not allow access to the requested resource (get pods kube-flannel-ds-z3xgn) 였λ₯˜μ™€ ν•¨κ»˜ CrashLoopBackOff에 λ“€μ–΄κ°”μŠ΅λ‹ˆλ‹€.

Centosμ—μ„œλ„ μ‹œλ„ν•  κ²ƒμ΄μ§€λ§Œ DaemonSet이 μ—¬κΈ°μ—μ„œ μ±…μž„μ΄ μžˆλ‹€κ³  μƒκ°ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. kubeadm이 μ—¬κΈ°μ—μ„œ μ€‘λ‹¨λ©λ‹ˆλ‹€.

그것은 rbac κΆŒν•œ 였λ₯˜μž…λ‹ˆλ‹€.

@jimmycuadra μ•” ν”„λ‘œμ„Έμ„œκ°€ μžˆλŠ” 라즈베리 νŒŒμ΄μ—μ„œ μ‹€ν–‰ν•˜κ³  μžˆλ‹€λŠ” 사싀을 방금 μ•Œμ•˜μŠ΅λ‹ˆλ‹€.

flannel 데λͺ¬ μ„ΈνŠΈμ˜ 경우 λ‹€μŒμ΄ μžˆμŠ΅λ‹ˆλ‹€.

``` λ…Έλ“œ 선택기:
beta.kubernetes.io/arch: amd64

but your node is labeled with: 

beta.kubernetes.io/arch=arm
```

λ”°λΌμ„œ DaemonSet은 이 λ…Έλ“œμ—μ„œ ν¬λ“œλ₯Ό 점심할 수 μ—†μœΌλ©° λ…Έλ“œ 선택기λ₯Ό λ³€κ²½ν•˜κΈ°λ§Œ ν•˜λ©΄ μž‘λ™ν•©λ‹ˆλ‹€.
μ—¬μ „νžˆ rbac κΆŒν•œμœΌλ‘œ 였λ₯˜κ°€ λ°œμƒν•˜μ§€λ§Œ λ‚΄κ°€ λͺ¨λ₯΄κΈ° λ•Œλ¬Έμ— @mikedanese κ°€ μˆ˜μ • 방법을 μ•Œλ €μ€„ κ²ƒμž…λ‹ˆλ‹€.

μ•„, κ°μ‚¬ν•©λ‹ˆλ‹€ @lukaszo! μ΄λ²ˆμ—λŠ” RPi κ΄€λ ¨ κ°€μ΄λ“œ(k8s 1.5에 μ‚¬μš©)λ₯Ό λ”°λ₯΄μ§€ μ•Šμ•˜κ³  κ·Έ 단계λ₯Ό μžŠμ–΄λ²„λ ΈμŠ΅λ‹ˆλ‹€. 데λͺ¬ μ„ΈνŠΈμ— 였λ₯˜κ°€ λ°œμƒν–ˆμ„ λ•Œ 그것을 λ°œκ²¬ν–ˆμ„ κ²ƒμž…λ‹ˆλ‹€. :}

여기에 μ„€λͺ…λœ 지침을 λ”°λ₯Ό λ•Œλ„ 이 λ¬Έμ œκ°€ λ°œμƒν•©λ‹ˆλ‹€.
https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/

μ˜¬λ°”λ₯Έ ν”Œλž€λ„¬ λ„€νŠΈμ›Œν¬ ν¬λ“œλ₯Ό μ„€μΉ˜ν•œ ν›„ μž‘λ™ν•˜λ„λ‘ κ΄€λ¦¬ν–ˆμŠ΅λ‹ˆλ‹€.

@jimmycuadra κ°€ @lukaszo λŒ“κΈ€κ³Ό ν•¨κ»˜ μž‘λ™ ν•˜λ„λ‘ ν•  수 μžˆλ‹€κ³  μƒκ°ν•©λ‹ˆλ‹€.

[apiclient] First node has registered, but is not ready yet λ©”μ‹œμ§€κ°€ ν”ŒλŸ¬λ”©μ„ μ‹œμž‘ν•˜λ©΄ kubernetes API μ„œλ²„κ°€ μ‹€ν–‰λ˜λ―€λ‘œ λ‹€μŒμ„ μˆ˜ν–‰ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml | kubectl create -f -

라즈베리 파이 μ„€μΉ˜μ˜ 경우:

curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

그러면 μ™„λ£Œλ©λ‹ˆλ‹€.

[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node has registered, but is not ready yet
[apiclient] First node is ready after 245.050597 seconds
[apiclient] Test deployment succeeded
[token] Using token: 4dc99e............
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 4dc99e........... 192.168.1.200:6443

λ‚˜λŠ” 같은 λ¬Έμ œκ°€ μžˆμ—ˆκ³  λ‹€μŒκ³Ό 같이 μˆ˜μ •ν–ˆμŠ΅λ‹ˆλ‹€.
당신은 루트 μ—¬μ•Όν•©λ‹ˆλ‹€

kubeadm 1.6.0μ—μ„œλŠ” μ‹œμŠ€ν…œ νŒŒμΌμ—μ„œ $KUBELET_NETWORK_ARGS ν™˜κ²½ λ³€μˆ˜λ₯Ό μ œκ±°ν•΄μ•Ό ν•©λ‹ˆλ‹€. /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

그런 λ‹€μŒ μ•…λ§ˆλ₯Ό λ‹€μ‹œ μ‹œμž‘ν•˜μ‹­μ‹œμ˜€.

systemctl daemon-reload

kubeadm init

쑰금 μ‹œκ°„μ΄ κ±Έλ¦½λ‹ˆλ‹€ ... 성곡 ν›„

μ‚¬μš©ν•˜λ €λŠ” λ„€νŠΈμ›Œν¬ μ• λ“œμ˜¨ λ‹€μš΄λ‘œλ“œ: http://kubernetes.io/docs/admin/addons/

μ˜₯μ–‘λͺ©μ΄ κ°€μž₯ 쒋은 것 κ°™μŠ΅λ‹ˆλ‹€. ν™•μ‹€ν•˜μ§€λŠ” μ•Šμ§€λ§Œ 아직 ν…ŒμŠ€νŠΈ μ€‘μž…λ‹ˆλ‹€.

@thelastworm
방금 ν•˜λ €κ³ ν–ˆλŠ”λ° μž‘λ™ν•˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.
μš°λΆ„νˆ¬ 16.04.2 LTS, kubeadm 1.6.0
λ‹€μŒ 단계λ₯Ό μˆ˜ν–‰ν–ˆμŠ΅λ‹ˆλ‹€.

  1. /etc/systemd/system/kubelet.service.d/10-kubeadm.conf νŽΈμ§‘ 및 $KUBELET_NETWORK_ARGS 제거
  2. kubeadm reset μ‹œμž‘ν•˜λ €λŠ” 이전 μ‹œλ„λ₯Ό μ •λ¦¬ν•©λ‹ˆλ‹€.
  3. kubeadm init --token=<VALUE> --apiserver-advertise-address=<IP>

[νŽΈμ§‘λ¨]
@srinat999 κ°€ systemctl daemon-reload 전에 kubeadm init systemctl daemon-reload λ₯Ό μ‹€ν–‰ν•΄μ•Ό ν•œλ‹€κ³  μ§€μ ν•œ ν›„ μž‘λ™ν–ˆμŠ΅λ‹ˆλ‹€.

@jcorral 의 μ†”λ£¨μ…˜μ€ μ•ˆμ „ν•˜μ§€ μ•Šμ€ API ν¬νŠΈκ°€ 더 이상 kubeadm μ˜ν•΄ μƒμ„±λ˜μ§€ μ•ŠκΈ° λ•Œλ¬Έμ— ν”Œλž€λ„¬ 배포에 λŒ€ν•œ ν•œ 번의 λ³€κ²½μœΌλ‘œ μ €μ—κ²Œ νš¨κ³Όμ μ΄μ—ˆμŠ΅λ‹ˆλ‹€ .

curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml | \
kubectl --kubeconfig /etc/kubernetes/admin.conf create -f -

@MaximF conf νŒŒμΌμ„ λ³€κ²½ν•œ ν›„ systemctl daemon-reload λ₯Ό ν•΄μ€˜μ•Ό ν•©λ‹ˆλ‹€. λ‚˜λ₯Ό μœ„ν•΄ μΌν–ˆλ‹€.

@jcorral κ·€ν•˜μ˜ μ†”λ£¨μ…˜μ΄ μ €μ—κ²Œ νš¨κ³Όμ μž…λ‹ˆλ‹€. 감사 ν•΄μš”.

@MaximF λ‚œ κ·Έλƒ₯ λ‹€μ‹œ μ‹œμž‘ μ•…λ§ˆ λͺ…령쀄을 μΆ”κ°€ν•©λ‹ˆλ‹€

kubeadm initκ°€ μ„±κ³΅μ μœΌλ‘œ μ™„λ£Œλ˜μ§€λ§Œ 버전을 확인할 λ•Œ λ‹€μŒ 였λ₯˜κ°€ λ°œμƒν•©λ‹ˆλ‹€.

ν΄λΌμ΄μ–ΈνŠΈ 버전: version.Info{μ£Ό:"1", λΆ€:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-163 33Z", GoVersion:"go1.7.5", 컴파일러:"gc", ν”Œλž«νΌ:"linux/amd64"}
μ„œλ²„ localhost:8080 에 λŒ€ν•œ 연결이 κ±°λΆ€λ˜μ—ˆμŠ΅λ‹ˆλ‹€. μ˜¬λ°”λ₯Έ 호슀트 λ˜λŠ” 포트λ₯Ό μ§€μ •ν–ˆμŠ΅λ‹ˆκΉŒ?

@ν•˜λ¦¬λ³Ό
KUBECONFIG ν™˜κ²½ λ³€μˆ˜λ₯Ό μ„€μ •ν•΄μ•Ό ν•©λ‹ˆλ‹€.

CNI와 κ΄€λ ¨λœ ν•΄κ²° 방법을 μ‹€ν–‰ν•œ ν›„ Flannel을 μ‹€ν–‰ν•œ μ‚¬λžŒμ΄ μžˆμŠ΅λ‹ˆκΉŒ? μ€€λΉ„λ˜μ§€ μ•Šμ€ 문제λ₯Ό 전달할 수 μžˆμ§€λ§Œ Flannel을 μ‹€ν–‰ν•˜λ©΄ λ‹€μŒκ³Ό 같은 였λ₯˜κ°€ λ°œμƒν•©λ‹ˆλ‹€.

Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-g5cbj': the server does not allow access to the requested resource (get pods kube-flannel-ds-g5cbj)

Pod μƒνƒœμ— "CrashLoopBackOff"κ°€ ν‘œμ‹œλ¨

flannel이 APIμ—μ„œ 읽을 수 μžˆλ„λ‘ κΆŒν•œμ„ λΆ€μ—¬ν•˜λ €λ©΄ rbac 역할을 μΆ”κ°€ν•΄μ•Ό ν•©λ‹ˆλ‹€.

flannel이 APIμ—μ„œ 읽을 수 μžˆλ„λ‘ κΆŒν•œμ„ λΆ€μ—¬ν•˜λ €λ©΄ rbac 역할을 μΆ”κ°€ν•΄μ•Ό ν•©λ‹ˆλ‹€.

λ‹€λ₯Έ μ‚¬λžŒμ΄ 이것이 μ˜λ―Έν•˜λŠ” λ°”κ°€ κΆκΈˆν•˜λ‹€λ©΄ flannel을 λ§Œλ“€κΈ° 전에 kube-flannel-rbac.yml λ₯Ό λ§Œλ“€μ–΄μ•Ό ν•˜λŠ” 것 κ°™μŠ΅λ‹ˆλ‹€.

kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

루트 λ¬Έμ œκ°€ ν•΄κ²°λ˜κ³  κ΄€λ ¨ 티켓이 λ‹«ν˜€ 있기 λ•Œλ¬Έμ— 이

μ •λ³΄μš©: Ubuntu 16.04μ—μ„œ μ—…λ°μ΄νŠΈλœ νŒ¨ν‚€μ§€λ‘œ ν˜„μž¬ μž‘λ™ν•˜κ³  μžˆμŠ΅λ‹ˆλ‹€.

1.6.1이 μ €μ—κ²Œ νš¨κ³Όμ μž…λ‹ˆλ‹€! 이 문제λ₯Ό ν•΄κ²°ν•˜λŠ” 데 도움을 μ£Όμ‹  λͺ¨λ“  λΆ„λ“€κ»˜ κ°μ‚¬λ“œλ¦½λ‹ˆλ‹€!

λ‹€μŒ 단계λ₯Ό μˆ˜ν–‰ν•˜μ—¬ centos-release-7-3.1611.el7.centos.x86_64에 Kubernetes ν΄λŸ¬μŠ€ν„°λ₯Ό μ„±κ³΅μ μœΌλ‘œ μ„€μ •ν–ˆμŠ΅λ‹ˆλ‹€(Dockerκ°€ 이미 μ„€μΉ˜λ˜μ–΄ μžˆλ‹€κ³  κ°€μ •).

1) (/etc/yum.repo.d/kubernetes.repoμ—μ„œ) baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64-unstable
=> μ΅œμ‹  Kubernetes 1.6.1μ—μ„œ λΆˆμ•ˆμ •ν•œ μ €μž₯μ†Œλ₯Ό μ‚¬μš©ν•˜λ €λ©΄
2) yum μ„€μΉ˜ -y kubelet kubeadm kubectl kubernetes-cni
3) (/etc/systemd/system/kubelet.service.d/10-kubeadm.conf) λ§ˆμ§€λ§‰ 쀄 끝에 "--cgroup-driver=systemd"λ₯Ό μΆ”κ°€ν•©λ‹ˆλ‹€.
=> DockerλŠ” cgroup-driver에 systemdλ₯Ό μ‚¬μš©ν•˜κ³  kubelet은 cgroup-driver에 cgroupfsλ₯Ό μ‚¬μš©ν•˜κΈ° λ•Œλ¬Έμž…λ‹ˆλ‹€.
4) systemctl은 kubelet을 ν™œμ„±ν™”ν•˜κ³  && systemctl은 kubelet을 μ‹œμž‘ν•©λ‹ˆλ‹€.
5) kubeadm μ΄ˆκΈ°ν™” --pod-network-cidr 10.244.0.0/16
=> --api-advertise-addressesλ₯Ό μΆ”κ°€ν–ˆλ‹€λ©΄ λŒ€μ‹  --apiserver-advertise-addressλ₯Ό μ‚¬μš©ν•΄μ•Ό ν•©λ‹ˆλ‹€.
6) cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
내보내기 KUBECONFIG=$HOME/admin.conf
=> 이 단계가 μ—†μœΌλ©΄ kubectl getμ—μ„œ 였λ₯˜κ°€ λ°œμƒν•  수 μžˆμŠ΅λ‹ˆλ‹€.
=> 1.5.2μ—μ„œλŠ” ν•˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.
7) kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
=> 1.6.0은 μ—­ν•  기반 μ•‘μ„ΈμŠ€ μ œμ–΄λ₯Ό λ„μž…ν•˜λ―€λ‘œ Flannel 데λͺ¬μ…‹μ„ μƒμ„±ν•˜κΈ° 전에 ClusterRole 및 ClusterRoleBinding을 μΆ”κ°€ν•΄μ•Ό ν•©λ‹ˆλ‹€.
8) kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
=> Flannel 데λͺ¬μ…‹ 생성
9) (λͺ¨λ“  슬레이브 λ…Έλ“œμ—μ„œ) kubeadm join --token (κ·€ν•˜μ˜ 토큰) (ip):(port)
=> kubeadm init의 결과와 같이

μœ„μ˜ λͺ¨λ“  λ‹¨κ³„λŠ” Kubernetes-1.6.0, 특히 kubeadmκ³Ό κ΄€λ ¨λœ λ‹€μ–‘ν•œ 문제의 μ œμ•ˆμ„ κ²°ν•©ν•œ κ²°κ³Όμž…λ‹ˆλ‹€.

μ‹œκ°„μ„ μ ˆμ•½ν•  수 있기λ₯Ό λ°”λžλ‹ˆλ‹€.

@eastcirclek @Sliim λŒ€λ‹¨ ν•˜μ‹­λ‹ˆλ‹€

@eastcirclek 이것은 μ—¬λŸ¬ ν¬λŸΌμ—λ„ μΏΌλ¦¬ν•˜μ—¬ 방금 μ‹€ν–‰ν•œ μ •ν™•ν•œ λ‹¨κ³„μž…λ‹ˆλ‹€. μ‹œκ°„λŒ€ 차이가 μ•„λ‹κΉŒμš”? λͺ¨λ‘ κ°μ‚¬ν•©λ‹ˆλ‹€. 이 μ£Όμ œλŠ” 정말 도움이 λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

AWS에 Ubuntu 16.04 μ„œλ²„κ°€ 있고 단계λ₯Ό λ”°λžμŠ΅λ‹ˆλ‹€.

  1. /etc/systemd/system/kubelet.service.d/10-kubeadm.conf νŽΈμ§‘ 및 $KUBELET_NETWORK_ARGS 제거
  2. kubeadm reset을 톡해 이전 μ‹œμž‘ μ‹œλ„λ₯Ό μ •λ¦¬ν•©λ‹ˆλ‹€.
  3. kubeadm μ΄ˆκΈ°ν™” --토큰=--apiserver-advertise-address=

λΆ„λͺ…νžˆ μ˜¬λ°”λ₯΄κ²Œ μž‘λ™ν–ˆμ§€λ§Œ Calicoλ₯Ό λ„€νŠΈμ›Œν¬ ν”ŒλŸ¬κ·ΈμΈμœΌλ‘œ μ„€μΉ˜ν•˜λ €κ³ ν•˜λ©΄ λ‹€μŒ 였λ₯˜κ°€ λ°œμƒν•©λ‹ˆλ‹€.
μ„œλ²„ localhost:8080 에 λŒ€ν•œ 연결이 κ±°λΆ€λ˜μ—ˆμŠ΅λ‹ˆλ‹€. μ˜¬λ°”λ₯Έ 호슀트 λ˜λŠ” 포트λ₯Ό μ§€μ •ν–ˆμŠ΅λ‹ˆκΉŒ?

k8s νŒ€μ΄ 패치 μž‘μ—…μ„ ν•˜κ³  μžˆμŠ΅λ‹ˆκΉŒ?

감사 ν•΄μš”

@overip μ €λŠ” νŒ¨μΉ˜κ°€ ν•„μš”ν•˜μ§€ μ•Šλ‹€κ³  μƒκ°ν•©λ‹ˆλ‹€... kubectl을 μ‚¬μš©ν•  λ•Œ μ˜¬λ°”λ₯Έ kubeconfig νŒŒμΌμ„ μ§€μ •ν•˜κΈ°λ§Œ ν•˜λ©΄ λ©λ‹ˆλ‹€. kubeadm은 /etc/kubernetes/admin.conf μž‘μ„±ν–ˆμ–΄μ•Ό ν•©λ‹ˆλ‹€.

@jimmycuadra κ·Έ 단계λ₯Ό μ„€λͺ…ν•΄ μ£Όμ‹œκ² μŠ΅λ‹ˆκΉŒ?

@overip kubeadm init 의 좜λ ₯μ—λŠ” λ‹€μŒ 지침이 μžˆμŠ΅λ‹ˆλ‹€.

To start using your cluster, you need to run (as a regular user):

  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf

개인적으둜 μ €λŠ” νŒŒμΌμ„ $HOME/.kube/config 에 λ³΅μ‚¬ν•˜λŠ” 것을 μ„ ν˜Έν•©λ‹ˆλ‹€. 이 μœ„μΉ˜μ—μ„œ kubectl이 기본적으둜 νŒŒμΌμ„ μ°ΎμŠ΅λ‹ˆλ‹€. 그러면 KUBECONFIG ν™˜κ²½ λ³€μˆ˜λ₯Ό μ„€μ •ν•  ν•„μš”κ°€ μ—†μŠ΅λ‹ˆλ‹€.

둜컬 μ»΄ν“¨ν„°μ—μ„œ kubectl을 μ‚¬μš©ν•  κ³„νšμ΄λΌλ©΄ scp (λ˜λŠ” λ‚΄μš©μ„ λ³΅μ‚¬ν•˜μ—¬ λΆ™μ—¬λ„£κΈ°λ§Œ 해도) μžμ‹ μ˜ μ»΄ν“¨ν„°μ—μ„œ ~/.kube/config 에 μ“Έ 수 μžˆμŠ΅λ‹ˆλ‹€.

μžμ„Έν•œ λ‚΄μš©μ€ 이 GitHub λ¬Έμ œμ—μ„œ "admin.conf"λ₯Ό κ²€μƒ‰ν•˜μ„Έμš”. λͺ‡ λ²ˆμ΄λ‚˜ μ–ΈκΈ‰λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

@eastcirclek - 단계λ₯Ό λ”°λžμ§€λ§Œ μ–΄λ–€ 이유둜 λ…Έλ“œκ°€ ν”Œλž€λ„¬μ„ μ œλŒ€λ‘œ μ„€μΉ˜ν•  수 μ—†μŠ΅λ‹ˆλ‹€.
(μ°Έκ³ : λ§ˆμŠ€ν„°μ—μ„œλŠ” λͺ¨λ“  것이 λΆ€λ“œλŸ½μŠ΅λ‹ˆλ‹€.)

Apr 13 22:31:11 node2 kubelet[22893]: I0413 22:31:11.666206   22893 kuberuntime_manager.go:458] Container {Name:install-cni Image:quay.io/coreos/flannel:v0.7.0-amd64 Command:[/bin/sh -c set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:cni ReadOnly:false MountPath:/etc/cni/net.d SubPath:} {Name:flannel-cfg ReadOnly:false MountPath:/etc/kube-flannel/ SubPath:} {Name:flannel-token-g65nf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath:}] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Apr 13 22:31:11 node2 kubelet[22893]: I0413 22:31:11.666280   22893 kuberuntime_manager.go:742] checking backoff for container "install-cni" in pod "kube-flannel-ds-3smf7_kube-system(2e6ad0f9-207f-11e7-8f34-0050569120ff)"
Apr 13 22:31:12 node2 kubelet[22893]: I0413 22:31:12.846325   22893 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/configmap/2e6ad0f9-207f-11e7-8f34-0050569120ff-flannel-cfg" (spec.Name: "flannel-cfg") pod "2e6ad0f9-207f-11e7-8f34-0050569120ff" (UID: "2e6ad0f9-207f-11e7-8f34-0050569120ff").
Apr 13 22:31:12 node2 kubelet[22893]: I0413 22:31:12.846373   22893 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2e6ad0f9-207f-11e7-8f34-0050569120ff-flannel-token-g65nf" (spec.Name: "flannel-token-g65nf") pod "2e6ad0f9-207f-11e7-8f34-0050569120ff" (UID: "2e6ad0f9-207f-11e7-8f34-0050569120ff").

λ‚΄ ν•΄κ²° 방법을 κ³΅μœ ν•˜μ‹­μ‹œμ˜€. λ¨Όμ € $KUBELET_NETWORK_ARGSκ°€ ν•„μš”ν•©λ‹ˆλ‹€. 그렇지 μ•ŠμœΌλ©΄ CNIκ°€ ν™œμ„±ν™”/κ΅¬μ„±λ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. $KUBELET_NETWORK_ARGS 제거 및 볡원이 λ„ˆλ¬΄ λ³΅μž‘ν•΄ λ³΄μž…λ‹ˆλ‹€.
kubeadm init에 "[apiclient] 첫 번째 λ…Έλ“œκ°€ λ“±λ‘λ˜μ—ˆμ§€λ§Œ 아직 μ€€λΉ„λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€"κ°€ ν‘œμ‹œλ˜λ©΄ k8s ν΄λŸ¬μŠ€ν„°κ°€ μ‹€μ œλ‘œ μš”μ²­μ„ μ²˜λ¦¬ν•  μ€€λΉ„κ°€ 된 κ²ƒμž…λ‹ˆλ‹€. μ΄λ•Œ μ‚¬μš©μžλŠ” λ‹€μŒκ³Ό 같이 https://kubernetes.io/docs/getting-started-guides/kubeadm/의 3/4λ‹¨κ³„λ‘œ μ΄λ™ν•˜λ©΄ λ©λ‹ˆλ‹€.

To start using your cluster, you need to run (as a regular user):

  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:  http://kubernetes.io/docs/admin/addons/

μ‚¬μš©μžκ°€ ν¬λ“œ λ„€νŠΈμ›Œν¬λ₯Ό μ„€μΉ˜ν•  λ•Œ ν¬λ“œ λ„€νŠΈμ›Œν¬ μ •μ±…μ˜ μ„œλΉ„μŠ€ 계정에 μΆ©λΆ„ν•œ κΆŒν•œμ΄ λΆ€μ—¬λ˜μ—ˆλŠ”μ§€ ν™•μΈν•˜μ‹­μ‹œμ˜€. ν”Œλž€λ„¬μ„ 예둜 λ“€μ–΄λ³΄κ² μŠ΅λ‹ˆλ‹€. λ‹€μŒκ³Ό 같이 flannel의 μ„œλΉ„μŠ€ 계정에 ν΄λŸ¬μŠ€ν„° κ΄€λ¦¬μž 역할을 λ°”μΈλ”©ν•©λ‹ˆλ‹€. 이상적이지 μ•Šμ„ 수 있으며 ν”Œλž€λ„¬ μ„œλΉ„μŠ€ 계정에 λŒ€ν•œ νŠΉμ • 역할을 μ •μ˜ν•  수 μžˆμŠ΅λ‹ˆλ‹€. BTW, μ‚¬μš©μžκ°€ λŒ€μ‹œλ³΄λ“œμ™€ 같은 λ‹€λ₯Έ μ• λ“œμ˜¨ μ„œλΉ„μŠ€λ₯Ό 배포할 λ•Œ κ΄€λ ¨ μ„œλΉ„μŠ€ 계정에 μΆ©λΆ„ν•œ κΆŒν•œλ„ λΆ€μ—¬ν•΄μ•Ό ν•©λ‹ˆλ‹€.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: flannel:daemonset
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: flannel
  namespace:  kube-system

Podnetwork μ„œλ²„κ°€ μ€€λΉ„λ˜λ©΄ kubeadm initλŠ” λ…Έλ“œκ°€ μ€€λΉ„λ˜μ—ˆμŒμ„ ν‘œμ‹œν•˜κ³  μ‚¬μš©μžλŠ” μ§€μ‹œλ₯Ό 계속할 수 μžˆμŠ΅λ‹ˆλ‹€.

ν”Œλž€λ„¬μ„ 예둜 λ“€μ–΄λ³΄κ² μŠ΅λ‹ˆλ‹€. λ‹€μŒκ³Ό 같이 flannel의 μ„œλΉ„μŠ€ 계정에 ν΄λŸ¬μŠ€ν„° κ΄€λ¦¬μž 역할을 λ°”μΈλ”©ν•©λ‹ˆλ‹€. 이상적이지 μ•Šμ„ 수 있으며 ν”Œλž€λ„¬ μ„œλΉ„μŠ€ 계정에 λŒ€ν•œ νŠΉμ • 역할을 μ •μ˜ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml이 이미 μžˆμŠ΅λ‹ˆλ‹€.

도움을 μ£Όμ…”μ„œ κ°μ‚¬ν•©λ‹ˆλ‹€.
λ§ˆμ§€λ§‰μœΌλ‘œ ν”Œλž€λ„¬μ΄ μžˆλŠ” k8s 1.6.1이 μ™„μ „νžˆ μž‘λ™ν•©λ‹ˆλ‹€. 이제 λͺ¨λ“  것이 κ°€λŠ₯ν•œ ν”Œλ ˆμ΄λΆμ— μžˆμŠ΅λ‹ˆλ‹€.
Centos/RHELμ—μ„œ ν…ŒμŠ€νŠΈλ˜μ—ˆμŠ΅λ‹ˆλ‹€. Debian 기반(예: Ubuntu)에 λŒ€ν•œ 쀀비도 μ‹œμž‘λ˜μ—ˆμ§€λ§Œ μ•½κ°„μ˜ κ°œμ„ μ΄ ν•„μš”ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

https://github.com/ReSearchITEng/kubeadm-playbook/blob/master/README.md

μΆ”μ‹ : sjenning/kubeadm-playbook 기반 μž‘μ—… - @sjenning에 감사

ν΄λŸ¬μŠ€ν„°μ— κ°€μž…ν•˜κΈ° μœ„ν•΄ λ‹€μŒμ„ μ–»μŠ΅λ‹ˆλ‹€.
[discovery] " https://10.100.2.158 :6443"μ—μ„œ 정보λ₯Ό μš”μ²­ν•˜λŠ” ν΄λŸ¬μŠ€ν„° 정보 검색 ν΄λΌμ΄μ–ΈνŠΈ 생성
[검색] ν΄λŸ¬μŠ€ν„° 정보λ₯Ό μš”μ²­ν•˜μ§€ λͺ»ν–ˆμŠ΅λ‹ˆλ‹€. λ‹€μ‹œ μ‹œλ„ν•©λ‹ˆλ‹€. [configmaps "cluster-info"λŠ” κΈˆμ§€λ¨: μ‚¬μš©μž " system:anonymous "λŠ” "kube-public" λ„€μž„μŠ€νŽ˜μ΄μŠ€μ—μ„œ configmap을 κ°€μ Έμ˜¬ 수 μ—†μŠ΅λ‹ˆλ‹€.]
[검색] ν΄λŸ¬μŠ€ν„° 정보λ₯Ό μš”μ²­ν•˜μ§€ λͺ»ν–ˆμŠ΅λ‹ˆλ‹€. λ‹€μ‹œ μ‹œλ„ν•©λ‹ˆλ‹€. [configmaps "cluster-info"λŠ” κΈˆμ§€λ¨: μ‚¬μš©μž " system:anonymous "λŠ” "kube-public" λ„€μž„μŠ€νŽ˜μ΄μŠ€μ—μ„œ configmap을 κ°€μ Έμ˜¬ 수 μ—†μŠ΅λ‹ˆλ‹€.]

λ…Έλ“œλ₯Ό SelfHosting으둜 μ‹œμž‘ν–ˆμŠ΅λ‹ˆλ‹€.

이 νŽ˜μ΄μ§€κ°€ 도움이 λ˜μ—ˆλ‚˜μš”?
0 / 5 - 0 λ“±κΈ‰