Kubernetes: `ν•„μš”ν•œ μ–‘μ˜ 이미지 가비지 μˆ˜μ§‘μ— μ‹€νŒ¨ν–ˆμŠ΅λ‹ˆλ‹€. 473842483 λ°”μ΄νŠΈλ₯Ό ν™•λ³΄ν•˜κ³  μ‹Άμ—ˆμ§€λ§Œ 0 λ°”μ΄νŠΈλ₯Ό ν™•λ³΄ν–ˆμŠ΅λ‹ˆλ‹€.

에 λ§Œλ“  2018λ…„ 12μ›” 08일  Β·  30μ½”λ©˜νŠΈ  Β·  좜처: kubernetes/kubernetes

버그λ₯Όλ³΄κ³  ν•  λ•Œμ΄ ν…œν”Œλ¦Ώμ„ μ‚¬μš©ν•˜κ³  κ°€λŠ₯ν•œ ν•œ λ§Žμ€ 정보λ₯Ό μ œκ³΅ν•˜μ‹­μ‹œμ˜€. κ·Έλ ‡κ²Œν•˜μ§€ μ•ŠμœΌλ©΄ 버그가 μ μ‹œμ— ν•΄κ²°λ˜μ§€ μ•Šμ„ 수 μžˆμŠ΅λ‹ˆλ‹€. 감사!

무슨 일이 일어 λ‚¬λŠ”κ°€ : μ΅œκ·Όμ— λ””μŠ€ν¬ μ••λ ₯으둜 μΈν•œ μ—¬λŸ¬ 퇴거λ₯Ό λͺ©κ²©ν–ˆμŠ΅λ‹ˆλ‹€.

$$$ kubectl get pod kumo-go-api-d46f56779-jl6s2 --namespace=kumo-main -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: 2018-12-06T10:05:25Z
  generateName: kumo-go-api-d46f56779-
  labels:
    io.kompose.service: kumo-go-api
    pod-template-hash: "802912335"
  name: kumo-go-api-d46f56779-jl6s2
  namespace: kumo-main
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: kumo-go-api-d46f56779
    uid: c0a9355e-f780-11e8-b336-42010aa80057
  resourceVersion: "11617978"
  selfLink: /api/v1/namespaces/kumo-main/pods/kumo-go-api-d46f56779-jl6s2
  uid: 7337e854-f93e-11e8-b336-42010aa80057
spec:
  containers:
  - env:
    - redacted...
    image: gcr.io/<redacted>/kumo-go-api<strong i="8">@sha256</strong>:c6a94fc1ffeb09ea6d967f9ab14b9a26304fa4d71c5798acbfba5e98125b81da
    imagePullPolicy: Always
    name: kumo-go-api
    ports:
    - containerPort: 5000
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-t6jkx
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-t6jkx
    secret:
      defaultMode: 420
      secretName: default-token-t6jkx
status:
  message: 'The node was low on resource: nodefs.'
  phase: Failed
  reason: Evicted
  startTime: 2018-12-06T10:05:25Z

kubectl get events λ₯Ό μ‚΄νŽ΄λ³΄λ©΄ λ‹€μŒ κ²½κ³ κ°€ ν‘œμ‹œλ©λ‹ˆλ‹€.

$$$ kubectl get events
LAST SEEN   FIRST SEEN   COUNT     NAME                                                                   KIND      SUBOBJECT   TYPE      REASON          SOURCE                                                         MESSAGE
2m          13h          152       gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s.156e07f40b90ed91   Node                  Warning   ImageGCFailed   kubelet, gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s   (combined from similar events): failed to garbage collect required amount of images. Wanted to free 473948979 bytes, but freed 0 bytes
37m         37m          1         gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s.156e3127ebc715c3   Node                  Warning   ImageGCFailed   kubelet, gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s   failed to garbage collect required amount of images. Wanted to free 473674547 bytes, but freed 0 bytes

μ’€ 더 깊이 파고 λ“€κΈ° :

$$$ kubectl get event gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s.156e07f40b90ed91 -o yaml
apiVersion: v1
count: 153
eventTime: null
firstTimestamp: 2018-12-07T11:01:06Z
involvedObject:
  kind: Node
  name: gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s
  uid: gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s
kind: Event
lastTimestamp: 2018-12-08T00:16:09Z
message: '(combined from similar events): failed to garbage collect required amount
  of images. Wanted to free 474006323 bytes, but freed 0 bytes'
metadata:
  creationTimestamp: 2018-12-07T11:01:07Z
  name: gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s.156e07f40b90ed91
  namespace: default
  resourceVersion: "381976"
  selfLink: /api/v1/namespaces/default/events/gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s.156e07f40b90ed91
  uid: 65916e4b-fa0f-11e8-ae9a-42010aa80058
reason: ImageGCFailed
reportingComponent: ""
reportingInstance: ""
source:
  component: kubelet
  host: gke-kumo-customers-n1-standard-1-pree-0cd7990c-jg9s
type: Warning

μ‹€μ œλ‘œ μ—¬κΈ°μ—λŠ” 거의 μ—†μŠ΅λ‹ˆλ‹€. 이 λ©”μ‹œμ§€λŠ” ImageGCκ°€ μ‹œμž‘λœ 이유 λ˜λŠ” 더 λ§Žμ€ 곡간을 볡ꡬ ν•  μˆ˜μ—†λŠ” μ΄μœ μ— λŒ€ν•΄μ„œλŠ” μ–ΈκΈ‰ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.

μ˜ˆμƒλ˜λŠ” 상황 : Image GCκ°€ μ˜¬λ°”λ₯΄κ²Œ μž‘λ™ν•˜κ±°λ‚˜ μ΅œμ†Œν•œ λ””μŠ€ν¬ 곡간이 λΆ€μ‘±ν•œ λ…Έλ“œμ— ν¬λ“œλ₯Ό μ˜ˆμ•½ν•˜μ§€ λͺ»ν•©λ‹ˆλ‹€.

μž¬ν˜„ 방법 (κ°€λŠ₯ν•œ ν•œ μ΅œμ†Œν•œμœΌλ‘œ μ •ν™•ν•˜κ²Œ) : λ””μŠ€ν¬ μ••λ ₯을 높이기 μœ„ν•΄ λ…Έλ“œμ—μ„œ κ°€λŠ₯ν•œ ν•œ λ§Žμ€ ν¬λ“œλ₯Ό μ‹€ν–‰ν•˜κ³  μ€‘μ§€ν•©λ‹ˆλ‹€. 그런 λ‹€μŒ μ΄λŸ¬ν•œ 였λ₯˜λ₯Ό κ΄€μ°°ν•˜μ‹­μ‹œμ˜€.

μš°λ¦¬κ°€ μ•Œμ•„μ•Ό ν•  λ‹€λ₯Έ 것이 μžˆμŠ΅λ‹ˆκΉŒ? : ν•΄λ‹Ή 사항 μ—†μŒ

ν™˜κ²½ :

  • Kubernetes 버전 ( kubectl version ) :
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.7-gke.11", GitCommit:"fa90543563c9cfafca69128ce8cd9ecd5941940f", GitTreeState:"clean", BuildDate:"2018-11-08T20:22:21Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
  • ν΄λΌμš°λ“œ 제곡 업체 λ˜λŠ” ν•˜λ“œμ›¨μ–΄ ꡬ성 : GKE
  • OS (예 : / etc / os-release) : macOS 10.14λ₯Ό μ‹€ν–‰ 쀑이고 λ…Έλ“œλŠ” Container-Optimized OS (cos)λ₯Ό μ‹€ν–‰ μ€‘μž…λ‹ˆλ‹€.
  • 컀널 (예 : uname -a ) : Darwin D-10-19-169-80.dhcp4.washington.edu 18.0.0 Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64 x86_64
  • 도ꡬ μ„€μΉ˜ : ν•΄λ‹Ή 사항 μ—†μŒ
  • 기타 : ν•΄λ‹Ή 사항 μ—†μŒ


/ μ’…λ₯˜μ˜ 버그

areprovidegcp kinbug sinode
좜처
picture samuela
👍21

κ°€μž₯ μœ μš©ν•œ λŒ“κΈ€

같은 λ¬Έμ œμ— μ§λ©΄ν–ˆμŠ΅λ‹ˆλ‹€.

kubectl drain --delete-local-data --ignore-daemonsets $NODE_IP && kubectl uncordon $NODE_IP 은 (λŠ”) λ””μŠ€ν¬ μ €μž₯μ†Œλ₯Ό μ •λ¦¬ν•˜κΈ°μ— μΆ©λΆ„ν–ˆμŠ΅λ‹ˆλ‹€.

picture KIVagant 에 2020λ…„ 01μ›” 31일
👍5 😕4

λͺ¨λ“  30 λŒ“κΈ€

/ sig gcp

samuela picture samuela 에 2018λ…„ 12μ›” 08일

방금 λ§ˆμŠ€ν„° 버전과 λ…Έλ“œλ₯Ό 1.11.3-gke.18둜 μ—…κ·Έλ ˆμ΄λ“œν•˜μ—¬ λ„μ›€μ΄λ˜λŠ”μ§€ ν™•μΈν–ˆμ§€λ§Œ μ—¬μ „νžˆ λ˜‘κ°™μ€ λ‚΄μš©μ΄ ν‘œμ‹œλ©λ‹ˆλ‹€.

samuela picture samuela 에 2018λ…„ 12μ›” 09일

FWIW "λΆ€νŒ… λ””μŠ€ν¬ 크기 (GB (λ…Έλ“œ λ‹Ή))"κ°€ μ΅œμ†Œ 10Gb둜 μ„€μ •λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

samuela picture samuela 에 2018λ…„ 12μ›” 09일
👀1 🚀1 ❤1 🎉1 👍1

@samuela λ¬Έμ œμ— λŒ€ν•œ μ—…λ°μ΄νŠΈκ°€ μžˆμŠ΅λ‹ˆκΉŒ? λ‚˜λŠ” 같은 문제λ₯Ό λ³Έλ‹€.

hgokavarapuz picture hgokavarapuz 에 2018λ…„ 12μ›” 11일

@hgokavarapuz λ‚΄κ°€ λ“€μ—ˆλ˜ ν•œ μ—…λ°μ΄νŠΈκ°€ μ—†μŠ΅λ‹ˆλ‹€. DefλŠ” GKE의 μ‹¬κ°ν•œ 문제인 것 κ°™μŠ΅λ‹ˆλ‹€.

samuela picture samuela 에 2018λ…„ 12μ›” 12일

@samuela AWSμ—μ„œμ΄ 문제λ₯Ό λ³΄μ•˜μ§€λ§Œ λ‹€λ₯Έ AMIλ₯Ό μ‚¬μš©ν•˜μ—¬ ν•΄κ²°ν•  μˆ˜μžˆμ—ˆμŠ΅λ‹ˆλ‹€. AMI의 차이점은 무엇인지 ν™•μΈν•΄μ•Όν•©λ‹ˆλ‹€.

hgokavarapuz picture hgokavarapuz 에 2018λ…„ 12μ›” 12일

@hgokavarapuz ν₯λ―Έ

samuela picture samuela 에 2018λ…„ 12μ›” 12일

이 문제의 원인을 μ •ν™•νžˆ λ””λ²„κΉ…ν•΄μ•Όν•©λ‹ˆλ‹€.

2018 λ…„ 12 μ›” 12 일 μˆ˜μš”μΌ μ˜€ν›„ 1:23 samuela [email protected] 은 λ‹€μŒκ³Ό 같이 μΌμŠ΅λ‹ˆλ‹€.

@hgokavarapuz https://github.com/hgokavarapuz ν₯미둜운 ...
λ…Έλ“œ OS / μ„€μ •κ³Ό 관련이 μžˆμŠ΅λ‹ˆλ‹€.

β€”
당신이 μ–ΈκΈ‰ λ˜μ—ˆκΈ° λ•Œλ¬Έμ— 이것을 λ°›κ³  μžˆμŠ΅λ‹ˆλ‹€.
이 이메일에 직접 λ‹΅μž₯ν•˜κ³  GitHubμ—μ„œ ν™•μΈν•˜μ„Έμš”.
https://github.com/kubernetes/kubernetes/issues/71869#issuecomment-446748663 ,
λ˜λŠ” μŠ€λ ˆλ“œ μŒμ†Œκ±°
https://github.com/notifications/unsubscribe-auth/AmWWLdQjFnWgM5jeutfY6YqJBQ9l2l8gks5u4XO2gaJpZM4ZJWSq
.

-
κ°μ‚¬ν•©λ‹ˆλ‹€
ν—€λ§Œ 슀

hgokavarapuz picture hgokavarapuz 에 2018λ…„ 12μ›” 12일

@hgokavarapuz λŠ” kubelet λ‘œκ·Έμ—μ„œ λ‹¨μ„œλ₯Ό ν™•μΈν•©λ‹ˆλ‹€.

dims picture dims 에 2018λ…„ 12μ›” 26일

일뢀 μ œν•œλœ 크기둜 EBS λ³Όλ₯¨μ— / var 폴더가 λ§ˆμš΄νŠΈλ˜μ–΄μžˆλŠ” AMIμ—μ„œ Docker μ»¨ν…Œμ΄λ„ˆ 생성에 λ¬Έμ œκ°€ λ°œμƒν•˜λŠ” 문제λ₯Ό ν•΄κ²°ν•  μˆ˜μžˆμ—ˆμŠ΅λ‹ˆλ‹€. λ‘œκ·Έμ—μ„œ μ§μ ‘μ μœΌλ‘œλŠ” λͺ…ν™•ν•˜μ§€ μ•Šμ•˜μ§€λ§Œ 곡간 및 기타 사항을 ν™•μΈν•˜λ©΄ λͺ…ν™•ν–ˆμŠ΅λ‹ˆλ‹€.

hgokavarapuz picture hgokavarapuz 에 2018λ…„ 12μ›” 26일

@hgokavarapuz 이것이 μ‹€μ œλ‘œ 문제λ₯Ό ν•΄κ²°ν•˜κ³  버그가 λ°œμƒν•˜κΈ° μœ„ν•΄ 더 λ§Žμ€ 이미지 λ‹€μš΄λ‘œλ“œκ°€ ν•„μš”ν•˜μ§€ μ•Šλ‹€κ³  ν™•μ‹ ν•©λ‹ˆκΉŒ?

제 κ²½μš°μ—λŠ” 이것이 GKE ν—ˆμš© λ””μŠ€ν¬ 크기 λ‚΄μ—μ„œ λ°œμƒν–ˆκΈ° λ•Œλ¬Έμ— 적어도 μ—¬κΈ° GKEμ—λŠ” μ—¬μ „νžˆ μΌμ’…μ˜ 버그가 μžˆλ‹€κ³  λ§ν•˜κ³  μ‹ΆμŠ΅λ‹ˆλ‹€.

이 였λ₯˜κ°€ λ°œμƒν•˜μ§€ μ•Šκ³  λ…Έλ“œμ—μ„œ kubernetesλ₯Ό μ‹€ν–‰ν•˜κΈ° μœ„ν•΄ ν•„μš”ν•œ μ΅œμ†Œ λ””μŠ€ν¬ 크기에 λŒ€ν•΄ μΌμ’…μ˜ 곡식적인 μž…μž₯을 κ°–λŠ” 것도 μ’‹μŠ΅λ‹ˆλ‹€. 그렇지 μ•ŠμœΌλ©΄ kubernetesλ₯Ό μ‹€ν–‰ν•˜κΈ°μœ„ν•œ 사양 내에 있으렀면 λ³Όλ₯¨μ΄ μ–Όλ§ˆλ‚˜ μ»€μ•Όν•˜λŠ”μ§€ μ •ν™•νžˆ μ•Œ 수 μ—†μŠ΅λ‹ˆλ‹€.

samuela picture samuela 에 2018λ…„ 12μ›” 27일

@samuela λ‚˜λŠ” GKEμ—μ„œ μ‹œλ„ν•˜μ§€ μ•Šμ•˜μ§€λ§Œ 일뢀 AMI의 AWS λ¬Έμ œμ˜€μŠ΅λ‹ˆλ‹€. GKE에 λ¬Έμ œκ°€μžˆμ„ 수 μžˆμŠ΅λ‹ˆλ‹€.

hgokavarapuz picture hgokavarapuz 에 2018λ…„ 12μ›” 27일

GKE v1.11.5-gke.4μ—μ„œλ„ λΉ„μŠ·ν•œ λ¬Έμ œκ°€ λ°œμƒν•©λ‹ˆλ‹€. λ‹€μŒ μ΄λ²€νŠΈμ—μ„œ λ³Ό 수 μžˆλ“―μ΄ GCκ°€ 따라 가지 λͺ»ν•˜λŠ” λ¬Έμ œκ°€μžˆλŠ” 것 κ°™μŠ΅λ‹ˆλ‹€.

Events:
  Type     Reason                 Age                 From                                               Message
  ----     ------                 ----                ----                                               -------
  Warning  FreeDiskSpaceFailed    47m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 758374400 bytes, but freed 375372075 bytes
  Warning  FreeDiskSpaceFailed    42m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 898760704 bytes, but freed 0 bytes
  Warning  ImageGCFailed          42m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 898760704 bytes, but freed 0 bytes
  Normal   NodeHasDiskPressure    37m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  Node gke-v11-service-graph-pool-c6e93d11-k6h6 status is now: NodeHasDiskPressure
  Warning  FreeDiskSpaceFailed    37m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 1430749184 bytes, but freed 0 bytes
  Warning  ImageGCFailed          37m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 1430749184 bytes, but freed 0 bytes
  Warning  EvictionThresholdMet   36m (x21 over 37m)  kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  Attempting to reclaim ephemeral-storage
  Warning  ImageGCFailed          32m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 1109360640 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed    27m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 1367126016 bytes, but freed 0 bytes
  Warning  ImageGCFailed          22m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 1885589504 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed    17m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 2438008832 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed    12m                 kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 2223022080 bytes, but freed 0 bytes
  Warning  ImageGCFailed          7m                  kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  failed to garbage collect required amount of images. Wanted to free 2358378496 bytes, but freed 0 bytes
  Normal   NodeHasNoDiskPressure  2m (x4 over 4h)     kubelet, gke-v11-service-graph-pool-c6e93d11-k6h6  Node gke-v11-service-graph-pool-c6e93d11-k6h6 status is now: NodeHasNoDiskPressure

kubelet 둜그λ₯Ό μŠ€μΊ”ν•˜λ©΄ λ‹€μŒ ν•­λͺ©μ΄ ν‘œμ‹œλ©λ‹ˆλ‹€.

Feb 07 21:15:31 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: I0207 21:15:31.447179    1594 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 99% which is over the high threshold (85%). Trying to free 2358378496 byte
Feb 07 21:15:31 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: E0207 21:15:31.452366    1594 kubelet.go:1253] Image garbage collection failed multiple times in a row: failed to garbage collect required amount of images. Wanted to free 2358378496 b
Feb 07 21:15:31 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: I0207 21:15:31.711566    1594 kuberuntime_manager.go:513] Container {Name:metadata-agent Image:gcr.io/stackdriver-agents/stackdriver-metadata-agent:0.2-0.0.21-1 Command:[] Args:[-o Kub
Feb 07 21:15:32 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: I0207 21:15:32.004882    1594 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "gke-v11-service-graph-pool-c6e93d11-k6h6"
Feb 07 21:15:32 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: I0207 21:15:32.008529    1594 cloud_request_manager.go:108] Node addresses from cloud provider for node "gke-v11-service-graph-pool-c6e93d11-k6h6" collected
Feb 07 21:15:34 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: I0207 21:15:34.817530    1594 kube_docker_client.go:348] Stop pulling image "gcr.io/stackdriver-agents/stackdriver-logging-agent:0.8-1.6.2-1": "e807eb07af89: Extracting [==============
Feb 07 21:15:34 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: E0207 21:15:34.817616    1594 remote_image.go:108] PullImage "gcr.io/stackdriver-agents/stackdriver-logging-agent:0.8-1.6.2-1" from image service failed: rpc error: code = Unknown desc
Feb 07 21:15:34 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: E0207 21:15:34.817823    1594 kuberuntime_manager.go:733] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to register layer: Error processing tar file(exi
Feb 07 21:15:35 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: W0207 21:15:35.057924    1594 kubelet_getters.go:264] Path "/var/lib/kubelet/pods/652e958e-2b1d-11e9-827c-42010a800fdc/volumes" does not exist
Feb 07 21:15:35 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: I0207 21:15:35.058035    1594 eviction_manager.go:400] eviction manager: pods fluentd-gcp-v3.1.1-spdfd_kube-system(652e958e-2b1d-11e9-827c-42010a800fdc) successfully cleaned up
Feb 07 21:15:35 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: E0207 21:15:35.091740    1594 pod_workers.go:186] Error syncing pod 7e06145a-2b1d-11e9-827c-42010a800fdc ("fluentd-gcp-v3.1.1-bgdg6_kube-system(7e06145a-2b1d-11e9-827c-42010a800fdc)"),
Feb 07 21:15:35 gke-v11-service-graph-pool-c6e93d11-k6h6 kubelet[1594]: W0207 21:15:35.179545    1594 eviction_manager.go:329] eviction manager: attempting to reclaim ephemeral-storage

μŠ€ν† λ¦¬μ§€λ₯Ό μΆ©λΆ„νžˆ λΉ λ₯΄κ²Œ νšŒμˆ˜ν•˜κΈ° μœ„ν•΄ λ­”κ°€ GCλ₯Ό λ³΄μœ ν•˜κ³ μžˆλŠ” 것 κ°™μŠ΅λ‹ˆλ‹€. λ…Έλ“œλŠ” κ²°κ΅­ λ³΅κ΅¬λ˜λŠ” κ²ƒμ²˜λŸΌ λ³΄μ΄μ§€λ§Œ ν”„λ‘œμ„ΈμŠ€μ—μ„œ 일뢀 ν¬λ“œκ°€ μ œκ±°λ©λ‹ˆλ‹€.

kyessenov picture kyessenov 에 2019λ…„ 02μ›” 07일

λ™μΌν•œ λ¬Έμ œκ°€ λ°œμƒν•©λ‹ˆλ‹€. AWS에 kopsλ₯Ό μ‚¬μš©ν•˜μ—¬ μŠ€νƒμ„ λ°°ν¬ν–ˆμœΌλ©° k8s 버전은 1.11.6μž…λ‹ˆλ‹€. λ¬Έμ œλŠ” λ””μŠ€ν¬ μ••λ ₯이 λ°œμƒν–ˆμ„ λ•Œ μ£Όλ‹Ή μ• ν”Œλ¦¬μΌ€μ΄μ…˜ λ‹€μš΄ νƒ€μž„μ΄ λ°œμƒν•œλ‹€λŠ” κ²ƒμž…λ‹ˆλ‹€.

angegar picture angegar 에 2019λ…„ 02μ›” 22일

여기에 같은 문제. λ‚˜λŠ” 그것을 κ³ μΉ  것이라고 μƒκ°ν•˜λ©΄μ„œ ebs λ³Όλ₯¨μ„ ν™•μž₯ν–ˆμŠ΅λ‹ˆλ‹€.
μ‚¬μš©
ami k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17 (ami-009b9699070ffc46f)

deathrizzo picture deathrizzo 에 2019λ…„ 03μ›” 04일
👍1

λ‚˜λŠ” λΉ„μŠ·ν•œ λ¬Έμ œμ— μ§λ©΄ν–ˆμ§€λ§Œ AKSμ—μ„œ. az cli 둜 ν΄λŸ¬μŠ€ν„°λ₯Ό μΆ•μ†Œ ν•œ λ‹€μŒ ν™•μž₯ν•˜λ©΄ μƒˆ λ…Έλ“œκ°€ κΉ¨λ—ν•˜λ‹€κ³  ​​생각할 수 μžˆμŠ΅λ‹ˆλ‹€.

$ kubectl get no
NAME                       STATUS   ROLES   AGE   VERSION
aks-agentpool-11344223-0   Ready    agent   77d   v1.12.4
aks-agentpool-11344223-1   Ready    agent   9h    v1.12.4
aks-agentpool-11344223-2   Ready    agent   9h    v1.12.4
aks-agentpool-11344223-3   Ready    agent   9h    v1.12.4
aks-agentpool-11344223-4   Ready    agent   9h    v1.12.4
aks-agentpool-11344223-5   Ready    agent   9h    v1.12.4

그쀑 ν•˜λ‚˜μ— sshν•˜λ©΄ λ‹€μŒκ³Ό 같은 였래된 이미지λ₯Ό 많이 λ³Ό 수 μžˆμŠ΅λ‹ˆλ‹€.

$ docker images | grep addon-resizer
k8s.gcr.io/addon-resizer                               1.8.4               5ec630648120        6 months ago        38.3MB
k8s.gcr.io/addon-resizer                               1.8.1               6c0dbeaa8d20        17 months ago       33MB
k8s.gcr.io/addon-resizer                               1.7                 9b0815c87118        2 years ago         39MB

λ˜λŠ”

$ docker images | grep k8s.gcr.io/cluster-autoscaler
k8s.gcr.io/cluster-autoscaler                          v1.14.0             ef6c40006faf        7 weeks ago         142MB
k8s.gcr.io/cluster-autoscaler                          v1.13.2             0f47d27d8e0d        2 months ago        137MB
k8s.gcr.io/cluster-autoscaler                          v1.12.3             9119261ec106        2 months ago        232MB
k8s.gcr.io/cluster-autoscaler                          v1.3.7              c711df426ac6        2 months ago        217MB
k8s.gcr.io/cluster-autoscaler                          v1.12.2             d67faca6c0aa        3 months ago        232MB
k8s.gcr.io/cluster-autoscaler                          v1.13.1             39c073d73c1e        5 months ago        137MB
k8s.gcr.io/cluster-autoscaler                          v1.3.4              6168be341178        6 months ago        217MB
k8s.gcr.io/cluster-autoscaler                          v1.3.3              bd9362bb17a5        7 months ago        217MB
k8s.gcr.io/cluster-autoscaler                          v1.2.2              2378f4474aa3        11 months ago       209MB
k8s.gcr.io/cluster-autoscaler                          v1.1.2              e137f4b4d451        14 months ago       198MB

μ•„λž˜μ— λ§Žμ€ 였λ₯˜κ°€ ν‘œμ‹œλ˜λ―€λ‘œ 미친 μ§“μž…λ‹ˆλ‹€.

  Type     Reason               Age    From                               Message
  ----     ------               ----   ----                               -------
  Warning  FreeDiskSpaceFailed  15m    kubelet, aks-agentpool-11344223-5  failed to garbage collect required amount of images. Wanted to free 1297139302 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed  10m    kubelet, aks-agentpool-11344223-5  failed to garbage collect required amount of images. Wanted to free 1447237222 bytes, but freed 0 bytes
  Warning  ImageGCFailed        10m    kubelet, aks-agentpool-11344223-5  failed to garbage collect required amount of images. Wanted to free 1447237222 bytes, but freed 0 bytes
widgetpl picture widgetpl 에 2019λ…„ 05μ›” 08일

@samuela :이 λ¬Έμ œμ—λŠ” μ‹œκ·Έ λ ˆμ΄λΈ”μ΄ μ—†μŠ΅λ‹ˆλ‹€. λ‹€μŒ 쀑 ν•˜λ‚˜λ₯Ό μ‚¬μš©ν•˜μ—¬ μ„œλͺ… λ ˆμ΄λΈ”μ„ μΆ”κ°€ν•˜μ‹­μ‹œμ˜€.

  1. μ„œλͺ… μ–ΈκΈ‰ : @kubernetes/sig-<group-name>-<group-suffix>
    예 : @kubernetes/sig-contributor-experience-<group-suffix> κΈ°μ—¬μž κ²½ν—˜ μ‹œκ·Έμ— μ•Œλ¦¬κΈ°, λ˜λŠ”

  2. μˆ˜λ™μœΌλ‘œ λ ˆμ΄λΈ” 지정 : /sig <group-name>
    예 : /sig scalability sig/scalability λ ˆμ΄λΈ” 적용

μ°Έκ³  : 방법 1은 그룹에 λŒ€ν•œ 이메일을 νŠΈλ¦¬κ±°ν•©λ‹ˆλ‹€. κ·Έλ£Ή λͺ©λ‘μ„ μ°Έμ‘°ν•˜μ‹­μ‹œμ˜€.
방법 1의 <group-suffix> λŠ” _ 버그, κΈ°λŠ₯ μš”μ²­, 홍보 κ²€ν† , ν…ŒμŠ€νŠΈ μ‹€νŒ¨, μ œμ•ˆ _ 쀑 ν•˜λ‚˜λ‘œ λŒ€μ²΄λ˜μ–΄μ•Ό

PR λŒ“κΈ€μ„ μ‚¬μš©ν•˜μ—¬ λ‚˜μ™€ μƒν˜Έ μž‘μš©ν•˜λŠ” 방법은 μ—¬κΈ°μ—μ„œ 확인할 수 kubernetes / test-infra μ €μž₯μ†Œμ— 문제λ₯Ό μ œμΆœν•˜μ„Έμš”.

k8s-ci-robot picture k8s-ci-robot 에 2019λ…„ 08μ›” 06일

λ‚˜λŠ” v1.11.10을 μ‚¬μš©ν•˜μ—¬ Openstackμ—μ„œ 이것을 치고 μžˆμŠ΅λ‹ˆλ‹€.

λ…Έλ“œμ— λ””μŠ€ν¬ 곡간이 μ™„μ „νžˆ λΆ€μ‘±ν•˜κ³  kubelet λ‘œκ·ΈλŠ” 이제 λ‹€μŒμ˜ λ£¨ν”„μž…λ‹ˆλ‹€.

E1029 06:41:37.397348    8907 remote_runtime.go:278] ContainerStatus "redacted" from runtime service failed: rpc error: code = Unknown desc = unable to inspect docker image "sha256:redacted" while inspecting docker container "redacted": no such image: "sha256:redacted"
Oct 29 06:41:37 node-name bash[8907]: E1029 06:41:37.397378    8907 kuberuntime_container.go:391] ContainerStatus for redacted error: rpc error: code = Unknown desc = unable to inspect docker image "sha256:redacted" while inspecting docker container "redacted": no such image: "sha256:redacted"
Oct 29 06:41:37 node-name bash[8907]: E1029 06:41:37.397388    8907 kuberuntime_manager.go:873] getPodContainerStatuses for pod "coredns-49t6c_kube-system(redacted)" failed: rpc error: code = Unknown desc = unable to inspect docker image "sha256:redacted" while inspecting docker container "redacted": no such image: "sha256:redacted"
Oct 29 06:41:37 node-name bash[8907]: E1029 06:41:37.397404    8907 generic.go:241] PLEG: Ignoring events for pod coredns-49t6c/kube-system: rpc error: code = Unknown desc = unable to inspect docker image "sha256:redacted" while inspecting docker container "redacted": no such image: "sha256:redacted"
rubencabrera picture rubencabrera 에 2019λ…„ 10μ›” 29일

μ €μ—κ²Œ λ¬Έμ œλŠ” 짧은 μ‹œκ°„μ— λ§Žμ€ λ””μŠ€ν¬ 곡간을 μ°¨μ§€ν•˜λŠ” μ»¨ν…Œμ΄λ„ˆλ‘œ 인해 λ°œμƒν–ˆμŠ΅λ‹ˆλ‹€. 이것은 μ—¬λŸ¬ λ…Έλ“œμ—μ„œ λ°œμƒν–ˆμŠ΅λ‹ˆλ‹€. μ»¨ν…Œμ΄λ„ˆκ°€ μ œκ±°λ˜μ—ˆμ§€λ§Œ (λ…Έλ“œμ˜ λͺ¨λ“  ν¬λ“œκ°€ 제거됨) kubelet이 λ””μŠ€ν¬λ₯Ό νšŒμˆ˜ν•˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.

이 μž‘μ—…μ„ μˆ˜ν–‰ν•˜λŠ” μ»¨ν…Œμ΄λ„ˆλ₯Ό μ°Ύκ³  μˆ˜λ™μœΌλ‘œ μ‚­μ œν•˜κΈ° μœ„ν•΄ du /var/lib/docker/overlay -h | sort -h ν•΄μ•Όν–ˆμŠ΅λ‹ˆλ‹€. 이것은 Disk Pressure μ—μ„œ λ…Έλ“œλ₯Ό κ°€μ Έ μ™”κ³  λ³΅κ΅¬λ˜μ—ˆμŠ΅λ‹ˆλ‹€ (κ·Έ 쀑 ν•˜λ‚˜λŠ” reboot -f ).

rubencabrera picture rubencabrera 에 2019λ…„ 10μ›” 30일
👍3 👀1

이것은 λ‚˜μ—κ²Œλ„ μΌμ–΄λ‚˜κ³  μžˆμŠ΅λ‹ˆλ‹€. EKS ν΄λŸ¬μŠ€ν„°μ— 8 개의 λ…Έλ“œκ°€ 있으며 μ–΄λ–€ 이유둜 ν•œ λ…Έλ“œμ—λ§Œμ΄ GC λ¬Έμ œκ°€ μžˆμŠ΅λ‹ˆλ‹€. 이것은 두 번 λ°œμƒν–ˆμœΌλ©° μ•„λž˜ λ‹¨κ³„λŠ” 문제λ₯Ό ν•΄κ²°ν•˜κΈ° μœ„ν•΄ μˆ˜ν–‰ ν•œ μž‘μ—…μž…λ‹ˆλ‹€. λˆ„κ΅¬λ“ μ§€ 이것을 μˆ˜ν–‰ν•˜λŠ” 더 λ‚˜μ€ / μ§€μ›λ˜λŠ” 방법을 μ•Œκ³  μžˆμŠ΅λ‹ˆκΉŒ? https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#maintenance -on-a-node

  1. EKS에 λŒ€ν•œ μžλ™ ν™•μž₯ 그룹을 +1 증가 (λ‚˜μœ λ…Έλ“œμ— λŒ€ν•œ ꡐ체 λ…Έλ“œ)
  2. λΆˆλŸ‰ λ…Έλ“œ 차단 (kubectl cordon)
  3. λΆˆλŸ‰ λ…Έλ“œ (kubectl drain)λ₯Ό λ°°μΆœν•˜μ—¬μ΄ λ…Έλ“œμ—μ„œ λ‹€λ₯Έ λ…Έλ“œ 쀑 ν•˜λ‚˜λ‘œ ν¬λ“œλ₯Ό ν‚₯ν•©λ‹ˆλ‹€.
  4. λΆˆλŸ‰ λ…Έλ“œλ₯Ό μ œμ™Έν•œ λͺ¨λ“  λ…Έλ“œ μΆ•μ†Œ 보호 μΆ”κ°€
  5. EKS에 λŒ€ν•œ μžλ™ 크기 μ‘°μ • 그룹을 -1둜 μ€„μž…λ‹ˆλ‹€ (λ³΄ν˜Έλ˜μ§€ μ•Šμ€ μœ μΌν•œ λ…Έλ“œμ΄λ―€λ‘œ λΆˆλŸ‰ λ…Έλ“œκ°€ μ‚­μ œλ¨).
  6. λͺ¨λ“  λ…Έλ“œμ—μ„œ μΆ•μ†Œ 보호 제거
dat-timfa picture dat-timfa 에 2020λ…„ 01μ›” 22일

같은 λ¬Έμ œμ— μ§λ©΄ν–ˆμŠ΅λ‹ˆλ‹€.

kubectl drain --delete-local-data --ignore-daemonsets $NODE_IP && kubectl uncordon $NODE_IP 은 (λŠ”) λ””μŠ€ν¬ μ €μž₯μ†Œλ₯Ό μ •λ¦¬ν•˜κΈ°μ— μΆ©λΆ„ν–ˆμŠ΅λ‹ˆλ‹€.

KIVagant picture KIVagant 에 2020λ…„ 01μ›” 31일
👍5 😕4

FWIW "λΆ€νŒ… λ””μŠ€ν¬ 크기 (GB (λ…Έλ“œ λ‹Ή))"κ°€ μ΅œμ†Œ 10Gb둜 μ„€μ •λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

λŒ€λ‹¨νžˆ κ°μ‚¬ν•©λ‹ˆλ‹€. 그것은 λ‚˜μ™€ ν•¨κ»˜ μΌν–ˆμŠ΅λ‹ˆλ‹€

HayTran94 picture HayTran94 에 2020λ…„ 03μ›” 03일

/ μ‹œκ·Έ λ…Έλ“œ

dims picture dims 에 2020λ…„ 04μ›” 12일

@ HayTran94 @samuela @KIVagant @dattim
realImageGCManager # freeSpaceλŠ” νŠΉμ • 이미지가 GC에 μ ν•©ν•˜μ§€ μ•Šμ€ 경우 레벨 5의 둜그λ₯Ό κ°–μŠ΅λ‹ˆλ‹€.
예

        if image.lastUsed.Equal(freeTime) || image.lastUsed.After(freeTime) {
            klog.V(5).Infof("Image ID %s has lastUsed=%v which is >= freeTime=%v, not eligible for garbage collection", image.id, image.lastUsed, freeTime)
            continue

둜그 μˆ˜μ€€μ„ 5둜 μ„€μ •ν•˜κ³  realImageGCManager # freeSpaceμ—μ„œ μ œκ³΅ν•˜λŠ” λ‹¨μ„œκ°€ μžˆλŠ”μ§€ 확인할 수 μžˆμŠ΅λ‹ˆκΉŒ?

감사

tedyu picture tedyu 에 2020λ…„ 04μ›” 13일

λΏ‘ 빡뀨
κ²Œμ‹œ ν•œ λ‘œκ·Έμ—μ„œ :

no such image: "sha256:redacted"

κΈ°λ³Έ 이미지가 μžˆλŠ”μ§€ μ—¬λΆ€λ₯Ό 확인할 κΈ°νšŒκ°€ μžˆμ—ˆμŠ΅λ‹ˆκΉŒ?

감사

tedyu picture tedyu 에 2020λ…„ 04μ›” 13일

이 λ£¨ν”„μ—μ„œ λ‚˜λ₯Ό μ§€μΌœμ£Όμ„Έμš”.
이 이메일에 볡사 된 이유λ₯Ό 잘 λͺ¨λ₯΄κ² μŠ΅λ‹ˆλ‹€.

κ°μ‚¬ν•©λ‹ˆλ‹€.
Ashutosh Singh

2020 λ…„ 4 μ›” 13 일 μ›”μš”μΌ 00:21 Zhihong Yu [email protected] μž‘μ„± :

@rubencabrera https://github.com/rubencabrera
κ²Œμ‹œ ν•œ λ‘œκ·Έμ—μ„œ :

ν•΄λ‹Ή 이미지 μ—†μŒ : "sha256 : redacted"

κΈ°λ³Έ 이미지가 μ‘΄μž¬ν•˜λŠ”μ§€ μ—¬λΆ€λ₯Ό 확인할 κΈ°νšŒκ°€ μžˆμ—ˆμŠ΅λ‹ˆκΉŒ?
μ•„λ‹ˆ?

감사

β€”
이 μŠ€λ ˆλ“œλ₯Ό κ΅¬λ…ν–ˆκΈ° λ•Œλ¬Έμ—μ΄ λ©”μ‹œμ§€κ°€ μ „μ†‘λ˜μ—ˆμŠ΅λ‹ˆλ‹€.
이 이메일에 직접 λ‹΅μž₯ν•˜κ³  GitHubμ—μ„œ ν™•μΈν•˜μ„Έμš”.
https://github.com/kubernetes/kubernetes/issues/71869#issuecomment-612684868 ,
λ˜λŠ” ꡬ독 μ·¨μ†Œ
https://github.com/notifications/unsubscribe-auth/ADS6CKHTR2QTDJOWNKMLX23RMI5FXANCNFSM4GJFMSVA
.

ashu2207 picture ashu2207 에 2020λ…„ 04μ›” 13일

λΏ‘ 빡뀨
κ²Œμ‹œ ν•œ λ‘œκ·Έμ—μ„œ :

no such image: "sha256:redacted"

κΈ°λ³Έ 이미지가 μžˆλŠ”μ§€ μ—¬λΆ€λ₯Ό 확인할 κΈ°νšŒκ°€ μžˆμ—ˆμŠ΅λ‹ˆκΉŒ?

감사

μ•ˆλ…•ν•˜μ„Έμš”, @tedyu

예, 일뢀 개인 μ €μž₯μ†Œλ₯Ό μ‚¬μš©ν•˜κ³  있으며 μ‚¬μš©ν•  μˆ˜μ—†λŠ” 이미지가 λΉˆλ²ˆν•œ λ¬Έμ œλΌλŠ” 것을 ν™•μΈν–ˆκΈ° λ•Œλ¬Έμ— κ·Έ 였λ₯˜λ₯Ό λ³Ό λ•Œ 처음 μƒκ°ν–ˆμŠ΅λ‹ˆλ‹€. 이미지가 μ‚¬μš© κ°€λŠ₯ν•˜κ³  λ™μΌν•œ ν΄λŸ¬μŠ€ν„°μ˜ λ‹€λ₯Έ λ…Έλ“œμ—μ„œ μ‹€ν–‰ μ€‘μž…λ‹ˆλ‹€.

rubencabrera picture rubencabrera 에 2020λ…„ 04μ›” 20일

루트 파일 μ‹œμŠ€ν…œμ΄ μ•„λ‹Œ λ””μŠ€ν¬μ—μ„œ k8s 가비지 μˆ˜μ§‘μ„ μ‹€ν–‰ν•˜λ„λ‘ μ„€λ“ν•˜λŠ” 방법을 μ•Œμ•„ λ‚Έ μ‚¬λžŒμ΄ μžˆμŠ΅λ‹ˆκΉŒ? EKS μ„±λŠ₯ 문제λ₯Ό ν•΄κ²°ν•˜λ €λ©΄ / var / lib / docker에 보쑰 (SSD) λ””μŠ€ν¬λ₯Ό μ‚¬μš©ν•΄μ•Όν•©λ‹ˆλ‹€ (https://github.com/awslabs/amazon-eks-ami/issues/454 μ°Έμ‘°). κ·ΈλŸ¬λ‚˜ 가비지 μˆ˜μ§‘μ€ μ‹€ν–‰λ˜μ§€ μ•ŠμœΌλ©° λ•Œλ•Œλ‘œ 보쑰 λ””μŠ€ν¬λ₯Ό μ˜€λ²„ν”Œλ‘œν•©λ‹ˆλ‹€.

jae-63 picture jae-63 에 2020λ…„ 07μ›” 13일

90 일 λ™μ•ˆ ν™œλ™μ΄ μ—†μœΌλ©΄ λ¬Έμ œκ°€ λΆ€μ‹€ν•΄μ§‘λ‹ˆλ‹€.
/remove-lifecycle stale ν•˜μ—¬ μ΅œμ‹  문제둜 ν‘œμ‹œν•©λ‹ˆλ‹€.
λΉ„ν™œμ„± λ¬Έμ œλŠ” μΆ”κ°€λ‘œ 30 일 λ™μ•ˆ μ‚¬μš©ν•˜μ§€ μ•ŠμœΌλ©΄ 썩고 κ²°κ΅­ μ’…λ£Œλ©λ‹ˆλ‹€.

이 문제λ₯Ό μ§€κΈˆ μ’…λ£Œν•΄λ„ μ•ˆμ „ν•˜λ‹€λ©΄ /close ν•˜μ—¬ μ’…λ£Œν•˜μ„Έμš”.

sig-testing, kubernetes / test-infra 및 / λ˜λŠ” fejta에 ν”Όλ“œλ°±μ„
/ lifecycle stale

fejta-bot picture fejta-bot 에 2020λ…„ 10μ›” 11일

/ remove-lifecycle stale

μš°λ¦¬λŠ” μ§€λ‚œμ£Όμ—μ΄ 문제λ₯Ό κ²ͺκΈ° μ‹œμž‘ν–ˆμŠ΅λ‹ˆλ‹€. K8s-1.17-debian-stretch-amd64-hvm-ebs-2020-01-17 AMI, docker 19.03.11을 μ‚¬μš©ν•˜μ—¬ AWSμ—μ„œ 자체 ν˜ΈμŠ€νŒ…λ˜λŠ” Kops 1.17.1둜 λΉŒλ“œ 된 Kubernetes 1.17.9.

μ΄λŠ” μ§€λ‚œμ£Όμ— 두 개의 κ°œλ³„ λ…Έλ“œμ—μ„œ λ°œμƒν–ˆμœΌλ©° 두 λ…Έλ“œ λͺ¨λ‘ λ‹€μŒκ³Ό 같이 ν‘œμ‹œλ˜μ—ˆμŠ΅λ‹ˆλ‹€.

Events:
  Type     Reason               Age                  From                                                Message
  ----     ------               ----                 ----                                                -------
  Warning  FreeDiskSpaceFailed  10m (x204 over 17h)  kubelet, ip-10-224-54-0.us-west-2.compute.internal  (combined from similar events): failed to garbage collect required amount of images. Wanted to free 5877565849 bytes, but freed 101485977 bytes
  Warning  ImageGCFailed        18s (x205 over 17h)  kubelet, ip-10-224-54-0.us-west-2.compute.internal  (combined from similar events): failed to garbage collect required amount of images. Wanted to free 5886654873 bytes, but freed 0 bytes

λ…Έλ“œμ˜ du 및 df κ°€ μ‚¬μš©λ˜λŠ” 곡간에 λ™μ˜ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.

admin@ip-10-224-54-0:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2   57G   48G  5.8G  90% /

admin@ip-10-224-54-0:~$ sudo du -sh /
du: cannot access '/proc/9856/task/9856/fd/3': No such file or directory
du: cannot access '/proc/9856/task/9856/fdinfo/3': No such file or directory
du: cannot access '/proc/9856/fd/4': No such file or directory
du: cannot access '/proc/9856/fdinfo/4': No such file or directory
11G     /

admin@ip-10-224-54-0:~$ sudo du -sh --one-file-system /
6.6G    /

마운트 된 λ‹€λ₯Έ 파일 μ‹œμŠ€ν…œμ„ μ œκ±°ν•˜κΈ° μœ„ν•΄ 루트 μž₯치λ₯Ό λ‹€λ₯Έ 마운트 지점에 λ§ˆμš΄νŠΈν•˜λ©΄ du 이 (κ°€) μ‚¬μš© 된 곡간에 μΌκ΄€λ˜κ²Œ λ™μ˜ν•˜μ§€λ§Œ df μ—¬μ „νžˆ λ™μ˜ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.

admin@ip-10-224-54-0:~$ mkdir tmproot
admin@ip-10-224-54-0:~$ sudo mount /dev/nvme0n1p2 /home/admin/tmproot
admin@ip-10-224-54-0:~$ df -h tmproot/
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2   57G   48G  5.8G  90% /home/admin/tmproot
admin@ip-10-224-54-0:~$ sudo du -sh tmproot/
6.6G    tmproot/
admin@ip-10-224-54-0:~$ sudo du -sh --one-file-system tmproot/
6.6G    tmproot/

λ‚˜λŠ” 이것이 μ—΄λ¦° μ‚­μ œ νŒŒμΌμ„ λ³΄μœ ν•˜λŠ” ν”„λ‘œμ„ΈμŠ€ λ•Œλ¬ΈμΌ 수 μžˆλ‹€κ³  μƒκ°ν•©λ‹ˆλ‹€. κ·ΈλŸ¬λ‚˜ kubelet을 λ‹€μ‹œ μ‹œμž‘ν•΄λ„μ΄ 곡간이 λΉ„μ›Œμ§€μ§€λŠ” μ•ŠμœΌλ©° 이것이 원인이 될 κ²ƒμœΌλ‘œ μ˜μ‹¬λ˜λŠ” ν”„λ‘œμ„ΈμŠ€μž…λ‹ˆλ‹€. Dockerλ₯Ό λ‹€μ‹œ μ‹œμž‘ν•΄λ„ 곡간이 ν™•λ³΄λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.

처음 이런 일이 λ°œμƒν–ˆμ„ λ•Œ λͺ‡ μ‹œκ°„μ˜ κ²°μ‹€μ—†λŠ” 쑰사 끝에 λ…Έλ“œλ₯Ό μ’…λ£Œν–ˆμ§€λ§Œ 이제 λ‹€μ‹œ λ°œμƒν•˜λ―€λ‘œ λ¬Έμ œμ— λŒ€ν•œ 영ꡬ적 인 해결책을 λ§Œλ“€ 수 μ—†μŠ΅λ‹ˆλ‹€.

ν₯미둜운 데이터 포인트 : containerdκ°€ μ‚­μ œ ν•œ νŒŒμΌμ„ μ—΄μ—ˆμŠ΅λ‹ˆλ‹€.

admin@ip-10-224-54-0:~$ sudo lsof 2>&1| grep -v "no pwd entry" |  grep deleted
container 12469           root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469           root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469           root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469           root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469           root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12470     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12470     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12470     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12470     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12470     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12471     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12471     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12471     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12471     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12471     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12472     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12472     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12472     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12472     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12472     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12473     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12473     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12473     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12473     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12473     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12474     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12474     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12474     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12474     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12474     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12475     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12475     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12475     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12475     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12475     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12476     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12476     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12476     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12476     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12476     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12477     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 12477     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12477     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 12477     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 12477     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 19325     root  cwd       DIR               0,19        40 1180407868 /run/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12 (deleted)
container 12469 19325     root    4u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 19325     root    6u     FIFO              259,2       0t0    2097336 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stdout.log (deleted)
container 12469 19325     root    7u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)
container 12469 19325     root    8u     FIFO              259,2       0t0    2097337 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/34089ad41629df20f181ed191acec724c79fc879dc49287d29184f2fedfaba12/shim.stderr.log (deleted)

containerd.serviceλ₯Ό λ‹€μ‹œ μ‹œμž‘ν•΄λ„ 곡간이 ν™•λ³΄λ˜κ±°λ‚˜ μ΄λŸ¬ν•œ 파일 핸듀이 μ œκ±°λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€.

geekofalltrades picture geekofalltrades 에 2020λ…„ 10μ›” 13일
이 νŽ˜μ΄μ§€κ°€ 도움이 λ˜μ—ˆλ‚˜μš”?
0 / 5 - 0 λ“±κΈ‰

κ΄€λ ¨ 문제

theothermike picture theothermike  Β·  3μ½”λ©˜νŠΈ
sanjana-bhat picture sanjana-bhat  Β·  3μ½”λ©˜νŠΈ
ttripp picture ttripp  Β·  3μ½”λ©˜νŠΈ
alexferl picture alexferl  Β·  3μ½”λ©˜νŠΈ
zetaab picture zetaab  Β·  3μ½”λ©˜νŠΈ