Helm: "prometheus-operator"๋ฆด๋ฆฌ์Šค ์‹คํŒจ : rpc ์˜ค๋ฅ˜ : ์ฝ”๋“œ = ์ทจ์†Œ๋จ

์— ๋งŒ๋“  2019๋…„ 07์›” 31์ผ  ยท  71์ฝ”๋ฉ˜ํŠธ  ยท  ์ถœ์ฒ˜: helm/helm

๋ฒ„๊ทธ ์„ค๋ช…
helm install stable/prometheus-operator --name prometheus-operator -f prometheus-operator-values.yaml AKS์— prometheus ์—ฐ์‚ฐ์ž๋ฅผ ์„ค์น˜ํ•˜๋ ค๊ณ ํ•˜๋ฉด์ด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

prometheus-operator "์‹คํŒจ : rpc ์˜ค๋ฅ˜ : ์ฝ”๋“œ = ์ทจ์†Œ๋จ

๋‚˜๋Š” ์—ญ์‚ฌ๋ฅผ ํ™•์ธํ–ˆ๋‹ค.

helm history prometheus-operator -o yaml
- chart: prometheus-operator-6.3.0
  description: 'Release "prometheus-operator" failed: rpc error: code = Canceled desc
    = grpc: the client connection is closing'
  revision: 1
  status: FAILED
  updated: Tue Jul 30 12:36:52 2019

์ฐจํŠธ
[์•ˆ์ • / ํ”„๋กœ ๋ฉ”ํ…Œ์šฐ์Šค ์—ฐ์‚ฐ์ž]

์ถ”๊ฐ€ ์ •๋ณด
์•„๋ž˜ ๊ตฌ์„ฑ์„ ์‚ฌ์šฉํ•˜์—ฌ ์ฐจํŠธ๋ฅผ ๋ฐฐํฌํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
 kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml

๊ฐ’ ํŒŒ์ผ์—์„œ : createCustomResource ๊ฐ€ false .

helm version :
ํด๋ผ์ด์–ธํŠธ : & version.Version {SemVer : "v2.14.3", GitCommit : "0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState : "clean"}
์„œ๋ฒ„ : & version.Version {SemVer : "v2.14.3", GitCommit : "0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState : "clean"}

kubectl version ์ถœ๋ ฅ :
ํด๋ผ์ด์–ธํŠธ ๋ฒ„์ „ : version.Info {Major : "1", Minor : "10", GitVersion : "v1.10.4", GitCommit : "5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState : "clean", BuildDate : "2018-06-06T08 : 13 : 03Z ", GoVersion :"go1.9.3 ", ์ปดํŒŒ์ผ๋Ÿฌ :"gc ", ํ”Œ๋žซํผ :"windows / amd64 "}
์„œ๋ฒ„ ๋ฒ„์ „ : version.Info {Major : "1", Minor : "13", GitVersion : "v1.13.7", GitCommit : "4683545293d792934a7a7e12f2cc47d20b2dd01b", GitTreeState : "clean", BuildDate : "2019-06-06T01 : 39 : 30Z ", GoVersion :"go1.11.5 ", ์ปดํŒŒ์ผ๋Ÿฌ :"gc ", ํ”Œ๋žซํผ :"linux / amd64 "}

ํด๋ผ์šฐ๋“œ ์ œ๊ณต ์—…์ฒด / ํ”Œ๋žซํผ (AKS, GKE, Minikube ๋“ฑ) :
AKS

questiosupport

๊ฐ€์žฅ ์œ ์šฉํ•œ ๋Œ“๊ธ€

readme.md์˜ 'Helm์ด CRD๋ฅผ ๋งŒ๋“ค์ง€ ๋ชปํ•จ'์„น์…˜์— ๋”ฐ๋ผ์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•  ์ˆ˜์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. ๋‚˜๋Š” ๊ทธ๋“ค์ด ์–ด๋–ป๊ฒŒ ๊ด€๋ จ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์‹คํ•˜์ง€ ์•Š์ง€๋งŒ ์ž‘๋™ํ–ˆ์Šต๋‹ˆ๋‹ค.

1 ๋‹จ๊ณ„ : CRDS ์ˆ˜๋™ ์ƒ์„ฑ

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/podmonitor.crd.yaml

2 ๋‹จ๊ณ„:
CRD๊ฐ€ ์ƒ์„ฑ ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฌ์‹ญ์‹œ์˜ค. ๋ช‡ ์ดˆ ๋ฐ–์— ๊ฑธ๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

3 ๋‹จ๊ณ„ :
์ฐจํŠธ๋ฅผ ์„ค์น˜ํ•˜๋˜ prometheusOperator.createCustomResource = false๋ฅผ ์„ค์ •ํ•˜์—ฌ CRD ํ”„๋กœ๋น„์ €๋‹์„ ๋น„ํ™œ์„ฑํ™”ํ•ฉ๋‹ˆ๋‹ค.

$ helm install --name my-release stable/prometheus-operator --set prometheusOperator.createCustomResource=false

๋ชจ๋“  71 ๋Œ“๊ธ€

์šฐ๋ฆฌ๋Š” minikube์— ๋Œ€ํ•ด ๋™์ผํ•œ ๋ฌธ์ œ๊ฐ€ ์žˆ์œผ๋ฏ€๋กœ AWS์—๋งŒ ๊ตญํ•œ๋˜์ง€ ์•Š๋Š” ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.

kubespray ๋ฐฐํฌ ๋œ ํด๋Ÿฌ์Šคํ„ฐ์—์„œ๋„ ๋™์ผํ•œ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

๋˜ํ•œ ์ž๋™ํ™” ๋œ ํŒŒ์ดํ”„ ๋ผ์ธ์—์„œ k8s 12.x ๋ฐ 13.x k8s kubespray ๋ฐฐํฌ ํด๋Ÿฌ์Šคํ„ฐ ๋ชจ๋‘์—์„œ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค (100 % ์‹คํŒจ์œจ). prometheus-operator (0.30.1)์˜ ์ด์ „ ๋ฒ„์ „์€ ๋ฌธ์ œ์—†์ด ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค.
์žฌ๋ฏธ์žˆ๋Š” ์ ์€ CD ํŒŒ์ดํ”„ ๋ผ์ธ ๋Œ€์‹  ์ˆ˜๋™์œผ๋กœ ๋ช…๋ น์„ ์‹คํ–‰ํ•˜๋ฉด ์ž‘๋™ํ•œ๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ์›์ธ์ด ๋ฌด์—‡์ธ์ง€์— ๋Œ€ํ•ด ์•ฝ๊ฐ„ ํ˜ผ๋ž€์Šค๋Ÿฌ์›Œํ•ฉ๋‹ˆ๋‹ค.

์˜ค๋Š˜ promethus ์ฐจํŠธ๊ฐ€ ์—…๋ฐ์ดํŠธ ๋œ ๊ฒƒ์„ ๋ณด์•˜์Šต๋‹ˆ๋‹ค. ๋‚˜๋Š” ๊ทธ๊ฒƒ์„ ๋ถ€๋”ช์ณค๋‹ค

NAME                            CHART VERSION   APP VERSION
stable/prometheus-operator      6.8.0           0.32.0     

๋” ์ด์ƒ ๋ฌธ์ œ๊ฐ€ ๋ณด์ด์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

@rnkhouse https://github.com/helm/helm/issues/6130#issuecomment -526977731์—์„œ @ dlevene1์ด ์–ธ๊ธ‰ ํ•œ ์ตœ์‹  ์ฐจํŠธ ๋ฒ„์ „์œผ๋กœ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

AKS์˜ ๋ฒ„์ „ 6.8.1์—์„œ ์ด์™€ ๋™์ผํ•œ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

NAME                        CHART VERSION   APP VERSION
stable/prometheus-operator  6.8.1           0.32.0
โฏ helm version 
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
 โฏ helm install -f prd.yaml --name prometheus --namespace monitoring stable/prometheus-operator 
Error: release prometheus failed: grpc: the client connection is closing
>>> elapsed time 1m56s

kubespray ๋ฐฐํฌ ๋œ ํด๋Ÿฌ์Šคํ„ฐ์—์„œ๋„ ๋™์ผํ•œ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

Kubernete ๋ฒ„์ „ : v1.4.1
Helm ๋ฒ„์ „ :

Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

Prometheus-operator ๋ฒ„์ „ :

NAME                            CHART VERSION   APP VERSION
stable/prometheus-operator      6.8.1           0.32.0  

aks์—๋„ ๋™์ผํ•œ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ˆ„๊ตฌ๋‚˜ Helm 3์—์„œ์ด ๋ฌธ์ œ๋ฅผ ์žฌํ˜„ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ? ์•„๋‹ˆ๋ฉด ๋‹ค๋ฅธ ์˜ค๋ฅ˜๋กœ ์ „ํŒŒ๋ฉ๋‹ˆ๊นŒ? ๋‚ด ๊ฐ€์ •์€ ๊ฒฝ์šด๊ธฐ๋ฅผ ์ œ๊ฑฐํ•˜๋ฉด ๋” ์ด์ƒ ๋ฌธ์ œ๊ฐ€๋˜์ง€ ์•Š๋Š”๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

@bacongobbler Helm 3์—์„œ ์—ฌ์ „ํžˆ ๋ฌธ์ œ์ž…๋‹ˆ๋‹ค.

bash$ helm install r-prometheus-operator stable/prometheus-operator --version 6.8.2 -f prometheus-operator/helm/prometheus-operator.yaml

manifest_sorter.go:179: info: skipping unknown hook: "crd-install"
Error: apiVersion "monitoring.coreos.com/v1" in prometheus-operator/templates/exporters/kube-controller-manager/servicemonitor.yaml is not available

ํ•˜์ง€๋งŒ OP๊ฐ€ ์ œ๊ธฐ ํ•œ ๋ฌธ์ œ์™€๋Š” ๋‹ค๋ฅธ ๋ฌธ์ œ์ธ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.

์„ค๋ช… : ' "prometheus-operator"๋ฆด๋ฆฌ์Šค ์‹คํŒจ : rpc ์˜ค๋ฅ˜ : ์ฝ”๋“œ = ์ทจ์†Œ ๋œ desc
= grpc : ํด๋ผ์ด์–ธํŠธ ์—ฐ๊ฒฐ์ด ๋‹ซํž™๋‹ˆ๋‹ค '

์ตœ์‹  ๋ฒ ํƒ€ ๋ฆด๋ฆฌ์Šค๋„ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธํ•˜๊ณ  ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ? ์ด ์˜ค๋ฅ˜๋Š” 3.0.0-beta.3์—์„œ ๋ฆด๋ฆฌ์Šค ๋œ # 6332์—์„œ ํ•ด๊ฒฐ ๋œ ๊ฒƒ์œผ๋กœ ๋ณด์ž…๋‹ˆ๋‹ค. ๊ทธ๋ ‡์ง€ ์•Š๋‹ค๋ฉด ์ƒˆ๋กœ์šด ๋ฌธ์ œ๋ฅผ ์—ด โ€‹โ€‹์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

@bacongobbler ์ตœ์‹  Helm v3.0.0-beta.3์„ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.

์ œ๋Œ€๋กœ ์„ค์น˜ํ•˜๋ ค๋ฉด --version 6.7.3์œผ๋กœ ๋Œ์•„ ๊ฐ€์•ผํ–ˆ์Šต๋‹ˆ๋‹ค.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์€ v0.31.1์—์„œ prometheus ์šด์˜์ž ์ด๋ฏธ์ง€๋ฅผ ์œ ์ง€ํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

helm.log
๋˜ํ•œ DockerEE kubernetes ์„ค์น˜ ์—์„œ์ด ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ–ˆ์Šต๋‹ˆ๋‹ค.

์„ค์น˜ ์˜ต์…˜ --debug ๋“ฑ์„ ์กฐ์ž‘ ํ•œ ํ›„ ์ด์ œ ๋‹ค์Œ์„ ์–ป์Šต๋‹ˆ๋‹ค.

Error: release prom failed: context canceled

ํŽธ์ง‘ : ํ˜„์žฌ v2.12.3์—์„œ ๋‚ด helm ๋ฒ„์ „์„ ์—…๋ฐ์ดํŠธ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
Edit2 : 2.14.3์œผ๋กœ ์—…๋ฐ์ดํŠธ๋˜์—ˆ์ง€๋งŒ ์—ฌ์ „ํžˆ ๋ฌธ์ œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
grpc: the client connection is closing
Edit3 : ์œ„์˜ ์ œ์•ˆ์— ๋”ฐ๋ผ ๋ฒ„์ „ 6.7.3์„ ์„ค์น˜ํ•˜์—ฌ ์ž‘์—…์„ ๋‹ค์‹œ ์ง„ํ–‰ํ•˜์‹ญ์‹œ์˜ค.
Edit4 : helm.log๋กœ ์‹คํŒจํ•œ ์„ค์น˜์— ๋Œ€ํ•œ ํ‹ธ๋Ÿฌ ๋กœ๊ทธ ์ฒจ๋ถ€

๊ด€๋ จ : https://github.com/helm/charts/issues/15977

@ cyp3d๋กœ ๋ช‡ ๊ฐ€์ง€ ์กฐ์‚ฌ๋ฅผ ์ˆ˜ํ–‰ ํ•œ ํ›„ ์ผ๋ถ€ ํด๋Ÿฌ์Šคํ„ฐ์—๋Š” ๋„ˆ๋ฌด ์งง์€ helm ์‚ญ์ œ ์‹œ๊ฐ„ ์ดˆ๊ณผ๋กœ ์ธํ•ด ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•  ์ˆ˜์žˆ๋Š” ๊ฒƒ์œผ๋กœ ๋ณด์ž…๋‹ˆ๋‹ค. ๋‚˜๋Š” ์–ด๋””์—์„œ๋‚˜ ๋ฌธ์ œ๋ฅผ ์žฌํ˜„ ํ•  ์ˆ˜ ์—†์œผ๋ฏ€๋กœ ์ด๊ฒƒ์„ ๊ฒฝํ—˜ํ•˜๋Š” ๋ˆ„๊ตฐ๊ฐ€๊ฐ€ ์—ฐ๊ฒฐ๋œ ํ’€ ๋ฆฌํ€˜์ŠคํŠธ ๋ธŒ๋žœ์น˜์—์„œ ์ž ์žฌ์  ์ธ ์ˆ˜์ •์„ ๊ฒ€์ฆ ํ•  ์ˆ˜ ์žˆ๋‹ค๋ฉด ๋Œ€๋‹จํžˆ ๊ฐ์‚ฌํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค!

https://github.com/helm/charts/pull/17090

AWS์—์„œ kops๋กœ ์ƒ์„ฑ ๋œ ์—ฌ๋Ÿฌ ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ๋™์ผํ•ฉ๋‹ˆ๋‹ค.
ํ•˜์ง€๋งŒ K3S์—์„œ ์‹คํ–‰ํ•  ๋•Œ ๋ฌธ์ œ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

@xvzf

์ด PR์—์„œ ์ž ์žฌ์  ์ธ ์ˆ˜์ • ์‚ฌํ•ญ์„ ์‹œ๋„ํ•ด ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ? https://github.com/helm/charts/pull/17090

๋‚˜๋Š” PR์— ์‹คํ–‰์„ํ–ˆ์ง€๋งŒ ์—ฌ์ „ํžˆ Error: release prom failed: context canceled
tiller.log

@vsliouniaev Nope, ์—ฌ๊ธฐ์„œ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

@xvzf ๋ฐ @pyadminn์„ ํ™•์ธํ•ด ์ฃผ์…”์„œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค. ๋™์ผํ•œ PR์—์„œ ๋˜ ๋‹ค๋ฅธ ๋ณ€๊ฒฝ์„ํ–ˆ์Šต๋‹ˆ๋‹ค. ์ด๊ฒƒ์ด ๋„์›€์ด๋˜๋Š”์ง€ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

์—…๋ฐ์ดํŠธ ๋œ PR์„ ํ™•์ธํ–ˆ์ง€๋งŒ ์—ฌ์ „ํžˆ ์ธํ”„๋ผ์— ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋‚ด์šฉ์ด ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค. Error: release prom failed: rpc error: code = Canceled desc = grpc: the client connection is closing

์ฐธ๊ณ ๋กœ ์šฐ๋ฆฌ๋Š” Kuber 1.14.3์— ์žˆ์Šต๋‹ˆ๋‹ค.
Helm ๋ฒ„์ „ v2.14.3

readme.md์˜ 'Helm์ด CRD๋ฅผ ๋งŒ๋“ค์ง€ ๋ชปํ•จ'์„น์…˜์— ๋”ฐ๋ผ์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•  ์ˆ˜์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. ๋‚˜๋Š” ๊ทธ๋“ค์ด ์–ด๋–ป๊ฒŒ ๊ด€๋ จ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์‹คํ•˜์ง€ ์•Š์ง€๋งŒ ์ž‘๋™ํ–ˆ์Šต๋‹ˆ๋‹ค.

1 ๋‹จ๊ณ„ : CRDS ์ˆ˜๋™ ์ƒ์„ฑ

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/podmonitor.crd.yaml

2 ๋‹จ๊ณ„:
CRD๊ฐ€ ์ƒ์„ฑ ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฌ์‹ญ์‹œ์˜ค. ๋ช‡ ์ดˆ ๋ฐ–์— ๊ฑธ๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

3 ๋‹จ๊ณ„ :
์ฐจํŠธ๋ฅผ ์„ค์น˜ํ•˜๋˜ prometheusOperator.createCustomResource = false๋ฅผ ์„ค์ •ํ•˜์—ฌ CRD ํ”„๋กœ๋น„์ €๋‹์„ ๋น„ํ™œ์„ฑํ™”ํ•ฉ๋‹ˆ๋‹ค.

$ helm install --name my-release stable/prometheus-operator --set prometheusOperator.createCustomResource=false

@vsliouniaev ์—ฌ์ „ํžˆ ๊ฐ™์€ ๋ฌธ์ œ์ž…๋‹ˆ๋‹ค! lethalwire์˜ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์ด ์ž‘๋™ํ•˜์ง€๋งŒ.

์น˜๋ช…์ ์ธ ๋ฌธ์ œ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์œผ๋กœ๋„ ํ•ด๊ฒฐ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

๊ทธ๋ž˜์„œ 4 ์ผ ๋™์•ˆ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์ด ์ž‘๋™ํ•˜๊ณ  ์ž‘๋™์„ ๋ฉˆ์ท„์Šต๋‹ˆ๋‹ค. ๋งˆ์Šคํ„ฐ๊ฐ€ ์•„๋‹Œ 0.32.0 ์˜ CRD ํŒŒ์ผ์„ ์‚ฌ์šฉํ•ด์•ผํ–ˆ์Šต๋‹ˆ๋‹ค.

๋‚˜๋Š” ํ˜„์žฌ ๋งˆ์Šคํ„ฐ์—์žˆ๋Š” CRD์—์„œ ๋™์ผํ•œ ๋ฌธ์ œ๋ฅผ ๊ฒฝํ—˜ํ–ˆ์Šต๋‹ˆ๋‹ค. ํ˜„์žฌ ์ด์ „ ๋ฒ„์ „์„ ์‚ฌ์šฉํ•˜๋„๋ก ์ œ์•ˆ ํ•ด ์ฃผ์‹  @Typositoire ์—๊ฒŒ ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค. CRD ์„ค์น˜๋ฅผ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์กฐ์ •ํ•˜๋ฉด ์ €์—๊ฒŒ ํšจ๊ณผ์ ์ด์—ˆ์Šต๋‹ˆ๋‹ค.

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/podmonitor.crd.yaml

๊ทธ๋ ‡๊ธฐ ๋•Œ๋ฌธ์— ๋ฒ„์ „์„ ์ˆ˜์ •ํ•˜๋Š” ๊ฒƒ์ด ์ข‹์€ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค.

๋˜ํ•œ์ด ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•˜๋ฉด admissionWebhooks ๋ฅผ) ๋น„ํ™œ์„ฑํ™”ํ•˜์‹ญ์‹œ์˜ค. ์ œ ๊ฒฝ์šฐ์— ๋„์›€์ด๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

prometheus-operator chart 6.0.0์„ ์„ค์น˜ํ•˜๊ณ  helm ์—…๊ทธ๋ ˆ์ด๋“œ๋ฅผ ์ˆ˜ํ–‰ํ•˜์‹ญ์‹œ์˜ค .-- force --version 6.11.0, ์ด๊ฒƒ์€ rancher kubernetes 1.13.10 ๋ฐ helm v2.14.3์—์„œ ์ž‘๋™ํ•˜๋Š” ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.

@Typositoire๊ฐ€ ์ œ์•ˆํ•œ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์€ kops ์ƒ์„ฑ 1.13.10 ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์ž˜ ์ž‘๋™ํ–ˆ์Šต๋‹ˆ๋‹ค.

kubernetes 1.13.10 ๋ฐ prometheus-operator-6.18.0์„ ์‚ฌ์šฉํ•˜์—ฌ helm v2.14.3์„ ์‚ฌ์šฉํ•˜์—ฌ Azure AKS์— ์„ค์น˜ํ•˜๋ ค๋Š” ๋™์ผํ•œ ๋ฌธ์ œ์ž…๋‹ˆ๋‹ค. ์–ด๋– ํ•œ ์ œ์•ˆ?

CRD๊ฐ€ ์ˆ˜๋™์œผ๋กœ ์„ค์น˜๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

์ด ๋ช…๋ น์ด ์‹คํŒจํ–ˆ์Šต๋‹ˆ๋‹ค.
helm install --name prometheus-operator stable/prometheus-operator --namespace=monitoring --set prometheusOperator.createCustomResource=false

์˜ค๋ฅ˜๋ฅผ ์ฃผ๋‹ค

์˜ค๋ฅ˜ : prometheus-operator ํ•ด์ œ ์‹คํŒจ : rpc ์˜ค๋ฅ˜ : ์ฝ”๋“œ = ์ทจ์†Œ๋จ desc = grpc : ํด๋ผ์ด์–ธํŠธ ์—ฐ๊ฒฐ์ด ๋‹ซํž™๋‹ˆ๋‹ค.

ํŽธ์ง‘ : ์ฐจํŠธ์˜ ๋ฒ„์ „ 6.11.0 (๋ฐ 6.7.3) ์„ค์น˜๊ฐ€ ์ž‘๋™ ์ค‘์ž…๋‹ˆ๋‹ค.

helm install --name prometheus-operator stable/prometheus-operator --namespace=monitoring --set prometheusOperator.createCustomResource=false --version 6.11.0

์ž…ํ•™ ์ปจํŠธ๋กค๋Ÿฌ ์›น ํ›„ํฌ๋ฅผ ๋น„ํ™œ์„ฑํ™” ํ•˜์‹œ๊ฒ ์Šต๋‹ˆ๊นŒ?

https://waynekhan.net/2019/10/09/prometheus-operator-release-failed.html

2019 ๋…„ 10 ์›” 15 ์ผ 19:32์— iMacX [email protected] ์€ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ผ์Šต๋‹ˆ๋‹ค.

,,,,,,,,, ๋ฌด,,,,,,,,,,,,,,,.
kubernetes 1.13.10 ๋ฐ prometheus-operator-6.18.0์„ ์‚ฌ์šฉํ•˜์—ฌ helm v2.14.3์„ ์‚ฌ์šฉํ•˜์—ฌ Azure AKS์— ์„ค์น˜ํ•˜๋ ค๋Š” ๋™์ผํ•œ ๋ฌธ์ œ์ž…๋‹ˆ๋‹ค. ์–ด๋– ํ•œ ์ œ์•ˆ?

โ€”
์ด ์Šค๋ ˆ๋“œ๋ฅผ ๊ตฌ๋…ํ–ˆ๊ธฐ ๋•Œ๋ฌธ์—์ด ๋ฉ”์‹œ์ง€๊ฐ€ ์ „์†ก๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
์ด ์ด๋ฉ”์ผ์— ์ง์ ‘ ๋‹ต์žฅํ•˜๊ฑฐ๋‚˜ GitHub์—์„œ ํ™•์ธํ•˜๊ฑฐ๋‚˜ ๊ตฌ๋…์„ ์ทจ์†Œํ•˜์„ธ์š”.

์ €๋„ ๊ฐ™์€ ๋ฌธ์ œ๋ฅผ ์‹ธ์šฐ๊ณ , ๋‚ด๊ฐ€ ์ˆ˜๋™์œผ๋กœ @JBosom์— ์˜ํ•ด ์ง€์ •๋œ CRDS๋ฅผ ์„ค์น˜ํ•˜๊ณ  ์‚ฌ์šฉํ•  ์›น ํ›„ํฌ๋ฅผ ์„ค์น˜ํ–ˆ๋‹ค.

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/podmonitor.crd.yaml

helm --tls --tiller-namespace=tiller install --namespace=monitoring --name prom-mfcloud stable/prometheus-operator --set prometheusOperator.createCustomResource=false --set prometheusOperator.admissionWebhooks.enabled=false --values values.yaml --versi on 6.18.0

v2.14.3 helm v8.0.0 ์„ (๋ฅผ) ์„ค์น˜ํ•˜๋ ค๊ณ ํ•˜๋ฉด ๋™์ผํ•œ ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ–ˆ์Šต๋‹ˆ๋‹ค. @lethalwire๊ฐ€ ์ œ์•ˆํ•œ๋Œ€๋กœ ๋จผ์ € CRD๋ฅผ ๋งŒ๋“  ํ›„์— ๋งŒ โ€‹โ€‹์„ค์น˜ํ•  ์ˆ˜์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.

๋‚˜๋Š” ์ด๊ฒƒ์ด prometheus-operator ์ฐจํŠธ์˜ ํŠน์ • ๋ฌธ์ œ๋ผ๊ณ  ํŒ๋‹จํ•˜๊ธฐ์— ์ถฉ๋ถ„ํ•œ ์‚ฌ๋ก€๊ฐ€ ์žˆ๋‹ค๊ณ  ์ƒ๊ฐํ•ฉ๋‹ˆ๋‹ค.

์šฐ๋ฆฌ ์ชฝ์—์„œ๋Š” ์‹คํ–‰ ๊ฐ€๋Šฅํ•œ ์‘๋‹ต์ด์—†๋Š” ๊ฒƒ์œผ๋กœ ๋งˆ๋ฌด๋ฆฌํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ๋Œ€ํ™”๋ฅผ ๊ณ„์† ์ง„ํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ถˆํ‰์„ํ•ด์„œ ์ฃ„์†กํ•ฉ๋‹ˆ๋‹ค. ์ตœ์‹  ํ—ฌ๋ฉง v2.15.2 ์—…๊ทธ๋ ˆ์ด๋“œ ํ•œ ํ›„์—๋„ ๋” ์ด์ƒ์ด ์˜ค๋ฅ˜๊ฐ€ ํ‘œ์‹œ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๐Ÿ‘

Helm์—์„œ ๋ฌด์Šจ ์ผ์ด ์ผ์–ด๋‚˜๊ณ  ์žˆ๋Š”์ง€์— ๋Œ€ํ•œ ์ •๋ณด๊ฐ€ ์—†๋‹ค๋Š” ๊ฒƒ์€ ๋งค์šฐ ์ด์ƒํ•ด ๋ณด์ž…๋‹ˆ๋‹ค.

์—ฌ๊ธฐ์—๋Š” ๋””๋ฒ„๊ทธ ๋กœ๊ทธ๊ฐ€ ๊ฒŒ์‹œ๋˜์ง€ ์•Š์•˜๊ณ  ์š”์ฒญ๋„ ์—†์œผ๋ฉฐ ์‚ฌ๋žŒ๋“ค์€ ์Šค์œ„์น˜๋ฅผ ๋’ค์ง‘๊ณ  ๋„์›€์ด๋˜๋Š”์ง€ ํ™•์ธํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.

์˜ค๋ฅ˜๋Š” ์‹ค์ œ๋กœ ๋ฌด์—‡์„ ์˜๋ฏธํ•ฉ๋‹ˆ๊นŒ? ๋Œ€๊ธฐ๋กœ ์ธํ•œ ๊ต์ฐฉ ์ƒํƒœ์˜ ํ‘œ์‹œ์ž…๋‹ˆ๊นŒ? ์ง‘๋‹จ์  ์–ด๊นจ๋ฅผ ์œผ์“ฑํ•˜๋Š” ๊ฒƒ ์™ธ์— ์ˆ˜ํ–‰ ํ•  ์ˆ˜์žˆ๋Š” ๋‹ค๋ฅธ ์กฐ์น˜๊ฐ€ ์žˆ์Šต๋‹ˆ๊นŒ?

์˜ˆ. ์›น ํ›„ํฌ๋ฅผ ๋น„ํ™œ์„ฑํ™”ํ•˜๋ฉด ์ฐจํŠธ๊ฐ€ ๋ฌธ์ œ์—†์ด ์„ค์น˜ ๋  ์ˆ˜ ์žˆ์œผ๋ฏ€๋กœ ์›๋ž˜ ์˜ค๋ฅ˜๋Š” ์Šน์ธ ์›น ํ›„ํฌ๊ฐ€ ์™„๋ฃŒ๋˜๊ธฐ๋ฅผ ๊ธฐ๋‹ค๋ฆฌ๋Š” ๊ต์ฐฉ ์ƒํƒœ๋กœ ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค. Tiller์˜ ๋กœ๊ทธ๋ฅผ ๋ณด๋ฉด ๋ฌธ์ œ๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Helm 3๋Š” ๋ฏน์Šค ํƒ€์ž„ ์•„์›ƒ์— gRPC ๋ ˆ์ด์–ด๊ฐ€์—†๊ณ  ํƒ€์ž„ ์•„์›ƒ์—์„œ ์š”์ฒญ์„ ์ทจ์†Œํ•˜๊ธฐ ๋•Œ๋ฌธ์— ์‚ฌ์šฉ์ž์—๊ฒŒ ์˜ฌ๋ฐ”๋ฅธ ์˜ค๋ฅ˜๋ฅผ๋ณด๊ณ ํ•ด์•ผํ•ฉ๋‹ˆ๋‹ค.

Helm 2์— ๋Œ€ํ•œ ํŒจ์น˜๋ฅผ ์ž์œ ๋กญ๊ฒŒ ์ œ๊ณตํ•˜์‹ญ์‹œ์˜ค. Helm 3์— ๋Œ€ํ•ด ๊ฐœ์„ ๋˜์—ˆ์œผ๋ฏ€๋กœ ๊ณ„์† ์ง„ํ–‰ํ•˜์—ฌ ์ตœ์‹  ๋ฆด๋ฆฌ์Šค์—์„œ ์ˆ˜์ • ๋œ๋Œ€๋กœ ์ข…๋ฃŒํ–ˆ์Šต๋‹ˆ๋‹ค.

๋„์›€์ด ๋˜์—ˆ๊ธฐ๋ฅผ ๋ฐ”๋ž๋‹ˆ๋‹ค.

์›น ํ›„ํฌ๋ฅผ ๋น„ํ™œ์„ฑํ™”ํ•˜๋ฉด ์ฐจํŠธ๊ฐ€ ๋ฌธ์ œ์—†์ด ์„ค์น˜ ๋  ์ˆ˜ ์žˆ์œผ๋ฏ€๋กœ ์›๋ž˜ ์˜ค๋ฅ˜๋Š” ์Šน์ธ ์›น ํ›„ํฌ๊ฐ€ ์™„๋ฃŒ๋˜๊ธฐ๋ฅผ ๊ธฐ๋‹ค๋ฆฌ๋Š” ๊ต์ฐฉ ์ƒํƒœ๋กœ ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค.

ํ•ด๊ฒฐ์ฑ…์€ ์ž‘์—…์„ ๋น„ํ™œ์„ฑํ™”ํ•˜๊ฑฐ๋‚˜ CRD ํ›„ํฌ ์„ค์น˜๋ฅผ ๋น„ํ™œ์„ฑํ™”ํ•˜๋Š” ๊ฒƒ์ด๊ธฐ ๋•Œ๋ฌธ์— ๊ฒฐ๋ก ์œผ๋กœ โ€‹โ€‹๊ฝค ์ด์ƒํ•˜๊ฒŒ ๋ณด์ž…๋‹ˆ๋‹ค. ์ด ๋‘ ๊ฐ€์ง€ ๋ชจ๋‘ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋Š” ๊ฒƒ์œผ๋กœ ๋ณด์ด๋ฏ€๋กœ ์ž‘์—…๊ณผ ๊ด€๋ จ๋œ ๋ฌธ์ œ๋Š” ์•„๋‹™๋‹ˆ๋‹ค.

์ด ๋ฌธ์ œ๋ฅผ ๊ฒช๊ณ ์žˆ๋Š” ๋‹ค๋ฅธ ์‚ฌ๋žŒ์—๊ฒŒ-์–ด๋–ค ์ž‘์—…์ด ์‹คํŒจํ•˜๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋„๋ก kubectl describe job ์˜ ์ถœ๋ ฅ์„ ์ œ๊ณตํ•ด ์ฃผ์‹œ๊ฒ ์Šต๋‹ˆ๊นŒ? ๋‚˜๋Š” ์ „์— ์ด๊ฒƒ์„ ์š”์ฒญํ–ˆ์ง€๋งŒ ๋ชจ๋“  ์‚ฌ๋žŒ๋“ค์ด ์ผ์ž๋ฆฌ๊ฐ€ ์—†๋‹ค๋Š” ๊ฒƒ์„ ๋‚˜ํƒ€๋‚ด๋Š” ๊ฒƒ์ฒ˜๋Ÿผ ๋ณด์ž…๋‹ˆ๋‹ค.

Tiller๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ฝ์Šต๋‹ˆ๋‹ค.

[kube] 2019/11/15 14:35:46 get relation pod of object: monitoring/PrometheusRule/prometheus-operator-node-time
[kube] 2019/11/15 14:35:46 Doing get for PrometheusRule: "prometheus-operator-kubernetes-apps"
[ A lot of unrelated updates in between... ]
2019/11/15 14:36:38 Cannot patch PrometheusRule: "prometheus-operator-kubernetes-apps" (rpc error: code = Canceled desc = grpc: the client connection is closing)
2019/11/15 14:36:38 Use --force to force recreation of the resource
[kube] 2019/11/15 14:36:38 error updating the resource "prometheus-operator-kubernetes-apps":
     rpc error: code = Canceled desc = grpc: the client connection is closing
[tiller] 2019/11/15 14:36:38 warning: Upgrade "prometheus-operator" failed: rpc error: code = Canceled desc = grpc: the client connection is closing
[storage] 2019/11/15 14:36:38 updating release "prometheus-operator.v94"
[storage] 2019/11/15 14:36:38 updating release "prometheus-operator.v95"
[ then rollback... ]

๊ทธ๋ž˜์„œ์ด ๋ฆฌ์†Œ์Šค๋ฅผ ์ˆ˜๋™์œผ๋กœ ์‚ญ์ œํ•ด์•ผํ–ˆ์Šต๋‹ˆ๋‹ค. apiserver์—๋Š” ๋” ๋งŽ์€ ์ •๋ณด๊ฐ€์žˆ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค (์‹ค์ œ๋กœ ์Šน์ธ ์ปจํŠธ๋กค๋Ÿฌ์™€ ๊ด€๋ จ๋œ ๊ฒƒ์ฒ˜๋Ÿผ ๋“ค๋ฆผ).

@desaintmartin ์ด๊ฒƒ์€ ์„ค์น˜๊ฐ€ ์•„๋‹Œ ์—…๊ทธ๋ ˆ์ด๋“œ์—์„œ ๋ฐœ์ƒํ•˜๋Š” ๊ฒƒ์ฒ˜๋Ÿผ ๋ณด์ž…๋‹ˆ๋‹ค.

Helm 3.0์€ ์ด์ œ GA์ด๊ณ  ์ฐจํŠธ๊ฐ€ ์ž‘๋™ํ•˜๋ฏ€๋กœ ๊ฑฐ๊ธฐ์—์„œ ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ๊ณ  ๋” ๋‚˜์€ ๋กœ๊ทธ๋ฅผ ์–ป์„ ์ˆ˜ ์žˆ๋Š”์ง€๋ณด๊ณ ํ•˜์‹ญ์‹œ์˜ค.

Helm3์— ์žˆ๊ณ  Azure AKS์—์„œ ์—ฌ์ „ํžˆ์ด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

๋‚˜๋Š” ์ฐจํŠธ v8.2.4์—์„œ ์‹œ๋„ํ–ˆ๋‹ค : prometheusOperator.admissionWebhooks=false , prometheus.tlsProxy.enabled=false ๋„.

๋˜ํ•œ vsliouniaev๊ฐ€ ๋งํ•œ ๊ฒƒ์ฒ˜๋Ÿผ --debug ๋ฐ --dry-run ๋Š” ๋ฌด์—‡์„ ๋งํ•ฉ๋‹ˆ๊นŒ?

@ truealex81 helm3์€ ์ด์— ๋Œ€ํ•œ ์ž์„ธํ•œ ์ •๋ณด๋ฅผ ์ œ๊ณตํ•˜๊ธฐ์œ„ํ•œ ๊ฒƒ์ด๋ฏ€๋กœ ์„ค์น˜ ํ”„๋กœ์„ธ์Šค์—์„œ ์ž์„ธํ•œ ๋กœ๊ทธ๋ฅผ ๊ฒŒ์‹œ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

Azure AKS์— 8.2.4๋ฅผ ๋ฐฐํฌํ•˜๋Š” ๊ฒƒ๊ณผ ๋™์ผํ•œ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

Helm ๋ฒ„์ „ :
version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}

Helm --debug๋Š” ๋‹ค์Œ ์ถœ๋ ฅ์„ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค.

install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /root/.cache/helm/repository/prometheus-operator-8.2.4.tgz
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
install.go:139: [debug] Clearing discovery cache
wait.go:51: [debug] beginning wait for 5 resources with timeout of 1m0s
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:245: [debug] serviceaccounts "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:245: [debug] podsecuritypolicies.policy "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:245: [debug] rolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:245: [debug] roles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:245: [debug] clusterrolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:245: [debug] clusterroles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:245: [debug] jobs.batch "prometheus-operator-admission-create" not found
client.go:87: [debug] creating 1 resource(s)
client.go:420: [debug] Watching for changes to Job prometheus-operator-admission-create with timeout of 5m0s
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:484: [debug] prometheus-operator-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:87: [debug] creating 120 resource(s)
Error: context canceled

์ด๊ฒƒ์„ ํ™•์‹คํžˆ ์žฌํ˜„ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋” ์ž์„ธํ•œ ๋กœ๊ทธ๋ฅผ ์–ป์„ ์ˆ˜์žˆ๋Š” ๋ฐฉ๋ฒ•์ด ์žˆ์œผ๋ฉด ์•Œ๋ ค์ฃผ์„ธ์š”. ์—ฌ๊ธฐ์— ์ถœ๋ ฅ์„ ๊ฒŒ์‹œํ•ฉ๋‹ˆ๋‹ค.

@ pather87 ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!

์ฐจํŠธ์—์„œ ๋ฐœ์ƒํ•˜๋Š” ์ˆœ์„œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  1. CRD๊ฐ€ ํ”„๋กœ๋น„์ €๋‹ ๋จ
  2. ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ์Šน์ธ ํ›„ํฌ์— ๋Œ€ํ•œ ์ธ์ฆ์„œ๋กœ ๋น„๋ฐ€์„ ์ƒ์„ฑํ•˜๋Š” ์‚ฌ์ „ ์„ค์น˜, ์‚ฌ์ „ ์—…๊ทธ๋ ˆ์ด๋“œ ์ž‘์—…์ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ์ž‘์—…๊ณผ ํ•ด๋‹น ๋ฆฌ์†Œ์Šค๋Š” ์„ฑ๊ณต์‹œ ์ •๋ฆฌ๋ฉ๋‹ˆ๋‹ค.
  3. ๋ชจ๋“  ๋ฆฌ์†Œ์Šค๊ฐ€ ์ƒ์„ฑ๋ฉ๋‹ˆ๋‹ค.
  4. ์ƒ์„ฑ ๋œ ์œ ํšจ์„ฑ ๊ฒ€์‚ฌ gwebhook ๊ตฌ์„ฑ์„ ํŒจ์น˜ํ•˜๊ณ  2 ๋‹จ๊ณ„์—์„œ ์ƒ์„ฑ ๋œ ์ธ์ฆ์„œ์˜ CA๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์›น ํ›„ํฌ ๊ตฌ์„ฑ์„ ๋ณ€๊ฒฝํ•˜๋Š” ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์‹คํ–‰ํ•˜๋Š” ์‚ฌํ›„ ์„ค์น˜; ์—…๊ทธ๋ ˆ์ด๋“œ ํ›„ ์ž‘์—…์ด ์žˆ์Šต๋‹ˆ๋‹ค.์ด ์ž‘์—…๊ณผ ํ•ด๋‹น ๋ฆฌ์†Œ์Šค๋Š” ์„ฑ๊ณต์‹œ ์ •๋ฆฌ๋ฉ๋‹ˆ๋‹ค.

์‹คํŒจํ•œ ์ž‘์—…์ด ์•„์ง ์žˆ๋Š”์ง€ ํ™•์ธํ•ด ์ฃผ์‹œ๊ฒ ์Šต๋‹ˆ๊นŒ? ๋กœ๊ทธ์—์„œ ๋ชจ๋‘ ์„ฑ๊ณตํ–ˆ๊ธฐ ๋•Œ๋ฌธ์— ๊ทธ๋ ‡๊ฒŒํ•˜๋ฉด ์•ˆ๋˜๋Š” ๊ฒƒ์ฒ˜๋Ÿผ ์ฝ์Šต๋‹ˆ๋‹ค.

Error: context canceled ๋ฐœ์ƒ ํ›„ ํด๋Ÿฌ์Šคํ„ฐ์— ๋‹ค๋ฅธ ๋ฆฌ์†Œ์Šค๊ฐ€ ์žˆ์Šต๋‹ˆ๊นŒ?

prometheus-operator๋ฅผ ์„ค์น˜ํ•  ๋•Œ๋„ ๋งˆ์ฐฌ๊ฐ€์ง€์ž…๋‹ˆ๋‹ค.

helm install prometheus-operator stable/prometheus-operator \
  --namespace=monitoring \
  --values=values.yaml

Error: rpc error: code = Canceled desc = grpc: the client connection is closing

@vsliouniaev ๋‹ต๋ณ€ ์ฃผ์…”์„œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!

  1. ๋ฐฐํฌ ํ›„์—๋Š” ์ผ์ž๋ฆฌ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.
  2. ๋ฐฐํฌ ํ›„ ํด๋Ÿฌ์Šคํ„ฐ์— ๋ฐฐํฌ ๋ฐ ์„œ๋น„์Šค๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. kubectl ์ถœ๋ ฅ์„ ์ฐธ์กฐํ•˜์„ธ์š”.

kubectl get all -lrelease = prometheus-operator

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/prometheus-operator-grafana-59d489899-4b5kd          2/2     Running   0          3m56s
pod/prometheus-operator-operator-8549bcd687-4kb2x        2/2     Running   0          3m56s
pod/prometheus-operator-prometheus-node-exporter-4km6x   1/1     Running   0          3m56s
pod/prometheus-operator-prometheus-node-exporter-7dgn6   1/1     Running   0          3m56s

NAME                                                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)            AGE
service/prometheus-operator-alertmanager               ClusterIP   xxx   <none>        9093/TCP           3m57s
service/prometheus-operator-grafana                    ClusterIP   xxx   <none>        80/TCP             3m57s
service/prometheus-operator-operator                   ClusterIP   xxx     <none>        8080/TCP,443/TCP   3m57s
service/prometheus-operator-prometheus                 ClusterIP   xxx   <none>        9090/TCP           3m57s
service/prometheus-operator-prometheus-node-exporter   ClusterIP   xxx    <none>        9100/TCP           3m57s

NAME                                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-operator-prometheus-node-exporter   2         2         2       2            2           <none>          3m57s

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-operator-grafana    1/1     1            1           3m57s
deployment.apps/prometheus-operator-operator   1/1     1            1           3m57s

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-operator-grafana-59d489899     1         1         1       3m57s
replicaset.apps/prometheus-operator-operator-8549bcd687   1         1         1       3m57s

NAME                                                             READY   AGE
statefulset.apps/alertmanager-prometheus-operator-alertmanager   1/1     3m44s
statefulset.apps/prometheus-prometheus-operator-prometheus       1/1     3m34s

๋””๋ฒ„๊ทธ๋กœ ์„ค์น˜ :

client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD alertmanagers.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD podmonitors.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD prometheuses.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD prometheusrules.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD servicemonitors.monitoring.coreos.com is already present. Skipping.
install.go:139: [debug] Clearing discovery cache
wait.go:51: [debug] beginning wait for 0 resources with timeout of 1m0s
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:245: [debug] clusterrolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:245: [debug] rolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:245: [debug] clusterroles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:245: [debug] serviceaccounts "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:245: [debug] podsecuritypolicies.policy "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:245: [debug] roles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:245: [debug] jobs.batch "prometheus-operator-admission-create" not found
client.go:87: [debug] creating 1 resource(s)
client.go:420: [debug] Watching for changes to Job prometheus-operator-admission-create with timeout of 5m0s
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:484: [debug] prometheus-operator-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:87: [debug] creating 122 resource(s)
Error: context canceled
helm.go:76: [debug] context canceled

๊ทธ๋Ÿฐ ๋‹ค์Œ kubectl get all -lrelease=prometheus-operator -A ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

NAMESPACE    NAME                                                     READY   STATUS    RESTARTS   AGE
monitoring   pod/prometheus-operator-grafana-d6676b794-r6cg9          2/2     Running   0          2m45s
monitoring   pod/prometheus-operator-operator-6584f4b5f5-wdkrx        2/2     Running   0          2m45s
monitoring   pod/prometheus-operator-prometheus-node-exporter-2g4tg   1/1     Running   0          2m45s
monitoring   pod/prometheus-operator-prometheus-node-exporter-798p5   1/1     Running   0          2m45s
monitoring   pod/prometheus-operator-prometheus-node-exporter-pvk5t   1/1     Running   0          2m45s
monitoring   pod/prometheus-operator-prometheus-node-exporter-r9j2r   1/1     Running   0          2m45s

NAMESPACE     NAME                                                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)            AGE
kube-system   service/prometheus-operator-coredns                    ClusterIP   None           <none>        9153/TCP           2m46s
kube-system   service/prometheus-operator-kube-controller-manager    ClusterIP   None           <none>        10252/TCP          2m46s
kube-system   service/prometheus-operator-kube-etcd                  ClusterIP   None           <none>        2379/TCP           2m46s
kube-system   service/prometheus-operator-kube-proxy                 ClusterIP   None           <none>        10249/TCP          2m46s
kube-system   service/prometheus-operator-kube-scheduler             ClusterIP   None           <none>        10251/TCP          2m46s
monitoring    service/prometheus-operator-alertmanager               ClusterIP   10.0.238.102   <none>        9093/TCP           2m46s
monitoring    service/prometheus-operator-grafana                    ClusterIP   10.0.16.19     <none>        80/TCP             2m46s
monitoring    service/prometheus-operator-operator                   ClusterIP   10.0.97.114    <none>        8080/TCP,443/TCP   2m45s
monitoring    service/prometheus-operator-prometheus                 ClusterIP   10.0.57.153    <none>        9090/TCP           2m46s
monitoring    service/prometheus-operator-prometheus-node-exporter   ClusterIP   10.0.83.30     <none>        9100/TCP           2m46s

NAMESPACE    NAME                                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
monitoring   daemonset.apps/prometheus-operator-prometheus-node-exporter   4         4         4       4            4           <none>          2m46s

NAMESPACE    NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
monitoring   deployment.apps/prometheus-operator-grafana    1/1     1            1           2m46s
monitoring   deployment.apps/prometheus-operator-operator   1/1     1            1           2m46s

NAMESPACE    NAME                                                      DESIRED   CURRENT   READY   AGE
monitoring   replicaset.apps/prometheus-operator-grafana-d6676b794     1         1         1       2m46s
monitoring   replicaset.apps/prometheus-operator-operator-6584f4b5f5   1         1         1       2m46s

NAMESPACE    NAME                                                             READY   AGE
monitoring   statefulset.apps/alertmanager-prometheus-operator-alertmanager   1/1     2m40s
monitoring   statefulset.apps/prometheus-prometheus-operator-prometheus       1/1     2m30s

์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋ ค๊ณ  ์‹œ๋„ํ•˜๋ฉด์„œ ๋ฐœ๊ฒฌ ํ•œ ๋‚ด์šฉ : ๋‚˜์ค‘์— ์ฐจํŠธ์™€ CRD๋ฅผ ์‚ญ์ œํ•˜๊ณ  ์ฐจํŠธ๋ฅผ ๋‹ค์‹œ ์„ค์น˜ํ•˜๋ฉด ๋ฌธ์ œ๊ฐ€ ์ง€์†๋˜์ง€๋งŒ crd๋ฅผ ์‚ญ์ œํ•˜์ง€ ์•Š์œผ๋ฉด ๋ฌธ์ œ๊ฐ€ ์ง€์†๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์‚ฌ์ „์— crds๋ฅผ ์‹œ๋„ํ•˜๊ณ  ์„ค์น˜ํ•˜๊ณ  helm install --skip-crds ์„ ์ˆ˜ํ–‰ํ–ˆ์ง€๋งŒ ์—ฌ์ „ํžˆ ๋ฌธ์ œ๊ฐ€ ์ง€์†๋ฉ๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ๋‹ค์†Œ ํ˜ผ๋ž€ ์Šค๋Ÿฝ์Šต๋‹ˆ๋‹ค.

์ด ํ›„ ์˜ˆ์ƒ๋˜๋Š” ๋‹ค์Œ ๋กœ๊ทธ ์ค„์€ ์„ค์น˜ ํ›„, ์—…๊ทธ๋ ˆ์ด๋“œ ํ›„ ํ›„ํฌ์— ๊ด€ํ•œ ๊ฒƒ์ด์ง€๋งŒ ๊ท€ํ•˜์˜ ๊ฒฝ์šฐ์—๋Š” ๋‚˜ํƒ€๋‚˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์—์„œ ์–ด๋–ค ์กฐํƒ€๊ฐ€ ๊ธฐ๋‹ค๋ฆฌ๊ณ  ์žˆ๋Š”์ง€ ์ž˜ ๋ชจ๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค

...
lient.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" RoleBinding
client.go:245: [debug] rolebindings.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" Role
client.go:245: [debug] roles.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRole
client.go:245: [debug] clusterroles.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ServiceAccount
client.go:245: [debug] serviceaccounts "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRoleBinding
client.go:245: [debug] clusterrolebindings.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" PodSecurityPolicy
client.go:245: [debug] podsecuritypolicies.policy "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission-patch" Job
client.go:245: [debug] jobs.batch "prom-op-prometheus-operato-admission-patch" not found
client.go:87: [debug] creating 1 resource(s)
client.go:420: [debug] Watching for changes to Job prom-op-prometheus-operato-admission-patch with timeout of 5m0s
client.go:445: [debug] Add/Modify event for prom-op-prometheus-operato-admission-patch: MODIFIED
client.go:484: [debug] prom-op-prometheus-operato-admission-patch: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:445: [debug] Add/Modify event for prom-op-prometheus-operato-admission-patch: MODIFIED
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" RoleBinding
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" Role
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRole
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ServiceAccount
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRoleBinding
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" PodSecurityPolicy
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission-patch" Job

์ˆ˜๋™ CRD ์ƒ์„ฑ์€ ์ตœ์†Œํ•œ Azure์—์„œ ๋„์›€์ด๋ฉ๋‹ˆ๋‹ค.
๋จผ์ € https://github.com/coreos/prometheus-operator/tree/release-0.34/example/prometheus-operator-crd ๋งํฌ์—์„œ crd๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
"kubectl create -f alertmanager.crd.yaml"๋“ฑ ๋ชจ๋“  ํŒŒ์ผ์— ๋Œ€ํ•ด
๊ทธ๋•Œ
helm install prometheus-operator stable / prometheus-operator --namespace monitoring --version 8.2.4 --set prometheusOperator.createCustomResource = false

๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค @ truealex81 ! Azure์—์„œ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค.

myenv :
k8s 1.11.2 ํˆฌ๊ตฌ 2.13.1 ๊ฒฝ์šด๊ธฐ 2.13.1
prometheus-operator-5.5 APP VERSION 0.29๋Š” ๊ดœ์ฐฎ์Šต๋‹ˆ๋‹ค !!!

๊ทธ๋Ÿฌ๋‚˜:
prometheus-operator-8 APP VERSION 0.32 hava ๋™์ผํ•œ ๋ฌธ์ œ :
"์ปจํ…์ŠคํŠธ ์ทจ์†Œ๋จ"๋˜๋Š” "grpc : ํด๋ผ์ด์–ธํŠธ ์—ฐ๊ฒฐ์ด ๋‹ซํž™๋‹ˆ๋‹ค"!!!

prometheus-operator์˜ ์ตœ์‹  ๋ฒ„์ „์ด ํ˜ธํ™˜๋˜์ง€ ์•Š๋Š” ๊ฒƒ ๊ฐ™์•„์š”? !!!

@bierhov ์‹คํŒจ ํ›„ ๋„ค์ž„ ์ŠคํŽ˜์ด์Šค์— ๋ฆฌ์†Œ์Šค๋ฅผ ๊ฒŒ์‹œ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

์˜ˆ!
shell execute "helm ls"๋‚ด prometheus-operator ๋ฆด๋ฆฌ์Šค ์ƒํƒœ๊ฐ€ "failed"์ธ ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ์ง€๋งŒ prometheus-operator๊ฐ€ ์„ค์น˜ ํ•œ ๋„ค์ž„ ์ŠคํŽ˜์ด์Šค์—๋Š” ๋ชจ๋“  prometheus-operator ๋ฆฌ์†Œ์Šค๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋Ÿฌ๋‚˜,
promethues ์›น์€ ๋ฐ์ดํ„ฐ๋ฅผ ์–ป์„ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค!

๊ทธ๋ž˜๋„ ๋ฆฌ์†Œ์Šค๋ฅผ ๊ฒŒ์‹œ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

๊ทธ๋ž˜๋„ ๋ฆฌ์†Œ์Šค๋ฅผ ๊ฒŒ์‹œ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

๋ฏธ์•ˆ ํ•ด์š”, ์•ˆ์ •๋œ ์กฐํƒ€ ํ™˜๊ฒฝ์„ ์ œ๊ฑฐํ•˜๊ณ  ๋‹ค์‹œํ•˜์ง€ ์•Š์œผ๋ฉด ๋‹ค์‹œ ๋‚˜ํƒ€๋‚  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค!

@bierhov ์„ค์น˜ ํ›„ ์‹คํŒจํ•œ ์ž‘์—…์ด ๋‚จ์•„ ์žˆ์Šต๋‹ˆ๊นŒ?

@bierhov ์„ค์น˜ ํ›„ ์‹คํŒจํ•œ ์ž‘์—…์ด ๋‚จ์•„ ์žˆ์Šต๋‹ˆ๊นŒ?

๋‚ด k8s ๋ฒ„์ „์€ 1.11.2 helm์ด๊ณ  ํƒ€๋ณ‘ ๋ฒ„์ „์€ 2.13.1์ž…๋‹ˆ๋‹ค.
prometheus-operator ๋ฒ„์ „ 8.x๋ฅผ ์„ค์น˜ํ•˜๋ฉด
shell exec ๋ช…๋ น "helm ls", ์ž‘์—… ์ƒํƒœ๊ฐ€ ์‹คํŒจ ํ•จ
ํ•˜์ง€๋งŒ prometheus-operator ๋ฒ„์ „ 5.x๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.
shell exec ๋ช…๋ น "helm ls", ์ž‘์—… ์ƒํƒœ๊ฐ€ ๋ฐฐํฌ๋˜์—ˆ์Šต๋‹ˆ๋‹ค !!!

๋‹ค์Œ์„ ์‚ฌ์šฉํ•˜์—ฌ ์žฌํ˜„ ํ•  ์ˆ˜ ์—†์Œ :

Kubernetes ๋ฒ„์ „ : v1.13.12"
Kubectl ๋ฒ„์ „ : v1.16.2
Helm ๋ฒ„์ „ : 3.0.1
Prometheus-operator ๋ฒ„์ „ : 8.3.3

  1. CRD๋ฅผ ์ˆ˜๋™์œผ๋กœ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/alertmanager.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/prometheus.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/prometheusrule.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/servicemonitor.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/podmonitor.crd.yaml

  1. Values.yaml์—์„œ ๋˜๋Š” ์„ค์น˜ํ•  ๋•Œ crd๋ฅผ ๋งŒ๋“ค์ง€ ์•Š๋„๋ก ์—ฐ์‚ฐ์ž๋ฅผ ๊ตฌ์„ฑํ•˜์‹ญ์‹œ์˜ค.

--set prometheusOperator.createCustomResource=false

prometheusOperator: createCustomResource: false

์•ˆ๋…•ํ•˜์„ธ์š”.
CRD๋ฅผ ์ˆ˜๋™์œผ๋กœ ์ƒ์„ฑํ•˜์ง€ ์•Š์œผ๋ฉด ์–ด๋–ป๊ฒŒ๋ฉ๋‹ˆ๊นŒ? ์ด๊ฒƒ์ด ๋ฌธ์ œ์— ๋Œ€ํ•œ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ• ์ค‘ ํ•˜๋‚˜์ž…๋‹ˆ๋‹ค.

@vsliouniaev๋ฅผ ๋งŒ๋“ค์ง€ ์•Š์œผ๋ฉด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.
๊ทธ๋Ÿฌ๋‚˜ Additional Info์˜ ์›๋ž˜ ๋ฌธ์ œ์—์„œ @rnkhouse ๋Š” CRD๋ฅผ ์ˆ˜๋™์œผ๋กœ ์ƒ์„ฑํ•œ๋‹ค๊ณ  ๋งํ–ˆ์Šต๋‹ˆ๋‹ค.

๋ฐฐํฌ์‹œ prometheus-operator๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ฐ„๋‹จํžˆ ๋งํ•ด์„œ prom-op์„ 6.9.3์—์„œ 8.3.3์œผ๋กœ ์—…๊ทธ๋ ˆ์ด๋“œํ–ˆ์œผ๋ฉฐ ํ•ญ์ƒ "์˜ค๋ฅ˜ : ์ปจํ…์ŠคํŠธ ์ทจ์†Œ๋จ"์œผ๋กœ ์‹คํŒจํ–ˆ์Šต๋‹ˆ๋‹ค.
๋˜ํ•œ ์šฐ๋ฆฌ๋Š” ํ•ญ์ƒ prometheus-operator๋ฅผ ์„ค์น˜ / ์—…๊ทธ๋ ˆ์ด๋“œํ•˜๊ธฐ ์ „์— crd๋ฅผ ์„ค์น˜ํ•˜๋ฉฐ, ofc๋Š” ์ด๋Ÿฌํ•œ crd-s๋ฅผ ๋ณ€๊ฒฝํ•˜๊ฑฐ๋‚˜ ์—…๋ฐ์ดํŠธํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค.

'github.com/helm/charts/tree/master/stable/prometheus-operator'์—์„œ ์–ธ๊ธ‰ ํ•œ crds๋ฅผ ์ƒˆ๋กœ ๊ณ ์น˜๋ ค๊ณ ํ•ฉ๋‹ˆ๋‹ค (์˜ˆ : kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator /master/example/prometheus-operator-crd/alertmanager.crd.yaml), ๊ทธ๋Ÿฌ๋‚˜ ์ด๋“ค์€ ๋” ์ด์ƒ ์กด์žฌํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
๊ทธ ํ›„ https://github.com/helm/charts/tree/master/stable/prometheus-operator/crds์—์„œ ์ด๊ฒƒ๋“ค์„ ์‹œ๋„ํ•ฉ๋‹ˆ๋‹ค.
ํ•˜์ง€๋งŒ ๋˜ ์‹คํŒจํ–ˆ์Šต๋‹ˆ๋‹ค.

๊ฑฐ์˜ ํฌ๊ธฐํ•  ๋ป”ํ–ˆ์ง€๋งŒ์ด crds๋กœ helm ๋ฐฐํฌ๊ฐ€ ์„ฑ๊ณตํ–ˆ์Šต๋‹ˆ๋‹ค! ์˜ˆ์ด
https://github.com/coreos/kube-prometheus/tree/master/manifests/setup

๋‚ด ์„ค์ • :

Kubernetes ๋ฒ„์ „ : v1.14.3
Kubectl ๋ฒ„์ „ : v1.14.2
Helm ๋ฒ„์ „ : 2.14.3
Prometheus-operator ๋ฒ„์ „ : 8.3.3

k8s์—์„œ prometheus-operator๋ฅผ ์ œ๊ฑฐํ•˜์‹ญ์‹œ์˜ค!

๊ทธ๋•Œ:

kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml   
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0podmonitorCustomResourceDefinition.yaml     
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml     
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0prometheusruleCustomResourceDefinition.yaml 
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0servicemonitorCustomResourceDefinition.yaml 
helm upgrade -i prom-op                               \
  --version 8.3.3                                     \
  --set prometheusOperator.createCustomResource=false \
  stable/prometheus-operator

๊ทธ๊ฒŒ ๋‹ค์•ผ !

์ด๊ฒƒ์€ ์ƒˆ๋กœ ์„ค์น˜๋ฅผ ์ˆ˜ํ–‰ํ•˜๊ณ  ๊ธฐ๋ก ๋ฉ”ํŠธ๋ฆญ ๋ฐ์ดํ„ฐ๋ฅผ ์žƒ์–ด์•ผ ํ•จ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๊นŒ?

AKS k8s๋ฅผ 1.15.5๋กœ, helm์„ 3.0.1๋กœ, Prometheus-operator chart๋ฅผ 8.3.3์œผ๋กœ ์˜ฌ๋ฆฌ๋ฉด ๋ฌธ์ œ๊ฐ€ ์‚ฌ๋ผ์กŒ์Šต๋‹ˆ๋‹ค.

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์€ v0.31.1์—์„œ prometheus ์šด์˜์ž ์ด๋ฏธ์ง€๋ฅผ ์œ ์ง€ํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

AKS v1.14.8 ๋ฐ helm + tiller v2.16.1 ์—์„œ ์ €๋ฅผ ์œ„ํ•ด ์ผํ–ˆ๊ณ  ์šด์˜์ž ์ด๋ฏธ์ง€๋ฅผ v0.31.1

์ˆ˜๋™ CRD ์ƒ์„ฑ์€ ์ตœ์†Œํ•œ Azure์—์„œ ๋„์›€์ด๋ฉ๋‹ˆ๋‹ค.
๋จผ์ € https://github.com/coreos/prometheus-operator/tree/release-0.34/example/prometheus-operator-crd ๋งํฌ์—์„œ crd๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
"kubectl create -f alertmanager.crd.yaml"๋“ฑ ๋ชจ๋“  ํŒŒ์ผ์— ๋Œ€ํ•ด
๊ทธ๋•Œ
helm install prometheus-operator stable / prometheus-operator --namespace monitoring --version 8.2.4 --set prometheusOperator.createCustomResource = false

ํ‘ธ๋ฅธ kubernetes์—์„œ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค.

readme.md์˜ 'Helm์ด CRD๋ฅผ ๋งŒ๋“ค์ง€ ๋ชปํ•จ'์„น์…˜์— ๋”ฐ๋ผ์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•  ์ˆ˜์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. ๋‚˜๋Š” ๊ทธ๋“ค์ด ์–ด๋–ป๊ฒŒ ๊ด€๋ จ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์‹คํ•˜์ง€ ์•Š์ง€๋งŒ ์ž‘๋™ํ–ˆ์Šต๋‹ˆ๋‹ค.

1 ๋‹จ๊ณ„ : CRDS ์ˆ˜๋™ ์ƒ์„ฑ

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/podmonitor.crd.yaml

2 ๋‹จ๊ณ„:
CRD๊ฐ€ ์ƒ์„ฑ ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฌ์‹ญ์‹œ์˜ค. ๋ช‡ ์ดˆ ๋ฐ–์— ๊ฑธ๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

3 ๋‹จ๊ณ„ :
์ฐจํŠธ๋ฅผ ์„ค์น˜ํ•˜๋˜ prometheusOperator.createCustomResource = false๋ฅผ ์„ค์ •ํ•˜์—ฌ CRD ํ”„๋กœ๋น„์ €๋‹์„ ๋น„ํ™œ์„ฑํ™”ํ•ฉ๋‹ˆ๋‹ค.

$ helm install --name my-release stable/prometheus-operator --set prometheusOperator.createCustomResource=false

๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ AKS ํด๋Ÿฌ์Šคํ„ฐ์—์„œ ์ €์—๊ฒŒ ํšจ๊ณผ์ ์ด์—ˆ์Šต๋‹ˆ๋‹ค. CRD์˜ URL์„ ๋ณ€๊ฒฝํ•ด์•ผํ–ˆ์Šต๋‹ˆ๋‹ค.

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml --validate = false

helm install stable / prometheus-operator --name prometheus-operator --namespace monitoring --set prometheusOperator.createCustomResource = false

ํ์‡„. ์ง€๋‚œ ์„ธ ๋ช…์˜ ๋Œ“๊ธ€ ์ž‘์„ฑ์ž์— ๋”ฐ๋ฅด๋ฉด์ด ๋ฌธ์ œ๋Š” ์ดํ›„ ํ•ด๊ฒฐ ๋œ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค. ๊ฐ์‚ฌ!

์ด ํŽ˜์ด์ง€๊ฐ€ ๋„์›€์ด ๋˜์—ˆ๋‚˜์š”?
0 / 5 - 0 ๋“ฑ๊ธ‰