๋ฒ๊ทธ ์ค๋ช
helm install stable/prometheus-operator --name prometheus-operator -f prometheus-operator-values.yaml
AKS์ prometheus ์ฐ์ฐ์๋ฅผ ์ค์นํ๋ ค๊ณ ํ๋ฉด์ด ์ค๋ฅ๊ฐ ๋ฐ์ํฉ๋๋ค.
prometheus-operator "์คํจ : rpc ์ค๋ฅ : ์ฝ๋ = ์ทจ์๋จ
๋๋ ์ญ์ฌ๋ฅผ ํ์ธํ๋ค.
helm history prometheus-operator -o yaml
- chart: prometheus-operator-6.3.0
description: 'Release "prometheus-operator" failed: rpc error: code = Canceled desc
= grpc: the client connection is closing'
revision: 1
status: FAILED
updated: Tue Jul 30 12:36:52 2019
์ฐจํธ
[์์ / ํ๋ก ๋ฉํ
์ฐ์ค ์ฐ์ฐ์]
์ถ๊ฐ ์ ๋ณด
์๋ ๊ตฌ์ฑ์ ์ฌ์ฉํ์ฌ ์ฐจํธ๋ฅผ ๋ฐฐํฌํ๊ณ ์์ต๋๋ค.
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml
๊ฐ ํ์ผ์์ : createCustomResource
๊ฐ false
.
helm version
:
ํด๋ผ์ด์ธํธ : & version.Version {SemVer : "v2.14.3", GitCommit : "0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState : "clean"}
์๋ฒ : & version.Version {SemVer : "v2.14.3", GitCommit : "0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState : "clean"}
kubectl version
์ถ๋ ฅ :
ํด๋ผ์ด์ธํธ ๋ฒ์ : version.Info {Major : "1", Minor : "10", GitVersion : "v1.10.4", GitCommit : "5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState : "clean", BuildDate : "2018-06-06T08 : 13 : 03Z ", GoVersion :"go1.9.3 ", ์ปดํ์ผ๋ฌ :"gc ", ํ๋ซํผ :"windows / amd64 "}
์๋ฒ ๋ฒ์ : version.Info {Major : "1", Minor : "13", GitVersion : "v1.13.7", GitCommit : "4683545293d792934a7a7e12f2cc47d20b2dd01b", GitTreeState : "clean", BuildDate : "2019-06-06T01 : 39 : 30Z ", GoVersion :"go1.11.5 ", ์ปดํ์ผ๋ฌ :"gc ", ํ๋ซํผ :"linux / amd64 "}
ํด๋ผ์ฐ๋ ์ ๊ณต ์
์ฒด / ํ๋ซํผ (AKS, GKE, Minikube ๋ฑ) :
AKS
์ฐ๋ฆฌ๋ minikube์ ๋ํด ๋์ผํ ๋ฌธ์ ๊ฐ ์์ผ๋ฏ๋ก AWS์๋ง ๊ตญํ๋์ง ์๋ ๊ฒ ๊ฐ์ต๋๋ค.
kubespray ๋ฐฐํฌ ๋ ํด๋ฌ์คํฐ์์๋ ๋์ผํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค.
๋ํ ์๋ํ ๋ ํ์ดํ ๋ผ์ธ์์ k8s 12.x ๋ฐ 13.x k8s kubespray ๋ฐฐํฌ ํด๋ฌ์คํฐ ๋ชจ๋์์ ๋ฌธ์ ๊ฐ ๋ฐ์ํฉ๋๋ค (100 % ์คํจ์จ). prometheus-operator (0.30.1)์ ์ด์ ๋ฒ์ ์ ๋ฌธ์ ์์ด ์๋ํฉ๋๋ค.
์ฌ๋ฏธ์๋ ์ ์ CD ํ์ดํ ๋ผ์ธ ๋์ ์๋์ผ๋ก ๋ช
๋ น์ ์คํํ๋ฉด ์๋ํ๋ค๋ ๊ฒ์
๋๋ค. ๊ทธ๋์ ์์ธ์ด ๋ฌด์์ธ์ง์ ๋ํด ์ฝ๊ฐ ํผ๋์ค๋ฌ์ํฉ๋๋ค.
์ค๋ promethus ์ฐจํธ๊ฐ ์ ๋ฐ์ดํธ ๋ ๊ฒ์ ๋ณด์์ต๋๋ค. ๋๋ ๊ทธ๊ฒ์ ๋ถ๋ช์ณค๋ค
NAME CHART VERSION APP VERSION
stable/prometheus-operator 6.8.0 0.32.0
๋ ์ด์ ๋ฌธ์ ๊ฐ ๋ณด์ด์ง ์์ต๋๋ค.
@rnkhouse https://github.com/helm/helm/issues/6130#issuecomment -526977731์์ @ dlevene1์ด ์ธ๊ธ ํ ์ต์ ์ฐจํธ ๋ฒ์ ์ผ๋ก ํ์ธํ ์ ์์ต๋๊น?
AKS์ ๋ฒ์ 6.8.1์์ ์ด์ ๋์ผํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค.
NAME CHART VERSION APP VERSION
stable/prometheus-operator 6.8.1 0.32.0
โฏ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
โฏ helm install -f prd.yaml --name prometheus --namespace monitoring stable/prometheus-operator
Error: release prometheus failed: grpc: the client connection is closing
>>> elapsed time 1m56s
kubespray ๋ฐฐํฌ ๋ ํด๋ฌ์คํฐ์์๋ ๋์ผํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค.
Kubernete ๋ฒ์ : v1.4.1
Helm ๋ฒ์ :
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}
Prometheus-operator ๋ฒ์ :
NAME CHART VERSION APP VERSION
stable/prometheus-operator 6.8.1 0.32.0
aks์๋ ๋์ผํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค.
๋๊ตฌ๋ Helm 3์์์ด ๋ฌธ์ ๋ฅผ ์ฌํ ํ ์ ์์ต๋๊น? ์๋๋ฉด ๋ค๋ฅธ ์ค๋ฅ๋ก ์ ํ๋ฉ๋๊น? ๋ด ๊ฐ์ ์ ๊ฒฝ์ด๊ธฐ๋ฅผ ์ ๊ฑฐํ๋ฉด ๋ ์ด์ ๋ฌธ์ ๊ฐ๋์ง ์๋๋ค๋ ๊ฒ์ ๋๋ค.
@bacongobbler Helm 3์์ ์ฌ์ ํ ๋ฌธ์ ์ ๋๋ค.
bash$ helm install r-prometheus-operator stable/prometheus-operator --version 6.8.2 -f prometheus-operator/helm/prometheus-operator.yaml
manifest_sorter.go:179: info: skipping unknown hook: "crd-install"
Error: apiVersion "monitoring.coreos.com/v1" in prometheus-operator/templates/exporters/kube-controller-manager/servicemonitor.yaml is not available
ํ์ง๋ง OP๊ฐ ์ ๊ธฐ ํ ๋ฌธ์ ์๋ ๋ค๋ฅธ ๋ฌธ์ ์ธ ๊ฒ ๊ฐ์ต๋๋ค.
์ค๋ช : ' "prometheus-operator"๋ฆด๋ฆฌ์ค ์คํจ : rpc ์ค๋ฅ : ์ฝ๋ = ์ทจ์ ๋ desc
= grpc : ํด๋ผ์ด์ธํธ ์ฐ๊ฒฐ์ด ๋ซํ๋๋ค '
์ต์ ๋ฒ ํ ๋ฆด๋ฆฌ์ค๋ ์ฌ์ฉํ๊ณ ์๋์ง ํ์ธํ๊ณ ํ์ธํ ์ ์์ต๋๊น? ์ด ์ค๋ฅ๋ 3.0.0-beta.3์์ ๋ฆด๋ฆฌ์ค ๋ # 6332์์ ํด๊ฒฐ ๋ ๊ฒ์ผ๋ก ๋ณด์ ๋๋ค. ๊ทธ๋ ์ง ์๋ค๋ฉด ์๋ก์ด ๋ฌธ์ ๋ฅผ ์ด โโ์ ์์ต๋๊น?
@bacongobbler ์ต์ Helm v3.0.0-beta.3์ ์ฌ์ฉํ๊ณ ์์ต๋๋ค.
์ ๋๋ก ์ค์นํ๋ ค๋ฉด --version 6.7.3์ผ๋ก ๋์ ๊ฐ์ผํ์ต๋๋ค.
ํด๊ฒฐ ๋ฐฉ๋ฒ์ v0.31.1์์ prometheus ์ด์์ ์ด๋ฏธ์ง๋ฅผ ์ ์งํ๋ ๊ฒ์ ๋๋ค.
helm.log
๋ํ DockerEE kubernetes ์ค์น ์์์ด ๋ฌธ์ ๊ฐ ๋ฐ์ํ์ต๋๋ค.
์ค์น ์ต์ --debug ๋ฑ์ ์กฐ์ ํ ํ ์ด์ ๋ค์์ ์ป์ต๋๋ค.
Error: release prom failed: context canceled
ํธ์ง : ํ์ฌ v2.12.3์์ ๋ด helm ๋ฒ์ ์ ์
๋ฐ์ดํธ ํ ์ ์์ต๋๋ค.
Edit2 : 2.14.3์ผ๋ก ์
๋ฐ์ดํธ๋์์ง๋ง ์ฌ์ ํ ๋ฌธ์ ๊ฐ ์์ต๋๋ค.
grpc: the client connection is closing
Edit3 : ์์ ์ ์์ ๋ฐ๋ผ ๋ฒ์ 6.7.3์ ์ค์นํ์ฌ ์์
์ ๋ค์ ์งํํ์ญ์์ค.
Edit4 : helm.log๋ก ์คํจํ ์ค์น์ ๋ํ ํธ๋ฌ ๋ก๊ทธ ์ฒจ๋ถ
@ cyp3d๋ก ๋ช ๊ฐ์ง ์กฐ์ฌ๋ฅผ ์ํ ํ ํ ์ผ๋ถ ํด๋ฌ์คํฐ์๋ ๋๋ฌด ์งง์ helm ์ญ์ ์๊ฐ ์ด๊ณผ๋ก ์ธํด ๋ฌธ์ ๊ฐ ๋ฐ์ํ ์์๋ ๊ฒ์ผ๋ก ๋ณด์ ๋๋ค. ๋๋ ์ด๋์์๋ ๋ฌธ์ ๋ฅผ ์ฌํ ํ ์ ์์ผ๋ฏ๋ก ์ด๊ฒ์ ๊ฒฝํํ๋ ๋๊ตฐ๊ฐ๊ฐ ์ฐ๊ฒฐ๋ ํ ๋ฆฌํ์คํธ ๋ธ๋์น์์ ์ ์ฌ์ ์ธ ์์ ์ ๊ฒ์ฆ ํ ์ ์๋ค๋ฉด ๋๋จํ ๊ฐ์ฌํ๊ฒ ์ต๋๋ค!
AWS์์ kops๋ก ์์ฑ ๋ ์ฌ๋ฌ ํด๋ฌ์คํฐ์์ ๋์ผํฉ๋๋ค.
ํ์ง๋ง K3S์์ ์คํํ ๋ ๋ฌธ์ ๊ฐ ์์ต๋๋ค.
@xvzf
์ด PR์์ ์ ์ฌ์ ์ธ ์์ ์ฌํญ์ ์๋ํด ๋ณผ ์ ์์ต๋๊น? https://github.com/helm/charts/pull/17090
๋๋ PR์ ์คํ์ํ์ง๋ง ์ฌ์ ํ Error: release prom failed: context canceled
tiller.log
@vsliouniaev Nope, ์ฌ๊ธฐ์ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ์ง ์์ต๋๋ค.
@xvzf ๋ฐ @pyadminn์ ํ์ธํด ์ฃผ์ ์ ๊ฐ์ฌํฉ๋๋ค. ๋์ผํ PR์์ ๋ ๋ค๋ฅธ ๋ณ๊ฒฝ์ํ์ต๋๋ค. ์ด๊ฒ์ด ๋์์ด๋๋์ง ๋ณผ ์ ์์ต๋๊น?
์
๋ฐ์ดํธ ๋ PR์ ํ์ธํ์ง๋ง ์ฌ์ ํ ์ธํ๋ผ์ ๋ค์๊ณผ ๊ฐ์ ๋ด์ฉ์ด ํ์๋ฉ๋๋ค. Error: release prom failed: rpc error: code = Canceled desc = grpc: the client connection is closing
์ฐธ๊ณ ๋ก ์ฐ๋ฆฌ๋ Kuber 1.14.3์ ์์ต๋๋ค.
Helm ๋ฒ์ v2.14.3
readme.md์ 'Helm์ด CRD๋ฅผ ๋ง๋ค์ง ๋ชปํจ'์น์ ์ ๋ฐ๋ผ์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ ์์์์ต๋๋ค. ๋๋ ๊ทธ๋ค์ด ์ด๋ป๊ฒ ๊ด๋ จ๋์ด ์๋์ง ํ์คํ์ง ์์ง๋ง ์๋ํ์ต๋๋ค.
1 ๋จ๊ณ : CRDS ์๋ ์์ฑ
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/podmonitor.crd.yaml
2 ๋จ๊ณ:
CRD๊ฐ ์์ฑ ๋ ๋๊น์ง ๊ธฐ๋ค๋ฆฌ์ญ์์ค. ๋ช ์ด ๋ฐ์ ๊ฑธ๋ฆฌ์ง ์์ต๋๋ค.
3 ๋จ๊ณ :
์ฐจํธ๋ฅผ ์ค์นํ๋ prometheusOperator.createCustomResource = false๋ฅผ ์ค์ ํ์ฌ CRD ํ๋ก๋น์ ๋์ ๋นํ์ฑํํฉ๋๋ค.
$ helm install --name my-release stable/prometheus-operator --set prometheusOperator.createCustomResource=false
@vsliouniaev ์ฌ์ ํ ๊ฐ์ ๋ฌธ์ ์ ๋๋ค! lethalwire์ ํด๊ฒฐ ๋ฐฉ๋ฒ์ด ์๋ํ์ง๋ง.
์น๋ช ์ ์ธ ๋ฌธ์ ํด๊ฒฐ ๋ฐฉ๋ฒ์ผ๋ก๋ ํด๊ฒฐ๋์์ต๋๋ค.
๊ทธ๋์ 4 ์ผ ๋์ ํด๊ฒฐ ๋ฐฉ๋ฒ์ด ์๋ํ๊ณ ์๋์ ๋ฉ์ท์ต๋๋ค. ๋ง์คํฐ๊ฐ ์๋ 0.32.0
์ CRD ํ์ผ์ ์ฌ์ฉํด์ผํ์ต๋๋ค.
๋๋ ํ์ฌ ๋ง์คํฐ์์๋ CRD์์ ๋์ผํ ๋ฌธ์ ๋ฅผ ๊ฒฝํํ์ต๋๋ค. ํ์ฌ ์ด์ ๋ฒ์ ์ ์ฌ์ฉํ๋๋ก ์ ์ ํด ์ฃผ์ @Typositoire ์๊ฒ ๊ฐ์ฌ๋๋ฆฝ๋๋ค. CRD ์ค์น๋ฅผ ๋ค์๊ณผ ๊ฐ์ด ์กฐ์ ํ๋ฉด ์ ์๊ฒ ํจ๊ณผ์ ์ด์์ต๋๋ค.
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/podmonitor.crd.yaml
๊ทธ๋ ๊ธฐ ๋๋ฌธ์ ๋ฒ์ ์ ์์ ํ๋ ๊ฒ์ด ์ข์ ๋ฐฉ๋ฒ์ ๋๋ค.
๋ํ์ด ๋ฌธ์ ๊ฐ ๋ฐ์ํ๋ฉด admissionWebhooks
๋ฅผ) ๋นํ์ฑํํ์ญ์์ค. ์ ๊ฒฝ์ฐ์ ๋์์ด๋์์ต๋๋ค.
prometheus-operator chart 6.0.0์ ์ค์นํ๊ณ helm ์ ๊ทธ๋ ์ด๋๋ฅผ ์ํํ์ญ์์ค .-- force --version 6.11.0, ์ด๊ฒ์ rancher kubernetes 1.13.10 ๋ฐ helm v2.14.3์์ ์๋ํ๋ ๊ฒ ๊ฐ์ต๋๋ค.
@Typositoire๊ฐ ์ ์ํ ํด๊ฒฐ ๋ฐฉ๋ฒ์ kops ์์ฑ 1.13.10 ํด๋ฌ์คํฐ์์ ์ ์๋ํ์ต๋๋ค.
kubernetes 1.13.10 ๋ฐ prometheus-operator-6.18.0์ ์ฌ์ฉํ์ฌ helm v2.14.3์ ์ฌ์ฉํ์ฌ Azure AKS์ ์ค์นํ๋ ค๋ ๋์ผํ ๋ฌธ์ ์ ๋๋ค. ์ด๋ ํ ์ ์?
CRD๊ฐ ์๋์ผ๋ก ์ค์น๋์์ต๋๋ค.
์ด ๋ช
๋ น์ด ์คํจํ์ต๋๋ค.
helm install --name prometheus-operator stable/prometheus-operator --namespace=monitoring --set prometheusOperator.createCustomResource=false
์ค๋ฅ๋ฅผ ์ฃผ๋ค
์ค๋ฅ : prometheus-operator ํด์ ์คํจ : rpc ์ค๋ฅ : ์ฝ๋ = ์ทจ์๋จ desc = grpc : ํด๋ผ์ด์ธํธ ์ฐ๊ฒฐ์ด ๋ซํ๋๋ค.
ํธ์ง : ์ฐจํธ์ ๋ฒ์ 6.11.0 (๋ฐ 6.7.3) ์ค์น๊ฐ ์๋ ์ค์ ๋๋ค.
helm install --name prometheus-operator stable/prometheus-operator --namespace=monitoring --set prometheusOperator.createCustomResource=false --version 6.11.0
์ ํ ์ปจํธ๋กค๋ฌ ์น ํํฌ๋ฅผ ๋นํ์ฑํ ํ์๊ฒ ์ต๋๊น?
https://waynekhan.net/2019/10/09/prometheus-operator-release-failed.html
2019 ๋ 10 ์ 15 ์ผ 19:32์ iMacX [email protected] ์ ๋ค์๊ณผ ๊ฐ์ด ์ผ์ต๋๋ค.
,,,,,,,,, ๋ฌด,,,,,,,,,,,,,,,.
kubernetes 1.13.10 ๋ฐ prometheus-operator-6.18.0์ ์ฌ์ฉํ์ฌ helm v2.14.3์ ์ฌ์ฉํ์ฌ Azure AKS์ ์ค์นํ๋ ค๋ ๋์ผํ ๋ฌธ์ ์ ๋๋ค. ์ด๋ ํ ์ ์?โ
์ด ์ค๋ ๋๋ฅผ ๊ตฌ๋ ํ๊ธฐ ๋๋ฌธ์์ด ๋ฉ์์ง๊ฐ ์ ์ก๋์์ต๋๋ค.
์ด ์ด๋ฉ์ผ์ ์ง์ ๋ต์ฅํ๊ฑฐ๋ GitHub์์ ํ์ธํ๊ฑฐ๋ ๊ตฌ๋ ์ ์ทจ์ํ์ธ์.
์ ๋ ๊ฐ์ ๋ฌธ์ ๋ฅผ ์ธ์ฐ๊ณ , ๋ด๊ฐ ์๋์ผ๋ก @JBosom์ ์ํด ์ง์ ๋ CRDS๋ฅผ ์ค์นํ๊ณ ์ฌ์ฉํ ์น ํํฌ๋ฅผ ์ค์นํ๋ค.
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/podmonitor.crd.yaml
helm --tls --tiller-namespace=tiller install --namespace=monitoring --name prom-mfcloud stable/prometheus-operator --set prometheusOperator.createCustomResource=false --set prometheusOperator.admissionWebhooks.enabled=false --values values.yaml --versi
on 6.18.0
v2.14.3
helm v8.0.0
์ (๋ฅผ) ์ค์นํ๋ ค๊ณ ํ๋ฉด ๋์ผํ ์ค๋ฅ๊ฐ ๋ฐ์ํ์ต๋๋ค. @lethalwire๊ฐ ์ ์ํ๋๋ก ๋จผ์ CRD๋ฅผ ๋ง๋ ํ์ ๋ง โโ์ค์นํ ์์์์ต๋๋ค.
๋๋ ์ด๊ฒ์ด prometheus-operator ์ฐจํธ์ ํน์ ๋ฌธ์ ๋ผ๊ณ ํ๋จํ๊ธฐ์ ์ถฉ๋ถํ ์ฌ๋ก๊ฐ ์๋ค๊ณ ์๊ฐํฉ๋๋ค.
์ฐ๋ฆฌ ์ชฝ์์๋ ์คํ ๊ฐ๋ฅํ ์๋ต์ด์๋ ๊ฒ์ผ๋ก ๋ง๋ฌด๋ฆฌํ๊ฒ ์ต๋๋ค. ๋ํ๋ฅผ ๊ณ์ ์งํํ ์ ์์ต๋๋ค.
๋ถํ์ํด์ ์ฃ์กํฉ๋๋ค. ์ต์ ํฌ๋ฉง v2.15.2
์
๊ทธ๋ ์ด๋ ํ ํ์๋ ๋ ์ด์์ด ์ค๋ฅ๊ฐ ํ์๋์ง ์์ต๋๋ค. ๐
Helm์์ ๋ฌด์จ ์ผ์ด ์ผ์ด๋๊ณ ์๋์ง์ ๋ํ ์ ๋ณด๊ฐ ์๋ค๋ ๊ฒ์ ๋งค์ฐ ์ด์ํด ๋ณด์ ๋๋ค.
์ฌ๊ธฐ์๋ ๋๋ฒ๊ทธ ๋ก๊ทธ๊ฐ ๊ฒ์๋์ง ์์๊ณ ์์ฒญ๋ ์์ผ๋ฉฐ ์ฌ๋๋ค์ ์ค์์น๋ฅผ ๋ค์ง๊ณ ๋์์ด๋๋์ง ํ์ธํ๊ณ ์์ต๋๋ค.
์ค๋ฅ๋ ์ค์ ๋ก ๋ฌด์์ ์๋ฏธํฉ๋๊น? ๋๊ธฐ๋ก ์ธํ ๊ต์ฐฉ ์ํ์ ํ์์ ๋๊น? ์ง๋จ์ ์ด๊นจ๋ฅผ ์ผ์ฑํ๋ ๊ฒ ์ธ์ ์ํ ํ ์์๋ ๋ค๋ฅธ ์กฐ์น๊ฐ ์์ต๋๊น?
์. ์น ํํฌ๋ฅผ ๋นํ์ฑํํ๋ฉด ์ฐจํธ๊ฐ ๋ฌธ์ ์์ด ์ค์น ๋ ์ ์์ผ๋ฏ๋ก ์๋ ์ค๋ฅ๋ ์น์ธ ์น ํํฌ๊ฐ ์๋ฃ๋๊ธฐ๋ฅผ ๊ธฐ๋ค๋ฆฌ๋ ๊ต์ฐฉ ์ํ๋ก ๋ํ๋ฉ๋๋ค. Tiller์ ๋ก๊ทธ๋ฅผ ๋ณด๋ฉด ๋ฌธ์ ๋ฅผ ํ์ธํ ์ ์์ต๋๋ค.
Helm 3๋ ๋ฏน์ค ํ์ ์์์ gRPC ๋ ์ด์ด๊ฐ์๊ณ ํ์ ์์์์ ์์ฒญ์ ์ทจ์ํ๊ธฐ ๋๋ฌธ์ ์ฌ์ฉ์์๊ฒ ์ฌ๋ฐ๋ฅธ ์ค๋ฅ๋ฅผ๋ณด๊ณ ํด์ผํฉ๋๋ค.
Helm 2์ ๋ํ ํจ์น๋ฅผ ์์ ๋กญ๊ฒ ์ ๊ณตํ์ญ์์ค. Helm 3์ ๋ํด ๊ฐ์ ๋์์ผ๋ฏ๋ก ๊ณ์ ์งํํ์ฌ ์ต์ ๋ฆด๋ฆฌ์ค์์ ์์ ๋๋๋ก ์ข ๋ฃํ์ต๋๋ค.
๋์์ด ๋์๊ธฐ๋ฅผ ๋ฐ๋๋๋ค.
์น ํํฌ๋ฅผ ๋นํ์ฑํํ๋ฉด ์ฐจํธ๊ฐ ๋ฌธ์ ์์ด ์ค์น ๋ ์ ์์ผ๋ฏ๋ก ์๋ ์ค๋ฅ๋ ์น์ธ ์น ํํฌ๊ฐ ์๋ฃ๋๊ธฐ๋ฅผ ๊ธฐ๋ค๋ฆฌ๋ ๊ต์ฐฉ ์ํ๋ก ๋ํ๋ฉ๋๋ค.
ํด๊ฒฐ์ฑ ์ ์์ ์ ๋นํ์ฑํํ๊ฑฐ๋ CRD ํํฌ ์ค์น๋ฅผ ๋นํ์ฑํํ๋ ๊ฒ์ด๊ธฐ ๋๋ฌธ์ ๊ฒฐ๋ก ์ผ๋ก โโ๊ฝค ์ด์ํ๊ฒ ๋ณด์ ๋๋ค. ์ด ๋ ๊ฐ์ง ๋ชจ๋ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๋ ๊ฒ์ผ๋ก ๋ณด์ด๋ฏ๋ก ์์ ๊ณผ ๊ด๋ จ๋ ๋ฌธ์ ๋ ์๋๋๋ค.
์ด ๋ฌธ์ ๋ฅผ ๊ฒช๊ณ ์๋ ๋ค๋ฅธ ์ฌ๋์๊ฒ-์ด๋ค ์์
์ด ์คํจํ๊ณ ์๋์ง ํ์ธํ ์ ์๋๋ก kubectl describe job
์ ์ถ๋ ฅ์ ์ ๊ณตํด ์ฃผ์๊ฒ ์ต๋๊น? ๋๋ ์ ์ ์ด๊ฒ์ ์์ฒญํ์ง๋ง ๋ชจ๋ ์ฌ๋๋ค์ด ์ผ์๋ฆฌ๊ฐ ์๋ค๋ ๊ฒ์ ๋ํ๋ด๋ ๊ฒ์ฒ๋ผ ๋ณด์
๋๋ค.
Tiller๋ ๋ค์๊ณผ ๊ฐ์ด ์ฝ์ต๋๋ค.
[kube] 2019/11/15 14:35:46 get relation pod of object: monitoring/PrometheusRule/prometheus-operator-node-time
[kube] 2019/11/15 14:35:46 Doing get for PrometheusRule: "prometheus-operator-kubernetes-apps"
[ A lot of unrelated updates in between... ]
2019/11/15 14:36:38 Cannot patch PrometheusRule: "prometheus-operator-kubernetes-apps" (rpc error: code = Canceled desc = grpc: the client connection is closing)
2019/11/15 14:36:38 Use --force to force recreation of the resource
[kube] 2019/11/15 14:36:38 error updating the resource "prometheus-operator-kubernetes-apps":
rpc error: code = Canceled desc = grpc: the client connection is closing
[tiller] 2019/11/15 14:36:38 warning: Upgrade "prometheus-operator" failed: rpc error: code = Canceled desc = grpc: the client connection is closing
[storage] 2019/11/15 14:36:38 updating release "prometheus-operator.v94"
[storage] 2019/11/15 14:36:38 updating release "prometheus-operator.v95"
[ then rollback... ]
๊ทธ๋์์ด ๋ฆฌ์์ค๋ฅผ ์๋์ผ๋ก ์ญ์ ํด์ผํ์ต๋๋ค. apiserver์๋ ๋ ๋ง์ ์ ๋ณด๊ฐ์์ ์ ์์ต๋๋ค (์ค์ ๋ก ์น์ธ ์ปจํธ๋กค๋ฌ์ ๊ด๋ จ๋ ๊ฒ์ฒ๋ผ ๋ค๋ฆผ).
@desaintmartin ์ด๊ฒ์ ์ค์น๊ฐ ์๋ ์ ๊ทธ๋ ์ด๋์์ ๋ฐ์ํ๋ ๊ฒ์ฒ๋ผ ๋ณด์ ๋๋ค.
Helm 3.0์ ์ด์ GA์ด๊ณ ์ฐจํธ๊ฐ ์๋ํ๋ฏ๋ก ๊ฑฐ๊ธฐ์์ ๋ฐ์ํ ์ ์๊ณ ๋ ๋์ ๋ก๊ทธ๋ฅผ ์ป์ ์ ์๋์ง๋ณด๊ณ ํ์ญ์์ค.
Helm3์ ์๊ณ Azure AKS์์ ์ฌ์ ํ์ด ์ค๋ฅ๊ฐ ๋ฐ์ํฉ๋๋ค.
๋๋ ์ฐจํธ v8.2.4์์ ์๋ํ๋ค : prometheusOperator.admissionWebhooks=false
, prometheus.tlsProxy.enabled=false
๋.
๋ํ vsliouniaev๊ฐ ๋งํ ๊ฒ์ฒ๋ผ --debug
๋ฐ --dry-run
๋ ๋ฌด์์ ๋งํฉ๋๊น?
@ truealex81 helm3์ ์ด์ ๋ํ ์์ธํ ์ ๋ณด๋ฅผ ์ ๊ณตํ๊ธฐ์ํ ๊ฒ์ด๋ฏ๋ก ์ค์น ํ๋ก์ธ์ค์์ ์์ธํ ๋ก๊ทธ๋ฅผ ๊ฒ์ ํ ์ ์์ต๋๊น?
Azure AKS์ 8.2.4๋ฅผ ๋ฐฐํฌํ๋ ๊ฒ๊ณผ ๋์ผํ ๋ฌธ์ ๊ฐ ๋ฐ์ํฉ๋๋ค.
Helm ๋ฒ์ :
version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}
Helm --debug๋ ๋ค์ ์ถ๋ ฅ์ ์์ฑํฉ๋๋ค.
install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /root/.cache/helm/repository/prometheus-operator-8.2.4.tgz
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
client.go:87: [debug] creating 1 resource(s)
install.go:139: [debug] Clearing discovery cache
wait.go:51: [debug] beginning wait for 5 resources with timeout of 1m0s
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:245: [debug] serviceaccounts "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:245: [debug] podsecuritypolicies.policy "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:245: [debug] rolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:245: [debug] roles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:245: [debug] clusterrolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:245: [debug] clusterroles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:245: [debug] jobs.batch "prometheus-operator-admission-create" not found
client.go:87: [debug] creating 1 resource(s)
client.go:420: [debug] Watching for changes to Job prometheus-operator-admission-create with timeout of 5m0s
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:484: [debug] prometheus-operator-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:87: [debug] creating 120 resource(s)
Error: context canceled
์ด๊ฒ์ ํ์คํ ์ฌํ ํ ์ ์์ต๋๋ค. ๋ ์์ธํ ๋ก๊ทธ๋ฅผ ์ป์ ์์๋ ๋ฐฉ๋ฒ์ด ์์ผ๋ฉด ์๋ ค์ฃผ์ธ์. ์ฌ๊ธฐ์ ์ถ๋ ฅ์ ๊ฒ์ํฉ๋๋ค.
@ pather87 ๊ฐ์ฌํฉ๋๋ค!
์ฐจํธ์์ ๋ฐ์ํ๋ ์์๋ ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
์คํจํ ์์ ์ด ์์ง ์๋์ง ํ์ธํด ์ฃผ์๊ฒ ์ต๋๊น? ๋ก๊ทธ์์ ๋ชจ๋ ์ฑ๊ณตํ๊ธฐ ๋๋ฌธ์ ๊ทธ๋ ๊ฒํ๋ฉด ์๋๋ ๊ฒ์ฒ๋ผ ์ฝ์ต๋๋ค.
Error: context canceled
๋ฐ์ ํ ํด๋ฌ์คํฐ์ ๋ค๋ฅธ ๋ฆฌ์์ค๊ฐ ์์ต๋๊น?
prometheus-operator๋ฅผ ์ค์นํ ๋๋ ๋ง์ฐฌ๊ฐ์ง์ ๋๋ค.
helm install prometheus-operator stable/prometheus-operator \
--namespace=monitoring \
--values=values.yaml
Error: rpc error: code = Canceled desc = grpc: the client connection is closing
@vsliouniaev ๋ต๋ณ ์ฃผ์ ์ ๊ฐ์ฌํฉ๋๋ค!
kubectl get all -lrelease = prometheus-operator
NAME READY STATUS RESTARTS AGE
pod/prometheus-operator-grafana-59d489899-4b5kd 2/2 Running 0 3m56s
pod/prometheus-operator-operator-8549bcd687-4kb2x 2/2 Running 0 3m56s
pod/prometheus-operator-prometheus-node-exporter-4km6x 1/1 Running 0 3m56s
pod/prometheus-operator-prometheus-node-exporter-7dgn6 1/1 Running 0 3m56s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/prometheus-operator-alertmanager ClusterIP xxx <none> 9093/TCP 3m57s
service/prometheus-operator-grafana ClusterIP xxx <none> 80/TCP 3m57s
service/prometheus-operator-operator ClusterIP xxx <none> 8080/TCP,443/TCP 3m57s
service/prometheus-operator-prometheus ClusterIP xxx <none> 9090/TCP 3m57s
service/prometheus-operator-prometheus-node-exporter ClusterIP xxx <none> 9100/TCP 3m57s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/prometheus-operator-prometheus-node-exporter 2 2 2 2 2 <none> 3m57s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/prometheus-operator-grafana 1/1 1 1 3m57s
deployment.apps/prometheus-operator-operator 1/1 1 1 3m57s
NAME DESIRED CURRENT READY AGE
replicaset.apps/prometheus-operator-grafana-59d489899 1 1 1 3m57s
replicaset.apps/prometheus-operator-operator-8549bcd687 1 1 1 3m57s
NAME READY AGE
statefulset.apps/alertmanager-prometheus-operator-alertmanager 1/1 3m44s
statefulset.apps/prometheus-prometheus-operator-prometheus 1/1 3m34s
๋๋ฒ๊ทธ๋ก ์ค์น :
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD alertmanagers.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD podmonitors.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD prometheuses.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD prometheusrules.monitoring.coreos.com is already present. Skipping.
client.go:87: [debug] creating 1 resource(s)
install.go:126: [debug] CRD servicemonitors.monitoring.coreos.com is already present. Skipping.
install.go:139: [debug] Clearing discovery cache
wait.go:51: [debug] beginning wait for 0 resources with timeout of 1m0s
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:245: [debug] clusterrolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:245: [debug] rolebindings.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:245: [debug] clusterroles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:245: [debug] serviceaccounts "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:245: [debug] podsecuritypolicies.policy "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:245: [debug] roles.rbac.authorization.k8s.io "prometheus-operator-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:245: [debug] jobs.batch "prometheus-operator-admission-create" not found
client.go:87: [debug] creating 1 resource(s)
client.go:420: [debug] Watching for changes to Job prometheus-operator-admission-create with timeout of 5m0s
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:484: [debug] prometheus-operator-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:445: [debug] Add/Modify event for prometheus-operator-admission-create: MODIFIED
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" RoleBinding
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ClusterRole
client.go:220: [debug] Starting delete for "prometheus-operator-admission" ServiceAccount
client.go:220: [debug] Starting delete for "prometheus-operator-admission" PodSecurityPolicy
client.go:220: [debug] Starting delete for "prometheus-operator-admission" Role
client.go:220: [debug] Starting delete for "prometheus-operator-admission-create" Job
client.go:87: [debug] creating 122 resource(s)
Error: context canceled
helm.go:76: [debug] context canceled
๊ทธ๋ฐ ๋ค์ kubectl get all -lrelease=prometheus-operator -A
์คํํฉ๋๋ค.
NAMESPACE NAME READY STATUS RESTARTS AGE
monitoring pod/prometheus-operator-grafana-d6676b794-r6cg9 2/2 Running 0 2m45s
monitoring pod/prometheus-operator-operator-6584f4b5f5-wdkrx 2/2 Running 0 2m45s
monitoring pod/prometheus-operator-prometheus-node-exporter-2g4tg 1/1 Running 0 2m45s
monitoring pod/prometheus-operator-prometheus-node-exporter-798p5 1/1 Running 0 2m45s
monitoring pod/prometheus-operator-prometheus-node-exporter-pvk5t 1/1 Running 0 2m45s
monitoring pod/prometheus-operator-prometheus-node-exporter-r9j2r 1/1 Running 0 2m45s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-system service/prometheus-operator-coredns ClusterIP None <none> 9153/TCP 2m46s
kube-system service/prometheus-operator-kube-controller-manager ClusterIP None <none> 10252/TCP 2m46s
kube-system service/prometheus-operator-kube-etcd ClusterIP None <none> 2379/TCP 2m46s
kube-system service/prometheus-operator-kube-proxy ClusterIP None <none> 10249/TCP 2m46s
kube-system service/prometheus-operator-kube-scheduler ClusterIP None <none> 10251/TCP 2m46s
monitoring service/prometheus-operator-alertmanager ClusterIP 10.0.238.102 <none> 9093/TCP 2m46s
monitoring service/prometheus-operator-grafana ClusterIP 10.0.16.19 <none> 80/TCP 2m46s
monitoring service/prometheus-operator-operator ClusterIP 10.0.97.114 <none> 8080/TCP,443/TCP 2m45s
monitoring service/prometheus-operator-prometheus ClusterIP 10.0.57.153 <none> 9090/TCP 2m46s
monitoring service/prometheus-operator-prometheus-node-exporter ClusterIP 10.0.83.30 <none> 9100/TCP 2m46s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
monitoring daemonset.apps/prometheus-operator-prometheus-node-exporter 4 4 4 4 4 <none> 2m46s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
monitoring deployment.apps/prometheus-operator-grafana 1/1 1 1 2m46s
monitoring deployment.apps/prometheus-operator-operator 1/1 1 1 2m46s
NAMESPACE NAME DESIRED CURRENT READY AGE
monitoring replicaset.apps/prometheus-operator-grafana-d6676b794 1 1 1 2m46s
monitoring replicaset.apps/prometheus-operator-operator-6584f4b5f5 1 1 1 2m46s
NAMESPACE NAME READY AGE
monitoring statefulset.apps/alertmanager-prometheus-operator-alertmanager 1/1 2m40s
monitoring statefulset.apps/prometheus-prometheus-operator-prometheus 1/1 2m30s
์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๋ ค๊ณ ์๋ํ๋ฉด์ ๋ฐ๊ฒฌ ํ ๋ด์ฉ : ๋์ค์ ์ฐจํธ์ CRD๋ฅผ ์ญ์ ํ๊ณ ์ฐจํธ๋ฅผ ๋ค์ ์ค์นํ๋ฉด ๋ฌธ์ ๊ฐ ์ง์๋์ง๋ง crd๋ฅผ ์ญ์ ํ์ง ์์ผ๋ฉด ๋ฌธ์ ๊ฐ ์ง์๋์ง ์์ต๋๋ค.
์ฌ์ ์ crds๋ฅผ ์๋ํ๊ณ ์ค์นํ๊ณ helm install --skip-crds
์ ์ํํ์ง๋ง ์ฌ์ ํ ๋ฌธ์ ๊ฐ ์ง์๋ฉ๋๋ค. ์ด๊ฒ์ ๋ค์ ํผ๋ ์ค๋ฝ์ต๋๋ค.
์ด ํ ์์๋๋ ๋ค์ ๋ก๊ทธ ์ค์ ์ค์น ํ, ์ ๊ทธ๋ ์ด๋ ํ ํํฌ์ ๊ดํ ๊ฒ์ด์ง๋ง ๊ทํ์ ๊ฒฝ์ฐ์๋ ๋ํ๋์ง ์์ต๋๋ค. ์ฌ๊ธฐ์์ ์ด๋ค ์กฐํ๊ฐ ๊ธฐ๋ค๋ฆฌ๊ณ ์๋์ง ์ ๋ชจ๋ฅด๊ฒ ์ต๋๋ค
...
lient.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" RoleBinding
client.go:245: [debug] rolebindings.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" Role
client.go:245: [debug] roles.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRole
client.go:245: [debug] clusterroles.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ServiceAccount
client.go:245: [debug] serviceaccounts "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRoleBinding
client.go:245: [debug] clusterrolebindings.rbac.authorization.k8s.io "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" PodSecurityPolicy
client.go:245: [debug] podsecuritypolicies.policy "prom-op-prometheus-operato-admission" not found
client.go:87: [debug] creating 1 resource(s)
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission-patch" Job
client.go:245: [debug] jobs.batch "prom-op-prometheus-operato-admission-patch" not found
client.go:87: [debug] creating 1 resource(s)
client.go:420: [debug] Watching for changes to Job prom-op-prometheus-operato-admission-patch with timeout of 5m0s
client.go:445: [debug] Add/Modify event for prom-op-prometheus-operato-admission-patch: MODIFIED
client.go:484: [debug] prom-op-prometheus-operato-admission-patch: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:445: [debug] Add/Modify event for prom-op-prometheus-operato-admission-patch: MODIFIED
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" RoleBinding
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" Role
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRole
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ServiceAccount
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" ClusterRoleBinding
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission" PodSecurityPolicy
client.go:220: [debug] Starting delete for "prom-op-prometheus-operato-admission-patch" Job
์๋ CRD ์์ฑ์ ์ต์ํ Azure์์ ๋์์ด๋ฉ๋๋ค.
๋จผ์ https://github.com/coreos/prometheus-operator/tree/release-0.34/example/prometheus-operator-crd ๋งํฌ์์ crd๋ฅผ ๋ง๋ญ๋๋ค.
"kubectl create -f alertmanager.crd.yaml"๋ฑ ๋ชจ๋ ํ์ผ์ ๋ํด
๊ทธ๋
helm install prometheus-operator stable / prometheus-operator --namespace monitoring --version 8.2.4 --set prometheusOperator.createCustomResource = false
๊ฐ์ฌํฉ๋๋ค @ truealex81 ! Azure์์ ์๋ํฉ๋๋ค.
myenv :
k8s 1.11.2 ํฌ๊ตฌ 2.13.1 ๊ฒฝ์ด๊ธฐ 2.13.1
prometheus-operator-5.5 APP VERSION 0.29๋ ๊ด์ฐฎ์ต๋๋ค !!!
๊ทธ๋ฌ๋:
prometheus-operator-8 APP VERSION 0.32 hava ๋์ผํ ๋ฌธ์ :
"์ปจํ
์คํธ ์ทจ์๋จ"๋๋ "grpc : ํด๋ผ์ด์ธํธ ์ฐ๊ฒฐ์ด ๋ซํ๋๋ค"!!!
prometheus-operator์ ์ต์ ๋ฒ์ ์ด ํธํ๋์ง ์๋ ๊ฒ ๊ฐ์์? !!!
@bierhov ์คํจ ํ ๋ค์ ์คํ์ด์ค์ ๋ฆฌ์์ค๋ฅผ ๊ฒ์ ํ ์ ์์ต๋๊น?
์!
shell execute "helm ls"๋ด prometheus-operator ๋ฆด๋ฆฌ์ค ์ํ๊ฐ "failed"์ธ ๊ฒ์ ๋ณผ ์ ์์ง๋ง prometheus-operator๊ฐ ์ค์น ํ ๋ค์ ์คํ์ด์ค์๋ ๋ชจ๋ prometheus-operator ๋ฆฌ์์ค๊ฐ ์์ต๋๋ค.
๊ทธ๋ฌ๋,
promethues ์น์ ๋ฐ์ดํฐ๋ฅผ ์ป์ ์ ์์ต๋๋ค!
๊ทธ๋๋ ๋ฆฌ์์ค๋ฅผ ๊ฒ์ ํ ์ ์์ต๋๊น?
๊ทธ๋๋ ๋ฆฌ์์ค๋ฅผ ๊ฒ์ ํ ์ ์์ต๋๊น?
๋ฏธ์ ํด์, ์์ ๋ ์กฐํ ํ๊ฒฝ์ ์ ๊ฑฐํ๊ณ ๋ค์ํ์ง ์์ผ๋ฉด ๋ค์ ๋ํ๋ ์ ์์ต๋๋ค!
@bierhov ์ค์น ํ ์คํจํ ์์ ์ด ๋จ์ ์์ต๋๊น?
@bierhov ์ค์น ํ ์คํจํ ์์ ์ด ๋จ์ ์์ต๋๊น?
๋ด k8s ๋ฒ์ ์ 1.11.2 helm์ด๊ณ ํ๋ณ ๋ฒ์ ์ 2.13.1์
๋๋ค.
prometheus-operator ๋ฒ์ 8.x๋ฅผ ์ค์นํ๋ฉด
shell exec ๋ช
๋ น "helm ls", ์์
์ํ๊ฐ ์คํจ ํจ
ํ์ง๋ง prometheus-operator ๋ฒ์ 5.x๋ฅผ ์ค์นํฉ๋๋ค.
shell exec ๋ช
๋ น "helm ls", ์์
์ํ๊ฐ ๋ฐฐํฌ๋์์ต๋๋ค !!!
๋ค์์ ์ฌ์ฉํ์ฌ ์ฌํ ํ ์ ์์ :
Kubernetes ๋ฒ์ : v1.13.12"
Kubectl ๋ฒ์ : v1.16.2
Helm ๋ฒ์ : 3.0.1
Prometheus-operator ๋ฒ์ : 8.3.3
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.34.0/example/prometheus-operator-crd/podmonitor.crd.yaml
--set prometheusOperator.createCustomResource=false
prometheusOperator: createCustomResource: false
์๋
ํ์ธ์.
CRD๋ฅผ ์๋์ผ๋ก ์์ฑํ์ง ์์ผ๋ฉด ์ด๋ป๊ฒ๋ฉ๋๊น? ์ด๊ฒ์ด ๋ฌธ์ ์ ๋ํ ํด๊ฒฐ ๋ฐฉ๋ฒ ์ค ํ๋์
๋๋ค.
@vsliouniaev๋ฅผ ๋ง๋ค์ง ์์ผ๋ฉด ์ค๋ฅ๊ฐ ๋ฐ์ํฉ๋๋ค.
๊ทธ๋ฌ๋ Additional Info์ ์๋ ๋ฌธ์ ์์ @rnkhouse ๋ CRD๋ฅผ ์๋์ผ๋ก ์์ฑํ๋ค๊ณ ๋งํ์ต๋๋ค.
๋ฐฐํฌ์ prometheus-operator๋ฅผ ์ฌ์ฉํฉ๋๋ค. ๊ฐ๋จํ ๋งํด์ prom-op์ 6.9.3์์ 8.3.3์ผ๋ก ์
๊ทธ๋ ์ด๋ํ์ผ๋ฉฐ ํญ์ "์ค๋ฅ : ์ปจํ
์คํธ ์ทจ์๋จ"์ผ๋ก ์คํจํ์ต๋๋ค.
๋ํ ์ฐ๋ฆฌ๋ ํญ์ prometheus-operator๋ฅผ ์ค์น / ์
๊ทธ๋ ์ด๋ํ๊ธฐ ์ ์ crd๋ฅผ ์ค์นํ๋ฉฐ, ofc๋ ์ด๋ฌํ crd-s๋ฅผ ๋ณ๊ฒฝํ๊ฑฐ๋ ์
๋ฐ์ดํธํ์ง ์์์ต๋๋ค.
'github.com/helm/charts/tree/master/stable/prometheus-operator'์์ ์ธ๊ธ ํ crds๋ฅผ ์๋ก ๊ณ ์น๋ ค๊ณ ํฉ๋๋ค (์ : kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator /master/example/prometheus-operator-crd/alertmanager.crd.yaml), ๊ทธ๋ฌ๋ ์ด๋ค์ ๋ ์ด์ ์กด์ฌํ์ง ์์ต๋๋ค.
๊ทธ ํ https://github.com/helm/charts/tree/master/stable/prometheus-operator/crds์์ ์ด๊ฒ๋ค์ ์๋ํฉ๋๋ค.
ํ์ง๋ง ๋ ์คํจํ์ต๋๋ค.
๊ฑฐ์ ํฌ๊ธฐํ ๋ปํ์ง๋ง์ด crds๋ก helm ๋ฐฐํฌ๊ฐ ์ฑ๊ณตํ์ต๋๋ค! ์์ด
https://github.com/coreos/kube-prometheus/tree/master/manifests/setup
๋ด ์ค์ :
Kubernetes ๋ฒ์ : v1.14.3
Kubectl ๋ฒ์ : v1.14.2
Helm ๋ฒ์ : 2.14.3
Prometheus-operator ๋ฒ์ : 8.3.3
k8s์์ prometheus-operator๋ฅผ ์ ๊ฑฐํ์ญ์์ค!
๊ทธ๋:
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0podmonitorCustomResourceDefinition.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0prometheusruleCustomResourceDefinition.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/setup/prometheus-operator-0servicemonitorCustomResourceDefinition.yaml
helm upgrade -i prom-op \
--version 8.3.3 \
--set prometheusOperator.createCustomResource=false \
stable/prometheus-operator
๊ทธ๊ฒ ๋ค์ผ !
์ด๊ฒ์ ์๋ก ์ค์น๋ฅผ ์ํํ๊ณ ๊ธฐ๋ก ๋ฉํธ๋ฆญ ๋ฐ์ดํฐ๋ฅผ ์์ด์ผ ํจ์ ์๋ฏธํฉ๋๊น?
AKS k8s๋ฅผ 1.15.5๋ก, helm์ 3.0.1๋ก, Prometheus-operator chart๋ฅผ 8.3.3์ผ๋ก ์ฌ๋ฆฌ๋ฉด ๋ฌธ์ ๊ฐ ์ฌ๋ผ์ก์ต๋๋ค.
ํด๊ฒฐ ๋ฐฉ๋ฒ์ v0.31.1์์ prometheus ์ด์์ ์ด๋ฏธ์ง๋ฅผ ์ ์งํ๋ ๊ฒ์ ๋๋ค.
AKS v1.14.8
๋ฐ helm + tiller v2.16.1
์์ ์ ๋ฅผ ์ํด ์ผํ๊ณ ์ด์์ ์ด๋ฏธ์ง๋ฅผ v0.31.1
์๋ CRD ์์ฑ์ ์ต์ํ Azure์์ ๋์์ด๋ฉ๋๋ค.
๋จผ์ https://github.com/coreos/prometheus-operator/tree/release-0.34/example/prometheus-operator-crd ๋งํฌ์์ crd๋ฅผ ๋ง๋ญ๋๋ค.
"kubectl create -f alertmanager.crd.yaml"๋ฑ ๋ชจ๋ ํ์ผ์ ๋ํด
๊ทธ๋
helm install prometheus-operator stable / prometheus-operator --namespace monitoring --version 8.2.4 --set prometheusOperator.createCustomResource = false
ํธ๋ฅธ kubernetes์์ ์๋ํฉ๋๋ค.
readme.md์ 'Helm์ด CRD๋ฅผ ๋ง๋ค์ง ๋ชปํจ'์น์ ์ ๋ฐ๋ผ์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ ์์์์ต๋๋ค. ๋๋ ๊ทธ๋ค์ด ์ด๋ป๊ฒ ๊ด๋ จ๋์ด ์๋์ง ํ์คํ์ง ์์ง๋ง ์๋ํ์ต๋๋ค.
1 ๋จ๊ณ : CRDS ์๋ ์์ฑ
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/podmonitor.crd.yaml
2 ๋จ๊ณ:
CRD๊ฐ ์์ฑ ๋ ๋๊น์ง ๊ธฐ๋ค๋ฆฌ์ญ์์ค. ๋ช ์ด ๋ฐ์ ๊ฑธ๋ฆฌ์ง ์์ต๋๋ค.3 ๋จ๊ณ :
์ฐจํธ๋ฅผ ์ค์นํ๋ prometheusOperator.createCustomResource = false๋ฅผ ์ค์ ํ์ฌ CRD ํ๋ก๋น์ ๋์ ๋นํ์ฑํํฉ๋๋ค.$ helm install --name my-release stable/prometheus-operator --set prometheusOperator.createCustomResource=false
๊ฐ์ฌํฉ๋๋ค. ์ด๊ฒ์ AKS ํด๋ฌ์คํฐ์์ ์ ์๊ฒ ํจ๊ณผ์ ์ด์์ต๋๋ค. CRD์ URL์ ๋ณ๊ฒฝํด์ผํ์ต๋๋ค.
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml --validate = false
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml --validate = false
helm install stable / prometheus-operator --name prometheus-operator --namespace monitoring --set prometheusOperator.createCustomResource = false
ํ์. ์ง๋ ์ธ ๋ช ์ ๋๊ธ ์์ฑ์์ ๋ฐ๋ฅด๋ฉด์ด ๋ฌธ์ ๋ ์ดํ ํด๊ฒฐ ๋ ๊ฒ ๊ฐ์ต๋๋ค. ๊ฐ์ฌ!
๊ฐ์ฅ ์ ์ฉํ ๋๊ธ
readme.md์ 'Helm์ด CRD๋ฅผ ๋ง๋ค์ง ๋ชปํจ'์น์ ์ ๋ฐ๋ผ์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ ์์์์ต๋๋ค. ๋๋ ๊ทธ๋ค์ด ์ด๋ป๊ฒ ๊ด๋ จ๋์ด ์๋์ง ํ์คํ์ง ์์ง๋ง ์๋ํ์ต๋๋ค.
1 ๋จ๊ณ : CRDS ์๋ ์์ฑ
2 ๋จ๊ณ:
CRD๊ฐ ์์ฑ ๋ ๋๊น์ง ๊ธฐ๋ค๋ฆฌ์ญ์์ค. ๋ช ์ด ๋ฐ์ ๊ฑธ๋ฆฌ์ง ์์ต๋๋ค.
3 ๋จ๊ณ :
์ฐจํธ๋ฅผ ์ค์นํ๋ prometheusOperator.createCustomResource = false๋ฅผ ์ค์ ํ์ฌ CRD ํ๋ก๋น์ ๋์ ๋นํ์ฑํํฉ๋๋ค.