μ¬μ© μ¬λ‘κ° μμ΅λλ€. Prometheusλ₯Ό StatefulSetμΌλ‘ λ°°ν¬νκ³ κ²½κ³ κ·μΉμ ConfigMapμΌλ‘ κ΅¬μ± ν©λλ€.
_alerting-rules_λ https://prometheus.io/docs/alerting/rules/#alerting -rulesμμ λ μμΈν μ 보λ₯Ό μ»μ μ μμ΅λλ€.
λ€μκ³Ό κ°μ΄ 보μ
λλ€.
ALERT InstanceMemoryOverload
IF node_memory_Active >= 1
FOR 1m
LABELS {
service = "k8s_metrics",
alertname = "InstanceMemoryOverload"
}
ANNOTATIONS {
summary = "Instance {{ $labels.instance }} memory overload",
description = "{{ $labels.instance }} memory overload for more than 1 minutes, now is {{ $value }}."
}
Helmμ΄ κ΅¬μ± μ©μ΄μ§λ§ λ λλ§μ΄ μλ {{expr}}μ 무μνλλ‘ μ§μν μ μλμ?
κ΅¬μ± λ§΅μ go μ€νμΌ ν νλ¦Ώμ μ¬μ©νλ μ μ¬ν μ¬μ© μ¬λ‘κ° μμ΅λλ€. νμ¬ helmμ ν νλ¦Ώμ νκ°νλ €κ³ μλνκ³ κ΅¬μ± λ§΅μμ μ€λ₯λ₯Ό λ°μμν΅λλ€.
μΌλ§ μ μμ΄ κ°μ / μ μ¬ν λ¬Έμ κ° λ°μνμ¬ κ²½κ³ κ·μΉμ λ³λμ νμΌλ‘ ν¬ν¨νμ¬ ν΄κ²°νμ΅λλ€.
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "fullname" . }}-rules
labels:
chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
role: {{ template "fullname" . }}-rulefiles
prometheus: {{ template "fullname" . }}
data:
{{ (.Files.Glob "files/kubernetes.rules").AsConfig }}
{{ (.Files.Glob "files/etcd2.rules").AsConfig }}
{{ (.Files.Glob "files/custom_alert.rules").AsConfig }}
AFAIK go ν
νλ¦Ώμμ {{ }}
μ΄μ€μΌμ΄ννλ κ²μ λ§€μ° μ΄λ ΅κ³ λ€λ₯Έ κ³³μμ νμΌμ νμ΄ λ³΄λ κ²μ ν¨μ¬ λ κ°λ¨ν©λλ€.
Go ν
νλ¦Ώμμ μ΄μ€ μ€κ΄νΈλ₯Ό μ΄μ€μΌμ΄ννλ λ°©λ²μ {{ "{{" }}
μ
λλ€. κ·Έκ²μ κ·Ήλλ‘ μ΄λ ΅μ§ μκ³ μΆμ
ν©λλ€.
λ¬Έμ μ λν λ κ°μ§ λ³λμ μ루μ μ΄ μμΌλ―λ‘μ΄ λ¬Έμ λ₯Ό μ’ λ£νκ² μ΅λλ€. λ¬Έμ κ° ν΄κ²°λμ§ μμ κ²½μ° λ€μμ¬μμμ€.
@bacongobbler μλ
νμΈμ Matthew, μ λ λ§ν¬ λ€μ΄μ Helm ν
νλ¦Ώμμ μ€κ΄νΈλ₯Ό μ΄μ€μΌμ΄ννλ λ°©λ²κ³Ό μΈμ°κ³ μμ΅λλ€. μλ§λ λΉμ μ ν΄κ²°μ±
μ μ΄ν΄νμ§ λͺ»ν μλ μμ΅λλ€. μ΄κ²μ μ¬λ°λ₯΄κ² λ λλ§νλ €λ©΄ μ΄λ»κ²ν΄μΌν©λκΉ?
(λ¨μ§ μ€ λν«). λλ raw / endraw, pre, single backslash, double backslashλ₯Ό μλνμ΅λλ€-μ±κ³΅νμ§ λͺ»νμ΅λλ€!
μ μ ν΄ μ£Όμ
μ κ°μ¬ν©λλ€ ... λ€μ¬ μ°κΈ° λλ λ°±ν±μΌλ‘ μ½λ λΈλ‘μ λ£μΌλ €κ³ ν©λλ€.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: {{ template "fullname" . }}
@bacongobbler ν΄κ²°μ±
μ μ°Ύμμ΅λλ€. preμ &123; &125;
λͺ¨λ μ¬μ©ν΄μΌν©λλ€.
@bacongobbler ν΄κ²°μ± μ μ°Ύμμ΅λλ€. preμ
&123; &125;
λͺ¨λ μ¬μ©ν΄μΌν©λλ€.
μλ₯Ό λ³΄μ¬ μ£Όμκ² μ΅λκΉ?
ν
νλ¦Ώ μμ§μμ νλμ yamlμ μ μΈνλ λ°©λ²μ΄ μμ΅λκΉ?
μ€ν κ°λ₯ν ν΄νΉμ Helmμ΄ ν νλ¦Ώμ μμ λ¬Έμμ΄λ‘ λ λλ§νλλ‘νλ κ²μ λλ€. λ°±ν±μ΄μλ μ¬λ μ€κ΄νΈμ λ«λ μ€κ΄νΈμ μ μνμμμ€.
# Excerpt from a Prometheus Alertmanager yaml
receivers:
- name: slack-receiver
slack_configs:
- text: |-
{{`{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
{{ end }}`}}
μ€ν κ°λ₯ν ν΄νΉμ Helmμ΄ ν νλ¦Ώμ μμ λ¬Έμμ΄λ‘ λ λλ§νλλ‘νλ κ²μ λλ€. λ°±ν±μ΄μλ μ¬λ μ€κ΄νΈμ λ«λ μ€κ΄νΈμ μ μνμμμ€.
# Excerpt from a Prometheus Alertmanager yaml receivers: - name: slack-receiver slack_configs: - text: |- {{`{{ range .Alerts }} *Alert:* {{ .Annotations.summary }} {{ end }}`}}
λλ¨ν κ°μ¬ν©λλ€. ν° λ¬Έμ κ° ν΄κ²°λμμ΅λλ€
μ΄λ₯Ό μν΄ printfλ₯Ό μ¬μ©ν μλ μμ΅λλ€.
{{ printf "{{ some value }}" }}
μ€ν κ°λ₯ν ν΄νΉμ Helmμ΄ ν νλ¦Ώμ μμ λ¬Έμμ΄λ‘ λ λλ§νλλ‘νλ κ²μ λλ€. λ°±ν±μ΄μλ μ¬λ μ€κ΄νΈμ λ«λ μ€κ΄νΈμ μ μνμμμ€.
# Excerpt from a Prometheus Alertmanager yaml receivers: - name: slack-receiver slack_configs: - text: |- {{`{{ range .Alerts }} *Alert:* {{ .Annotations.summary }} {{ end }}`}}
μλν©λλ€, thx
apiVersion : v1
μ’ λ₯ : ConfigMap
λ©ν λ°μ΄ν° :
μ΄λ¦ : {{ν νλ¦Ώ "fullname". }}-κ·μΉ
λΌλ²¨ :
μ°¨νΈ : "{{.Chart.Name}}-{{.Chart.Version | replace"+ ""_ "}}"
μν : {{template "fullname". }}-κ·μΉ νμΌ
prometheus : {{template "fullname". }}
λ°μ΄ν°:
{{(.Files.Glob "files / kubernetes.rules"). AsConfig}}
{{(.Files.Glob "files / etcd2.rules"). AsConfig}}
{{(.Files.Glob "files / custom_alert.rules"). AsConfig}}
μ΄κ²μ λλ₯Ό μν΄ μλνμ§ μμμ΅λλ€.
2 μΌ λμ 머리λ₯Ό λλλ¦° ν λ€μ μμ
μ ν μμμμ΅λλ€!
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "fullname" . }}-rules
labels:
chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
role: {{ template "fullname" . }}-rulefiles
prometheus: {{ template "fullname" . }}
data:
{{- (.Files.Glob "files/kubernetes.rules").AsConfig | nindent 2}}
μ€ν κ°λ₯ν ν΄νΉμ Helmμ΄ ν νλ¦Ώμ μμ λ¬Έμμ΄λ‘ λ λλ§νλλ‘νλ κ²μ λλ€. λ°±ν±μ΄μλ μ¬λ μ€κ΄νΈμ λ«λ μ€κ΄νΈμ μ μνμμμ€.
# Excerpt from a Prometheus Alertmanager yaml receivers: - name: slack-receiver slack_configs: - text: |- {{`{{ range .Alerts }} *Alert:* {{ .Annotations.summary }} {{ end }}`}}
μ΄κ²μ΄ μ¬μ ν μλνλ€λ νμΈμ ν©μ°½μ μΆκ° ν μ μμ΅λλ€.
κ·Έλ¦¬κ³ μ΄κ²μ μλμν€κΈ° μν΄ νμν μ μμ΅λλ€.
https://www.vaultproject.io/docs/platform/k8s/injector/examples/
cat <<EOF >> ./patch.yaml
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/db-app"
vault.hashicorp.com/agent-inject-template-db-creds: |
{{- with secret "database/creds/db-app" -}}
postgres://{{ .Data.username }}:{{ .Data.password }}<strong i="13">@postgres</strong>:5432/appdb?sslmode=disable
{{- end }}
vault.hashicorp.com/role: "db-app"
vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
vault.hashicorp.com/client-cert: "/vault/tls/client.crt"
vault.hashicorp.com/client-key: "/vault/tls/client.key"
vault.hashicorp.com/tls-secret: "vault-tls-client"
EOF
spec:
groups:
- name: alertrules.kafkalag
rules:
- alert: AssessmentAggregator lag
expr: sum(samza_pipeline_metrics_consumer_lag{job_name= "AssessmentAggregator"}) > {{ .Values.assessment_aggregator_threshold }}
for: 5m
labels:
severity: critical
annotations:
message: {{`"AssessmentAggregator lag is {{$value}}"`}}
summary: AssessmentAggregator lag is Critical
````
You can add
{{ .......
}}
```
μ λλ‘ ν
νλ¦Ώμ
μ€ν κ°λ₯ν ν΄νΉμ Helmμ΄ ν νλ¦Ώμ μμ λ¬Έμμ΄λ‘ λ λλ§νλλ‘νλ κ²μ λλ€. λ°±ν±μ΄μλ μ¬λ μ€κ΄νΈμ λ«λ μ€κ΄νΈμ μ μνμμμ€.
# Excerpt from a Prometheus Alertmanager yaml receivers: - name: slack-receiver slack_configs: - text: |- {{`{{ range .Alerts }} *Alert:* {{ .Annotations.summary }} {{ end }}`}}
λΆννλ μ΄κ²μ promtool
μ¬μ©νμ¬ μλνμ§ μμ΅λλ€. https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/
spec: groups: - name: alertrules.kafkalag rules: - alert: AssessmentAggregator lag expr: sum(samza_pipeline_metrics_consumer_lag{job_name= "AssessmentAggregator"}) > {{ .Values.assessment_aggregator_threshold }} for: 5m labels: severity: critical annotations: message: {{`"AssessmentAggregator lag is {{$value}}"`}} summary: AssessmentAggregator lag is Critical
μΆκ° ν μ μμ΅λλ€.
{{` ....... `}}
μ λλ‘ ν νλ¦Ώμ
λλ°!!!. ν¨κ³Όκ°μμμ΅λλ€. κ°μ¬.
κ°μ₯ μ μ©ν λκΈ
μ€ν κ°λ₯ν ν΄νΉμ Helmμ΄ ν νλ¦Ώμ μμ λ¬Έμμ΄λ‘ λ λλ§νλλ‘νλ κ²μ λλ€. λ°±ν±μ΄μλ μ¬λ μ€κ΄νΈμ λ«λ μ€κ΄νΈμ μ μνμμμ€.