Grafana: ๊ธฐ๋Šฅ: ์ƒ‰์ƒ ์ž„๊ณ„๊ฐ’์— ๋Œ€ํ•ด null์„ 0์œผ๋กœ ์ฒ˜๋ฆฌ

์— ๋งŒ๋“  2015๋…„ 07์›” 24์ผ  ยท  24์ฝ”๋ฉ˜ํŠธ  ยท  ์ถœ์ฒ˜: grafana/grafana

ํŠนํžˆ Singlestat ํŒจ๋„์˜ ๊ฒฝ์šฐ null์„ 0์œผ๋กœ ์ฒ˜๋ฆฌํ•˜๋Š” ๊ฒƒ์ด ์œ ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ ‡์ง€ ์•Š์œผ๋ฉด null ๊ฒฐ๊ณผ๋กœ ํŒจ๋„์„ ์ƒ‰์น ํ•  ๋ฐฉ๋ฒ•์ด ์—†์Šต๋‹ˆ๋‹ค.

๊ฐ€์žฅ ์œ ์šฉํ•œ ๋Œ“๊ธ€

OR ์—ฐ์‚ฐ์ž๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Prometheus์—์„œ null ์— ๋Œ€ํ•ด 0์„ ๋ฐ˜ํ™˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฉ”ํŠธ๋ฆญ ํ‘œํ˜„์‹์ด ์•„๋ฌด ๊ฒƒ๋„ ๋ฐ˜ํ™˜ํ•˜์ง€ ์•Š์œผ๋ฉด exression OR on() vector(0) ๋Š” 0์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. @pdf @torkelo

๋ชจ๋“  24 ๋Œ“๊ธ€

transformNull ํ•จ์ˆ˜๋Š” Graphite๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ ์ž˜ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค.

๋‚ด ๋ฐฑ์—”๋“œ์— ๋Œ€ํ•ด ์ง€๊ธˆ ๋ณผ ์ˆ˜ ์žˆ๋Š” ์ฑ„์šฐ๊ธฐ ๋ฉ”์ปค๋‹ˆ์ฆ˜์ด ์—†์Šต๋‹ˆ๋‹ค.

๋ฌด์—‡์ž…๋‹ˆ๊นŒ?

ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค

ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค์— ๋ณ€ํ˜• ๊ธฐ๋Šฅ์ด ์žˆ์Šต๋‹ˆ๊นŒ? ์•„์ง ์‹œ๋„ํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค(์ •๋ง ํ•˜๊ณ  ์‹ถ์€ ๋งˆ์Œ, Prometheus๋ฅผ ๊ณต์‹ ๋ฐ์ดํ„ฐ ์†Œ์Šค ํ”Œ๋Ÿฌ๊ทธ์ธ์œผ๋กœ ์ถ”๊ฐ€ํ•  ์ˆ˜ ์žˆ์Œ)

๊ฝค ๊ดœ์ฐฎ์€ ์„ ํƒ: http://prometheus.io/docs/querying/functions/

grafana 2.1์—์„œ 'null' ๊ฐ’์€ ์ฑ„์ƒ‰์— ๊ด€ํ•œ ํ•œ 0 ๊ฐ’์œผ๋กœ ์ฒ˜๋ฆฌ๋˜๋Š” ๊ฒƒ์ฒ˜๋Ÿผ ๋ณด์ž…๋‹ˆ๋‹ค(null์ด #1166์—์„œ์™€ ๊ฐ™์ด ํ…์ŠคํŠธ์— ๋งคํ•‘๋˜๋”๋ผ๋„).

์ข€ ๋” ๋„์ „์ ์œผ๋กœ ๋งŒ๋“ค๋ ค๋ฉด: ๋‚ฎ์„์ˆ˜๋ก ์ข‹์ง€๋งŒ 0/1/3์—์„œ ๋…น์ƒ‰/์ฃผํ™ฉ์ƒ‰/๋นจ๊ฐ„์ƒ‰์— ๋Œ€ํ•œ ์ž„๊ณ„๊ฐ’์„ ์„ค์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฆ‰, >3 ๊ฐ’์„ ์ฝ์Šต๋‹ˆ๋‹ค(ํ•˜์ง€๋งŒ null๋„ ๋นจ๊ฐ„์ƒ‰์ด์–ด์•ผ ํ•จ). ๋”ฐ๋ผ์„œ null ๊ฐ’์€ ๋…น์ƒ‰์ด ๋˜์ง€๋งŒ ๋นจ๊ฐ„์ƒ‰์ด์–ด์•ผ ํ•œ๋‹ค๊ณ  ์ฃผ์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•  ๋•Œ #1319์™€ ํ•จ๊ป˜ ๊ณ ๋ คํ•ด ๋ณผ ๊ฐ€์น˜๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด ํ…์ŠคํŠธ ๋งคํ•‘์— ๋Œ€ํ•œ ๋ฒ”์œ„๋ฅผ ๊ตฌํ˜„ํ•˜๊ณ  ์ด๋Ÿฌํ•œ ํ…์ŠคํŠธ์— ๋Œ€ํ•œ ์ƒ‰์ƒ ์ฝ”๋”ฉ์„ ์ง€์ •ํ•˜์—ฌ ๋™์‹œ์— ํ•ด๊ฒฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค(๊ฐ’์˜ ์ƒ‰์ƒ์€ ๊ฐ„์ ‘์ ์œผ๋กœ ์ง€์ •๋˜๋ฏ€๋กœ ๋” ์œ ์—ฐํ•˜์ง€๋งŒ ํ‹€๋ฆผ์—†์ด ๋” ๋ณต์žกํ•ฉ๋‹ˆ๋‹ค).

@torkelo @feliksik์— ๋™์˜ํ•˜๊ณ  ์ด๊ฒƒ์„ ๋ฒ„๊ทธ๋ผ๊ณ  ๋ถ€๋ฅด๊ธฐ๊นŒ์ง€ ํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋‹ˆํ„ฐ๋ง๋˜๋Š” ๋…ธ๋“œ ์ค‘ ํ•˜๋‚˜๊ฐ€ ์˜คํ”„๋ผ์ธ ์ƒํƒœ๊ฐ€ ๋˜๋ฉด ๋กœ๋“œ ํ‰๊ท ์„ ์ฟผ๋ฆฌํ•˜๋ฉด null์ด ๋ฐ˜ํ™˜๋ฉ๋‹ˆ๋‹ค. singlestat ํŒจ๋„์€ null์„ 0์œผ๋กœ ๋Œ€์ฒดํ•˜๊ธฐ ๋•Œ๋ฌธ์— ํŒจ๋„์€ ๋…น์ƒ‰์œผ๋กœ "N/A"๋ฅผ ํ‘œ์‹œํ•˜๋Š”๋ฐ, ์‹ค์ œ๋กœ๋Š” ๋นจ๊ฐ„์ƒ‰์ด๋‚˜ ๋ฌด์ƒ‰์ด์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค(๊ตฌ์„ฑ ์˜ต์…˜์— ๋”ฐ๋ผ ๋‹ค๋ฆ„).


ํŽธ์ง‘: ์ด ๊ธฐ๋Šฅ์€ ์ด์ „์— ์กด์žฌํ–ˆ์ง€๋งŒ(#1130) cc21c66b3a128ea680be2e491cb89f8da90b2677์—์„œ ์ œ๊ฑฐ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ํ† ๋ก ์„ #1130์œผ๋กœ ์ด๋™ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

+1 @n-st ๋ฐ @feliksik์— ๋™์˜ํ•ฉ๋‹ˆ๋‹ค. ํ˜„์žฌ ์šฐ๋ฆฌ ๋Œ€์‹œ๋ณด๋“œ๋Š” N/A์— ๋Œ€ํ•ด '๋นจ๊ฐ„์ƒ‰'์œผ๋กœ ํ‘œ์‹œ๋˜๊ณ  ์žˆ๋Š”๋ฐ ์‚ฌ์‹ค N/A๋Š” ๋ฐ์ดํ„ฐ ์—†์Œ/ํ™œ๋™ ์—†์Œ์„ ์˜๋ฏธํ•˜๋ฏ€๋กœ ์ด์ƒ์ ์œผ๋กœ๋Š” ๋ฌด์ƒ‰์ด์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

OR ์—ฐ์‚ฐ์ž๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Prometheus์—์„œ null ์— ๋Œ€ํ•ด 0์„ ๋ฐ˜ํ™˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฉ”ํŠธ๋ฆญ ํ‘œํ˜„์‹์ด ์•„๋ฌด ๊ฒƒ๋„ ๋ฐ˜ํ™˜ํ•˜์ง€ ์•Š์œผ๋ฉด exression OR on() vector(0) ๋Š” 0์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. @pdf @torkelo

๋Œ€๋ถ€๋ถ„์˜ TSDB ๋ฐฑ์—”๋“œ์—๋Š” ์ฟผ๋ฆฌ์—์„œ null์„ ์ฒ˜๋ฆฌํ•˜๋Š” ๋ฐฉ๋ฒ•์ด ์žˆ์œผ๋ฏ€๋กœ ์ด๊ฒƒ์„ ๋‹ซ์Šต๋‹ˆ๋‹ค.

@grobie

ํ‘œํ˜„์‹์ด null์ธ ๊ฒฝ์šฐ OR ์—ฐ์‚ฐ์ž๊ฐ€ ์ด๋ฅผ ์ˆ˜์ •ํ•˜๋Š” ๊ฒƒ์€ ์‚ฌ์‹ค์ž…๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ ๋‹จ์ผ ํ†ต๊ณ„ ํŽ˜์ด์ง€์—์„œ ์ด๊ฒƒ์€ null ๊ฐ’์˜ ์ƒ‰์ƒ์„ ์ˆ˜์ •ํ•˜์ง€๋งŒ null์ด ์•„๋‹Œ ์ธก์ •ํ•ญ๋ชฉ์˜ ๊ฒฝ์šฐ ์ด๋ฅผ ์ค‘๋‹จํ•ฉ๋‹ˆ๋‹ค. ์‹ค์ œ ๊ฐ’๊ณผ prometheus ํ‘œํ˜„์‹์— ์˜ํ•ด ์ถ”๊ฐ€๋œ {} ๊ฐ’์ด ์—ฌ๋Ÿฌ ๊ฐœ ์žˆ๋Š” ๊ฒƒ์œผ๋กœ ๋ณด๊ณ ๋˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค.

์ด ๋ฌธ์ œ๋ฅผ ๋‹ค์‹œ ์—ด๋„๋ก ์ œ์•ˆํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

@boeboe ์œ„์˜ ๋Œ“๊ธ€์—์„œ ๋‚ด ํ‘œํ˜„์„ ์—…๋ฐ์ดํŠธํ–ˆ์Šต๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ๋‹น์‹ ์„ ์œ„ํ•ด ์ž‘๋™ํ•ด์•ผํ•ฉ๋‹ˆ๋‹ค.

์„ค๋ช…: Prometheus๋Š” ํ‘œํ˜„์‹์—์„œ ๋ ˆ์ด๋ธ” ์ผ์น˜๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. expression ๊ฐ€ ๋ ˆ์ด๋ธ”์ด ์žˆ๋Š” ํ•ญ๋ชฉ์„ ๋ฐ˜ํ™˜ํ•˜๋Š” ๊ฒฝ์šฐ vector(0) ์ƒ์„ฑํ•œ ์‹œ๊ณ„์—ด๊ณผ ์ผ์น˜ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ด๋ฅผ ๊ฐ€๋Šฅํ•˜๊ฒŒ ํ•˜๋ ค๋ฉด on() ๋ฅผ ์ถ”๊ฐ€ํ•˜์—ฌ ๋ ˆ์ด๋ธ”๊ณผ ์ผ์น˜ํ•˜์ง€ ์•Š๋„๋ก ๋ช…์‹œ์ ์œผ๋กœ Prometheus์— ์ง€์‹œํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

@grobie๋‹˜ , ์„ค๋ช… ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค. ๋ถˆํ–‰ํžˆ๋„ Prometheus ๋ฌธ์„œ๋Š” on() ์—ฐ์‚ฐ์ž์˜ ์‚ฌ์šฉ์— ๋Œ€ํ•œ ๋ช…ํ™•ํ•œ ์˜ˆ๋ฅผ ์ œ๊ณตํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค(https://prometheus.io/docs/querying/operators)

๋‚ด ํ˜„์žฌ ์ฟผ๋ฆฌ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.
counter_services_active_sessions_responses_4xx_total{kubernetes_namespace="dev-appservicebase"} OR vector(0)

๋ฉ”ํŠธ๋ฆญ์ด ์กด์žฌํ•˜๋Š” ๊ฒฝ์šฐ(์ด ๊ฒฝ์šฐ ๋งˆ์ดํฌ๋กœ์„œ๋น„์Šค์— HTTP 4xx ์‘๋‹ต์ด ๋ฐ˜ํ™˜๋จ) ์ค‘๋ณต์„ ํ”ผํ•˜๊ธฐ ์œ„ํ•ด ์ด๊ฒƒ์„ ๋‹ค์‹œ ์ž‘์„ฑํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ํ˜„์žฌ ์ฟผ๋ฆฌ๋Š” ๋‹ค์Œ 2๊ฐœ์˜ ๊ฒฐ๊ณผ๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

counter_services_active_sessions_responses_4xx_total{instance="172.24.27.130:8081",job="kubernetes-pods",kubernetes_namespace="dev-appservicebase",kubernetes_pod_name="active-sessions-dp-32102991495" -24-24",pod_template_hash="3210291495",type="ํ™œ์„ฑ ์„ธ์…˜ ์œ ํ˜•"} 1173
{} 0

@grobie๋Š” ๋‚ด ๋งˆ์ง€๋ง‰ ์˜๊ฒฌ์„

๋ˆ„๊ตฌ๋“ ์ง€ MySQL์— ๋Œ€ํ•œ ์ฟผ๋ฆฌ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•์ด ์žˆ์Šต๋‹ˆ๊นŒ?

@TheFrogDaddy ๋ณ‘ํ•ฉ ๊ธฐ๋Šฅ ๊ณผ ํ•จ๊ป˜ ๋ฐ/๋˜๋Š” ํ•จ๊ป˜ ์„ ํƒ ์ผ€์ด์Šค๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

@TheFrogDaddy ๋ณ‘ํ•ฉ ๊ธฐ๋Šฅ ๊ณผ ํ•จ๊ป˜ ๋ฐ/๋˜๋Š” ํ•จ๊ป˜ ์„ ํƒ ์ผ€์ด์Šค๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์•ˆ๋…•, ๋นจ๋ฆฌ ๋Œ์•„์™€์ค˜์„œ ๊ณ ๋งˆ์›Œ. ๋‚˜๋Š” ์ด๊ฒƒ์ด ์ž‘๋™ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋‚ด๊ฐ€ ๋ญ”๊ฐ€๋ฅผ ๋†“์น˜๊ณ  ์žˆ์Šต๋‹ˆ๊นŒ?

์ด๊ฒƒ์„ ์‹œ๋„:

SELECT $__timeGroup(updated, '1h', 0) AS time, COALESCE(SUM(1),0) as value, 'Errors' AS metric FROM mytable WHERE $__timeFilter(updated) AND error_message IS NOT NULL GROUP BY 1;

์ด

SELECT $__timeGroup(updated, '1h', 0) AS time, CASE ISNULL(SUM(1)) WHEN TRUE THEN 0 ELSE SUM(1) END AS value, 'Errors' AS metric FROM myTable WHERE $__timeFilter(updated) AND error_message IS NOT NULL GROUP BY 1;

๊ฐ์‚ฌ ํ•ด์š”

@TheFrogDaddy sum(1) ๋กœ ์š”์•ฝํ•˜๋ ค๋Š” ๊ฒƒ์€ ๋ฌด์—‡์ž…๋‹ˆ๊นŒ? ๋‚˜์—๊ฒŒ ์˜๋ฏธ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค. ์ปค๋ฎค๋‹ˆํ‹ฐ ์‚ฌ์ดํŠธ , ํŠนํžˆ mysql ํƒœ๊ทธ์— ๋Œ€ํ•œ ํ† ๋ก ์„ ๊ณ„์†ํ•  ๊ฒƒ์„ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ๊ฑฐ๊ธฐ์—์„œ ์ƒˆ ์ฃผ์ œ๋ฅผ ์—ด๊ณ  @ + mefraimsson์œผ๋กœ ์ €์—๊ฒŒ ํ•‘์„ ๋ณด๋‚ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

OR ์—ฐ์‚ฐ์ž๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Prometheus์—์„œ null ์— ๋Œ€ํ•ด 0์„ ๋ฐ˜ํ™˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฉ”ํŠธ๋ฆญ ํ‘œํ˜„์‹์ด ์•„๋ฌด ๊ฒƒ๋„ ๋ฐ˜ํ™˜ํ•˜์ง€ ์•Š์œผ๋ฉด exression OR on() vector(0) ๋Š” 0์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. @pdf @torkelo

ํ™œ์„ฑ ์ธ์Šคํ„ดํŠธ ์˜ต์…˜์„ ์žŠ์ง€ ๋งˆ์„ธ์š” !!

๋ฉ‹์ง€๋‹ค

OR ์—ฐ์‚ฐ์ž๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Prometheus์—์„œ null ์— ๋Œ€ํ•ด 0์„ ๋ฐ˜ํ™˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฉ”ํŠธ๋ฆญ ํ‘œํ˜„์‹์ด ์•„๋ฌด ๊ฒƒ๋„ ๋ฐ˜ํ™˜ํ•˜์ง€ ์•Š์œผ๋ฉด exression OR on() vector(0) ๋Š” 0์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. @pdf @torkelo

์ข‹์€. ํ˜•์ œ

OR ์—ฐ์‚ฐ์ž๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Prometheus์—์„œ null ์— ๋Œ€ํ•ด 0์„ ๋ฐ˜ํ™˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฉ”ํŠธ๋ฆญ ํ‘œํ˜„์‹์ด ์•„๋ฌด ๊ฒƒ๋„ ๋ฐ˜ํ™˜ํ•˜์ง€ ์•Š์œผ๋ฉด exression OR on() vector(0) ๋Š” 0์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. @pdf @torkelo

์ด๊ฒƒ์€ by (label) ์—์„œ๋Š” ์ž‘๋™ํ•˜์ง€ ์•Š์ง€๋งŒ ์ž‘๋™ํ•˜๋Š” ๊ฒƒ์ฒ˜๋Ÿผ ๋ณด์ž…๋‹ˆ๋‹ค.

expression OR up*0
sum by (label) (data_metric) - sum by (label) (nodata_metric or up*0)

์—ฌ๊ธฐ ๋‚ด 2์„ผํŠธ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

sum(expression) by (label) -
sum(expression offset $__interval OR expression * 0) by (label)

์ด๊ฒƒ์€ ๋ชฉํ‘œ ๊ฐ„๊ฒฉ์—์„œ 2 ํ‘œํ˜„์‹์˜ ์ฐจ์ด๋ฅผ ์ค„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‹œ๊ณ„์—ด์ด ์—†์œผ๋ฉด ๊ฐ’์„ 0์œผ๋กœ ๊ฐ„์ฃผํ•˜์—ฌ ๊ณ„์‚ฐ์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

์ด ํŽ˜์ด์ง€๊ฐ€ ๋„์›€์ด ๋˜์—ˆ๋‚˜์š”?
0 / 5 - 0 ๋“ฑ๊ธ‰