Grafana: Alerting: Elasticsearch support

ref https://github.com/grafana/grafana/issues/5858

Since #5948 is merged is this issue closed?

@luigiberrettini its not merged to master just to the alerting-elasticsearch branch. It far from completed yet.

not more far then graphite support ;)
missing graphite query features in alerting:

• template variables
• functions referencing other queries (e.g. asPercent)

Hi, will complete elasticsearch support come after the 4.0.0 deliverable (planned for November), because I see this issue marked with the 4.1.0 milestone?
If so: when is 4.1.0 planned for delivery?

4.1 does not have a release date yet, but would guess January / early feb.

Elasticsearch alerting might not make it into Grafana v4, we tried (the work is started), but requires a lot more work to complete and some more high prio issues has pushed it out of v4

In that case, would it be possible to enable/disable the alert-menu items and panel-tabs based on an Organization preference (or even better: on individual user level definitions within the organization)?

Has there been any progress on this? Is the elasticsearch-alerting branch still active? I am wondering if there is something that I could start testing and giving feedback on.

Looking forward to this getting into 4.2 so we can fully move to grafana + elasticsearch (metricbeat).

Having this functionality would be a huge win for my company

@andytsnowden enough to buy a support plan? https://grafana.net/support/plans :stuck_out_tongue_winking_eye:

We hope get a chance to continue work on this soon.

Maybe silly (please don't answer "yes, you are") but what about kind of feature-crowdfunding ? sure many will support

This was the whole reason I downloaded this tool, in hopes of having alerting for elastic.

There is a branch which was at least working for most queries. But it is a bit outdated - you would need to align some things.

Von meinem Samsung Gerät gesendet.

-------- Ursprüngliche Nachricht --------
Von: wirecutter313 notifications@github.com
Datum: 30.01.2017 20:56 (GMT+01:00)
An: grafana/grafana grafana@noreply.github.com
Cc: "A. Binzxxxxxx" alexander@binzberger.de, Manual manual@noreply.github.com
Betreff: Re: [grafana/grafana] Alerting: Elasticsearch support (#5893)

This was the whole reason I downloaded this tool, in hopes of having alerting for elastic.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/grafana/grafana","title":"grafana/grafana","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/grafana/grafana"}},"updates":{"snippets":[{"icon":"PERSON","message":"@wirecutter313 in #5893: This was the whole reason I downloaded this tool, in hopes of having alerting for elastic. "}],"action":{"name":"View Issue","url":"https://github.com/grafana/grafana/issues/5893#issuecomment-276172036"}}}

Is there a strong chance this is going to make the 4.2.0 mark? Trying to come up with temporary alternatives for the alerting part while we wait for this.

its not looking like that. No one is working on it that we know and we have a lot on our plate right now.

That's pretty unfortunate! Considering using Graphite as a backend, until we can support ES. But trying to understand what limitations that in terms of the queries (and thus alerts) and such, as opposed to ES

Is anyone aware of any way of getting some kind of alerting into Grafana whilst having ES as a data source? Or for anyone who also uses Kibana to visualise data before Grafana are you aware of any solutions to have alerting on this side of things?

we are working on it in this branch, https://github.com/grafana/grafana/tree/alerting-elasticsearch

Check out the experimental branch. You might need to align some things to get it working again.

Von meinem Samsung Gerät gesendet.

-------- Ursprüngliche Nachricht --------
Von: Alexander Payne notifications@github.com
Datum: 22.02.2017 15:48 (GMT+01:00)
An: grafana/grafana grafana@noreply.github.com
Cc: "A. Binzxxxxxx" alexander@binzberger.de, Manual manual@noreply.github.com
Betreff: Re: [grafana/grafana] Alerting: Elasticsearch support (#5893)

Is anyone aware of any way of getting some kind of alerting into Grafana whilst having ES as a data source? Or for anyone who also uses Kibana to visualise data before Grafana are you aware of any solutions to have alerting on this side of things?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/grafana/grafana","title":"grafana/grafana","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/grafana/grafana"}},"updates":{"snippets":[{"icon":"PERSON","message":"@Alexander-Payne in #5893: Is anyone aware of any way of getting some kind of alerting into Grafana whilst having ES as a data source? Or for anyone who also uses Kibana to visualise data before Grafana are you aware of any solutions to have alerting on this side of things?"}],"action":{"name":"View Issue","url":"https://github.com/grafana/grafana/issues/5893#issuecomment-281689640"}}}

What's a rough estimate of how far you are along with this? 50%? 90%?

Hello
We are very interested about this feature.
Is there a release date?

Thanks

Thumbs up for this feature and good work!

Thank you guys for the work on this one. Will be huge when it's ready!

Was super happy to upgrade Grafana to 4.1.2 just to realize that ES data source is not supported :(. I guess I have to stick with Powershell script for alerting for now.

Guys, it will be nice to have Elastic alerting in Grafana - that is the fact. But obviously it will not happen faster if every one will leave exactly the same comment here ;)
And obviously alert rules will be quite unsophisticated, at least at first, as they are now for metrics. Most likely simple threshholds.
If you NEED alerting for ElasticSearch - use ElastAlert. It's there for long time, support all Elastic versions and quite complex rules.

Unfortunately, ElastAlert does not support aggregation, The only aggregation supported currently is a terms aggregation, by setting use_terms_query. But I'm glad that Alert finally comes to Grafana.

Hi, can I know when this would be completed please? I am eagerly looking forward to alerting from elasticsearch in grafana

People, please stop spamming this issue with pointless comments. It will be done when it's done. If you want to contribute, by all means submit a PR to improve the implementation.

If you wish to show your support for the issue, use the reaction button on the first post. If you wish to receive an update when the issue is completed, the subscribe button is on the right.

Making a useless comment spams _every_ subscriber on the issue, and won't cause it to be resolved any faster, so please stop (apologies for my contribution the noise here everyone!).

I got "tsdb.HandleRequest() error type assertion to string failed" on alerting-elasticsearch branch
(Same problem as #7909)
Grafana v4.2.0-pre1
Datasource : Elasticsearch version 2.x and 5.x (both error)

json from query

{"search_type":"count","ignore_unavailable":true,"index":"moa-log-alias"}
{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1492718034259","lte":"1492719834259","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"type:\"oauth2_request_log\""}}]}},"aggs":{"3":{"terms":{"field":"method","size":500,"order":{"_term":"desc"},"min_doc_count":0},"aggs":{"2":{"date_histogram":{"interval":"1m","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":"1492718034259","max":"1492719834259"},"format":"epoch_millis"},"aggs":{}}}}}}

Test Request payload

{"dashboard":{"annotations":{"list":[]},"editMode":false,"editable":true,"gnetId":null,"graphTooltip":0,"hideControls":false,"id":15,"links":[],"refresh":"10s","rows":[{"collapse":false,"height":420,"panels":[{"alert":{"conditions":[{"evaluator":{"params":[150],"type":"gt"},"operator":{"type":"and"},"query":{"params":["A","1m","now"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"executionErrorState":"alerting","frequency":"60s","handler":1,"name":"API Requests / Min alert","noDataState":"no_data","notifications":[]},"aliasColors":{},"bars":true,"datasource":"moa-log","decimals":null,"fill":1,"hideTimeOverride":false,"id":1,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":true,"min":true,"rightSide":false,"show":true,"sort":"total","sortDesc":true,"total":true,"values":true},"lines":false,"linewidth":1,"links":[{"targetBlank":true,"title":"View Detail","type":"absolute","url":"http://kibana.exe.in.th/goto/ff87151449b8ed32d9492a59701b2a56"}],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"span":6,"stack":true,"steppedLine":false,"targets":[{"bucketAggs":[{"fake":true,"field":"method","id":"3","settings":{"min_doc_count":0,"order":"desc","orderBy":"_term","size":"0"},"type":"terms"},{"field":"@timestamp","id":"2","settings":{"interval":"1m","min_doc_count":0,"trimEdges":0},"type":"date_histogram"}],"dsType":"elasticsearch","metrics":[{"field":"select field","id":"1","type":"count"}],"query":"type:\"oauth2_request_log\"","refId":"A","timeField":"@timestamp"}],"thresholds":[{"colorMode":"critical","fill":true,"line":true,"op":"gt","value":150}],"timeFrom":null,"timeShift":"30s","title":"API Requests / Min","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"ops","label":null,"logBase":1,"max":null,"min":null,"show":false}]}],"repeat":null,"repeatIteration":null,"repeatRowId":null,"showTitle":false,"title":"Dashboard Row","titleSize":"h6"}],"schemaVersion":14,"style":"dark","tags":[],"templating":{"list":[]},"time":{"from":"now-30m","to":"now"},"timepicker":{"refresh_intervals":["1s","5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"test alert","version":0},"panelId":1}

we are also started testing elasticsearch alerting using alerting-elasticsearch branch and we see same issue what @kerlos is seeing.

Grafana v4.2.0
Datasource : Elasticsearch 5.x

from logs:

EROR[05-03|23:12:01] Alert Rule Result Error logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error type assertion to string failed" changing state to=alerting
EROR[05-03|23:13:01] Alert Rule Result Error logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error type assertion to string failed" changing state to=alerting
EROR[05-03|23:14:01] Alert Rule Result Error logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error type assertion to string failed" changing state to=alerting
EROR[05-03|23:15:00] Alert Rule Result Error logger=alerting.evalHandler ruleId=0 name=Test error="tsdb.HandleRequest() error type assertion to string failed" changing state to=alerting
EROR[05-03|23:15:01] Alert Rule Result Error logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error type assertion to string failed" changing state to=alerting
EROR[05-03|23:16:01] Alert Rule Result Error logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error type assertion to string failed" changing state to=alerting
EROR[05-03|23:17:01] Alert Rule Result Error logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error type assertion to string failed" changing state to=alerting

Hi there,

Just two questions:

• I see that the last commit in the alerting-elasticsearch branch is from Feb 2017. Is it already mature enough to fit into 4.4.0?
• Does this alerting module require X-Pack on the elasticsearch server? If that's the case, I would like to propose another candidate for the list of supported alert datasources: elastalert.

• It's not mature I think (have not tested it much).
• X-Pack is not required.

Are we getting any closer for this?

We are waiting for this figure for several months. When released?

How could I patch this to grafana_4.3.2 ?? it is really important & useful

Please do not discard this feature.

We are desperately waiting for this feature..

https://github.com/grafana/grafana/pull/8934 this is my edition about this issue. hope to be helpful

Could it be merged to one of nearest releases ?

Elastic is still not supported? @torkelo

Should we expect to have this released soon?

Why no answer @playqdrew

Please include the feature to the nearest release. It would be really cool!

@lvheyang I've pulled your changes locally and it appears the Alerting is not respecting the Size set in the Metric Query. I've set the Size to no limit but the alerting only evaluates 10 objects and the 10 evaluated seem to be random as I've set order to Bottom on my query showing the lowest values on top of my legend but none of those bottom values are evaluated. Maybe, this is a limitation or I'm doing something incorrectly? What do you think?

@dustinvanbuskirk I'd be glad to help you figuring this out. Maybe if the work was _largely_ done we can just finish

Hi guys,
Any updates on this ? Waiting for this functionality for over year ....

Really sorry that this always ends up being pushed, a bit overwhelmed by other issues & making progress on Grafana v5 (new grid, user groups & dashboard folders). Can't promise when we can get this in, hopefully in v5.1 (early Q1 2018)

Hoping to have it soon. Fingers crossed !

We'd benefit from this too :)

+1 Also missing this feature

Hey @torkelo,
I'm just curious if you made any progress in elasticsearch alerting ? :)

KR

+1

Q1 almost end..
so waiting for this Elastic alerting to be ready

I'm curious - do ELK "watcher" feature that can do elastic alerting which is also one of the main feature in payed X-Pack have something to do with stalling the progress on this feature?

@yossiv @MichaelLogutov @vijaychd @Shiinii https://www.bountysource.com/

i still wait for the "alert feature for es". it is import for me .but i can't see some more advance about this issue

• 1 would be very useful

I'd still trade all these newfangled dashboard rearrangements for this

I would also love to see ES alerting on Grafana.

Any timeline when it will be ready ?

I wouldn't be waiting for this functionality, given that the ticket has been open for 18 months and there appears to be no developers working on it for over a year. If you need alerting, I'd suggest you look at migrating to a backend that supports it.

So... making likes to issues is totally useless? This is the third issue sorted by votes... it's disappointing to see it ignored.

I'd suggest you look at migrating to a backend that supports it.

@HeWhoWas I would but they're all terrible.

We created our own alerting solution in a durable azure function instead, continuously polling the logs.

^ right, for basic use cases this definitely isn't rocket science. Just go to Kibana and run your query, then copy the raw Elasticsearch query that it generates. Then run some code like this (incomplete, but you get the idea):

# Get results from ES
result=$(
    curl -sS -X POST \
    "http://${ES_HOST}:9200/logstash-*/_search" \
    -H "cache-control: no-cache" \
    -H "content-type: application/json" \
    -d @/app/es_query.json
)
count=$(echo ${result} | jq -r '.hits.total')
echo "[DEBUG] Found ${count} hits"

# Send alert if necessary
if [ "${count}" -ge "${COUNT_THRESHOLD}" ]; then
    echo "[INFO] Found ${count} hits on search, which is over the threshold"

    alert_text="{\"text\": \"[${APP_NAME}] Found ${count} *${ALERT_TEXT}* events in the last ${REPORTING_PERIOD}. See ${INSPECT_LINK} .\"}"
    echo ${alert_text} | curl "${SLACK_HOOK_URL}" -d @-
fi

moving from Elasticsearch to other backend system is not an option and not a solution.
its a core system and familiar same like all other systems that already supported by Grafana alerting.
they promise it will be on 5.1 ..
i really hope it will be on time

Yeah or just use x-pack monitoring, or pass them through Riemann and alert that way.

Alerting isn't the issue, having the neat Grafana functionality of visualisation and tweaking it is the desire.

All your solutions above are basically "forget grafana" - well this is the grafana project, and this issue is here to improve it. So lets focus on solutions that do that.

This PR is huge, and partly because it includes an entire 3rd party ElasticSearch library, and support for ES 2.x and 5.x series.

If someone would take the time to simplify this PR drastically it could be reduced from ~5k lines of code to possibly 800-1000 tops, which would make it much more feasible to include.

As far as I've gathered this branch basically works, but there's a missing aggregate function that prevents certain kinds of queries working, maybe that doesn't have to be a deal breaker, and we can merge the foundations of the support rather quickly, and then work on fixing buggy or missing features in retrospect.

Does someone volunteer to get the branch simplified and prepare it, and champion it towards being merged?

Not sure why it would need a 3rd party library to send JSON to an HTTP/S service.

ES 2.x should just be discarded, no one should be running it anymore (I know lots of people are. Stop it!)

Is there a possible workaround to ES alerts in Grafana?

Thanks.

What I did to workaround this was to stand up a stand-alone InfluxDB instance and have anything that I wanted to alert on go there instead of ES. It's not too bad as InfluxDB isn't much of a resource hog and since you don't need all of your data in there, you can keep data usage low by not keeping a lot of historical data and only ingest the required data that you need into it.

That's a solid workaround

I have experimented with this in the past, I took all logs from a small windows environment that were already going to elasticsearch and sent them to telegraf, then to influxdb. This worked amazingly well because fields in the event log became fields in influxdb, so in grafana I could query for event ids, hostnames and usernames for example.

The best part about this is that the disk space this all took was tiny, I mean really tiny. Trying to remember from memory but it was something like 500mb a day in elasticsearch, this became less than 300mb in influxdb... For data covering over 6 months.

This was just an experiment, never thought about solving the alerting problem with this.

I encourage everyone to try this out.

Spent all this time trying to move away from logz.io (expensive) to Grafana for the company I work for...to only find out I can not setup alerting for my elasticsearch logs.

I need to have an ETA? I am working on a project which requires this feature.

@nikskiz Grafana is open source project, and everyone can contribute to it. Also you can contribute!

Many people work on open source projects for free in their free time, and I think it is not ok to be rude and criticize their work.

There is a smaller PR in progress: https://github.com/WPH95/grafana/pull/2 by @WPH95

@nikskiz right now you can use influxdb

@nikskiz or you can simply use elastalert

Hi everyone!

We all know that alerting for Elasticsearch is an amazing, very-welcome and long-waited feature.
But I think we should keep this thread as clean and organized as possible.

So, questions and comments like these...

Any timeline?
It would be very useful!
Can't wait for it!
Has there been any progress on this?

doesn't help in evolution of the feature, generates a lot of useless emails to people who subscribed to the thread and makes the discussion too much confusing to new participants.

Even questions about workarounds and other alerting solutions (like ElastAlert) shouldn't been posted here anymore, as it has already discussed a lot in previous comments.

So, please, stop trivializing this space! Let's use them to spread useful comments and to share the progress you've made on this feature.

I just finished a PR #11380 to make grafana support Alerting [Elasticsearch]

Compared to some early implementations e.g. #8943 , #10343

• base on grafana-5.0.0
• source code less than 1000 lines (and a third lib [leibowitz/moment]~=1000 line, and some test code)
• Code logic is consistent with the front

this week i will continue to improve this PR, more comprehensive test, clearer abstract model.
I'm looking forward to someone who can try to use this PR and improve it :)

@WPH95 - I will be trying your PR this week for sure, will create issue in your repo, if that's ok with you, and look into contributing there.

Merge that already please. Thats a hot feature.

anything we can help in order to promote this ?

What is the estimation to merge it? Shall I wait or use X-Pack?

+1, What is the estimation to merge it? 5.2.x?

Hi i know we pushing you too much on this , but this issue was open almost 2 years ago (Aug2016) .
elasticsearch + Grafana commutiy is wide and big like others i guess.
i really appreciate the great job this community doing for us, but seems like this issue stayed behind :(
is there anything we can do in order to promote this?

Why is it taking so long to merge the PR?

+1

+1

+1

Stop writing comments with only "+1" because you're spamming other people. You're not adding anything to the topic. Click thumbs up button on the main post or click Subscribe button instead.

+1

+1

:tada: :tada: :tada: :sparkling_heart:

🎉

Wow, it happened. I honestly thought this was going to go the way of duke nukem. :+1:

Thanks a lot for your efforts!