Grafana: Grafana Meta Queries (Generic Transforms on Time Series Query Results)

I hope this feature is released not so far :-)

:+1:

When this feature is expected to go in release for elasticsearch data source?

not ETA, hopefully this year :)

If we were interested/possible to help with development - where would we start? Does some code exist as a starting point?

I don't think there is any code for this, yet. A general decision has to be made wether the meta queries are performed server side or client side. I'm in favor of doing this server side so that it can be used for alerting as well. The disadvantage is that this would only work for proxied datasources but I think this is what most people are doing anyway.

If it's helpful at all - I had thought through this problem and had a discussion with Rashid before timelion was release (the first private commits were from before our discussion so I can't claim any credit :)

https://drive.google.com/drive/folders/0B9HCLnVMhNurTVY1UUJWZGxOdEU

I don't know if this is the right vision, but I had been thinking of modular components some of which could potentially be user client or server side. The advantages of server include caching and minimizing data transfer for larger queries/combinations. An advantage for client side is for other data sources where the permissions are granted via the browser or potentially to add a new series to an existing graph - the client could just query it directly.

I would think the timelion code would be an interesting start. We use the iframe sharing of timelion and the templating in grafana to create templated timelion queries in grafana - a bit hacky but solve the problem until a better solution comes along.

Thanks!

@yehosef Hey thanks for all this material. I had started work on this too, but later, after going through these discussions, realized that adding timeshift just for ES was not so much better an idea, as adding meta queries like Torkeo suggests above. If you guys have started coding this part, I'd be able to help you guys out. If not, I'm happy to take this forward with the reviews for your designs coming in later..

cheers.

@arcolife We haven't started working on it yet - would be happy if someone is able to pick it up.

I don't know how these kinds of things work, but it's possible we would would be able/willing to pay for development of these features if that made it more feasible. I don't know if there a bounty system for these kinds of things - eg the raintank or some other company says it'll cost "$x" to develop the feature and different interested parties could contribute. For us, this and the ability to do SQL queries are really really valuable.

@yehosef Hey thanks! I'm just glad to be able to contribute. :)
PS: if setting up a meeting soon (to discuss those mockups) helps, let's do it?

I'm happy to do whatever I can to help this move forward. contact me at [email protected] if you want to try to set up a meeting - or just just correspond about idea to proceed.

@yehosef "The email account that you tried to reach does not exist." :)
I tried it on your [github_id] @ google [dot] com.

sorry - its gmail.com

Hi, so we have an ETA of when this feature would be available for OpenTSDB datasource.

no ETA yet, maybe in 4.1 or 4.2 which means hopefully within 6months

I am not sure if this issue is similar to "Feature request: Two different time-range series data are shown in the same graph #3235" as i am concerned about #3235 .
So if you could tell me specifically to this that when this feature will be available for OpenTSDB datasource.

@torkelo whenever this issue is being reviewed, please copy me and @yehosef for we had some email exchanges around this thought and the designs Yehosef had proposed in this thread, and I've realized that it's better to bring those ideas back here for wider exposure. Thanks.

quite interested in updates on this!

Any update on release date for this feature? I am really looking forward to use timeshift option for Elasticsearch data source.

We would also be really interested in this. Atm there is only Timelion for Kibana

+1

It's possible to create a datasource plugin that has access to the other datasources?
This way could be possible to wrap datasource responses and transform then.

yes,

the included -- Mixed -- data source does this:

https://github.com/grafana/grafana/blob/master/public/app/plugins/datasource/mixed/datasource.ts#L13

On Wed, Nov 23, 2016 at 3:26 PM, Bruno Meneguello [email protected]
wrote:

It's possible to create a datasource plugin that has access to the other
datasources?
This way could be possible to wrap datasource responses and transform then.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grafana/grafana/issues/3677#issuecomment-262526082,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAq94YeRbQxzw_CDfEiDjjbZj4pJLmTks5rBE0wgaJpZM4HAses
.

This is perfect! I think I'll try to build something over this, to create some language to take series from other datasources and process then with functions.
Anyone else interested?

@bkmeneguello be sure to read my comment: https://github.com/grafana/grafana/issues/3677#issuecomment-230233579. Before implementing anything we should agree on if this should be done client side (js) or server side.

Sure, if this implementation will be core, but for preview it could be made as a datasource plugin.

@torkelo Will this allow "dividing" two queries?

Very interested in this functionality as well.

I would also be interested in this as well. I could use this to have the ability to show a "combined total" in the hover-over of all metrics on a panel that are stacked.

@torkelo Hi, so do we have an ETA now of when this feature would be available for OpenTSDB datasource.

@shivam009 no ETA currently

@torkelo Hello,

Is there any ETA for #3677 ? It will be very helpful when combining sum/avg/etc on results of the multiple metrics in same way as Graphite data source does.
Using Elasticsearch as data source here.

no ETA yet, but it's something we hope to get to this year.

It could be usefull to compare with an arbitrary time range (for example, a "data reference")

Hi @torkelo,

Any updates on #3677 ??

no updates :( I would post update here if i had any

Hi @torkelo,

Thanks for the quick response. Essentially, my requirement is to average out data points returned from ES and bucketize them over a 1 min time interval. Is there any other alternative to achieve summarize and sumSeries functionalities in Elasticsearch?

@shiv6146 can't you use Elasticsearch date histogram interval function for that?

@torkelo I can do a nested group by, using date histogram but unfortunately Grafana does not allow me to hide the inner group by's output and plot only the aggregated data points :(

Can you at least implement something simple like adding and subtracting?

• Metric C: #A - #B
• Metric D: #A + #B

@mstipanov it is on our roadmap but we are busy with other features until after the summer.

@mstipanov
Looking forward to see this ASAP also. Using influxdb

•Metric C: #A - #B
•Metric D: #A + #B

would like to definitely help on this one if there is a spec. Any place I can start working for this one ? Also there could be some metrics that might require more data that is available on the existing "A" for example, moving average of 7 days will require (current series + 7 days earlier data)

@torkelo do you have any ideas on how do you think we should go about implementing it ? I am willing to invest some time on building this. Need a frontend on top of Druid.

looked into extending via plugin. Would have to pass the datasourceSrv to the meta-queries-plugin and then let the plugin do calls back to the datasource. Planning to work on Moving average & Time Shift first.

Hi folks, I had an issue like @f1-outsourcing and @mstipanov (metric A (+-) metric B). And then, I developed a panel plugin to do this, called "Calculated Stat plugin". If you are interested, this my github https://github.com/fabiojose/grafana-calcltdstat-plugin.

@fabiojose thanks, I looked at your plugin. Thanks for putting it out. I think it might be better to implement it as a data source that way we can use any panel like table or graph. I am done with Time Shift. Will open source it in a day or two. Will be done with moving average by then as well

Poshmark has open sourced https://github.com/GoshPosh/grafana-meta-queries, it supports computed columns, Moving Average & Timeshift

Impressive! good job guys!

@torkelo if you get some time can you please give your feedback on the plugin ?

@Gauravshah yes! sorry, the hole team has been busy preparing and travelling to a developer conference that we are sponsoring this week so been slow on plugin reviews & feedback. Maybe next week or week after

@torkelo no worries, please take your time. Thanks

@torkelo Is there any update?

I would like to see this feature implemented asap as well

Hi,
using grafana v4.5.2 with Elasticsearch.
Need to convert timestamp to boolean value based on whether the time elapsed since the timestamp is less than 5minutes.
script field value (Time is a function that return current epoch):
boolean( Time - _value > 300 )
Is that also in your plans?
Also, we need the option to generate a graph that is the division of two values from different queries, over time.
Thanks,

@lilachmaliniak
you could do ((new Date()).getTime() - A['_value']) > 300
two different data sources would work

Thanks @Gauravshah
I couldn't do exactly what you've suggested, probably due to the value type, which is a number/int.
I got error:500
that is when I set the Options script to: (((new Date()).getTime() - _value) < 300000 )
This is the query:
{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"beaconindex"} {"size":0,"query":{"bool":{"filter":[{"range":{"private_data.timestamp":{"gte":"1513673225106","lte":"1513676825106","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"private_data.kes_hostname.keyword:(\"lilach\-centos\-vm.com\")"}}]}},"aggs":{"5":{"terms":{"field":"private_data.kes_hostname.keyword","size":10,"order":{"_term":"desc"},"min_doc_count":1},"aggs":{"1":{"max":{"field":"private_data.timestamp","missing":0,"script":{"inline":"(((new Date()).getTime() - _value) < 300000 )"}}},"3":{"min":{"field":"private_data.systemMetrics.systemData.system.uptime"}}}}}} "
Running same query from Kibana, I got:
{
"error": {
"root_cause": [
{
"type": "class_cast_exception",
"reason": null
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "beaconindex",
"node": "TGW3SyrEQHSnIi8qQt3rNw",
"reason": {
"type": "class_cast_exception",
"reason": null
}
}
]
},
"status": 500
}

So I used following, in Options, script: (((new Date()).getTime() - _value) < 300000 ) ? 1 : 0
And it works converting the values to rang: [0,1]

My question is: how can I convert the value further in Grafana, to display token instead (e.g. up/down)

I think that has to be supported by grafana and not by this plugin as such. You might be able to assign (((new Date()).getTime() - _value) < 300000 ) ? '↑' : '↓' haven't tried though

@Gauravshah, thanks
Can you be more specific, where do I set this?
If I write it in the Metric tab under Metric->Options->Script, the script is included in Lucene query as inline property and the query fail with error status 500.
"script":{"inline":"(((new Date()).getTime() - _value) < 300000 ) ? 'up' : 'down'"}

I think I have misunderstood your question. I do not know much on elastic search queries. I thought you were asking on how to do it on meta-queries plugin

@Gauravshah, yes I am trying to do it with ES queries. I already installed the meta-queries plugin but for some reason failed to create dashboard with it. When created new dashboard the option of MetaQuery type was not available. I'll give it another try, thanks.

@Gauravshah, how do I use MetaQuery with ES datasource?
where can I find the plugin docs?
as I couldn't find doc in git: https://github.com/GoshPosh/grafana-meta-queries
thanks

Hey @Gauravshah,
I managed to create a new dashboard after installing the MetaQueries datasource plugin.
after creating a new table and set table Data Source to -- Mixed --
added 2 queries:
A ($datasource) Metrics: Max(private_data.timestamp), Group by:Term(private_data.kes_hostname.keyword)
B (MetaQuery Datasource) Type Arithmetic Expression ((new Date()).getTime() - A['_value']) < 300000 ? 'up' : 'down'
Query A returns expected value and it is successfully displayed in the table.
but, I fail to display a column with the expression value from query B, and I can't see what is the expression output, neither whether it succeeded or failed. There is no indication.
My question: how do I debug the query (the one using metaQuery DS), and display the expression value in the table as an additional column?
Cheers

data source should be set to Meta Queries & not mixed. Will add it to the readme of plugin :(

Hey @Gauravshah,
Thanks.
Changed table datasource to "MetaQuery Datasource" as you advised.
Now I get error "cannot read property 'length' of undefined" and the table fail to display thought, as before the query from ES succeeds with expected value.

@lilachmaliniak can you create an issue on https://github.com/GoshPosh/grafana-meta-queries, also please attach stack trace from your browser

Hey @Gauravshah,
Added new issue: https://github.com/GoshPosh/grafana-meta-queries/issues/9

👍

This issue is >2 years old. Did this feature request get rolled into a different one?

👍

Many time series data stores can do this for you, but some (looking at you cloudwatch) don't and might never support this sort of thing.

In the mean time we can at least do timeShift with two panels, but per-query timeshift to do comparison graphs in the same panel would be pretty fantastic.

+1

@matschaffer & @JJMVG you should be able to use https://github.com/GoshPosh/grafana-meta-queries

@SwathiMuppalla you should be creating issues in plugin repo not here.

+1

+1 this would be yuge -- I haven't had much success with the grafana-meta-queries plugin with 6.4 :(

As a visualization layer that supports a multitude of data sources, this could be the crown jewel of monitoring features. We combine data from MySQL, Elasticsearch, and Prometheus in our dashboards, and normalizing data from one source based on another could be a huge leap forward in the product's importance in our monitoring stack.

This is not another +1 comment.

This is a +100 🙏🏼✌🏼

@torkelo Perhaps it would be possible to use something like https://github.com/wesm/feather or https://arrow.apache.org/docs/python/ipc.html as a kind of "column-store cache" that can then be used for additional processing or manipulations.

You might get other ideas from the work at https://www.dremio.com/ - it seems like a slightly related problem-space (querying a variety of data sources and caching in Arrow etc. for further manipulation and querying.)

@torkelo Perhaps it would be possible to use something like https://github.com/wesm/feather or https://arrow.apache.org/docs/python/ipc.html as a kind of "column-store cache" that can then be used for additional processing or manipulations.

While it's nice to have external tools for advanced functions, this does not address the issue in terms of simplicity or integration with Grafana. Providing the very basic arithmetic operations on two series can solve 80% of the problems with only 20% of the time.

@redlus @yehosef why is https://github.com/GoshPosh/grafana-meta-queries not a solution ?

@torkelo if you think https://github.com/GoshPosh/grafana-meta-queries solves the problem mentioned in this issue, we should close the issue addressing that it is being solved by a plugin for grafana. It is difficult for people to scan through long issue to find out that there is a plugin that solves the problem.

@Gauravshah
Would love to see a working plugin, but we've had issues using this plugin with newer versions of Grafana; i.e.
https://github.com/GoshPosh/grafana-meta-queries/issues/82

@redlus you should give it another shot, it does work on the grafana 6.4. Also if latest grafana version was the only issue , there were options to go back one minor version. Not sure why would you call would love to see a working plugin

@Gauravshah unfortunately still not working with Grafana 6.5.1 - I didn't try with 6.4 but switching to an older release may not be acceptable for most users.

no progress ...

I see a lot of people bouncing here trying to aggregate multiple data sources. I wonder if the blendstat panel solves some peoples problem and whether that design could be reused in other panels?

https://grafana.com/grafana/plugins/farski-blendstat-panel

Blendstat does fill a gap, but it doesn't work with repeating or looping for all variables. If there's a way I don't know how.

I just came across another use case I hope this solves. I am getting back CloudWatch metrics with percent (0.0-0.1) which gives me whole numbers 5, 7, etc. I was expecting 0.5%, 0.7% and I don't see a way to transform that.

Hi, apologies if I'm treading old ground here, but this is quite a long thread:

Am I right in thinking this feature request is to support graphs produced as a result of combining multiple queries? For example, if this work were done, you could take a time series from Prometheus and sum it with a time series from, say, Stack Driver?

If so: is this work still planned, what is the road map and do you need help? :)

that, plus (I hope) time shifting any graph so that I can compare todays graph with yesterday, Or the 7-day average.
Yes, some data stores can already do that, but this should be a generic feature.