Grafana: Feature Request: template variable string manipulations

Would be nice to be able to grab parts of variable, too, e.g. $(regex(some_variable, "/^([^-]+)\-([^-]+)$/", "foo: $1 bar:$2")).

This would allow showing much nicer labels with repeat panels/rows (no need to show full value like ew1-cosmos-prd-prod-admin_api-AutoScalingGroup-1FC4QCWSNZZFV but just admin_api (or even Admin API in combination also text transforms were supported).

any work around for this request yet?

I'm trying to understand if this can be solved already via http://docs.grafana.org/reference/templating/#query-options
Could you describe your problem with a concrete example, and possibly a screenshot on where you hit a wall here?

In my case I have a $Host variable of type _Query_, whose possible values are my server host names: I use it in my queries to only fetch data of this/theses host(s) and in the _Title_ or _Description_ field.

But theses hostnames can be rather long: "_srv42-nyc-prod.example.com_" which can create line wrapping on small "Singlestat" panel for example.

The ability to perform a substring/regex operation on my $Host variable so that it strips the "_-prod.example.com_" and only keep "_srv42-nyc_" would be great.

I have a similar use case to C-Duv.
I pull back a variable in the form "http://server1.example.com" and would like to use just the "server1" element of the string.

@billyaustin84 use the variable regex support for that

I'm still uncertain where the manipulated variables are being used: in panel queries or somewhere else? I screenshot would help immensely.

Also helpful, if @billyaustin84 and @C-Duv could confirm that they tried using http://docs.grafana.org/reference/templating/#query-options and failed.

it would probably be used in series alias formatting & panel titles.

When I initially added the support for ${varname:raw} syntax, my thought was that it could be extended to be a more general-purpose processing language for munging variables as needed, so a syntax like ${varname:replace(/(.*)_/,'$1 '):toLower} could represent a pipeline for modifying the value at interpolation time (as distinct from applying a regex to modify values at query time, since the same value might be used in different contexts within a dashboard).

For particular example I faced impossibility to just use simple conversions like [[variable:toLower]] or [[variable:toUpper]] since in different queries we need different flavours of variable value while keeping grouping (i.e for repeating rows by variable)

I also have a use-case in which template variables represent applications, and these applications can be namespaced using / (slash) characters in a container orchestration platform (namely Marathon).

These characters are removed in the metrics collection process and replaced by _ (underscore).

But then if you want to propose dynamic links in text panels for ex. to the app / API / whatever, you've to do the opposite transformation, ie. s/_/\./.

@davkal, I already used regex option for my $Host variable of type "Query". The query fetches all Prometheus known instances and the regex strips the port part so my variable only contains the server's FQDN.

It works for querying data (using some_metric{instance=~"$Host"}).

But for labels in graphs's legend or in graph's title/description I sometimes want to display a shorter hostname (eg. srv42-nyc instead of srv42-nyc-prod.example.com).
So a Bash's variable mangling kind of manipulation is fine for me.

I know I could change the variable's regex so that it only contains srv42-nyc and use the following query some_metric{instance=~"$Host-prod.example.com}. But I should adapt all queries, and what if I change/add a domain?. Also feel a FQDN-role variable $Host is proper and variable content/meaning should not be changed for display purpose.

my example:
i have a statsd-exporter for nginx-ingress-controller that changes all non-alphanumeric characters in metrics names and labels to underscore _

as a result i can't query both nginx metrics and kubernetes metrics in one dashboard,
because application deployment label is example-app and nginx handler label example_app

it would be very useful if we could have some basic string manipulation in variables section,
or right in a query string, for example change underscore _ to dash - in variable value for this query

When I initially added the support for ${varname:raw} syntax, my thought was that it could be extended to be a more general-purpose processing language for munging variables as needed, so a syntax like ${varname:replace(/(.*)_/,'$1 '):toLower} could represent a pipeline for modifying the value at interpolation time (as distinct from applying a regex to modify values at query time, since the same value might be used in different contexts within a dashboard).

This is exactly one of my use-cases. I have metrics with part of the path in upper-case, and part in lower-case. It would be great to be able to create a single variable that could be transformed to fit either use-case, on a given dashboard.

Also my usecase. I have some metrics that are tagged with the hosts IP address and some that are tagged with the hostname (which contains the ip address ip-x-x-x-x.blah) I'd like a way to use one variable to drive both queries.

My issue: I filter queries using lowercase environment names but I want environments to show in UPPERCASE in template panel titles. So generally something like ${variable:toUpper} would be great :-)
Other transformation ideas:

• from camelCase to underscore_case
• and opposite from underscore_case to camelCase
• first letter Upper case
• first letter lOWER case

My situation is I'm templating switches using the Zabbix plugin and what I need is the title will contain the interface name and description from the dashboard variable $ifalias.

The metrics will use the interface name extracted from the variable and catenate with strings like ":Bits received :Bits sent etc" and dynamically create a panel for every interface on the switch. The only piece missing is having the interface description in the title as so far I'm just using a variable with the interface name. If I could just drop everything after the first space in the variable. The full string will be: (Ethernet3/1 Alias _myinterfacedescription_). I can use "Interface ${ifalias _regex manipulation_}|: Bits received"

Is this possible now?

I'm going to redo the Zabbix item prototype to drop the prepended IF and space so I don't need to worry about that.

In the above example "item" field, replace $ifname with ${ifalias _variable manipulation_}

I'm so close it would be amazing if I could do this. We're about to roll out a lot of switches and this will be a huge time saver.

I also have a need for this. I take a list or measurements from InfluxDB as my variable, i need the first two parts of the measurement name to build consistent panels.
By way of example:

compA_orders_east
compB_orders_west
compB_orders_east

I wish for my variable list to be ['compA_orders', 'compB_orders']
I will then add west and east to the query in 2 panels in dashboard

The result should be I can pick compA_orders and have two panels for west and east for this company.

!!!!!
EDIT -- Ignore me, Grafana already does this with Regex in variable load /^(.*?_.*?)_/ was my answer
!!!!!

My use-case is repeating rows over 'tuples' of tag values. If I could create variables like:

action-controller,action-controller,action-controller...

And then use only part of the variable in right part of the query (two different tags). This would allow me to create the kind of repeating dashboards that I want to.

My use case is turning "host_example_com" to "host.example.com", something I thought would be easy to do...

My use case is to be able to define a list of http endpoints to monitor, like:
endpoint=GET /path1, POST /path2
and then be able to create a repeated graph for each of the values, like:
http_server_requests_seconds_count{method="${endpoint[0]}", uri="${endpoint[1]}"}

My use case is the the variables is ip_port , but we only want ip

see this the variable $host is 192.168.x.x_3307, but we want 192.168.x.x only . if there is a way like
'192.168.x.x_3307'.split('_')[0] it would be perfect!

@Fanduzi can't you use regex to remove the _<port> in the variable?

+1 the ability to manipulate the strings like this would be immensely useful!

For me, i'd like to maintain a single list of router/interface pairs e.g.

BYO-XXX-XXX_LZWXXXXXX,BYO-XXX-XXX_GigabitEthernet0/X/X.XXXX

I'd like to have a repeating panel on this variable list, that then uses the first part before _ to populate the Host field (Zabbix plugin) and the second part the Item field within a larger regex itself

if i can create nested variables that extract the parts of the parent variable to achieve this effect then that would work for me too

I think it would be nice to have maintainers views on this instead of adding use-cases and use-cases year after year :smile:
@davkal @marefr what do you think? Skeptical about validity of use-cases? Generally speaking, people don't always have the full control on their metric ingestion, metric transformation, metric history and would like to change their text representation in drop-down lists and panels.

@marefr

@Fanduzi can't you use regex to remove the _<port> in the variable?

How would one do that? Do You have an example?

EDIT: Just found the solution for me. Im using this regex in the query to get the variable:
'/(.*)_traffic_in/' select's everything without the "_traffic_in" from the metric.

@marefr

@Fanduzi can't you use regex to remove the _<port> in the variable?

How would one do that? Do You have an example?

EDIT: Just found the solution for me. Im using this regex in the query to get the variable:
'/(.*)_traffic_in/' select's everything without the "_traffic_in" from the metric.

Thank you! Exactly what I was looking for!

I am using table.
render value as link
adding ${__cell} into URL.

but , before call the URL, I have to transform ${_cell}
and remove domain.

like. regex: /([^.]+)/

like: hostname.cade.com or hostname.aqui.com or hostname , loaded to ${_cell}

tranform: ${_cell:regex=/([^.]+)/}

result: hostname

refresh=10s&var-job=All&var-host=${__cell:TRANSFORM}&var-interval=$__auto_interval_interval
refresh=10s&var-job=All&var-host=hostname&var-interval=$__auto_interval_interval

is there any solution for it ?

@davkal, I already used regex option for my $Host variable of type "Query". The query fetches all Prometheus known instances and the regex strips the port part so my variable only contains the server's FQDN.

It works for querying data (using some_metric{instance=~"$Host"}).

But for labels in graphs's legend or in graph's title/description I sometimes want to _display_ a shorter hostname (eg. srv42-nyc instead of srv42-nyc-prod.example.com).
So a Bash's variable mangling kind of manipulation is fine for me.

I know I could change the variable's regex so that it only contains srv42-nyc and use the following query some_metric{instance=~"$Host-prod.example.com}. But I should adapt all queries, and what if I change/add a domain?. Also feel a FQDN-role variable $Host is proper and variable content/meaning should not be changed for display purpose.

@davkal This method only works when a single value is selected for $Host. If you are looking for a way working for multiple selected values, try a query like some_metric{instance=~"${Host:regex}-prod.example.com"}.

${var_name:regex} formats variables with multiple values into a regex string [1], so when you have both srv41-nyc and srv42-nyc selected, the query translates to some_metric{instance=~"(srv41-nyc|sv42-nyc)-prod.example.com"}.

[1] https://grafana.com/docs/grafana/latest/variables/advanced-variable-format-options/#regex

+1 for this
My most desired option would be to use something like nunjucks to provide jinja type templating.

Yet another +1 for this - we use Grafana as a dashboard to look at hundreds of individual entities - each of them has a unique SHA that is 64+ characters long, and to uniquely identify one in the context that exists in, you have to click in to the graph to see the whole title, because the last 8 or 10 unique characters are cut off by the display.

TL;DR: I need the individual unique ID titles of HUNDREDS of graphs that look like:
x31089005522ubz7qhc3mldewguc45wrku3eqsxsr64bhbmvwrgeqsxhaffa Bandwidth
to look like:
qsxhaffa Bandwidth

Something like the issue that was closed as duplicate above would have been perfect.

Hello!

I ran into this issue while trying to use Prometheus metrics, specifically $__rate_interval.

It's defined as a string, so we get something to the tune of 60s.

This is fine when being used as a range vector selector, but our metric is aggregated via a record rule, so at the end of the day I would like to multiply the aggregated metric, but I'm stuck trying to remove the trailing s.

Full example is:

• basic counter:
  
  
  
  • seconds_service_used
  
  
• local aggregation:
  
  
  
  • sum(increase(seconds_service_used[$__range])) BY (email)
  
  
• source for record ruleagg_seconds_service_used:rate10m:
  
  
  
  • sum(rate(seconds_service_used[10m])) BY (email)
  
  
• rough estimate for aggregated sum:
  
  
  
  • sum_over_time(agg_seconds_service_used:rate10m[$__range]) * 60
  
  
• impasse that points to this issue as a fast path forward:
  
  
  
  • sum_over_time(agg_seconds_service_used:rate10m[$__range]) * $__rate_interval
  
  

The last query breaks, due to a trailing s.

Thanks for looking into the problem!

my issue: I wan't to use substring on the selected value
but the formatters are only for multi-selectors
maybe there is a way to add a single-value formatter

In my issue I have pods in k8s with names like

pod=auth-78df597855-f2hxq
pod=auth-redis-b499985c4-h2rx8

I have a variable

service_name= label_values(kube_deployment_labels{namespace=~"$namespace"}, deployment)

I'm looking for a way to have it as a pattern

so I could use pod=~"$service_name" instead of pod=~"$service_name-[a-zA-Z0-9]-[a-zA-Z0-9]"
The problem with the former is that the ALL pattern won't match some pods I have in kubernetes
as with $service_name~="." it would be pod=~".-[a-zA-Z0-9]-[a-zA-Z0-9]" instead of pod~=".*"

Is there a way to add postfix to a Variable in Grafana, eg with regex?
It could be a constant postfix "-[a-zA-Z0-9]-[a-zA-Z0-9]"
Eg like this?

This of course does not work.

The intended result would be

service_name:
  .*    <<<<<<<<<<<<<<<<<<<<<<- from Include All option
  auth-[a-zA-Z0-9]*-[a-zA-Z0-9]*       <<<<< pattern in variable name
  auth-redis-[a-zA-Z0-9]*-[a-zA-Z0-9]*
  some_other_name-[a-zA-Z0-9]*-[a-zA-Z0-9]*

Or is there a way label_replace in Variable Query??

This is how it looks in Prometheus

label_replace(kube_deployment_labels{namespace=~".*"}, "deploymentregex", "$1-[a-zA-Z0-9]*-[a-zA-Z0-9]*", "deployment", "(.*)" )

Solved it with
query_result

```
query_result(label_replace(kube_deployment_labels{namespace=~"."}, "deploymentregex", "$1-[a-zA-Z0-9]-[a-zA-Z0-9]", "deployment", "(.)" ))

/.deploymentregex="([^,]).*/
````

Does not work in a filter :/

Edit - works

As we discuss Variable manipulation - I'm missing an option to add "Custom Variables" to a Query variable
It would be great if we could combine few variable Types into single entity

A Query Variable with an option for an ad-hoc user input (Text Variable) for example

I have the need to concatenate string and replace characters. Imagine my variable values are like

• Apache Tomcat
• Apache Webserver
• ...

So they contain space characters. Later I need to query a MariaDB database, which contains separate tables for each of the values. In the database I find tables like

• instances_Apache_Tomcat
• instances_Apache_Webserver
• ...

As you can see, with simple text replacement (space->underscore) and the concatenation of the prefix 'instances_' it is easy to create the correct SQL table name. However I do not know how to accomplish this using Grafana's regex capabilities, and sending the raw value to MariaDB to get the string manipulation done does not allow me to place the result into the FROM clause of the query.

How can this be solved? BTW, I am running Grafana 7.3.1.1.

This is a messy combination of issues. First I want Grafana to take care of my quite basic problem but it does not (https://github.com/grafana/grafana/issues/8259#issuecomment-722990863), then I try to pass the string manipulation to MariaDB (https://stackoverflow.com/a/64714297/4222206) which also does not work (https://github.com/grafana/grafana/issues/8755).
It seems Grafana is not as veratile as I hoped it to be.

Any progress on the OP's request?
Why is it so hard to integrate this? It's been 3 years already

@torkelo @bergquist @marefr @ryantxu @dprokop

What do you think about this feature?
We want variable Regex to support substitions.
s/(.*)/\1_postfix/

I implemented a fix for this issue in https://github.com/grafana/grafana/pull/29754. I'd love feedback.