Prometheus: Rule formatting tool.

:+1: * :100:

Yep, that'd be great. Preserving comments is the biggest (and basically only) problem. And agreement on how to split expressions over multiple lines.

Aren't we in the middle of adding this to promtool?

Related: https://github.com/prometheus/prometheus/pull/1779

Some observations:

• With the 2.x rule format (YAML embedding PromQL), we want this to act on YAML files, too.
• It should preserve both comments in YAML and in PromQL.
• Indentation is crucial (not just line breaks).
• It has to be seen if certain line breaks in the input should be preserved in the output (cf. how gofmt does it).
• Wherever a rule is rendered, the same formatting should apply, in particular on the _Alerts_ and _Rules_ tabs of the Prometheus web UI. This implies that the AST needs to keep comments (and possibly certain line breaks, see previous point).

It should preserve both comments in YAML

Our current YAML library doesn't do this, but is hoping to "soon".

I'd like to include this in my gsoc proposal. @brian-brazil, can you link to the YAML library you're referring to?

https://github.com/go-yaml/yaml, but I'd scope this to just the PromQL as this stands.

@brian-brazil But since the PromQL from rules is always embedded in YAML files, it at least has to preserve YAML comments, right? Otherwise the tool wouldn't be very useful as nobody wants to lose their comments the benefit of formatting.

That'd be nice, but we can get benefits without that. I'd rather also not depend on something with an unclear timeline for a gsoc project.

Let's start with the PromQL part. The YAML part should be almost trivial once the library preserves comments and things like the |2 indentation hint for multiline strings.

Ok!

v3 of the YAML library is out, which should allow for all of this.

FYI perfectly readable rule in configuration

(
    kube_job_status_failed > 0
    UNLESS kube_job_status_succeeded > 0
)
* on (namespace, job_name) group_left(maintainer) label_replace(kube_job_labels, "maintainer", "$1", "label_maintainer", "(.*)")
* on (namespace, job_name) group_left(pager) label_replace(kube_job_labels, "pager", "$1", "label_pager", "(.*)")
* on (namespace, job_name) group_left(paging) label_replace(kube_job_labels, "paging", "$1", "label_paging", "(.*)")

VS its garbage representation in the Web UI

(kube_deployment_status_replicas_available
  / kube_deployment_spec_replicas * 100) * on(namespace, deployment) group_left(maintainer)
  label_replace(kube_deployment_labels, "maintainer", "$1", "label_maintainer",
  "(.*)") * on(namespace, deployment) group_left(pager) label_replace(kube_deployment_labels,
  "pager", "$1", "label_pager", "(.*)") * on(namespace,
  deployment) group_left(paging) label_replace(kube_deployment_labels, "paging",
  "$1", "label_paging", "(.*)") < 100

@sylr I think you posted some other rule from the web UI. also @beorn7 I am starting to work on this issue.

here's what it shows for me for your rule:

(kube_job_status_failed
  > 0 unless kube_job_status_succeeded > 0) * on(namespace, job_name) group_left(maintainer)
  label_replace(kube_job_labels, "maintainer", "$1", "label_maintainer",
  "(.*)") * on(namespace, job_name) group_left(pager) label_replace(kube_job_labels,
  "pager", "$1", "label_pager", "(.*)") * on(namespace,
  job_name) group_left(paging) label_replace(kube_job_labels, "paging", "$1",
  "label_paging", "(.*)")

@geekodour Yes, I mixed up two rules.

since this is still open and added in project ideas for upcoming GSOC (https://github.com/cncf/soc#rule-formatting-tool) i would like to work on the completion of this during GSOC. Any recent update and additional info regarding this would be appreciated @codesome @brian-brazil @juliusv @geekodour @simonpasquier @sylr .
From reading the comment and other reference link it seems the next action point is building a formatter for promQL expression with the promQL AST?

@haibeey There already has been some progress in that direction.

When building such a formatter it probably would be best to start with what is already implemented here and add proper white space and comment handling.

It would be also nice, when such features could be integrated with the upcoming PromQL language server. Inside the language server code a lot of the stuff that is needed to format YAML files is already implemented, and it might make sense to reuse some of it.

alright! Thanks @slrtbtfs . i will start perusing the links shared ASAP .

Can I take up this issue? Would love to implement this.

It looks like @haibeey is already somehow planning working on this.

@roidelapluie I have already implemented a part of it way before. It's just that I had not mentioned here.

FYI perfectly readable rule in configuration

(
    kube_job_status_failed > 0
    UNLESS kube_job_status_succeeded > 0
)
* on (namespace, job_name) group_left(maintainer) label_replace(kube_job_labels, "maintainer", "$1", "label_maintainer", "(.*)")
* on (namespace, job_name) group_left(pager) label_replace(kube_job_labels, "pager", "$1", "label_pager", "(.*)")
* on (namespace, job_name) group_left(paging) label_replace(kube_job_labels, "paging", "$1", "label_paging", "(.*)")

VS its garbage representation in the Web UI

(kube_deployment_status_replicas_available
  / kube_deployment_spec_replicas * 100) * on(namespace, deployment) group_left(maintainer)
  label_replace(kube_deployment_labels, "maintainer", "$1", "label_maintainer",
  "(.*)") * on(namespace, deployment) group_left(pager) label_replace(kube_deployment_labels,
  "pager", "$1", "label_pager", "(.*)") * on(namespace,
  deployment) group_left(paging) label_replace(kube_deployment_labels, "paging",
  "$1", "label_paging", "(.*)") < 100

Is this formatting style acceptable? @codesome @juliusv @brian-brazil @beorn7
I'm writing my proposal based on this style.

The final representation after formatting can be discussed and decided during the GSoC period (maybe even now if any maintainer wants to chime in, but I don't have the bandwidth to review it at the moment). I would suggest (to all GSoC aspirants) to focus on _how_ would you achieve this.

https://prometheus.io/docs/practices/rules/ uses the best practices.

i wrote a rough prototype to preserve comments by augmenting the goyacc grammar rules a little. commit here https://github.com/haibeey/prometheus/commit/885566786ac20bfd400b2e7db470c92545919690

Rules file
recording rule in web ui

That doesn't look right, the operators aren't on their own line

oh yes. that commit doesn't do formmating. it is just to preserve comments in promql expr after evaluation for printing.
previous versions ignores all comments after evaluation.

i wrote a rough prototype to preserve comments by augmenting the goyacc grammar rules a little. commit here [haibeey@8855667]

Putting the comments in the AST seems like a reasonable idea.

I've left some comments about the implementation there.

https://prometheus.io/docs/practices/rules/ uses the best practices.

Please don't enforce:

sum without (instance)(instance_path:requests:rate5m{job="myjob"})

over

sum(instance_path:requests:rate5m{job="myjob"}) without (instance)

Having grouping modifiers first is the best practice, and is already how we print it - it's very difficult to read non-trivial expressions otherwise.

Yeah, I had strong opinions about this initially. At first Prometheus only supported sum(x) by(y) because it read more like English to me and thus I much preferred it, but for any non-trivial x it becomes really hard to see what dimensions you are aggregating over / keeping. I wish I had just made it sum by(y) (x) from the beginning and made that the only legal variant.

I still don't like at all the exact formatting sum by (y)(x) and much prefer sum by(y) (x) (makes much more sense to me), not sure why we chose to go for the latter one in the docs.

I think everyone is entitled to its own opinion about this, I personally find the "best practice" to be hideous, but, as all syntaxes are legal, please do not enforce one over the others.

The whole point of a promfmt tool is to, if not enforce, at least "endorse" a certain "canonical" formatting, very much like gofmt, with a whole string of arguments behind it.

Yeah, gofmt's formatting is nobody's favorite, but at least it creates one consistent style rather than many different ones, which is a value in itself.

As a side note, this is actively been sought after for the GSoC. @haibeey I see that you are also interested in GSoC. While I appreciate the upfront work that you have put now, I would appreciate some open discussions and design sharing, in the GSoC proposal or otherwise, for easier coordination of projects in GSoC :)

Comment preserving is already supported very well in the new go-yamlv3 library. The implementation is so flawless that there is no loss of comments. Hence, we can keep preserving mechanism independent from the Prometheus code without any worries.

alright. i guess it is time share my proposal draft GSoC proposal .

Reviews would be appreciated before the final draft. Thanks

@Harkishen-Singh There are two levels of comments: YAML comments and comments within a PromQL expression. https://github.com/prometheus/prometheus/issues/21#issuecomment-604213849 was talking about the PromQL ones (but YAML ones would be good to preserve as well).

Comment preserving is already supported very well in the new go-yamlv3 library. The implementation is so flawless that there is no loss of comments. Hence, we can keep preserving mechanism independent from the Prometheus code without any worries.

During evaluation of a PromQL expression is when the comments are removed.
That said i think the formatting is not only to be done in rules files but in some places too like the web ui. is this true @juliusv ?
So as pointed out by @slrtbtfs most of the formatting and preserving would be done in the Printer module.

In relation to YAML comments, there's problems with v3 that's causing issues for Cortex so we should avoid further rollout of that until that's all resolved.

That said i think the formatting is not only to be done in rules files but in some places too like the web ui. is this true

The web ui is the only place it's done.