Helm: Proposal: Allow templating in values.yaml

Are you proposing interpolating the literal values from this hypothetical _values.yaml_ file into the chart's templates, and then interpolating those whole templates again to expand the release name (to use your example), or do you imagine interpolating the _values.yaml_ file first, and then treating its expanded form as the input for interpolating the rest of the chart's templates?

This has been discussed at length before, and for a variety of technical reasons it is problematic. But a far less problematic solution would be to support expansion of things that look and behave like environment variables.

So, for example, you'd be able to do myval: ${RELEASE_NAME}.foo.bar in your values.yaml.

On the Golang side, it should be as simple as calling os.Exand() with a custom list of variables that should be expanded.

To @seh's point, it is a little tricky to decide _when_ you want to run the expansion. I would think you'd want to do it after the values are coalesced, but before the values are sent into the template engine. That would bring with it the caveat that variables would have to be represented as well-formed YAML. But I think that is workable

@seh I personally imagined it as something closer to the first option, wherein the literal values are placed into the template before rendering.

@technosophos given that, I think what you are suggesting sounds like a good solution.

One challenge with systems that work like this is that simple interpolation is often not enough. The replacement value has to come in as a lexically valid replacement, which then leads to wanting some functions available transform the supplied value into something suitable.

I'm not arguing against the idea just yet; we just need to be suspicious that the simple substitutions to which @technosophos alluded above will turn out to be enough.

But if it is coming in through the values.yaml file, the transformation and validation can still happen in the templates, right? For example: myPort: ${port} can still be accessed in the template like this: {{ .myPort | int }}.

The bigger constraint is that the values.yaml file MUST always be a valid YAML file.

It's that constraint I had in mind: What happens if the "port" variable holds the value

"a

or some other sequence that makes for invalid YAML? As you said, there are syntactic constraints to keep in mind, though one can work around them:

myport: '${port}'

Right, so the values would have to be escaped or quoted in some way, eg in values.yaml:

portValue: "{{ .myPort | int }}"

Based on the cases where I have wanted this so far, I suspect full templating is required, not just simple interpolation.

The idea of interpolating the literal values from values.yaml into the templates, and then interpolating the templates again in case the literal values were actually templates is interesting, but seems like it would be dependent on how the templates are written. e.g. this isn't going to work as expected:

values.yaml:

otherServiceName: "{{ .Release.Name }}-my-service"

deployment.yaml

otherService: {{ .Values.otherServiceName | lower }}

Are there problems with interpolating values.yaml first (obviously without access to .Values) and then using the resulting values.yaml to interpolate the other templates?

@braedon Yes, there are several problems:

1. The values.yaml file MUST be valid YAML in its unparsed state. This places interesting limitations on template directives.
2. Values are merged in stages, and some stages do not have access to the template subsystem (namely, --set resolution, requirements.yaml constraints, and -f merge operations.)
3. All values must be computed before certain parts of the template system can be initialized. Otherwise, {{ define }} and {{ include }} do not work predictably, and {{ template }} doesn't either.

Essentially, an env var syntax is easy enough to embed in values.yaml that we could do it without introducing much complexity. But using full-fledged templates is a major undertaking, and even if successful it would result in an interminable series of bug reports filed when people hit the edge cases.

But I must be missing something... I don't understand your example above. What is being proposed is:

values.yaml

otherServiceName: "${RELEASE_NAME}-my-service"

deployment.yaml

otherService: {{ .Values.otherServiceName | lower }}

The order of operations would have values.yaml rendered after the Release object is created, but before the templates are compiled or rendered. If the Release name is FOO, the outcome of the above would be otherService: foo-my-service.

@technosophos I was envisioning running the templating engine on the values.yaml files before any other parsing/coalescing/etc. of the files was done (if this feature doesn't make it, I've considered running a separate templating system to generate the values.yaml files to then pass into Helm).

It seems like that should avoid issues 1 and 2, and be a pretty intuitive system to grok? I have no idea how hard that is to implement in Helm's codebase, however, and 3 would still be an issue (not sure how limiting disallowing those functions would be?).

My examples were in reference to the solution @seh and @benjigoldberg were discussing, rather than your proposal.
In their proposal (assuming I understood it correctly) my example would result in this after the first pass:

otherService: {{ .release.name }}-my-service

(i.e. the raw value of otherServiceName from values.yaml would be passed to deployment.yaml, and lowercased)
Which then wouldn't run as intended on the second pass.

What I was thinking is:

If a /templates/values.yaml file exists, then the logic is to read variables from /values.yaml to render the template, and then the rendered values.yaml takes over completely, before all other templates are rendered. In other words, /values.yaml bootstraps the template.

I have also run into a use case that I think would benefit from this. I suspect it might be similar to @braedon's otherService example.

The problem I am running into is that it's currently not possible to include the .Release.Name when overriding a service's name via values.yaml. The use case for this is when building completely decoupled charts that can then be wired together using a parent chart.

For example, an elasticsearch chart could have a serviceNameOverride value in its values.yaml file that would allow the user to override the service name. Similarly, a kibana chart could have an elasticsearch.host value in its values.yaml file that the user can use to set the location of the elasticsearch cluster. These charts know nothing about each other's implementation details. They are independent and would allow a user to, for example, deploy kibana to monitor an existing elasticsearch cluster that was not deployed with helm.

Ideally, these independent charts can also be used as building blocks for creating a parent "elasticsearch-kibana" chart that "wires" them together via subcharts. The parent chart would connect the services together by setting the following in values.yaml:

elasticsearch.serviceNameOverride: {{ .Release.Name }}-elasticsearch
kibana.elasticsearch.host: {{ .Release.Name }}-elasticsearch

Currently it's possible to do this without including the {{ .Release.Name }} in the values, but it's inconvenient as it would prevent the user from installing multiple "parent" charts in the same namespace without changing the values file for each installation.

@alexbrand You could use the tpl function to render values that contain variables:

values.yaml

elasticsearch.serviceNameOverride: {{ .Release.Name }}-elasticsearch
kibana.elasticsearch.host: {{ .Release.Name }}-elasticsearch

template.yaml:

name: {{.Values.elasticsearch.serviceNameOverride | quote | tpl }}

@eicnix To do that requires having control of the template.yaml in the subchart, and wanting to explicitly add support for this wherever it is required.

What I'm after is a way to write parent charts that do this kind of thing while treating the subcharts as black boxes. I think that's what @alexbrand was saying as well?

Someone above has already referred to the tpl function that was fairly recently added (#1978). I may be misunderstanding some complexities that are involved here, but for my use case and some others described above, all that seems to be required is an implicit pass over values evaluating them via tpl or equivalent, prior to rendering templates. (I guess you would potentially need to keep doing passes until no changes are made, to cope with recursive references.) I suppose you might want to require this to be enabled via, say, an --expand-values argument on the command line, but is there any other issue with such a solution?

@eicnix Thanks for the suggestion!

I was unable to get the pipeline in template.yaml working but the below workerd for me (with values.yaml above):

template.yaml

name: {{ tpl .Values.elasticsearch.serviceNameOverride . }}

I think that this and #2133 are the same issue, BTW. (Not sure which one is best to keep open though.)

We seem to be OK in 2.6.1 of Helm server (but not in 2.3.0)

@obriensystems can you elaborate on that thought just a bit more? :)

What seems to be OK about this proposal for v2.6 but not in v2.3? Perhaps you are referring to another ticket?

I like the approach proposed by @travishaagen and @braedon . I could also imagine using multiple -f values.yaml arguments, e.g.

helm install -f raw_values.yaml -tf values_rendered_with_templating.yaml ...

First it would read raw_values.yaml which would have to be valid yaml. Then helm would read values_rendered_with_templating.yaml and run this through the template engine which has access to the variables in raw_values.yaml. The rendered template is then fed to the next stage and so on.

I've used tf in the command line above to signal that it's a values file that should be rendered with templates before being used. The file structure proposed by @travishaagen would be the default setup.

This issue along with #2399 are quite big blockers that really affect composing parent charts from child charts. As things are right now, helm does not allow composition of charts for anything non-trivial. I wonder if there is a better solution for composing charts that would solve the issue at hand as well as #2399.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

I guess we don't necessarily complicate helm to do this. For example, a simple tool like https://github.com/roboll/helmfile can be used for templating in values.

@mumoshu I ran into a couple of cases where I'd need access to the release name in values.yaml (most recently the Kafka chart which can be set up not to use the Zookeeper Chart as a requirement, but then needs a Zookeeper location as a Value, which in our deployments is based on the release name...). I'd however like not to require yet some extra tooling and enforce it on our users to achieve this :smile:

@NicolasT Thanks for sharing your use-case 👍

You may already know but for anyone not there yet - you can name your Zookeeper release a concrete name by helm install incubator/zookeeper --name myzookeeper and then reuse that name as a value of Kafka release.

And your use-case doesn't seem to be supported by solutions suggested in this thread. Sry if I'm not following you, but you can't access another release's name from an another, independent release, right?

Yours seems to be more about "extending what can be referenced from templates" rather than "allow templating in values".

I'd however like not to require yet some extra tooling and enforce it on our users to achieve this 😄

I can't agree more 😄
An all-in-one tool to support everyone's use-cases would really a godsend.

Actually, I once wished if helm could integrate with a declarative workflow manager with dynamic service dependency resolution(your use case) feature like atlassian/smith.

Imagine an integrated solution of Helm CRD + Smith - would hopefully elegantly solve your issue?

However, on the other hand, I feel like that integrating/implementing a serious workflow engine like that into a tool for other purposes would be nightmare in regard to maintenance.

So, I rather wish helm core to remain simple and the helm plugin system is extended somehow to support an extension point like "values transformer" which is able to modify values.yaml before passed from one to another. Hmm, that sounds like almost smith though!

My primary use case is #2506. In the general case, I need a parent chart to be able to pass its own values down to a subchart without having to know those values at package time.

Hello gurus, I'm confused of this discussion. The solution proposed can not solve problem of my use case:

values.yml:
ui_secret: # Using "{{ tpl randAlpha 8 }}" here reports error.

template1.yml:
UI_SECRET: {{ .Values.ui_secret }}

template2.yml:
UI_SECRET: {{ .Values.ui_secret }}

I know the param ui_secret can be generated outside helm and passes it in via --set 'ui_secret=xxx'. But why not let helm generate a default random value ?
The default random value can be used for salt/password/ca_certificate/certificate_pair, etc.

But why not let helm generate a default random value ?

Hi! How would you like to achieve that?

For me, templating in values.yaml doesn't seem to solve it. What if you had an IMAGINARY values.yaml like:

ui_secret: "{{ randAlpha 8 }}"

and you run helm upgrade --install yourrelease yourchart -f values.yaml multiple times?
For me, there seems no way for helm/tiller when to do/dont generate the ui_secret.

So, I believe you should just provide a pregenerated secret via --set like you noted, regardless of this proposal is accepted or not.

Instead, would it be possible for you to just generate a random value inside a k8s secret?
So that all the pods created via your helm chart could refer the secret by its name.

@mumoshu for the upgrade scenario:
1) if ui_secret is set in values.yaml, then no impact;
2) else helm should retrieve existing values.yaml from somewhere (probably Tiller), then merge this existing values.yaml with the '-f values.yaml'

helm should be self-contained for generating random salt/password/ca_certificate/certificate_pair, and do not requires the end user to generate outside helm. This provides much better usability.

"generate a random value inside a k8s secret" is doable, but not straightforward comparing to 'ui_secret: {{ randAlpha 8 }}'

else helm should retrieve existing values.yaml from somewhere (probably Tiller), then merge this existing values.yaml with the '-f values.yaml'

Not really offending but it seems to introduce more edge cases.
How would helm/tiller differentiate the value is already set or not, and when to override if from --set or -f values.yaml or not, given your values.yaml always contain ui_secret: {{ randAlpha 8 }}?

"generate a random value inside a k8s secret" is doable, but not straightforward comparing to 'ui_secret: {{ randAlpha 8 }}'

Agree. But anyway, templating doesn't seem to be a solution for me.

You'd need a workflow engine like https://github.com/atlassian/smith to deal with service dependencies(with ability to pass values to be consumed by consequtive, dependent helm installs).

I guess you can submit a feature request to helm for supporting this use-case(workflow engine, service dependency management). However, I don't know whether that's feasible in scope of helm/tiller.

when installing the chart, '--set' > '-f values.yaml' > values.yaml in the chart
when upgrading the chart, '--set' > '-f values.yaml' > generated values.yaml for the installed chart

What I proposed is implemented in BOSH https://bosh.io/docs/variable-types.html. BOSH supports explicit variable type definition, while values.yaml in helm chart supports plain yaml. So it would be great to support template function in values.yaml, e.g. 'ui_secret: {{ randAlpha 8 }}'.

While this would provide better usability for some users it would also bloat up helm.

My recommendation of dealing with any kind of secrets in combination with helm is to manage them in a secret storage(kube secrets, vault, etc.) and just reference them from the chart.

What I proposed is implemented in BOSH https://bosh.io/docs/variable-types.html. BOSH supports explicit variable type definition, while values.yaml in helm chart supports plain yaml

I've reviewed Bosh's documentation about Variable Interpolation. It turns out to me that Bosh doesn't support both (1) populating variables from outputs of your apps(k8s objects created by helm/tiller in our case), and (2) generating a persistent value via template. See the notion of -v internal_ip=192.168.56.6 in https://bosh.io/docs/cli-int.html.

Even Bosh doesn't seem to support your use-case out of box.
Then, my best guess is still that you need something like a workflow engine as I've explained previously.
Or am I still missing something..?

@johngmyers Hi! Sorry to say but your use-case seems to be not supported by templating alone.

TO support your use-case, not only templating, helm/tiller also needs to manage coordination of charts and its releases, out/in propagations from precedent release(s) to dependent chart(s).

AFAICS, what's not helm/tiller is currentlyaware of. Doing it in the scope of helm/tiller would considerably bloat it up, as @eicnix pointed out.

To separate discussions, I can suggest you to submit a another issue. But on the other hand, honestly, I don't have a feasible way to implement it inside helm/tiller without bloating it.

So your best bet would be investigating into an another workflow engine like atlassian/smith(i'm not paid by authors of smith. please tell me if there's any alternative 😉) + probably upcoming helm CRD.

@mumoshu I believe my use case is adequately described by #2506. I don't see how submitting another issue helps.

I don't see how templating values.yaml doesn't handle my use case. Helm supports subcharts, which are coordinated by either requirements.yaml or embedding the subchart in the charts directory. The parent chart could pass down the value with something like:

subchart:
  serviceName: {{.Chart.Name}}
  serviceVersion: {{.Chart.AppVersion}}

Of course, templating values.yaml isn't the only way to implement allowing a chart to pass values not known at package time to a subchart. But if it's "bloating" Helm, then adding pretty much any feature would be "bloating" Helm.

Maybe we can add a limited value rendering before the templating process. I would only allow access to .Chart and .Release in order to avoid complex issues like for example cyclic references between values. But this would still solve a lot of usecases where people want to reference the release or chart instance from the values.

I have a use case where I need a parent chart to pass down a value sourced from its .Values. This allows separation of concerns into subcharts without the parent exposing the fact that it is implemented using a subchart.

For example, we would have a "platform" subchart which encapsulates common stuff needed by services implemented in our inner platform library. It would want to have a value "platform.oauthClient" which the service chart would pass in to indicate whether the service needs to be configured to be an OAuth client. The implementation of that might be to pass that .Values.oauthClient down to a subchart of the "platform" subchart. In the future we might want to change that implementation of that feature of the "platform" chart to something else.

The templated values might want to live somewhere other than the top-level values.yaml (perhaps templates/_values.yaml) so that they don't appear to be part of the parent chart's external API. That could also address the cyclic references issue.

@johngmyers The parent chart values should also be available in the child chart. I believe the method I proposed earlier would also solve your issue. You could create a templated value in the parent chart like you described in earlier and let it be passed to the child chart.

FYI, my PR that's already associated (ie. #3252) implements this feature and I think probably satisfies these various use cases. We have been using it internally for a while, and passing parent values to child charts is one of the problems we solve with it.

The PR hasn't made much progress so far. Any feedback, either on the PR itself or from trying out the feature in a build, is obviously very welcome.

@eicnix The Helm documentation states:

But lower level charts cannot access things in parent charts, so MySQL will not be able to access the title property. Nor, for that matter, can it access apache.port.

so that doesn't seem possible.

@johngmyers Child charts cannot access parent values but the parent chart can pass values to the child chart. Assuming you have a child chart called mychild you can pass to it values from the parent values.yaml this way:

mychild:
  overridenValue: true

@eicnix But such values have to be known at helm package time. Did you even read my use cases?

Does Helm Team be aware of this issue/proposal? It was created on May 25, 2017, 10 month ago. There has been a lot of discussion here, and several duplicated/similar issues created.
This issue is not assigned to anyone yet.

We're seeing comments 'it will bloat up helm'. So which solution does not bloat up? Could Helm Team provides a plan and ETA for this proposal and at least give a reasonable workaround/solution ?

Seems the PR https://github.com/kubernetes/helm/pull/3252 can support templating in values.yml. If Helm Team thinks so, please speed up on merging it.
Thanks.

@mparry's solution in https://github.com/kubernetes/helm/pull/3252 seems to handle my use case elegantly. It'd be amazing if it could be merged!

It doesn't look like the pull request related to this issue has been accepted yet, but I've just published a package that takes away the pain points I was seeing that led me to this issue. It's called Mercator and can be found here. I know it doesn't fit all these use cases, but hopefully someone might find it useful!

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

@johngmyers Did you ever find a good solution for https://github.com/helm/helm/issues/2492#issuecomment-370895073? I'm running into the same encapsulation issue. I really would like to avoid writing a wrapper for helm to recursively pre-render my templated values into the values.yaml files.

I do not have a good solution. I have a workaround which uses a private build of Helm with #3471 applied, allowing me to (redundantly) pass parent chart name and version to subcharts at package time.

This and the poor merge support are basically blocking me from doing any reasonable encapsulation. I'm hoping Helm 3's use of LUA will help with this issue.

For various technical and design reasons behind how Tiller's template rendering engine works, we do not intend to support pre-processing values.yaml as a Go template for either Helm 2 or Helm 3. However, we highly encourage the community to write wrappers or Helm plugins that support this feature for their use case! One such example we've heard from users is gomplate, but we'd love to see other pre-processors that can provide this functionality prior to running helm install.

To re-iterate a previous comment in this thread: adding support for expanding environment variables in values.yaml would be an easy problem to tackle without introducing much complexity to the rendering engine, but using full-fledged templates would be a major undertaking.

Even if we were able to support this feature, it would result in a sizable increase of bug reports filed when people eventually hit the edge cases. Given the current volume size of the issue queue and the limited number of maintainers and community members chipping in to help weigh in on issues, we simply do not have the human resources to support this undertaking.

We apologize for the inconvenience, and thank you for understanding!

@bacongobbler -- I originally opened this issue before I think I actually fully understood how to effectively use helm in my workflow. I completely agree with the decision to close the ticket. I left it open primarily because there ended up being a good amount of community discussion on the issue. In the time since I opened this ticket and I learned more "standard" workflows, I dont actually think I've ever found myself wanting for this particular feature. So, in any case, 👍

@benjigoldberg Can you clarify how you managed to solve this issue? What did you mean by a "standard" workflow? Our issue is one of simply being able to set the same password across a number of subcharts, yet only expose one password value in a top level umbrella chart. To clarify, we have a top-level chart for our app, which includes subcharts for prometheus and grafana. We want the passwords for all three to be the same. Right now, we have to expose the innards of the chart, and set the password three times, individually, which badly breaks encapsulation.

@nuwang in that type of scenario we would typically use a top level umbrella chart as you have described, create a secret at the top level, and then have each of the child charts use that secret to access the credential. Hopefully that helps -- its also possible I misunderstood your description. Let me know!

@benjigoldberg What if you don't control the subcharts? grafana is 3rd party for example. We can't easily change its templates to use a secret defined in the umbrella chart.

@dtshepherd we've definitely run into those issues -- in those scenarios we've forked the helm charts repo and contributed the option back upstream and then used our own forked chart until the change is merged (then we migrate back to the main repo's chart). There are moving parts, but these days it doesn't happen that much anymore. The main charts repo has matured quite a bit and a lot of charts have this flexibility out of the box.

Yes, most 3rd party charts templates things just fine. However, if i have a wrapper chart to encapsulate those 3rd party charts, I can’t normalize on some common variable without putting tpl in the 3rd party charts.

@dtshepherd you can control the name of the variable/secret from that top level values file and then push it down to the subcharts, right?

In the Elasticsearch/Kibana case, the service name is derived from the release name. But it's not _equal_ to the release name. Can someone suggest how to pass the resulting service name to Kibana just using variable renaming?

@benjigoldberg Yes, but that breaks encapsulation since the top level chart will need to know the variable names in the 3rd party chart. The chart in the middle should be the abstraction layer, but can’t be without templated values.yaml file.

@benjigoldberg Thanks for reply. The reason that we could not use a secret, is because the secret name must be passed to the subcharts as a single variable. However, since the values file doesn't support templating, you have to pass a hard-coded, constant name, without the release name prefixed to it (since the release name is a variable, naturally, that needs to be templated). This means that you can only have a single instance of the chart installed per namespace, which is an unacceptable restriction for us. That pretty much eliminates the use of secrets as an option, and we fall back into the trap of having to modify a third party chart that we do not control, or individually passing in the password three times in our case, fully exposing the implementation details of the umbrella chart.

So one way or the other, this either breaks encapsulation, breaks unique releases, breaks DRY, and often, a combination of all of the above.

A few more we ran into that come to my mind:

• Persistent volume claims that need to be shared among multiple subcharts, but must also support multiple releases.
• A shared variable that a user provides, which needs to be pushed to several subcharts.

This inability of Helm to encapsulate is a major pain point for us. Just the other week I warned a large cloud services company off of Helm over this issue.

This ticket is a bit overconstrained. The requirement is to permit a parent chart to pass a value to a subchart where the value is not known at package time, but derived from the parent chart's own .Value, .Chart, and/or .Release. I believe templating is required, but it does not have to be in values.yaml.

My proposal above of a templates/_values.yaml might be one way to achieve this without the edge conditions and complexity of #2133. The maintainers have stated they will not support templating of values.yaml, but it is unclear whether they would consider such an alternative values templating proposal.

There has been a lot of good comments around this issue. I wanted to provide some more clarity around this decision and try to amalgamate all of the information that is spread across various issues and PRs. First off, I completely get that this would solve some painful use cases. Not only am I a maintainer of Helm, but I write _a lot_ of charts and have run into this problem before. However, there are several big reasons we do not want to implement this:

The tl;dr
At its most simple, the reason behind not wanting to do this is you don’t template the file that provides the values that you template with. Furthermore, this feature will be made obsolete by Helm 3

Rendering order and logic
If you template the file you provide values with, it introduces a large amount of complexity in rendering order, circular value dependencies, and logic bugs/failures. When you start adding full logic blocks to values, that is yet another point of failure when trying to render a manifest. This makes things much harder to debug and diagnose (hence the comment about the number of bugs it would produce). It also makes the templates harder to write because you have to account for the variability that a templated values file introduces

Backwards compatibility
Templating values files introduces non-YAML syntax into a YAML file. This is likely to break existing templating solutions that generate values.yaml. Speaking from experience, I have used generated values files where having non-YAML in the file would have broken our workflow. Given that many people already have tooling around Helm values, this would make a templated values file a backward compatibility break. I’ll admit we haven’t been perfect with maintaining backward compatibility, but for the most part we have done a decent job at maintaining it, and a feature such as this would most definitely break existing workflows

Language complexity
A templated values file will require a custom dialect of go-templates because the YAML itself cannot be a template when it is passed as part of the release. This also introduces the complexity of creating a specialized context to evaluate the template before other steps.

Obsolete by Helm 3
In Helm 3 as part of the eventing/hook system, we will allow Lua-based modification of values in a much easier way than having to template them out. Seeing as we are well underway with development for Helm 3, adding a feature with all of the drawbacks mentioned above would cause more churn than the value added. So we aren’t rejecting this outright, but are implementing it (albeit in a different way) for Helm 3

Hopefully this clarifies it some more for everyone!

Thanks for taking the time to provide this detailed clarification @thomastaylor312 , that clears up this decision a lot better.

I think the core issue is one of separating the external interface of a chart, from its internal implementation details. That would appear to require some simple mechanism to separate what the chart exposes as the external values file, from how those values are transformed and distributed to third-party subcharts, even in Helm 3. Lua event hooks may well be the answer, or perhaps that will be too complicated for what should perhaps be an innate feature, but it's hard to evaluate that from my position of ignorance. So please consider this a feature request for Helm 3 and thanks for all the hard work that's going into Helm - it's a pleasure to use!

xref: https://github.com/helm/community/blob/master/helm-v3/002-events.md

Thanks for the great feedback @nuwang! We'll make sure to take that into account with all of the Helm 3 work

Here is a workaround for this.

Just writing some template expression in values.yaml as plain yaml string.
like.

helloworld: Hello World!!
sometemplate: '{{ .Values.helloworld }}'

And using tpl to evaluate sometemplate in the template file.

{{ $global := . }}
{{ tpl (trimAll "\"'" .Values.sometemplate) $global }}

The output will be

Hello World!

@wpc009 yes, that works, the only inconvenience is that tpl evaluates the string only once which is not a bad thing but something what you sould keep in mind:

publicHost: "{{ .Release.Name }}-{{ .Release.Namespace }}-paymentgw.{{ .Values.global.publicDomain }}"
callbackUrl: "{{ .Values.global.publicDomainSchema }}://{{ .Release.Name }}-{{ .Release.Namespace }}-paymentgw.{{ .Values.global.publicDomain }}/validate" # this will work
# callbackUrl: "{{ .Values.global.publicDomainSchema }}://{{ paymentgw.publicHost }}/validate" # this will not

@wpc009 yes, that works, the only inconvenience is that tpl evaluates the string only once which is not a bad thing but something what you sould keep in mind:

publicHost: "{{ .Release.Name }}-{{ .Release.Namespace }}-paymentgw.{{ .Values.global.publicDomain }}"
callbackUrl: "{{ .Values.global.publicDomainSchema }}://{{ .Release.Name }}-{{ .Release.Namespace }}-paymentgw.{{ .Values.global.publicDomain }}/validate" # this will work
# callbackUrl: "{{ .Values.global.publicDomainSchema }}://{{ paymentgw.publicHost }}/validate" # this will not

That's true. the value be referenced must be a constant.
Or you will need writing a nestedtpl.

@eicnix Tried {{ .Values.labels | quote | tpl | toYaml | indent 4 }}, but got this error: executing "financial-accounting/templates/worker-deployment.yaml" at <tpl>: wrong number of args for tpl: want 2 got 0.

values.yaml

labels:
  revision: revision-{{ .Release.Revision }}-{{ .Release.Revision | quote | b64enc }}

development.yaml

metadata:
  labels:
{{ .Values.labels | quote | tpl | toYaml | indent 4 }}

I got it working with:

values.yaml

dbName: test_{{ .Release.Namespace }}

deployment.yaml

env:
  - name: DB_NAME
    value: {{ tpl .Values.dbName . | quote }}

I'm using fabric8.io expose controller, which gives a default URL to the exposed service. Now I want to create a custom URL based on the namespace, Something like
fabric8.io/exposeUrl: https://{{ tpl .Release.Namespace }}.jx.org.co, something which the expose controller is going to pick, so I don't have an option to use {{ tpl ... }}. Is there a way such that the name resolution can be done in the values.yaml itself?

Couple of workarounds that others have pointed out in the various sub-threads:

1. use Helm global values to pass variables from parent to child (ref).
2. use YAML anchors to improve DRYness within your values.yaml file (ref, thanks @szwed).

These don't cover all the use cases, nonetheless they're nice to have in your tool belt.

I too want to template in my values.yml Mostly to stack complexity and specific configuration using charts. So I start with the generic chart, build my own chart on top having most configuration and leaving some final details to be defined when deploying. YAML anchors are not sufficient: they cannot replace inside YAML values (like a multiline configuration-file string), and lack the Helm dependency logic like subchart values.

I believe we need more powerful features for layering charts without changing the underlying charts to build the Helm chart community. The upstream community charts are now required to enable all the various use-cases in complex logic, rather than in layering. And this logic requires upstream changes to enable much features, wheres layering would require far less upstream changes. Just look at all the properties exposed in most community charts, the inconsistency in these properties and workings, and the duplication in the _helpers.tpl files. And worst of all each community chart is required to explicitly enable these settings on its own. Such a waste of effort.

@nicorikken Sry for the shameless plug but I'd suggest trying helmfile for templating values and layering templated helm releases and helm-x for modifying charts without forking, and even the combination of helmfile + helm-x.

I agree that we need something to deal with issues you've mentioned. On the other hand I don't want to scope-creep helm. That's why I started maintaining the two out-of-tree projects myself.

@mumoshu Thanks for the plug. I saw it mentioned earlier as a potential solution. I'll have a look at it. It will bring the cost of additional tooling, which we might have to integrate with Argo-CD as well (is that even possible?). For now I've forked the repo and parameterized the namespace, and make a PR to the upstream stable charts repo. Let's see what the maintainers make of it.

@nicorikken Thanks for replying!

I think I understand the cost of additional tooling. But on the other hand I'm questioning myself - if there was a huge tool that covers all the helm + helmfile + helm-x use-cases, would it be easy to learn/use?

IMHO it's almost documentation issue. If the additional tool has good-enough documentation to integrate it with the upstream tool and third-party tool(helm and argo cd in this specific case), it is comparable to an in-tree solution.

is that even possible?

Yep :) helmfile template plays nicely with argo-cd's "config management plugin" feature.

For the specific use-case of specifying the namespace, I've made an upstream PR to the Jenkins Helm chart https://github.com/helm/charts/pull/15202 This adds an namespaceOverride parameter. I hope this pattern will be adopted more broadly, to enable cluster-wide Helm chart deployments.
I'll have to take some time to look more into Helmfile. It certainly seems to fill our requirement by enabling advanced cluster-wide templating on top of upstream charts, and integrating with Argo-CD.

Does anyone know how to do this when setting something for an affinity rule? For example, on the NGINX Ingress chart at stable/nginx-ingress, I want to set the controller.affinity value to be the following:

values.yaml

controller:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 5
        podAffinityTerm:
          topologyKey: "kubernetes.io/hostname"
          labelSelector:
            matchLabels:
              app: {{ template "nginx-ingress.name" . }}
              component: "{{ .Values.controller.name }}"
              release: {{ .Release.Name }}

Using the {{ tpl .Values.controller.name }} for the componentkey works, but I also need the app and release portions, which don't work as they appear above.

Does anyone else need to call downstream template functions and Release.Name values?

@sc250024 Hey! Where do you define the nginx-ingress.name template in your values.yaml template?

It needs to be defined beforehand with {{define ... as explained in https://golang.org/pkg/text/template/#hdr-Actions

Also - {{.Release.Name}} works only within strings under releases[].values in helmfile.yaml.

This should work:

releases:
- ...
  values:
  - {{`{{.Release.Name}}`}}/values.yaml

This doesn't work:

releases:
- ...
  values:
  - foo: {{`{{.Release.Name}}`}}/values.yaml

I think it worth a dedicated feature request if you need one of those.

@mumoshu it's not defined in my values file, it's something that's included in the NGINX Helm chart located at stable/nginx-ingress. That's my point actually, I'm trying to use that template that's already in that chart.

For the record, I'm not using stable/nginx-ingress as a subchart in a chart I'm developing; I'm just using it directly to deploy NGINX ingress.

it's not defined in my values file, it's something that's included in the NGINX Helm chart located at stable/nginx-ingress. That's my point actually, I'm trying to use that template that's already in that chart.

Unfortunately It's impossible as Helmfile (values) templates works independently from Helm (chart) templates, and nginx-ingress.name is a template defined within the Helm chart for use in chart templates.

Probably the only way would be to copy the {{define "nginx-ingress.name"}} block defined in the nginx chart into your helmfile values template values.yaml?

I have thought about a workaround about this issue and I made it. It may be bad practice but I will share it just in case. You can put a placeholder in the text that you want to template and then replace that placeholder with the template that you would like to use in yaml files in the template. So here is an example:

In values.yaml:

keys:
  - name: key_for_test
    value: CHANGEFORRELEASE-my-value

And I will use this in a secrets yaml:

secrets_kv.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-test-kv
  annotations:
    description: Key/Value pairs to save in test datastore
  labels:
    app: test
    tier: backend
    vendor: test
    support: {{ template "supportMethod" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  {{- $var := .Release.Name }}
  kv.yaml: {{ toYaml .Values.keys | replace "CHANGEFORRELEASE" $var | b64enc | quote }}

I hope that it will be helpful to someone.

This is the first thread I ran into, than also comment here...
See also #2514

:) Thankfully, latest Helm manual says how to achieve this.
https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function

The trick is enclosing variable in " or in a yaml block |-, and than referencing it in a template as {{ tpl .Values.variable . }}
This seems to make Helm happy.

Example:

$ cat Chart.yaml | grep appVersion
appVersion: 0.0.1-SNAPSHOT-d2e2f42


$ cat platform/shared/t/values.yaml | grep -A2 image:
image: 
  tag: |-
    {{ .Chart.AppVersion }}


$ cat templates/deployment.yaml | grep image:
          image: "{{ .Values.image.repository }}:{{ tpl .Values.image.tag . }}"


$ helm template . --values platform/shared/t/values.betradar.yaml | grep image
          image: "docker-registry.default.svc:5000/namespace/service:0.0.1-SNAPSHOT-d2e2f42"
          imagePullPolicy: Always
      image: busybox

Otherwise there is an error thrown..

$ cat platform/shared/t/values.yaml | grep -A1 image:
image: 
  tag: {{ .Chart.AppVersion }}

1 $ helm template . --values platform/shared/t/values.yaml | grep image
Error: failed to parse platform/shared/t/values.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".Chart.AppVersion":interface {}(nil)}

For now, what I've done in the CI job is run helm template on the values.yaml file. It works pretty well atm.

cp values.yaml templates/
helm template $CI_BUILD_REF_NAME ./ | sed -ne '/^# Source: templates\/values.yaml/,/^---/p' > values.yaml
rm templates/values.yaml

helm upgrade --install ...

This breaks if you have multiple -f values.yml files, but I'm thinking of writing a small helm wrapper that runs essentially runs that bash script for each values.yaml file.

There is a use case where you would need to pass deployment name to dependency charts where you have no control.

For example I am trying to set podAffinity for zookeeper. And I have an application helm chart which sets zookeeper as a dependency. In this case, I am passing pod antiaffinity to zookeeper via values. So in my apps values.yaml file I have a zookeeper.affinity section. If I had the ability to get the release name inside the values yaml I would just set this as default and be done with it.

But now for every deployment I have to override this value, which is a big problem.

@thomastaylor312 and others:

I've read through numerous related issues on this general topic, and it seems the notion of templating values.yaml is impractical, non-trivial, etc.

Would it be reasonable to instead support some additional intermediate phase during chart templating/deployment/etc. where some arbitrary (i.e. user-provided) script (i.e. Python, BASH, Lua, etc.) receives a copy of the static values.input.yaml (aka values.yaml) and has access to various variables (i.e. .Release.*), and the script can be responsible for dynamically creating a "final" values.output.yaml file (which is actually directly used by the rest of the chart templates)? If Helm is able to supply the "hooks" for this, it seems like it might be a viable compromise, i.e. "here's all the data and a hook to trigger your script to create values.output.yaml, have fun!".

The one thing of concern that comes to mind that is if this approach could properly handle sub-charts and the scope/meaning of . changing, so I'd have to defer to the Helm maintainers to decide if that's solvable or just another can of worms (i.e. would this phase need to be triggered recursively for sub-charts).

I currently do something like this right now, but it requires me to provide a "launcher" script for my charts, so downstream consumers can't simply "helm install" my charts (somewhat similar use case to helmfile). This arose from a need to templatize values for third-party charts/containers (which I cannot modify for technical and legal reasons). In these charts, I have hard-coded values equating to .Release.Namespace appearing literally everywhere (inside URI/URL strings for the in-cluster FQDN for K8S services, arrays of host names, env vars, etc.). Every time I need to deploy the top-level chart to a different namespace, if it weren't for my helper script, I'd have to modify roughly 30 entries in values.yaml. If there were some way to standardize on this (even if it isn't via the implementation I described above), it would be helpful.

@IAXES Would the new post-render stuff possibly help solve this?

@thomastaylor312 In my own specific use case, definitely yes. In the use cases others have described (i.e. that are non-trivial and require some degree of dynamic generation of the values.yaml file, as well as access to deploy-time variables such as .Release.Namespace, etc.), I strongly suspect it could solve those problems too (would require some additional feedback from others that want/need the original feature described at the top of this thread/issue).

While I'm not versed in the Helm code base, I could potentially chip in or provide design/testing support (or chip away at this as a proof-of-concept if I could ping the collaborators/maintainers for advice on where to inject my code snippets to create the "hook" system).

If this is something deemed worth exploring, it might also be a good idea to +CC the helmfile maintainers/owners (consultation/opinions/feedback/etc), as it might be possible to hook in an existing solution for proof-of-concept. If one could setup these hooks and have any arbitrary solution step in for "filling in the blanks", and the final consumer/user still just has to execute helm install (plus maybe apt/yum/apk install some dependencies), that would be awesome!

@benjigoldberg @fsniper Would the use case I described above potentially address your use cases in a reasonable manner?

@IAXES If I understand correctly you are suggesting, helm providing hooks to external applications and then using the output of them as statics.

This is an interesting idea to explore, at the same time it's already possible without the help of helm at all. You already admit you have been using this process. And I believe all the interested parties could have implemented this themselves.

Considering helm including a templating engine which is more than capable enough to do this, I am not really happy with this approach. First this requires having external applications setup and integrated into probably already running processes, like CI/CD, and second more importantly requires an out of process templating, which is not well defined. As I stated before this definition, and standardization is already in place with helm. So why not using this in helm and provide the means for already requested feature?

Or implementing the already discussed and documented lua scripting could be used for this. Of course it is not available either.

Currently my approach is using helmsman values file environment value substitution. It's limited, but usable with some hops.

This issue makes it more challenging to use blue-green deployments with a parent helm chart for all your apps, at least if any of your values are environment-specific URLs or database connections. For now I'm going to try @leox-phq's workaround.

I also wonder if creating a templates/values.yaml file instead of a values.yaml file could be used as a convention to tell Helm to do value substitutions...not sure if this would this would help address some of the problems with implementing this natively in Helm that were mentioned...

Update: @leox-phq's workaround can be improved to avoid the need for a replacement using sed by using the --show-only option of helm template:

helm template . --show-only templates/values.yaml > values.yaml

I am looking for a way to use Chart.yaml appVersion in values.yaml files. Not sure if there already is a solution, or this issue is required.

@nodesocket It already exists in .Chart! https://docs.helm.sh/docs/chart_template_guide/builtin_objects/

@thomastaylor312 I thought you can only use .Chart.AppVersion inside of templates not in values.yaml files?

Ah sorry, I missed that part. Yes, there is no templating of values files due to the reasons mentioned here.

After reading about templating in values.yaml and trying out different solutions it seems that this are the cleanest solutions for the time being (in terms of being most readable by not having to look into too many places):

• Deploy Helm charts through Terraform, provide an external file values.yaml.tmpl, and use templatefile("values.yaml.tmpl",..) to render Terraform variables.
• Deploy Helm charts through Helmfile, provide all values in-line, and use {{ requiredEnv "..." }} to render environment variables.

There's also #6876 which is a newer approach to this issue, resolving the concerns raised by https://github.com/helm/helm/issues/2492#issuecomment-413635332. However, it's large, and is taking a long time to get reviewed.