Helm: Helm should preform deep merge on multiple values files

I think i found the issue and ill try to open a PR

If this is implemented, it should be optional, i. e. the merge mode should be specified as a command-line flag. Anything else would be a breaking change and, I think, in most cases not desirable. The problem is that you would not be able to override defaults for lists.

Here's an example: https://github.com/kubernetes/charts/blob/5453fc144f6f9d722c6bb433790f90d7943a4fc3/stable/docker-registry/values.yaml#L19

For those coming from the Java world, here's how Maven solves this for plugin configurations: https://blog.sonatype.com/2011/01/maven-how-to-merging-plugin-configuration-in-complex-projects/

Hi @connormckelvey

Sorry you did not get the expected results and I agree that its a frustrating limitation. If you are interested in contributing a feature please create a proposal spelling out the implementation which should include a flag as @unguiculus suggested.

Thanks @unguiculus! I see now how this would be considered a breaking change.

The problem is that you would not be able to override defaults for lists.

Are you saying you wouldn't be able to override an entire list (for example overriding a list of 3 items with a list of 1 item)? What I'm proposing would allow you to override individual defaulted items in a list. Giving you more control.

I guess Im not seeing why merging maps is desirable and merging lists is not desirable, except that at the current time some projects expect a certain behavior while others expect another.

@jascott1 Thanks, I will definitely think about the proposal with my team, but what I really needed was a quick fix that I thought could be resolved with a quick PR. At this time we need to focus on our immediate problem.

Was this functionality ever added?

I also see this as a limitation, especially for projects like prometheus where the whole prometheus configuration is defined in values file. Not being able to merge on sub-keys, I have to copy almost the same config per environment with is a huge repetition of code.

https://github.com/helm/helm/pull/4806 might be relevant, in which case this should exist for 2.12.0 (not yet released; try out the canary release to test).

Just ran into this and I'm confused as to why this was closed. As @mlushpenko states this seems like a pretty major limitation for managing any reasonably sized deployment. In our case we would like to be able to combine affinity rules from multiple levels -- providing a default set of rules in the base chart that can be augmented when deployed. However, as it stands we have to repeat the default affinity rules in every values override file. Not even close to DRY.

This is compounded by the fact that it doesn't seem possible to specify default values that will be based on things like the release name which aren't known until the deployment is performed (granted that's unrelated, but it means that overriding affinity essentially has to be done on every deployment).

@sfitts have you taken a look at #4806?

@bacongobbler I have and I don't see how it applies. AFAICT, it only deals with updates made during upgrade of an already deployed chart and ensuring that existing values aren't clobbered by new ones. Certainly related, but I don't see how it helps avoid the need to repeat information when combining values files during the initial install.

We'd really like this feature as well. We have a hierarchy of values files (eg: base.yaml <- qa.yaml) which are passed to helm via -f values/base.yaml,values/qa.yaml. As others have mentioned it would be great if we could merge the qa values into the base values without having to repeat everything.

I could also use this

I know this issue is closed, but as a point of reference for those looking here and thinking of resolving the issue I'll supply a comparison with chef.

There this issue is resolved by a fixed set of priorities for values (which they call attributes). Arrays set with the same priority will be merged, while a higher priority value will replace lower priority values.

https://docs.chef.io/attributes.html#about-deep-merge

If implementing something similar in helm I think it would make sense to introduce new versions of --set and --values, say, --overrideSet and --overrideValues for high priority values. As for setting values in a chart another file called, say, overrideValues.yaml could be allowed which sets values at a higher priority than in the values.yaml file.

We could also use this. +1

I'm a bit confused. The open end of this thread gave me the impression that this merging thing is still missing in helm. After trying it out (we do helm upgrade ... -f basic.yaml -f profile_extends_basic.yaml), it seems to work correctly. At least for me the #4806 seems to have nailed it.

I'm a bit confused. The open end of this thread gave me the impression that this merging thing is still missing in helm. After trying it out (we do helm upgrade ... -f basic.yaml -f profile_extends_basic.yaml), it seems to work correctly. At least for me the #4806 seems to have nailed it.

@janotav The #4806 addressed deep merge for the upgrade scenario. This issue was about merging values during normal install.

Yes. This is feature is needed. Another use case is to specify/merge/override a subset of environment variables across different deployment environments.

This continues to be a problem with helm 2.x, does anyone know if helm 3.x have a solution for (optionally) merging arrays during install?

I too would love to see this functionality!

For the folks here who need to modify the list of environment variables you're passing into your deployment template, the solution is to define the vars and defaults as a dict rather than as a list!

values.yaml:

env:
  key1: default1
  key2: default2

values-production.yaml:

env:
  key1: prod-value

deployment.yaml:

{{- range $key, $value :=  .Values.env }}
- name: {{ $key }}
  value: {{ $value | quote }}
{{- end }}

Rendered, using values-production:

- name: key1
  value: "prod-value"
- name: key2
  value: "default2"

@tarrall Can you supply the helm commands you used as well for the merge?

I'm trying to have my helm chart contain default values which are put into a configmap, then have a environment-specific file that I merge into the defaults, but it's overriding instead.

I'm using the range $key,$value in my configmap template.

Does it matter if you're using helm2 or helm3 btw?

puppet with hiera also manages this, giving options of which kind of merge do you want. I am also repeating code because I have several pods that need to mount the exact same file from a configmap and another specific for each pod. The common should be declared in an upper hierarchy and then give the option to merge the values below

@bacongobbler isn't possible to reopen this issue taking in consideration that https://github.com/helm/helm/pull/4806 doesn't solves this problem because it does deep merge in maps and in an upgrade process. What @connormckelvey mentioned was the ability to do deep merge in arrays. This kind of deep merge should work for install and upgrade situations.

I'm surprised helm has no solution for this. The workaround @tarrall suggested doesn't scale well into complex lists involving many keys & sub-lists. @connormckelvey please could you reopen?

I am using the following script as a workaround:

import yaml
from deepmerge import always_merger

fileA = “tmp.yaml"
fileB = “feature.yaml"

with open(fileA,'r+') as f:
fileAdictionary= yaml.load(f)

with open(fileB,'r+') as f:
fileBdictionary = yaml.load(f)

result = always_merger.merge(fileAdictionary, fileBdictionary)
with open(‘newFile.yaml’,'w+') as f:
yaml.dump(result,f)

What @connormckelvey mentioned was the ability to do deep merge in arrays. This kind of deep merge should work for install and upgrade situations.

The current behaviour with merging two arrays is likely going to stay the way it is. I do not believe we plan to change the current behaviour, which is why this issue is closed.

Let's take an example. Here I have a basic values.yaml with one value as an array:

myList: [ "foo" ]

Then I run the following:

helm install [...] --set myList={"bar"}

What's the expected behaviour?

If we were to consider a "deep merge" of the two arrays, the resulting array would be [ "foo", "bar" ].

However, what if the user wanted to null out the existing list, resulting in the output being [ "bar" ]?

The problem here with a deep merge is that we cannot determine what behaviour the user expects. There is also no way to null out existing values.

This is why the current behaviour exists as-is today. The user has a way to null out values set by the chart author in the default values as well as anything set by other values files on the command line (either via --values/-f or --set, --set-string, --set-file, etc.). If they want to perform a deep merge they can do so out-of-band.

You can do this programmatically, of course, and @aknakshay describes one way to do it. But a CLI is not well-designed to convey complex cases like whether --set myList={"bar"} means the user wants to perform a replace or a deep merge.

Note that the same message was conveyed back in 2018 in an earlier comment here. To change the behaviour now to a deep merge for arrays would introduce backwards compatibility issues and could cause more harm than good.

Hope this clarifies where this proposal stands.

Merging could always be allowed via use of a command line flag just as comment 2 described it (https://github.com/helm/helm/issues/3486#issuecomment-364534501). Normally user's values would override defaults but when a command line flag is provided (e.g. --deepMergeInputs) then a deep merge is done. That's how user would handle the need to null out existing values. Is there another complication to this approach?

Not sure what "out-of-band" means.

Merging could always be allowed via use of a command line flag just as comment 2 described it (#3486 (comment)). Normally user's values would override defaults but when a command line flag is provided (e.g. --deepMergeInputs) then a deep merge is done. That's how user would handle the need to null out existing values. Is there another complication to this approach?

An all or nothing approach like that is very dangerous when having big and/or many values files. Another problem with this is that you then can't control the behaviour in a chart. I prefer the solution I outlined in https://github.com/helm/helm/issues/3486#issuecomment-477099409

I noticed recently that using toYaml will only use the yaml from the highest priority yaml file instead of doing the deep merge like it does for range.

Helm version:
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

In fact, only list does not support merge. We agree that all list is rewritten as map， (list in k8s generally has a key called name, and this name can be used as the key of the map)

value.yaml before

# value.yaml 

volumeMounts:
    - mountPath: /data/log/
      name: logs

volumes:
    - emptyDir: {}
      name: logs

initContainers: []

envFrom: []

env:
  - name: NODE_ENV
    value: production
  - name: PORT
    value: "3333"
  - name: SOME_SECRET
    valueFrom:
      secretKeyRef:
        key: SOME_SECRET
        name: some-secret

value.yaml after

# value.yaml

volumeMounts:
  logs:
    - mountPath: /data/log/
      name: logs

volumes:
  logs:
    - emptyDir: {}
      name: logs

initContainers: {}

envFrom: {}

env:
  NODE_ENV:
    - name: NODE_ENV
      value: production
  PORT:
    - name: PORT
      value: "3333"
  SOME_SECRET:
    - name: SOME_SECRET
      valueFrom:
        secretKeyRef:
          key: SOME_SECRET
          name: some-secret

deployment.yaml before

{{- with .Values.env }}
    env:
    {{- toYaml . | nindent 8 }}
{{- end }}

deployment.yaml after

{{- with .Values.env }}
    env:
    {{- range $key, $value := . }}
    {{- toYaml $value | nindent 8 }}
    {{- end }}
{{- end }}

This will support the merge of list.

If you want to delete a configuration

env:
  NODE_ENV: null

@tarrall 's solution didn't work for map-type values, such as valueFrom. The below template code also handles this case:

{{- range $ev_key, $ev_value :=  .Values.deployment.env }}
{{- if (typeIs "string" $ev_value) }}
          - name: {{ $ev_key }}
            value: {{ $ev_value | quote }}
{{- else }}
          - name: {{ $ev_key }}
{{ toYaml $ev_value | indent 12 }}
{{- end }}
{{- end }}