Helm: Nested `null` values don't remove keys as expected

@scottrigby, do you by any chance want to take a gander at this one?

Marking as question/support until it's been confirmed that this is indeed a bug according to the original author.

I'm seeing this as well on 2.12.3. I'm seeing while trying to set the value to null via both -f and --set. Either way it just sets it to "null" as a string (without the quotes) rather than removing the default value.
My situation:

> helm get values grafana
admin:
  existingSecret: ""
chownDataImage:
  pullPolicy: null
  repository: null
  tag: null
<...>

> helm upgrade grafana stable/grafana --tls --reuse-values --set chownDataImage.pullPolicy=null
<output indicating it works>

> helm get values grafana
admin:
  existingSecret: ""
chownDataImage:
  pullPolicy: null
  repository: null
  tag: null
<...>

Use case is to reset these values to use the defaults from the template instead of the defaults now stuck in helm.

stable/filebeat

With null or nil I'm seeing the same thing... the key isn't removed... it's overwritten as null/nil but not removed and when you are replacing it with different key it causes problem:

---values.yml in the Chart
output.file:
  path: /var/log/foo.log

```yaml
---my overrides
output.elasticsearch:
hosts:
- 'http://localhost:9200'
output.file: null

or
```yaml
---my overrides
output:
  elasticsearch:
     host:
      - 'http://localhost:9200'
  file: null

or

---my overrides
output:
  elasticsearch:
     hosts:
      - 'http://localhost:9200'
output.file: null

» k get secret filebeat -o jsonpath="{.data.filebeat\.yml}" | base64 -D
    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
      prospectors:
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: false
    filebeat.prospectors:
    - enabled: true
      fields:
        apenv: dev
        app: kubernetes
        log_category: kubernetes
      fields_under_root: true
      paths:
      - /var/log/*.log
      - /var/log/messages
      - /var/log/syslog
      type: log
    - containers.ids:
      - '*'
      fields:
        apenv: dev
        app: kubernetes
        log_category: kubernetes
      fields_under_root: true
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_event:
          when:
            equals:
              kubernetes.container.name: filebeat
      type: docker
    http.enabled: true
    http.port: 5066
    output:
      elasticsearch:
        hosts:
        - http://localhost9200
    output.file: null
    processors:
    - add_cloud_metadata: null

pod errors as:

Exiting: error unpacking config data: more than one namespace configured accessing 'output' (source:'filebeat.yml')

I'm also encountering this exact issue with the filebeat chart.

@cdenneen You can get around file output specifically with config.output.file.enabled=false.

I'm encountering an issue where I want to use the new inputs key for filebeat but cannot remove the prospectors key.

Existing values:

config:
  filebeat.prospectors:
    - type: log
      enabled: true
      paths:
        - /var/log/*.log
        - /var/log/messages
        - /var/log/syslog

My override: --set config.filebeat.prospectors=null

Result: (config is used to set a Secret value)

filebeat:
  prospectors: null

I also encountered this problem with stable/kibana and stable/filebeat the keys will default to the chart values even when !!null is specified.

@aeijdenberg believe your PR just requires labels updated to be reviewed

Hey @bacongobbler sorry I missed your question in https://github.com/helm/helm/issues/5184#issuecomment-456138448. I can verify it is a bug. In #2648 I really should have added a test that matched the documented example. I had meant to get back to this but #5185 looks promising! 👏 will respond over there

Hi @scottrigby . May I know which release will include #5185's fix? I just encountered this problem when trying to remove resources definitions on istio chart.

This is supposed to be in 2.14.2 - but I can't get it to work.

I'm following a very similar example to that at the top of the issue - the only difference I can see is that I'm trying to delete a value set in a subchart (in my case, logstash).

I have tried:

• setting logstash.livenessProbe.httpGet to null, ~ and {} in the main chart's values.yaml, or on the command line. In both cases, "helm template" shows that the original value is still there (i.e. I'm still getting httpGet set) - and this is consistent with what I see for "helm install"
• setting logstash.livenessProbe.httpGet to "nil" - in this case the value does get overwritten, but to the string "nil" which isn't a valid value to use here, so kubernetes rejects the template.

Have I misunderstood and this didn't make it into 2.14.2 - or is there still an issue?

(Chart demonstrating this: demo.zip)

The patch was cherry-picked into 2.14.2 according to the release notes so it should be available.

In which case, I'm not sure this bug is fixed - should it be re-opened or a new issue created? Can someone else have a look at the chart I attached to the above and see if they see different?

@cc-stjm @bacongobbler - I am able to reproduce the issue, and I think this test demonstrates it:

func TestSubchartCoaleseWithNullValue(t *testing.T) {
    v, err := CoalesceValues(&chart.Chart{
        Metadata: &chart.Metadata{Name: "demo"},
        Dependencies: []*chart.Chart{
            {
                Metadata: &chart.Metadata{Name: "logstash"},
                Values: &chart.Config{
                    Raw: `livenessProbe: {httpGet: {path: "/", port: monitor}}`,
                },
            },
        },
        Values: &chart.Config{
            Raw: `logstash: {livenessProbe: {httpGet: null, exec: "/bin/true"}}`,
        },
    }, &chart.Config{})
    if err != nil {
        t.Errorf("Failed with %s", err)
    }
    result := v.AsMap()
    expected := map[string]interface{}{
        "logstash": map[string]interface{}{
            "global": map[string]interface{}{},
            "livenessProbe": map[string]interface{}{
                "exec": "/bin/true",
            },
        },
    }
    if !reflect.DeepEqual(result, expected) {
        t.Errorf("got %+v, expected %+v", result, expected)
    }
}

The problem, so far as I can tell, is that CoalesceValues() results in more than one call to the underlying function that coaleces the values:
https://github.com/helm/helm/blob/e04fa72f6f211cae68c362f9b7c62f06dc51493e/pkg/chartutil/values.go#L164-L180

ie line 173 above is called with the httpGet set to null, but the value it returns has deleted that key from the map (as intended). But then that output is passed a 2nd time as input to the 2nd set of coalescing (line 179), and then since key no longer exists, it defaults to the value in the chart.

Unfortunately I'm unlikely to have time available in the near future to go any further - I've changed roles and am not currently using Helm, and the answer on how to solve isn't obvious to me. Hopefully the above is helpful in resolving.

In fact, I just upgraded 2.14.0 => 2.14.2. Not only does the nulled key still exist, but it also contains the previous value. The previous behavior just made it null, so I believe this has actually regressed.

@aeijdenberg can you please look into @sgillespie's inquiry? If there's a regression, then it's probably safer to back that PR out unless you can determine a fix. If you're unable to help out then it's probably safer to revert your commit and go back to square 1. Let me know how you'd like to proceed.

@bacongobbler , while I haven't explictly tested 2.14.0, what @sgillespie matches the behaviour I referred to in https://github.com/helm/helm/issues/5184#issuecomment-517059748. Sadly I think it's a case of the unit tests passing, but the end to end failing - as the individual components are being used in series, which makes removing a key in an earlier stage makes it have no effect in later stages (and is why the original values are coming through).

I agree it's a regression, though backing it out is also a regression for anyone relying on the new behaviour.

I've had a quick go at a relatively minor patch which I think will reduce the impact (and fixes the test I added above) in #6146 - I'm sorry we got this wrong the in last attempt.

This might've been fixed in https://github.com/helm/helm/pull/6080. That PR fixes the case where coalesceDeps is called twice. Can someone confirm with master?

If so, can #6146 be closed, or is it attempting to fix a separate issue? I'm trying to grasp my head around the problem space here but it's unclear what these PRs are attempting to solve. Sorry.

This does indeed seem to fix the issue. A quick naive test:

Subchart values:

prop:
  nested:
    val: true

Parent chart values:

sub:
  prop:
    nested: null

Output as expected is {}

Hi,

I'm using helm 2.15.0 and I've still met this issue.

Given a subchart with values:

securityContext:
  runAsUser: 65534
  fsGroup: 65534

Where the values are used with toYaml

And in a parent chart values:

sub:
  securityContext:
  runAsUser: null

Then the actual output is:

securityContext:
  runAsUser: 65534
  fsGroup: 65534

While it should have been:

securityContext:
  fsGroup: 65534

Not trying to rain on your parade, but I'm still seeing this with helm v3.0.1

version.BuildInfo{Version:"v3.0.1", GitCommit:"7c22ef9ce89e0ebeb7125ba2ebf7d421f3e82ffa", GitTreeState:"clean", GoVersion:"go1.13.4"}

When trying to install stable/ignite I have to unset a value, but helm appears to just set the value to null/nil, causing k8s to balk in the validation step.

To reproduce, save this as bug5184-ignite.yaml (to override values from the chart's defaults: https://github.com/helm/charts/blob/master/stable/ignite/values.yaml):

persistence:
  enabled: true   # <-- without this, the keys in question are ignored
  persistenceVolume:
    provisionerParameters:
      type: null  # <-- I want to unset this key
  walVolume:
    provisionerParameters:
      type: null  # <-- I want to unset this key

Then use it as values file for an ignite install:

helm install runtimedb stable/ignite --version 1.0.1 --values bug5184-ignite.yaml --debug --dry-run | less

The result I'm getting is this error message, showing that the value I wanted to unset has been set to nil:

install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /home/creinig/.cache/helm/repository/ignite-1.0.1.tgz

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: unknown object type "nil" in StorageClass.parameters.type
helm.go:76: [debug] error validating "": error validating data: unknown object type "nil" in StorageClass.parameters.type
helm.sh/helm/v3/pkg/kube.scrubValidationError
        /home/circleci/helm.sh/helm/pkg/kube/client.go:520
helm.sh/helm/v3/pkg/kube.(*Client).Build
        /home/circleci/helm.sh/helm/pkg/kube/client.go:135
helm.sh/helm/v3/pkg/action.(*Install).Run
        /home/circleci/helm.sh/helm/pkg/action/install.go:229
[...]

What I've also tried:

• Setting the value to something different / not overriding it at all
  
  
  
  • => The PersistentVolumeClaim is stuck in "Pending" because there's an extra "type" option that's not supported by the provider
  
  
  • => it's definitely that value causing the problems
  
  
• Setting the value(s) to null via --set
  
  
  
  • Same behavior as when setting it in the file
  
  

When only setting a single value to null via the command line, while at the same keeping ignite persistence disabled (so that the storageclass template is not generated and the parameter in question is ignored)...

helm install myignite stable/ignite --version 1.0.1 --set persistence.persistenceVolume.provisionerParameters.type=null --debug --dry-run | less

... there is proper debug output instead of an error, but it shows that the value is just set to null (comments added by me):

NAME: myignite
LAST DEPLOYED: Tue Dec 10 09:43:40 2019
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
persistence:
  persistenceVolume:
    provisionerParameters:
      type: null   # <-- Set via the command line

COMPUTED VALUES:
affinity: {}
dataStorage:
  config: ""
env:
  IGNITE_QUIET: "false"
  JVM_OPTS: -Djava.net.preferIPv4Stack=true
  OPTION_LIBS: ignite-kubernetes,ignite-rest-http
fullnameOverride: ""
image:
  pullPolicy: IfNotPresent
  repository: apacheignite/ignite
  tag: 2.7.6
nameOverride: ""
nodeSelector: {}
peerClassLoadingEnabled: false
persistence:
  enabled: false
  persistenceVolume:
    provisioner: kubernetes.io/aws-ebs
    provisionerParameters:
      fsType: ext4
      type: null  # <-- Set to null instead of removed
    size: 8Gi
  walVolume:
    provisioner: kubernetes.io/aws-ebs
    provisionerParameters:
      fsType: ext4
      type: gp2  # <-- This is what the default looks like
    size: 8Gi
rbac:
  create: true
replicaCount: 2
resources: {}
serviceAccount:
  create: true
  name: null
tolerations: []

I have ran into the exact same issue using 3.0.1, it might be good to reopen this issue ?

After moving from helm v2.16.1 to v3.0.2. I have had this issue with annoations unsetting and cpu limits.

I just encounter this issue in 2.14.1 and 3.0.2. Any update on this?

In cases where I know the helm chart is simply using and true/false if test, I override the value with a non-table value like 0. I get a lot of warnings like Overwriting table item 'x', with non table value: 0, but I at least get a workaround in cases where I don't want a stanza included. My preference would be for null to work, but this is an ugly workaround.

I have set values to null (and left them null in my values files) and ignore the long list of table warnings which is OK for now but I'd really just like to do a one-time cleanup of the old incorrect values stored in kubernetes. Until this value deletion bug is fixed, is there a manual way to edit the values on an existing deployment with no downtime?

Running into the same issue with liveness and readiness probes.
Deleting Default Key from the template guide does not work.

Please give #7743 a try. It looks like the commits made to Helm 2 were not ported over to Helm 3, which is why many users would be seeing this behaviour over there.

Still seeing this problem with helm 2.16.3 but only when there's a requirements.yaml listing the subchart as dependency.

boo is the subchart and foo is the parent chart:

vibu@item-ax35755:~/work$ cat boo/values.yaml 
object:
  fromSubchart:
    hello: from boo
vibu@item-ax35755:~/work$ cat foo/values.yaml 
boo:
  object:
    fromParent:
      hello: from foo
    fromSubchart: null
vibu@item-ax35755:~/work$ cat boo/templates/test.yaml 
{{ toYaml .Values.object }}
vibu@item-ax35755:~/work$ cat foo/requirements.yaml 
dependencies:
- name: boo
  repository: file://../boo
  version: 0.1.0

vibu@item-ax35755:~/work/foo$ helm version -c
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
vibu@item-ax35755:~/work/foo$ helm dep up
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
Saving 1 charts
Deleting outdated charts

vibu@item-ax35755:~/work/foo$ helm template .
---
# Source: foo/charts/boo/templates/test.yaml
fromParent:
  hello: from foo
fromSubchart:
  hello: from boo


vibu@item-ax35755:~/work/foo$ mv requirements.yaml{,.bak}
vibu@item-ax35755:~/work/foo$ helm template .
---
# Source: foo/charts/boo/templates/test.yaml
fromParent:
  hello: from foo

Both charts here:

helm_5184.zip

@bacongobbler is there any intention of further addressing this in 2.x?

We also hit this issue (similar to one described by @vbuciuc) with not working "null" for values override when subchart is listed as a dependency in requirements.yaml with 3.2.x. Previous versions (3.1.x) work as expected.

@bacongobbler @technosophos I'm running into this issue as well with Helm 3.3.4, can you please re-open;

I can confirm that it worked in 3.1.2 and it stopped working after 3.2.x as @paleg mentions. For now, it appears that @tuzla0autopilot4 's workaround of setting it to a non-map value like false will produce the intended behavior but does give output warnings when that is done.

@Chili-Man , I have just tried with 3.1.3 and it doesn't seem to work. It will apply with null but I get a warning and it doesn't do anything. With anything else (false, 0, []) it refuses with (e.g.,)

```coalesce.go:196: warning: cannot overwrite table with non table for resources (map[requests:map[cpu:250m memory:256Mi]])
Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
postgresql:

• resources: Invalid type. Expected: object, given: boolean
  `` (or what the type that I try that is not a dict/hash). With3.3.4I get just silence and it does nothing withnull` and likewise everything else refuses to apply. Most annoying...

I'm still experiencing this issue in 3.4.2.

The issue is still present in 3.4.2, but only in certain situations.
I have a chart with multiple sub-charts.
When putting default into sub chart values null override does not work. If you move the same values into parent chart values it works as expected.
This is reproducible only with sub-charts. When you put it in a single (flat) chart everything works fine.

I am experiencing the same problem with 3.4.2 when using subcharts (as described by @BohdanKalytka above).
In my case I want to overwrite the securityContext when using the ElasticSearch helm chart in an OpenShift cluster.
Is this issue going to be reopened, or should we create a new one?

Raised #9136