Helm: Allow helm to add external files to chart deployment

Hi Lena,

Current way:
The current way to implement adding file-content is to use the "--values" n-dimensional-parameter when calling helm install. Placing every file (normal config values, SSL-stuff, license) in a separate YML file should provide better readability.

Lets assume we use "firstspirit" as chart repository name, that provides then the chart(s) to the customer. Then the customer can deploy your chart via the call:

helm install firstspirit/caas --values myvalues.yml,ssl.yml,foobar-license.yml

Within your chart you can now validate if all required keys have values set (ssl cert(s), license, ..).
The license file can be stored locally, and when sending to the kubernetes cluster they should be stored as kubernets secrets. For faster development rampup you can provide a pregenerated ssl zert as default value.

As another pros, the customer can automize the SSL cert generation by third-party tools and generate by himself the .yml files for setting your kubernetes secret values outside of Helm and passing the generated files to Helm for the "helm install" command.

Best regards,
Markus

Hi Markus,
unsure if we understand each other correctly. To elaborate a little on my use case, this is what my chart contains currently:

...
{{ if .Files.Get "config/fs-license.conf" }}
file.fs-license.conf:               {{ .Files.Get "config/fs-license.conf"                                | b64enc }}
{{ else }}
{{ fail "you need to supply your license file! add 'fs-license.conf' to your chart 'config' directory." }}
{{ end }}
...

I wanted to not force users to provide base64-encoded strings in yaml files, but instead give the files directly. However, the files need to be part of the chart itself and cannot be given from the outside, hence this ticket.

I have this exact need. My chart publishes a secret read from file at /keybase. This file is deliberately not in the chart.

I believe files for .Files.Get should not be assumed to be inside the chart:

https://github.com/kubernetes/helm/blob/master/pkg/chartutil/files.go#L32

I am quite sure .Files.Get not being able to access the file system arbitrarily is a security feature, so I don't think the current behaviour is wrong - it just does not fulfill all use cases.

One alternative solution is to distribute your certs and licences as a separate helm release. Where you put them into config maps with a name that your service deployment knows about, use a configmap/secrets volume mount in your service deployment.

Maybe that was what @cr4igo already suggested :)

We're having the same problem with a very similar use case:

We have certificates for Apple and Googles push notification services; we have different certificates for our test and production environments so we can't just bundle the certificate into the Chart (ignoring the security concerns around doing so); we would ideally like to be able to put the file path into the values file and have Helm load that file.

I don't have any good suggestions to how to tell Helm that the string in the values file is a relative file path - perhaps wrapping it in some sort of $file("path/goes/here") construct but it seems messy.

e.g.
values.yaml

apns:
  certificatePath: "./mycert.p12"

secret.yaml

...
data:
  apnsCertificate.p12: |-
    {{ Files.Get .Values.apns.certificatePath | b64enc }}

I think, this is a duplicate of #1754

I don't think it's a duplicate, this is about allowing helm to access files on absolute paths outside of helm chart "root".

Example:

data:$
  dynamicvalues.json: |-$
{{- if .Files.Get (required "Helm Error: .Values.valuesfile is required" .Values.valuesfile) }}$
    {{ .Files.Get (.Values.valuesfile) }}$
{{- else }}$
{{ fail "Helm Error: specified .Values.valuesfile could not be read" }}$
{{- end }}$

This fails to render if pointing to files inside helm repo, ie:
helm --set valuesfile=/tmp/my-dynamic-file [...]

If the same file copied to helm root and pointed with relative path, the same command works.
Example below only works if pointing to files inside helm repo, anything outside fails,

Hi,

I would like to suggest an approach similar to C-preprocessor... by default have the search path within the chart, however add --search-directories= parameter to allow additional directories. This will meet security requirements as sysadmin must specify where additional files resides, it will make the Files::glob work properly.

What do you think?

USE CASE

include additional set of files to configmap/secrets using .Files.Glob from directory outside of the chart package, without extracting the package.

PROBLEM

package must not escape authorized search directories.

SOLUTION

add --search-directories parameter to search files in additional directories specified by sysadmin, the files will be loaded into the fileset of the chart, available to the Files object.

Apologies for the delay on a response. The core team (including myself) has been focusing solely on getting an alpha release of Helm 3 out the door, so any development on significant contributions to Helm 2 have been much harder to dedicate resources for peer review and design work at this time.

The proposal in the OP (--include-file=foobar-license.conf) sounds like a reasonable solution if someone wants to tackle the feature! I'd highly suggest waiting until we have an alpha release of Helm 3 available before working on new feature work though; Helm 2 and Helm 3 development has diverged to a point where we will not be able to port over new Helm 2 features to Helm 3, which is why we've been hesitant about merging new features into recent releases and only working on bugfixes. Hope this helps!

Hello,

Any update on the status of this issue? How is it going for Helm 3?

I am asking too.. would very much like it for traefik ssl stuff f.ex. :)

AFAIK nobody from the community has started tackling this work for Helm 2.

Hi, What's the latest on this topic? We are also looking for the requested behaviour. Thanks

Same as earlier. No update.

Still waiting for an update on this

It should also be great if you provide an option like --include-folder=<path to folder> and Helm behaves like that folder is at the root of Helm chart. including a single file is not solving the issue in our case. there might hundreds of files and sub directories in that folder.

I'm marking this feature as "Help Wanted". It'd be great to have this feature, but we're busy re-working the architecture for Helm 3 that we don't have the time to focus on implementing new features like this one.

If you're interested in contributing this feature for Helm 3, please let us know so we can try to co-ordinate efforts/merge conflicts with the rest of the team.

Still waiting for an update on this

If there are further updates, we'll share them here.

Thanks!

In case it helps anyone, i've managed to work around this limitation for my needs.
I'm kinda new to helm, so it might be a well known approach

My folder structure:

├───mainfests
│   └───[app_name]
│       ├───charts
├───[app_name]
│   └───resources
│       ├───config-prod.json
│       ├───config-staging.json

My problem:

I was trying to copy the content of the config files into a k8s configMap, but i've hit the limitation where i couldn't reference the config files, as they are external to the helm chart.

My solution:

I've used the set-file to copy the content of the config file

helm install --set-file ``configValues=./[app-name]/config/config_prod.json

And in my helm deployment yaml it's referenced like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: <config-name>
data:
  {{ printf "config_%s.json" .Values.envType }}: |-
  {{ .Values.configValues }}`

(edited multiple time to fix line breaks without success)

Such a needed feature for certificates primarily.. Else it gets far too complex for the job

You could create a chart with the extra resources and then use the actual chart as a subchart through requirements.yaml

In case it helps anyone, i've managed to work around this limitation for my needs.
I'm kinda new to helm, so it might be a well known approach

My folder structure:

├───mainfests
│   └───[app_name]
│       ├───charts
├───[app_name]
│   └───resources
│       ├───config-prod.json
│       ├───config-staging.json

My problem:

I was trying to copy the content of the config files into a k8s configMap, but i've hit the limitation where i couldn't reference the config files, as they are external to the helm chart.

My solution:

I've used the set-file to copy the content of the config file

helm install --set-fileconfigValues=./[app-name]/config/config_prod.json ``

And in my helm deployment yaml it's referenced like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: <config-name>
data:
  {{ printf "config_%s.json" .Values.envType }}: |-
  {{ .Values.configValues }}`

(edited multiple time to fix line breaks without success)

what is resources files were not text or json format, in my case i have resources files as war files used for configuration of my application

I currently do it with a configmap:

 data:                                                                                                                                                                                                                              
   my-configuration.php: |-                                                                                                                                                                                                      
     {{- .Files.Get "admin-configs/my-configuration.php" | nindent 4 }}  
   other-configuration.yaml: |-                                                                                                                                                                                                    
     {{- .Files.Get "admin-configs/other-configuration.yaml" | nindent 4 }} 

and I just place the config files in the admin-configs folder (below chart) - before running helm, and it adds the content of those files to the configmap - which can then be mounted inside the pods, and land as files.

I was aiming to take a stab at this as my first contribution, since on the surface it looks like a decent, useful feature request. But now, I feel this can be addressed with existing functionality.

I've read through the comments and people talk about two use-cases: Certificates and Chart Overrides.

1. Certificates.
These go in Secrets or ConfigMaps. I am of the belief that you should set these as strings when you call helm, e.g. from a bash script. Going beyond that, you can write a script that fetches secrets from storage, e.g. HashiCorp's Vault, and populates via values at runtime.

For an example of what can be done, take a look at the awesome Helmfile project. I've used this to fetch secrets from vault into secret.yaml files, which then get passed to helm via helmfile, one per Chart, and deleted at the end.

So, this solution is flexible and scriptable enough. In my mind, Helm should not try to manage secrets from outside its immediate scope, which is a Chart; this includes having to fetch secrets from anywhere, even files.

2. Chart Overrides.
The new library chart feature of Helm should solve this use-case. A library chart, together with an example "install" chart, should be enough to get people started with using charts that require customizations beyond what can be done with simple values. I believe this would be an elegant solution to the OPs problem.

@bacongobbler, you set this issue as Help Wanted, so I'm tagging you. Any thoughts?

Hey @vladfr - I partly agree, but not 100% :). I am talking about the certificates use-case now, and it is purely based on personal experience, which means that it may not be widely applicable. For an internal project, I'd agree. You want to store your certificates in something like vault. Writing a script that fetches data from there and simply adds the base64-coded value via parameters is fine. You have control, no problem.

For our software, the distribution works via helm charts. People buy our product and install it on-premises in their kubernetes cluster. It's not that people installing the software usually have a load of experience with kubernetes, and they use a variety of operating systems (have seen all of linux, mac, windows) when calling helm. We do not provide certificates ourselves, customers buy them elsewhere; we supply helm charts and they add certificates. We do not have control over their CI/CD pipelines either. It already is problematic complexity-wise with k8s and helm itself.

I have had support use-cases where people did not succeed in converting their certificate to base64 correctly - may it be line feed problems or alike. We have less problems telling people to unzip the chart, place the certificate at a certain location and then install the chart from the directory. I know this is very far from ideal, but it works - by far - better than saying "make sure you add the base64-encoded certificate here". But it is cumbersome.

To add another use-case we have, besides certificates: licenses (e.g license key files) that restrict software usage. That case is easier in that we could provide licenses in a base64-encoded format already to allow to set it via --set, or as a preconfigured values.yaml that you could just apply.

So, I still think the feature would be very useful.

@lenalebt thanks for describing this use case in such detail. I agree, in this case it's cumbersome. I still feel this should be handled outside helm, but if the maintainers feel otherwise, I'd be happy to take a stab at implementing this.

Thank you, @vladfr.

I'm siding with @lenalebt on this one. It's possible to work around this, but if you are actually selling a product based on k8s/helm, the added complexity to the end solution could render negative points to your show case. You either demand the customer to handle some technical details manually, or hide them behind wrapper scripts or GUI interface that ammount to maintenance cost overtime.

I'm sure there are a few ways to implement this, and community feedback can help determine the most elegant way to go.

Personally, I would provide a new function on the lines of .Files.GetExternal, which could be well documented as an unsafe alternative to .Files.Get.

Please consider implementing it. Even kustomize is providing command line flag to allow external files to be accessed.

Seems like there's more than enough consensus for this feature. We are looking for volunteers willing to put the time forward to implement this. We don't really have the time to write this ourselves but we are willing to help others through the learning process. @solacens would you be willing to put in the time to implement this?

@bacongobbler I'd like to take a stab at it. I've been looking for a good first issue to start contributing.

Great!

I'd check in with @ankul to make sure nobody's stepping on each other's toes. He messaged me on Slack earlier today expressing interest in implementing the feature as well.

For the proposed --include-file=foobar-license.conf flag:

• Do we expect users to collect all files in one directory and name them appropriately? If so, how are appropriate names communicated to the user?
  Quoting from https://github.com/helm/helm/issues/1754#issuecomment-271381902

  I see values.yml (and --set) as sort of the "interface" in which inputs are passed to helm.

  In this approach, filenames would now be an arbitrary input to the chart.

• Should we have a files construct similar to values to list what file names the chart expects? Users can pass those files as --include-file key1=path1 key2=path2 where expected keys are listed in helm show files sourced from files.yml.

  .. Or just have the user specify filenames as values?

@juliohm1978 I'm not sure how Files.GetExternal would be unsafe (or why it's needed at all). These files could be made available through the same methods existing on the Files object (Get, GetBytes, Glob, Lines etc).

I hope I'm not missing something that has already been discussed so far.

@anukul you make a good point about discovering what file names to expect. Documentation is one way but it's not ideal.

I would like to suggest a different approach.

The use-case is to get file contents into a value. How about reading the file directly into a value variable? Something like: --set certificate={{file://path/to/your/file}}. Helm would need to parse every value and see if it fits the file:// scheme, then just read the file as a string and set the key. This would work via yaml as well.

This way, file names are no longer relevant. The chart authors would have control over where to receive the file contents, and users can easily pass arbitrary files as values, anywhere they like. Files.Get wouldn't be used anymore.

Does that make sense?

that sounds more like --set-file, which already exists today. In your example, you'd use --set-file certificate=/path/to/your/file, then call .Values.certificate in your templates to access the file's data.

https://github.com/helm/helm/blob/07a3d7299c7fa9e0e36953623b6a4917f06bbe53/cmd/helm/flags.go#L37

Hi,
What I had in mind is like -I switch for cc or -L switch to ld, add additional directories into file search path.
Currently, only the chart directories are searched for files when iterating/reading generic files, while the requirement is to search out-of-tree files in the same manner.
For example, a chart may have a directory in which stores configuration files *.yaml, the chart iterate all files in this directory in order to create a configmap with entry for each file.
The requirement is to allow the chart to search for additional files outside of the chart root or tarball, so that in this case the sysadmin can add additional *.yaml files to be injected into the configmap.
This can be easily established with -I<directory> (with option to multiple locations) to search for additional shadow directories, without changing the chart logic.
Thanks!

Is it possible to specify a path for config files in my values.yaml and use that path to read the contents to create multiple config maps?
I basically want it to generate the following yaml
Say I have another git repo where my config files are stored and I want to pull them down in a directory outside the helm chart. I want to be able to reference that directory and grab all of the files within it. We could probably just do a single layer first. no need to get recursive yet.

apiVersion: v1
kind: ConfigMap
metadata:
  name: conf
data:
  boot.properties: |
    <paste file content here>
  jndi.properties: |
    <paste file content here>

I understand it is a security feature, but some of us are actually authors of the helm charts and we trust ourselves that we are not trying to attack oruselves. So I suggest adding possibility to disable this security feature (--allow-all-files)

FWIW in case someone is looking for ConfigMap creation: https://helm.sh/docs/chart_template_guide/accessing_files/

@ravindranathakila

Helm Documentation does not work.

Sad to see such an important feature being still un-implemented. I expected this to be one of these basic things.

@theAkito, as I've mentioned a few times earlier in this thread, we are looking for volunteers willing to put the time forward to implement this. As core maintainers, a large majority of our time is spent helping others through the process of contributing fixes and features to Helm. We simply do not have the time to write new features ourselves, but we are more than willing to help others through the learning process.

So far, I have not seen any pull requests implementing this feature as suggested. We are still looking for a volunteer.

I see that you feel quite passionate about this feature. How would you feel about working on implementing this?

Even now you can add any external files to an installation since Post-Render Hooks are introduced

https://github.com/helm/helm/issues/7260

I see that you feel quite passionate about this feature. How would you feel about working on implementing this?

Sure, if it wasn't written in Go. It feels like every other Github user knows how to Go, so shouldn't be a problem to find someone interested. Actually, I already have a couple of people here on Github on my mind that I could ask, as they are involved in Kubernetes-related work and all of them do Go a lot. Finding a Go programmer is like looking for a sand grain on a beach.

Here is an example of how to use the kustomize via Post-Render Hooks
https://github.com/helm/helm/issues/7260

Here is an example of how to create config from external file:
https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/#configmapgenerator

I believe this issue can be closed.

Here is an even simpler example:

sh.sh

#!/bin/bash

sed "s/TOKEN_TO_REPLACE/$(cat file.txt)/"

file.txt

some content

enjoy:
helm install --post-renderer ./sh.sh stable/some_helm_chart

you just need to create config map with fake content TOKEN_TO_REPLACE

_but can be problem with line indentation in yaml_

@R-omk, as said in the doc:
"_The resources that a hook creates are currently not tracked or managed as part of the release._"

IMHO, the solution is the one suggested previsouly, ie have a flag to add directories to the search path of the ".Files" function.
IMHO there is no security issue as the user of the chart delibirately specifies the place where the chart can play with files, and this is not the chart that could manage files in arbitrary places

What about a new "--search-folder" flag that could be set multiple times? And as for the files filtering issues, it could implement exactly the same rules as currently implemented by .File.Globetc, so no new complexity or descripencies here

@titou10titou10

"The resources that a hook creates are currently not tracked or managed as part of the release."

Its about hooks: pre/post - install/delete/upgrade/rollback

post-renderer is a completely different kind of hook

@R-omk, fair enough. The doc is lagging on this
But I still can't see how I can solve our problems with post-renderer hoosk.
One of our use case:

• we use a jenkins pipeline to build a java app and deploy it into OpenShift
• we were using OpenShift "templates" for this and trying to move to Helm because it is way more flexible and we created a custom Chart for deployment
• the deployment process includes creating one ConfigMap with various classic properties files dynamically created by the build, they are all in the same directory (the build can put them in any place..)
• this is a perfect candidate for the "Files.Glob(<ext dir>) / AsConfig()" function, unfortunately as the properties files are created outside the Chart, it is not currtently possible
• remarks: the names of the properties files is dynamic, the number of files is also dynamic, the build groups them in one directory

Maybe using post-renderer hook could help us but I don't see how, and IMHO it would just be a workaround but all "workarounds" are welcome at this stage
And either adding a "--search-folder" flag or adding a .Files.GlobExternal() / .Files.GetExternal()functions as someone suggest seems to be a valid solution but the first is more secure IMHO as this is nit the Chart that aloow access to local folders

Quick reminder: we will not be accepting PRs that introduce a .Files.GetExternal or a —search-folder flag. Those can introduce security issues on the installer’s side. One could potentially write a malicious chart that copies a sensitive file and ship it elsewhere with a chart hook.

However, an explicit —include-file or —include-dir would be accepted, as mentioned in an earlier comment: https://github.com/helm/helm/issues/3276#issuecomment-412942372

There are cases where you’d want to include files as part of the rendering stage for those templates. A post-render may not handle all cases.

@bacongobbler good enough.
Let's hope an --include-dir new flag will allow to use the .Files.Glob(<dir>).AsConfig() or an equivalent feature to fullfill this use case, which I guess, is very common in a pipeline

Or maybe Helm in not the right tool in CICD pipeline scenario where many "data" can not be expressed in a pure "value" way (via -f, --set or--set-file) and is held outside the chart

For now the only workaround I see is to fetch and untar the chart somewhere, copy the externals files under the chart tree, the run helm from that local copy

I don’t understand why the raw files inside the release?

helm v3 no longer run server side template rendering. Therefore, you cannot use Files.Glob.
This issue actual only for helm v2.

@bacongobbler I'm 100% down to add this feature. My proposed implementation (inspired by kubectl create secret generic --from-file=...) would be:

helm install --include-file [<key>=]<path>[,[<key2>=]<path2>...] --include-dir [<prefix>=]<dirpath>

where <key> is the path that will be accepted as an argument to .Files.Get, and defaults to the basename of <path>. E.g., --include-file foo.txt=../dir/bar.txt would make the contents of ../dir/bar.txt accessible via .Files.Get "foo.txt".

For directories, all contents of dirpath will be made available under their path relative to <dirpath>, with an optional prefix appended. E.g., if mydir/ contains foo.txt and bar.txt, and I invoke this as --include-dir mydir/, files foo.txt and bar.txt become visible; when invoked as --include-dir baz/qux=mydir/, those files are visible as baz/qux/foo.txt and baz/qux/bar.txt.

If that sounds reasonable, I can get started this weekend.

(The main downside is that we/my employer won't be able to use this for a while anyway, as we'll have to support older versions of helm as well, but I guess better late than never)

That sounds great, thanks! I look forward to the PR.

@misberner can you confirm that using--include-dir <prefix>=<dirpath>will allow us to use .Files.Glob(<prefix>).AsConfig(), and so create a ConfigMap with one entry in the CM per file in<dirpath>?

Yeah that's the idea. An open question from my point of view is whether an --include-dir with a specified <prefix> introduces an overlay, or shadows everything under <prefix>/ from previous args and from the bundle itself. I'm not super opinionated on that one but would prefer the former.

ping @misberner. Any progress?

Hey! I just found another case where this feature would be useful.
@bacongobbler I see this is still open - is this issue unassigned? I may have some time next week to take a look.

Feel free

Spent some time, I have the file flag working, and planning to add a fir flag as well.

I have worked around this by deploying chart incubator/raw as 2nd chart

The PR is now open for review.

@vladfr seems the PR is slotted for the 3.4.0 release.

We talked about it last week on dev call. @vladfr was present and raised the question whether we could get this in for 3.3.0.

With our current workload for 3.3.0 we don't have enough time to invest reviewing this PR. Right now we're reaching out to community members who have written PRs for 3.3.0 and asking them to fix their PRs before the merge window closes and we cut the release. E_TOO_MANY_PRS_TO_REVIEW

Once 3.3.0 is out the door we can revisit this conversation.

following

following too

work around was:
{{- range $datakey, $dataval := $cm.fileList }}
{{- $datakey | nindent 2 }}: |-
{{- $fullPath := printf "configFiles/%s" $dataval }}
{{- $.Files.Get $fullPath | nindent 4 }}
{{- end }}

any updates on this, given we are post 3.3.1

Using --include-dir seems like a good approach. Looking forward to v3.4.0

Hi. How is this feature coming along? I see there is a closed and an open PR, last reference about a month ago?

this functionality would be so useful - can we please have it soon ?

Just to add my +1 to this please. I've just spent 4 hours trying to figure out how my syntax was wrong; when in fact it was silently ignoring the fact I was passing in an absolute path outside of the chart deployment folder. Didn't work that out until I found this issue!

In unix systems access to files in directories outside of the helm chart directory tree can be achieved using symbolic links (at least using Helm 3.3.4)

Just to add my +1 to this please. I've just spent 4 hours trying to figure out how my syntax was wrong; when in fact it was silently ignoring the fact I was passing in an absolute path outside of the chart deployment folder. Didn't work that out until I found this issue!

@catdevnull -- Went through the same situation. +1 to the issue.