Firebase-tools: Accept JSON file for functions:config:set

I agree with @jpreynat, we're dreading the day we forget to set a config var before deploying to our production. Tying config:set into our custom deploy scripts with a config.json would be a nice feature to see added.
Are there any workarounds right now, maybe piping in the config?

Seems like you're trying to replicate config across projects. Would firebase functions:config:clone work for you?

To give some context, we've intentionally leaned away from file-based config because we think it encourages bad practices (namely, keeping a file full of secrets stored with your code, potentially even checked into your source repository).

What would you think about some way (mechanics TBD) to declare "required" config variables -- and if you tried to deploy without required config in the current project it would error out before anything was changed. Would that solve your main concerns? If not, why is having a file specifically useful to you?

@mbleigh That's exactly what I didn't know I wanted. Nice idea!
Probably a given/what you meant by 'Mechanics TBD', but: Defining the keys/config structure in some file within our project rather than remotely, so that it goes through the git/PR review process like everything else.
@jpreynat Sorry for hijacking issue

@laurenzlong Thanks for the advice, but we are not trying to duplicate a config.
Our main concern is to update different projects configuration depending on our deployment stage.
So basically, we have a config file per environment (dev, staging and prod), and set the Firebase config respectively (for PRs, push on master and tags).

@mbleigh Thanks for the information.
However, we aren't exactly sure how we can store all these envs another way than using config files.
They are currently formatted as JSON for reading simplicity, but we are thinking of formatting them as key/value pairs to pass them directly to the functions:config:set command.
I'll agree with you that this is no more secure than passing a JSON file.
Do you have any recommendations on how we could store our environment variables for different stages other than using files?

@ahaverty No worries, I'm glad that this can be debated here.

@jpreynat when you run functions:config:set it is stored with the current project. If you're using firebase use to switch between projects, your config will be persistent. So you can run set once per project and then it will always be there. This should solve the "different config for different environments" issue, so maybe I'm not understanding the real probleM?

@jpreynat I'm going to close this issue for now, but please re-open if Michael and I have misunderstood your request.

@laurenzlong @mbleigh Sorry that I didn't answer quickly on this thread.
We found a workaround for this issue for now, but here is some context to clarify my request:

We are deploying our app using Travis (note that this would be great with any CI or even locally). Since we deploy our app to different stages depending on the branch and tag, it would be great to be able to use a different JSON file based on the environment.
For instance, this would allow to do either firebase config:set -f staging.json or firebase config:set -f prod.json depending on this condition.
Our workaround was to use firebase as a node module, but we still have to inline our JSON configuration, which is prone to errors.

Hey @jpreynat I'm still a bit confused by your answer. Once you set the config once, it always stays there, even across deployments, unless you explicty unset the config values by running firebase functions:config:unset. So there's no need to have this as part of the CI process.

Can somebody reopen this? %) @jpreynat @mbleigh That's really one of the things which is missing. Atm I have near 15 config variables (and more coming) and that will be very handy if functions:config:set can accept JSON in some way.

Big vote from us for @mbleigh suggestion on "declare "required" config variables -- and if you tried to deploy without required config in the current project it would error out before anything was changed."
Becoming a pain point for us too, but that would solve all our issues and would fit nicely into the version control/review process 👍

https://medium.com/@AllanHasegawa/setting-config-for-firebase-cloud-functions-with-json-136f455e7c69

Hey @yaronyosef You can directly supply JSON to the firebase functions:config:set command, it's just that you have to format it in a way that's appropriate for your platform. For Linux, the following should work:

firebase functions:config:set foo='{"bar":"something","faz":"other"}'

Thank you for your sharing! @laurenzlong
I wanted to import from a file, so I used command like this.

firebase functions:config:set service_account="$(cat service-account.json)"

I imported a service account json file to Firebase Cloud Functions, because I wanted to use the user management API of Firebase Admin SDK.

Cool use case! Thanks for sharing!

We would also prefer tracking non-secret configs through Git, where the committed JSON runtime config files (for multiple environments) will be authoritative and the CI environment would then apply functions:config:set from the file corresponding to a particular environment (and each environment corresponds to a separate Firebase project).

If you keep configs in git, it's trivial to read them into your functions runtime. Just put them at configs/<project_id>.json and then:

let config = {};
try {
  config = require(`./configs/${process.env.GCLOUD_PROJECT}.json`);
} catch (e) {
  console.log('No config found for project', process.env.GCLOUD_PROJECT);
}

The runtime config used by Firebase is specifically designed to avoid having config checked into git, as (especially with secrets) we've found it to be a pattern that causes more problems than it solves.

That's true, although then we no longer use Config API for this purpose at all. It may be beneficial to still go through Config API for storing sensitive values (which seems to be an appropriate use case, based on Firebase docs) and load both source-controlled and untracked values in the same exact way (i.e. through config() provided by firebase-functions).

For the time being, I'm using the following one-liner to feed the entire .runtimeconfig to functions:config:set:

firebase functions:config:set $(jq -r 'to_entries[] | [.key, (.value | tojson)] | join("=")' < .runtimeconfig/${FIREBASE_ENV}.json)

where FIREBASE_ENV is the name of the environment used in deployment (e.g. dev). It would be nice to be able to shorten this further, e.g.

firebase functions:config:set -f .runtimeconfig/${FIREBASE_ENV}.json

Ultimately though, even that may not be necessary if Config API enabled simple-to-use audit trails of all changes (please correct me if that's already the case, I'd like to explore that).

So how is this coming along? 😄

Would love to see @mbleigh suggestion be implemented still! We're constantly having issues forgetting to set configs on our non-dev targets

Thanks for the thread bump. Nothing to report yet, but I'll bring this up again in our planning meetings to see if we can start to make some progress on it.

I validate my config with a JSON schema file to ensure it has all the values I expect. This validation is part of my CI pipeline and prevents deployment if it fails. If like me you use TypeScript to write your functions, you may generate the JSON schema from your types.

It works great, with the limitation that the functions config only accepts strings, so the schema cannot validate that a config value is a number. For booleans it's OK because I can use an enum with "true" and "false" as the only valid values.

See my CircleCI config and this blog post for reference.

@hgwood thanks for sharing! We will definitely be adding this to our CI too.

Just wanted to leave my solution to this "problem", and my thoughts on the discussion:

1. I understand that secret/sensitive values should NOT be stored in the repo. My solution is only for non-secret values. Secret values are set manually, but a way to mark config values as required would be wonderful for those secret values.
2. I think this is more important than some seem to think. It's a PITA to have to manually set individual configuration values for different deployment environment projects, and not be able to simultaneously view/compare them easily. Defining configurations in files makes the process a LOT less of a hassle.
3. I think adding a functions config editor to the web console would improve this a lot, but it would be really nice if there was a way to simultaneously view configurations for multiple projects, so you could easily compare configurations for different deployment environments.
4. I have been saying forever that firebase needs to allow environments to be defined within a single project, so you don't need multiple projects to use as deployment environments. If this existed, it'd be very easy to simultaneously view configurations for multiple environments within a single project.
5. My solution probably has issues, don't trust my code verbatim, but I do like the general concept.

My solution

I use cascading yaml configuration files, which is useful for default values and values that apply to all configurations regardless of deployment environment.

./config/default.yml

app:
  name: My App
featureFlags:
  someFeature:
    enabled: false

./config/dev.yml (similar config files for other deployment environments)

imports:
  - {resource: default.yml}
app:
  environmentName: Development
services:
  someOtherService:
    baseUrl: https://dev.someotherservice.com/api
featureFlags:
  someFeature:
    enabled: true

Then, I've written a small node script that loads the resolved configuration, and returns a space-delimited list of key=val pairs

yaml_to_keyval.ts

import * as configYaml from "config-yaml"
import * as flat from "flat"


const yamlFile = process.argv[2]
const config = configYaml(yamlFile)
const flatConfig = flat(config)

const key_val_list = Object.keys(flatConfig).map(function(key){
    return `${key}=${flatConfig[key]}`
})

console.log(key_val_list.join(' '))

During CI deployment, I use this bash shell script to obtain the key=val pairs and set them with functions:config:set

# env is set to the desired deployment environment (eg "dev"), which is passed as an argument to the CI script command
config_in="$(readlink -f ./config/$env.yml)"
key_vals="$(ts-node ./scripts/node/yaml_to_keyval.ts $config_in)"
firebase functions:config:set $key_vals

@kanemotos I like your cat solution, but did you not encounter a Precondition check failed error when trying to upload your cert file?

We ended up using Deployment Manager for these reasons.

We set all configuration through "config Yaml" for DM. Then DM creates appropriate Runtime Configs and Variables, and firebase deploy retrieves and sets their values at deploy time without having to use firebase config:set at all. This also allows us to set up other resources (e.g. buckets and their access control) and set their values in RC without having to supply them externally to the template. Additionally, this allows us to use camelCase config and variable names.

The only time we set a value externally is when we need to pass an (encrypted) service account key through runtime config, because DM cannot encrypt values securely. But we still use gcloud beta runtime-config configs variables set for that (after the DM deployment and prior to firebase deploy) and not firebase config:set, because the former accepts stdout from KMS.

I just wanted to add - reading through this it seems that some people missed a crucial point that leads to some confusion. It would be really useful if this was cleared up in the documentation too.

Basically, the functions.config() persists in between _deployments_, but can ALSO be set separately between different firebase projects!

This means that for dev, staging and prod environments, you must set up multiple identical firebase projects (rather than using multi-site hosting, or deploying to the SAME project... 👎 ) which each persistently hold their own - different - environment variables.

Once you've set aliases (or just switched between projects with firebase use) you can deploy the same repo to multiple, separate firebase projects. This is how you set up a dev environment, staging, and production - as separate firebase projects.

Then, you can set separate environment configurations for each, like this:
firebase -P dev config:set - set variables for your development env
firebase -P prod config:set - set variables for your production env...

Then, you can deploy the same repo twice, like so:
firebase deploy -P dev
firebase deploy -P prod
and they will both deploy, to their respective firebase project, and will contain their own, separate environment configuration that will persist in-between deploys.

Yep, that's how we do it too. Separate projects for dev/test/prod.

Just sharing my vanilla JS solution which works regarding which platform you develop on (Windows, MacOS, Linux) :

Using separate projects for dev/test/prod.

/.firebaserc

{
  "projects": {
    "dev": "my-project-name-dev",
    "test": "my-project-name-test",
    "prod": "my-project-name-prod"
  }
}

Config files

JSON-files in /functions/config folder:

/functions/config/config.dev.json 
/functions/config/config.test.json 
/functions/config/config.prod.json 
/functions/config/config.SAMPLE.json <-- only file tracked in git

Example content:
{
    "google": {
        "key": "my-api-key",
        "storage_bucket": "firebase-storage-bucket"
    },
    "another_vendor": {
        "my_prop": "my value"
    },
    ...
}

/functions/set-config.js

const fs = require('fs');
const env = process.argv[2];

const configPath = `./config/config.${env}.json`;

if (!(configPath && fs.existsSync(configPath))) {
    return;
}

const collectConfigLines = (o, propPath, configLines) => {
    propPath = propPath || '';
    configLines = configLines || [];
    for (const key of Object.keys(o)) {
        const newPropPath = propPath + key;
        if (typeof o[key] === 'object') {
            collectConfigLines(o[key], newPropPath + '.', configLines);
        } else if (o[key] != null && o[key] !== '') {
            configLines.push(`${newPropPath}=${JSON.stringify(o[key])}`);
        }
    }
}

const config = require(configPath);
const configLines = [];
collectConfigLines(config, '', configLines);

const cp = require('child_process');
cp.execSync(`firebase -P ${env} functions:config:set ${configLines.join(' ')}`);

/functions/package.json

...
"scripts": {
    "config:set:dev": "node set-config dev",
    "config:set:test": "node set-config test",
    "config:set:prod": "node set-config prod",
    ...
},
...

If anyone is still looking, there's the method used in this medium article https://medium.com/indielayer/deploying-environment-variables-with-firebase-cloud-functions-680779413484. Saved my life.

What I am personally missing is the ability to set a config variable without doing a deploy. We have CICD pipeline which does deployment to our different environments and we don't want a developer to build the code locally and deploy it, especially not to production. That's how things break.

Also if we want to change an existing variable later, with some feature being developed, right now you have checkout the last released tag, build it, deploy, checkout back to your feature branch, continue. That really makes this unusable.

Essentially we would want remote config for cloud functions

@bezysoftware I think you need to use environment variables (instead of enviroment configuration), you can skip the firebase layer and do it on google cloud platform, check it out https://cloud.google.com/functions/docs/env-var

Thank you for your sharing! @laurenzlong
I wanted to import from a file, so I used command like this.

firebase functions:config:set service_account="$(cat service-account.json)"

I imported a service account json file to Firebase Cloud Functions, because I wanted to use the user management API of Firebase Admin SDK.

It didn't work

@Md-Abdul-Halim-Rafi

firebase functions:config:set service_account="$(cat service-account.json)"

Worked for me, I can see the config var with firebase functions:config:get.
It did not work the first time because I was not in the project folder and did not select the firebase project on which to set the config variable, to select the firebase project use firebase use --add, the CLI will prompt you with the list of projects on your firebase console (assuming you are logged in the CLI with firebase login)

@dnhyde

firebase functions:config:set service_account="$(cat service-account.json)"

Seems this will not work while using in your json file any other value types except strings (e.g. boolean or number):

{
   "test": {
        "hmm": true
    }
}

fails with:

Error: HTTP Error: 400, Invalid value at 'variable.text' (TYPE_STRING), true

It seems reasonable to me that I should be able to do:

firebase functions:config:get > config.json

and then later:

firebase functions:config:set < config.json

It just makes sense to have these two commands be complementary.

Having the config in a file allows for me to version control it and see how it has changed over time.

It's unfortunate that the only way we have to accomplish this right now (using env=$(cat config.json)) also results in breaking my ability to have config.json be the actual values since I cannot wrap them in { env: ... }.

Note for myself: this is where I need to start for a feature pull request: https://github.com/firebase/firebase-tools/blob/b17611a4ff0d36e157ed06a24f6c81d4e146d9e2/src/functionsConfig.js#L142

Just sharing my vanilla JS solution which works regarding which platform you develop on (Windows, MacOS, Linux) :

Using separate projects for dev/test/prod.

/.firebaserc

{
  "projects": {
    "dev": "my-project-name-dev",
    "test": "my-project-name-test",
    "prod": "my-project-name-prod"
  }
}

Config files

JSON-files in /functions/config folder:

/functions/config/config.dev.json 
/functions/config/config.test.json 
/functions/config/config.prod.json 
/functions/config/config.SAMPLE.json <-- only file tracked in git

Example content:
{
    "google": {
        "key": "my-api-key",
        "storage_bucket": "firebase-storage-bucket"
    },
    "another_vendor": {
        "my_prop": "my value"
    },
    ...
}

/functions/set-config.js

const fs = require('fs');
const env = process.argv[2];

const configPath = `./config/config.${env}.json`;

if (!(configPath && fs.existsSync(configPath))) {
    return;
}

const collectConfigLines = (o, propPath, configLines) => {
    propPath = propPath || '';
    configLines = configLines || [];
    for (const key of Object.keys(o)) {
        const newPropPath = propPath + key;
        if (typeof o[key] === 'object') {
            collectConfigLines(o[key], newPropPath + '.', configLines);
        } else if (o[key] != null && o[key] !== '') {
            configLines.push(`${newPropPath}=${JSON.stringify(o[key])}`);
        }
    }
}

const config = require(configPath);
const configLines = [];
collectConfigLines(config, '', configLines);

const cp = require('child_process');
cp.execSync(`firebase -P ${env} functions:config:set ${configLines.join(' ')}`);

/functions/package.json

...
"scripts": {
    "config:set:dev": "node set-config dev",
    "config:set:test": "node set-config test",
    "config:set:prod": "node set-config prod",
    ...
},
...

Just sharing my vanilla JS solution which works regarding which platform you develop on (Windows, MacOS, Linux) :

Using separate projects for dev/test/prod.

/.firebaserc

{
  "projects": {
    "dev": "my-project-name-dev",
    "test": "my-project-name-test",
    "prod": "my-project-name-prod"
  }
}

Config files

JSON-files in /functions/config folder:

/functions/config/config.dev.json 
/functions/config/config.test.json 
/functions/config/config.prod.json 
/functions/config/config.SAMPLE.json <-- only file tracked in git

Example content:
{
    "google": {
        "key": "my-api-key",
        "storage_bucket": "firebase-storage-bucket"
    },
    "another_vendor": {
        "my_prop": "my value"
    },
    ...
}

/functions/set-config.js

const fs = require('fs');
const env = process.argv[2];

const configPath = `./config/config.${env}.json`;

if (!(configPath && fs.existsSync(configPath))) {
    return;
}

const collectConfigLines = (o, propPath, configLines) => {
    propPath = propPath || '';
    configLines = configLines || [];
    for (const key of Object.keys(o)) {
        const newPropPath = propPath + key;
        if (typeof o[key] === 'object') {
            collectConfigLines(o[key], newPropPath + '.', configLines);
        } else if (o[key] != null && o[key] !== '') {
            configLines.push(`${newPropPath}=${JSON.stringify(o[key])}`);
        }
    }
}

const config = require(configPath);
const configLines = [];
collectConfigLines(config, '', configLines);

const cp = require('child_process');
cp.execSync(`firebase -P ${env} functions:config:set ${configLines.join(' ')}`);

/functions/package.json

...
"scripts": {
    "config:set:dev": "node set-config dev",
    "config:set:test": "node set-config test",
    "config:set:prod": "node set-config prod",
    ...
},
...

Sorry but when do you run the script "config:set:dev" ? I don't get it... thanks !