Compose: --env-file option

Good idea, this one.

Current plans for environment files are spelled out here: https://github.com/docker/compose/issues/745#issuecomment-346492587

While I'm not entirely excluding the possibility of an --env-file flag at some point in the future, I think we'd first want to see whether the mentioned changes alleviate the issues with the current model. But at this time, I don't think the added complexity is worth it.

From what I understand that solution doesn't really solve an other requirement (which I assume many have): having many docker-compose file in the same folder and being able to launch them using different .env files.

To bring a concrete example, I have a docker-compose.yml and a docker-compose.test.yml. The second is based on the first one. When developing I just want to use docker-compose.yml with my local unchecked .env file (development environment), but docker-compose.test.yml is run during CI and should use a checked .env.test file.

How would you deal with this scenario?

For reference:

docker-compose.yml

version: '3'
services:
  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${MYSQL_DATABASE}
    ports:
      - '3306:3306'

version: '3'
services:
  mysql:
    container_name: ch.migros.reactions-api-test-mysql
  mockserver:
    image: jamesdbloom/mockserver
    ports:
      - '1080:1080'
      - '1090:1090'
  api:
    build: ./api
    env_file: ./api/.env.test
    command: ['start:integration']
    ports:
      - '4000:4000'
    depends_on:
      - 'mockserver'
      - 'mysql'
  test:
    build: ./test
    env_file: ./test/.env.test
    depends_on:
      - 'api'

On a related note, the '.env' filename is really way too generic to have as a default.
Should be something like .docker-compose.env instead.

Where .env is used for different things, and without the ability to override it makes the whole environment variable mechanism unusable.

Am I missing something ? Is the --env-file option already valid? If it is not, the documentation says otherwise
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers

Am I missing something ? Is the --env-file option already valid? If it is not, the documentation says otherwise
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers

No that's different. With env_file you pass variables to containers, but not to the process that generates the containers. For example it wouldn't be enough to put MYSQL_ROOT_PASSWORD in the example above via env_file because somehow it is needed at the time the container is created.

At least according to my understanding. Maybe someone can add more details.

To expand on what @nmaro is saying and provide a summary of the ticket status as I understand it, @codextremist what this issue is discussing is being able to do something like so:

project folder:

dev.env
beta.env
prod.env
docker-compose.yml

```bash
docker-compose up --env-file=dev.env

Which would pass some variables from `dev.env` into the context of the `docker-compose.yml` file, not directly to the containers like `docker --env-file=dev.env` would do.
`docker-compose.yml` then defines which variables get passed down to containers and how, allowing only the settings needed for that specific image to be passed down e.g.:

```yml
version: '3'
services:
  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}     (passed in from dev.env or prod.env, etc)
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      ...

Right now only one .env file is allowed in the root of the project, which allows for defining settings separately from the docker-compose file, but it doesn't allow for having separate sets of settings for different environments (potentially risky if people put prod passwords by accident into dev environments). With --env-file, people could explicitly require passing the environment file they want to run with, instead of implicitly looking for a .env.

This proposal is distinct from the --env-file=<filename> option that already exists for the docker command, we're asking for a similar option to pass env variables to the docker-compose command, I think most users would find it intuitive and useful given the existing usage pattern with docker.

I strongly support this proposal since this would make many people's beta/prod/dev setup easier given the ability to pass env files by command line :) Current hacks involving symlinking .env to the env file you want to have active are error-prone and less clear and explicit than a CLI flag would be. To maintain backwards compatibility, .env could still be loaded, future users would be encouraged to use --env-file=<filename> primarily, and .env for machine-specific overrides not added to version control.

@shin- just wondering what is the current status on the --env-file option.
I see it is part of the doc but I am wondering if one can specify multiple --env-file to have them aggregated. (as talk in other issues). Would you mind commenting please?

@nolazybits Assuming you're talking about the env_file field inside the Compose file, since we don't have an --env-file flag in docker-compose:

Yes, you can specify multiple env files for a service and have them be combined. See the documentation.

Ah yeah got mixed up. Thanks for your reply though :)

So to come back to this issue (no actual --env-file)
@shin-

But at this time, I don't think the added complexity is worth it.

docker-compose offers a -f to get the docker-compose.yaml and to me those yaml file are going in par with the env files, so it would be nice to be able to specify the env files those docker-compose file are needing.

Copied from another issue

Hello, I would really like to see a --env-file option that could be use the same way as -f (i.e multiple --env-file could be loaded). My use case is the following: I have a monorepo each packages has a docker-compose and a .env file Now in my CI I would like to do something like `docker-compose -f ./Dockerfile.base.yml -f ./packages//Dockerfile -f ./packages//Dockerfile --env-file ./packages//.env --env-file ./packages//.env && docker run -it lerna run test --scope moduleA,moduleB` This will spin the needed environments for my modules and test only those modules Without the -env-file option I guess I can merge those .env myself with a CI script...

Thanks again @shin-

I'm interested in using --env-file as well.

Can we please have this essential feature? The --env-file CLI option would complement the docker compose's ability to replace ${ENV_VARIABLE} with the variable's value. This somehow is essential in our many use cases.
Regards,
Vijayant

@shin- is there possibility to load multiple env files ONLY for processing docker-compose.yml while starting project? In Symfony 4.2+ there is Ruby's behavior for common .env file with ability to override values with .env.local. The problem is, we want to make docker-compose.yml configurable per developer (Traefik host, database local port etc), so we can't use .env for it and if we do:

env_file:
  - .env
  - .env.local

the values will be populated into container and there won't be possibility to dynamically change application's behavior with environmental variables from .env/.env.local because they won't be loaded since APP_ENV will be already loaded.

So what I would like to do is loading env files only for creating environment, but without passing them to container.. Any suggestions?

Another case. I moved docker-related environment variables to docker/docker.env (ignored in Git) and changed docker-compose.yml:

version: '3.4'

services:
  # ...

  php:
    env_file: docker/docker.env
    user: "${RUN_AS_USER:?You must define user:group for permissions handling, look at README}"
    build:
      dockerfile: docker/Dockerfile
      target: php-dev
      context: .

Now, when running project, I get ERROR: Missing mandatory value for "user" option in service "php": You must define user:group for permissions handling, look at README even if RUN_AS_USER is set in docker/docker.env. When I remove :? part, I get WARNING: The RUN_AS_USER variable is not set. Defaulting to a blank string..

I know I can run export RUN_AS_USER=$UID or RUN_AS_USER=user docker-compose up -d --build but it would be soooooo good if docker-compose could pre-populate environment variables used within docker-compose.yml.

What if we can have a direct key value in docker-compose which specifies which file to use for substituting values in docker-compose file.

Example -

version: '3.7'
services:
  service-1:
    substitute_env_file:
      - ${MACHINE_ENV}.env

So for every different machine MACHINE_ENV can be defined in topmost environment. For example, developer can define MACHINE_ENV=dev for his machine, MACHINE_ENV=prod for production, MACHINE_ENV=staging for staging and so on, on respective machines. And according we can have dev.env, prod.env, staging.env, all checked into version control and using a single command of docker-compose up will load all environments according to the respective machine.

The practical use case that I face daily is while deploying kafka cluster, where environment variable KAFKA_ADVERTISED_LISTENERS must be passed for each kafka broker, and it cannot be put in .env because there are multiple kafka brokers in same machine, which need different values for each broker (but environment variable cannot be changed). Now this cannot be used in dev.env and prod.env because of the same name of every broker environment variable.

Guys, please, let's create the way how load specific .env file for each docker-compose.yml file.

Now I have clean project with folder structure:

app
db
doc
docker
|--- /dev/
|--------- docker-compose.yml
|--- /prod/
|------ docker-compose.yml
|--- Dockerfile
prod.env
dev.env
start_dev.sh
build_prod.sh
start_prod.sh
README.md
.gitignore

And I need just pass variables to each environment.
It can be argument for docker-compose command or definition in yaml.

My temporary solution: https://gist.github.com/landsman/514731b1cd94d379589a533a6b2d663f

This functionality missing me here a lot! 🙏

Please add the --env-file flag, for the sake of consistency with the docker command at least.

For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

My preference would also be for the --env-file option.

I have a use case where this would be great i.e. deploying alfresco into containers.

Alfresco uses a properties file of its own to configure the services called alfresco-global.properties. This is just a PROP=VAL file the same format as the .env file.

It would be great to use this same global.properties file to also enable the configuration for docker-compose rather than having multiple files with repeated properties for both the compose setup and the service setup.

Steve

My preference would also be for the --env-file option.

Please add the --env-file flag, for the sake of consistency with the docker command at least.

I agree

Other more specific filenames should be also read by default to avoid collisions with other tools parsing .env file (symfony/dotenv, vlucas/phpdotenv), maybe docker-compose.env or .docker-compose.env. If this file exists, .env file should be skipped.

And why not adding COMPOSE_ENV_FILE environnement variable to this list : https://docs.docker.com/compose/reference/envvars/ ? It seems to be really related to COMPOSE_FILE environnement variable.

AFAIS #6535 does not cover all use cases mentioned here. It brings basic functionality of overriding default env file which is loaded during build. If I'm correct, it doesn't allow specifying multiple env files to be loaded, which was described above in the example where env_file option inside docker-compose.yml was used.

Am I correct?

A generic way to solve this is to implement something like kustomize but for Docker, then users are free to patch their YAML files as they please.

I also came here looking for a way to specify an env file for different environments and I think the workaround seems to be duplicating my docker-compose.yml files in an ugly way.

Is it possible to also load env variables from a .yaml file. Like this node library does?

just saying i'm doing this via systemd:

[Unit]
Description=docker-compose
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
StandardError=null
StandardOutput=null

Environment=ENVIRONMENT=test.env
EnvironmentFile=/usr/local/src/sonarqube/test.env

# pull images
ExecStartPre=/usr/bin/docker-compose -f /usr/local/src/sonarqube/docker-compose.yml pull

# Compose up
ExecStart=/usr/bin/docker-compose -f  /usr/local/src/sonarqube/docker-compose.yml up -d

# Compose down, remove containers and volumes
ExecStop=/usr/bin/docker-compose -f /usr/local/src/sonarqube/docker-compose.yml stop

[Install]
WantedBy=multi-user.target

compose:

services:
  sonarqube:
    image: docker.registry.at/sonarqube:${SONARQUBE_VERSION:-latest}
    ports:
      - "${SONARQUBE_PORT:-9000}:9000"
    env_file: ${ENVIRONMENT:-local.env}

which was the only way i found to use the env vars from a file directly in the compose file while executing it (i dont like the .env file, i want multiple env files named by my convention for multiple environments)

And why not adding COMPOSE_ENV_FILE environnement variable to this list : https://docs.docker.com/compose/reference/envvars/ ? It seems to be really related to COMPOSE_FILE environnement variable.

That's actually a great idea although although you'd have to somehow resolve the cyclic dependency when someone sets COMPOSE_ENV_FILE inside the .env file as it is commonly done for COMPOSE_FILE

eg.

$cat .env

# COMPOSE CLI VARS
COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml

# EXTRA VARS
DNS=8.8.8.8

Please add the --env-file flag, for the sake of consistency with the docker command at least.

For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

This is my charm and working. Thank you!

Please add the --env-file flag, for the sake of consistency with the docker command at least.

For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

To resolve the error env: #The: No such file or directory which is caused by env files with comments in change the above to:

env $(cat .env.test | grep "#" -v) docker-compose up

Please add the --env-file flag, for the sake of consistency with the docker command at least.
For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

To resolve the error env: #The: No such file or directory which is caused by env files with comments in change the above to:

env $(cat .env.test | grep "#" -v) docker-compose up

just saying that all of those solutions with env in front are breaking auto-completion of docker-compose which i dont wanna miss

Please add the --env-file flag, for the sake of consistency with the docker command at least.
For now, for those looking for a workaround, I'm using the env command, like this:

env $(cat .env.test) docker-compose up

To resolve the error env: #The: No such file or directory which is caused by env files with comments in change the above to:

env $(cat .env.test | grep "#" -v) docker-compose up

just saying that all of those solutions with env in front are breaking auto-completion of docker-compose which i dont wanna miss

i think lets just make a VSCode extension to do this for us and call it a day

This is my workaround that I put in my .bashrc:

export COMPOSE_HOME=/home/jgrote/docker
alias dc="env PWD=$COMPOSE_HOME $(cat $COMPOSE_HOME/.env | grep -v '#' | tr '\n' ' ') docker-compose"
alias dcup="dc up -d"
complete -F _docker_compose dc

I can use dc for my special one, and then docker-compose if I need to use a non-special one.

@Elyytscha this doesn't break autocompletion :)

The feature has been implemented already https://github.com/docker/compose/pull/6535

The feature has been implemented already #6535

Has it been included in a release yet or is there a roadmap for when it will be? This still doesn't work for me and I don't see it anywhere in the docs, it's not listed here: https://docs.docker.com/compose/reference/up/

@adamerose
works for me but did not find it in the documentation either

pwirth@frenzy ~ % docker-compose --version
docker-compose version 1.25.4, build unknown

Is --env-file option available only with docker-compose up command? Because the command docker-compose run --env-file .env.test ... is not executed :confused:

Docker Compose version 1.26.2

--EDIT--

My mistake, sorry. I stupidity messed up the options order... :disappointed:

It should be docker-compose --env-file .env.test run service_name and not docker-compose run --env-file .env.test service_name

@iraklisg Check out this link https://docs.docker.com/compose/environment-variables/.

@MexsonFernandes the link doesn't mention --env-file for the docker-compose binary, which is what the discussion is about, given the merged PR.

Furthermore, pointing people to documentation in issues is rude, especially without an in-page anchor, and especially since few posts above the matter of documentation being outdated was raised. It's rude for two reasons. First, documentation changes over time and there is no guarantee that the information you saw will be there when someone else follows the link, and second, it's implies on your part an assumption that someone was either stupid or lazy as to not check relevant documentation before seeking answers in a Github issue.

@MexsonFernandes the link doesn't mention --env-file for the docker-compose binary, which is what the discussion is about, given the merged PR.

Furthermore, pointing people to documentation in issues is rude, especially without an in-page anchor, and especially since few posts above the matter of documentation being outdated was raised. It's rude for two reasons. First, documentation changes over time and there is no guarantee that the information you saw will be there when someone else follows the link, and second, it's implies on your part an assumption that someone was either stupid or lazy as to not check relevant documentation before seeking answers in a Github issue.

@bmarkovic my bad. I will make sure to read off and mention proper links out here. 👍🏼

--env-file was added with version 1.25.0, it is not mentioned in CHANGELOG.md.