I'm having the same problem, when piping docker-compose output via gzip to a file. When I run the command myself in Bash, it works just fine. However, as a cron job, I'm getting the same error as you — and the same stack trace file & line numbers & function names, as you, and return code -1.

Here's the command:

/usr/local/bin/docker-compose exec postgres pg_dumpall --username=postgres \
   | gzip > $postgres_backup_path

(I have that line in a backup script that I run sometimes manually (works fine) and sometimes as a cron job (won't work)).

Now I've worked around my problem by calling Docker directly like so:

/usr/bin/docker exec edm_postgres_1 pg_dumpall --username=postgres \
    | gzip > $postgres_backup_path

— works fine from both Cron and manually in Bash.

I do have the same issue when using docker-compose exec in a cronjob:

something like

cd ~/piwik && docker-compose exec piwik sudo -u www-data /usr/bin/php5 /app/console core:archive --url=https://piwik-host/ > ~/piwik-archive.log

does not work - neither as a one-liner in the crontab, nor as a shell script called from crontab

I could create a workaround with bash script like:

#!/bin/bash
piwikid=$(docker ps -f "ancestor=piwik" -q)

/usr/bin/docker exec ${piwikid} /usr/bin/php5 /app/console core:archive --url=https://piwik-host/ > ~/piwik-archive.log

I could get it to work by adding the -T parameter to not create a Pseudo-TTY.

docker-compose exec -T container_name ./build.sh for example.

The -T option worked for me, but the script doesn't return to command prompt even after it finishes the last command.

I'm having the same issue when running:
docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content

Everything seems to work after all (the command executes as expected), it's just that nasty crash that docker-compose gives me afterwards:

$ docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 61, in main
  File "compose/cli/main.py", line 113, in perform_command
  File "compose/cli/main.py", line 441, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

$ echo $?
255

When I use the -T flag as recommended though, my command doesn't work and I even get stuck on the docker-compose command (doesn't return to my shell but just hangs instead).

Traceback (most recent call last):
File "", line 3, in
File "compose/cli/main.py", line 62, in main
File "compose/cli/main.py", line 114, in perform_command
File "compose/cli/main.py", line 442, in exec_command
File "site-packages/dockerpty/pty.py", line 334, in start
File "site-packages/dockerpty/pty.py", line 370, in _hijack_tty
File "site-packages/dockerpty/io.py", line 164, in do_write
OSError: [Errno 32] Broken pipe
docker-compose returned -1

I have the same problem, but just by launching this from jenkins (it fails just there):
docker-compose exec drupal bash -c "bin/behat --profile=ci", now changed to docker exec ${COMPOSE_PROJECT_NAME}_drupal_1 bash -c "bin/behat --profile=ci".

Same here. docker-compose exec fails when running as part of a Jenkins job (works locally), but docker exec works. Thanks @paolomainardi !

Have you tried adding the -T flag to docker-compose exec?

Tried it now, adding -T to docker compose exec solve this for me as well.

-T isn't working for me. My only workaround is to use docker exec directly but means I have to parse the container ID from docker ps which isn't ideal.

To test I have a basic python script that just prints stdin

$ cat myScript.py
#!/usr/bin/python
import sys
print "started"
for line in sys.stdin:
    print line

$ echo test | docker-compose exec -T myContainer /usr/src/myScript.py
.. nothing happens, just waits ..

$ echo test | docker-compose exec myContainer /usr/src/myScript.py
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 65, in main
  File "compose/cli/main.py", line 117, in perform_command
  File "compose/cli/main.py", line 462, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

$ echo test | docker exec -i d6380314bb89  /usr/src/myScript.py
started
test

What happens when you add a -u to the first line in your Python script
(#!/usr/bin/python -u)? I guess your output is buffered.

On Mon, 9 Jan 2017, 19:38 Leon Blakey, notifications@github.com wrote:

-T isn't working for me. My only workaround is to use docker exec
directly but means I have to parse the container ID from docker ps which
isn't ideal.

To test I have a basic python script that just prints stdin

$ cat myScript.py

!/usr/bin/python

import sys
print "started"
for line in sys.stdin:
print line

$ echo test | docker-compose exec -T myContainer /usr/src/myScript.py
.. nothing happens, just waits ..

$ echo test | docker-compose exec myContainer /usr/src/myScript.py
Traceback (most recent call last):
File "", line 3, in
File "compose/cli/main.py", line 65, in main
File "compose/cli/main.py", line 117, in perform_command
File "compose/cli/main.py", line 462, in exec_command
File "site-packages/dockerpty/pty.py", line 338, in start
File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

$ echo test | docker exec -i d6380314bb89 /usr/src/myScript.py
started
test

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/3352#issuecomment-271366842,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABI3M4d_POesl4j9JdLn8eY4Z4K1DVGmks5rQn6pgaJpZM4ILWjn
.

>

Michaël Arnauts

I'd say that the problem is that -T in docker-compose exec is mostly equivalent to not using any of -t or -i flags in docker run, so with -T there is not tty but also not attached stdin, and it seems that there is no way of attaching stdin without tty in docker-compose.

docker-compose exec should probably have different flags for tty and for attaching stdin as docker run has.

env: ansible playbook + ubuntu(vagrant)

error:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 65, in main
    command()
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 117, in perform_command
    handler(command, command_options)
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 462, in exec_command
    pty.start()
  File "/usr/local/lib/python2.7/dist-packages/dockerpty/pty.py", line 338, in start
    io.set_blocking(pump, flag)
  File "/usr/local/lib/python2.7/dist-packages/dockerpty/io.py", line 32, in set_blocking
    old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)

add -T fix the issue.

for all those curious about getting around this while still getting information from docker-compose...

docker exec -i $(docker-compose ps -q cassandra) < someexample.cql

The -i keeps stdin open while not allotting a tty and the ps -q on the docker-compose provides us with the container ID we want.

I assume this will provide multiple IDs if you have many running, you'll have to do some munging to use this method.

It would appear docker-compose isn't implicitly using a -i when passing the exec along.

Just want to leave this minimal test-case for docker-compose:

echo 123 | docker-compose exec container_name tee

It triggers error:

Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 88, in main
  File "compose/cli/main.py", line 140, in perform_command
  File "compose/cli/main.py", line 486, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose

I can reproduce this bug just when docker-compose gets called from jenkins or gitlab-ci

Same here in Jenkins.

reproduced by gitlab-ci

reproduced in gitlab-ci for me too

same problem on jenkins, solved by using -T:
docker-compose exec -T container_name cmd

I do have the same problem when using docker-compose exec in a cronjob

Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 67, in main
  File "compose/cli/main.py", line 117, in perform_command
  File "compose/cli/main.py", line 453, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose
Command exited with non-zero status 255

solved by using -T

/usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec -T php ./yii command

where php is service name

I have no input
and
I have output from command

In our case (we are running these scripts by Makefile) docker-compose command returns code 2, solved by wrapping this running in $(shell docker-compose <whatever you want>)

@YurkinPark may be absolute path needed?

I have reproduced the bug on GitLab CI. Note, that adding -T flag solved it.

Here is the full .gitlab-ci.yml:

image: docker:17.04.0-ce

variables:
  # See https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#using-the-overlayfs-driver
  DOCKER_DRIVER: overlay

services:
  # See https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-in-docker-executor
  - docker:dind

before_script:
  # Install docker-compose
  - apk add --no-cache py-pip
  - pip install docker-compose
  - docker --version
  - docker-compose --version
  # Verify that Docker CE is installed correctly
  - docker run hello-world
  # Prepare environment
  - cp .env.example .env
  - docker-compose up -d
  - docker-compose ps
  # If you're perplexed as to why we took a detour using bash -c
  # see this issue https://github.com/docker/compose/issues/3379
  - docker-compose exec php bash -c 'yarn install --non-interactive --no-progress --no-emoji; (exit $?)'
  - docker-compose exec php bash -c 'yarn run production; (exit $?)'

jest:
  script: docker-compose exec php bash -c 'yarn run test; (exit $?)'

Output after the encounter with the first docker-compose exec:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 67, in main
    command()
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 117, in perform_command
    handler(command, command_options)
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 453, in exec_command
    pty.start()
  File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
    io.set_blocking(pump, flag)
  File "/usr/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
    old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)

Adding -T flag is solving problem, but command output become corrupted, it doesn't show all text as the normal command execution suppose to. I'm using docker-compose commands inside make file on gitlab ci.

So basically, you need -i for docker exec when you are piping in data to the command.

And you should not use -t for docker exec when you are piping data out from the command.

And docker-compose exec only have -T, which means neither -i or -t. But you cannot represent -i without -t in docker-compose.

That's the issue, right?

Usecase | TTY | Stdin | Option |
---|---|---|---
< | Does not matter | Yes | N/A (This issue!)
> | No | Does not matter | -T
Neither | Does not matter | Does not matter | Does not matter
Both | No | Yes | N/A (This issue!)

We're executing a test suite within jenkins that uses pytest-sugar for color-coded test failures and this issue(?) seems to be preventing this from working. Without using -T to disable pseudo-tty (and therefore strip out all color), docker-compose exec fails with ValueError: file descriptor cannot be a negative integer (-1)

I'm running into this exact issue as well. Would love to be able to pipe data into containers via docker-compose.

For people who are stumbling into this while trying to run docker-compose exec on Jenkins pipeline like myself:

The -T flag mentioned bypasses the error, but it does not actually execute the command for me. Jenkins just thinks the command executed successfully and moves on.

I had to fall back to docker exec, like so:

environment {
  PROJECT_NAME = "${env.BRANCH_NAME}.${env.BUILD_ID}"
  CONTAINER_PREFIX = "${PROJECT_NAME.replace('-', '').replace('.', '').toLowerCase()}"
}
stages {
  stage('run') {
    steps {
      sh 'docker-compose -p $PROJECT_NAME up -d --build'
      sh "docker exec ${CONTAINER_PREFIX}_app_1 bash -c \"hello world\""
    }
  }
}

The $CONTAINER_PREFIX is a workaround to get the docker compose prefix. It's not pretty, and a little too verbose, but it's the only way I could get it to work for me.

you can just use 'docker-compose -q app' to the container id to pass to
docker exec command

Il giorno gio 14 set 2017 alle 06:51 Tri Nguyen notifications@github.com
ha scritto:

For people who are stumbling into this while trying to run docker-compose
exec on Jenkins pipeline like myself:

The -T flag mentioned bypasses the error, but it does not actually
execute the command for me. Jenkins just thinks the command executed
successfully and moves on.

I had to fall back to docker exec, like so:

environment {
PROJECT_NAME = "${env.BRANCH_NAME}.${env.BUILD_ID}"
CONTAINER_PREFIX = "${PROJECT_NAME.replace('-', '').replace('.', '').toLowerCase()}"
}
stages {
stage('run') {
steps {
sh 'docker-compose -p $PROJECT_NAME up -d --build'
sh "docker exec ${CONTAINER_PREFIX}_app_1 bash -c \"hello world\""
}
}
}

The $CONTAINER_PREFIX is a workaround to get the docker compose prefix.
It's not pretty, and a little too verbose, but it's the only way I could
get it to work for me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/3352#issuecomment-329370946,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAiK34hAtSJ3RLGjAy0-5fTxvt51vhVks5siLDEgaJpZM4ILWjn
.

>

SPARKFABRIK

Paolo Mainardi
CTO & Founder

tel: +39-340-1678089
skype: paolo_mainardi

http://www.sparkfabrik.com
ᐧ

What is docker-compose -q app?

I think he means you can use docker-compose ps -q {app} to get the container ID. For example,

docker exec $(docker-compose ps -q {app}) bash -c ...

docker-compose exec container_name cat logfile > logfile

also fails for me, but with -T

docker-compose exec -t container_name cat logfile > logfile

it just produce empty file, so no any output! Even there no error, it is unusable.

So, as I understand, docker exec is the solution for now?

echo 123 | docker exec -i $(docker-compose ps -q service) cat - for now...

I would add a bit of changes for someone who have a container-name set .Based on @michaelarnauts
This worked for me, docker-compose exec -T app ./build.sh where app is the service defined in the docker-compose

now I try in cron (again)
crontab -e

*/10 * * * * time -p /usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec php ./yii command

now I have same problem in cron (again)

[17362] Failed to execute script docker-compose
Traceback (most recent call last):
  File "site-packages/dockerpty/pty.py", line 334, in start
  File "site-packages/dockerpty/pty.py", line 367, in _hijack_tty
  File "site-packages/dockerpty/io.py", line 59, in select
  File "site-packages/dockerpty/io.py", line 351, in fileno
  File "site-packages/dockerpty/io.py", line 103, in fileno
  File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 519, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
  File "site-packages/dockerpty/io.py", line 351, in fileno
  File "site-packages/dockerpty/io.py", line 103, in fileno
  File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.
Command exited with non-zero status 255
real 2.09
user 0.98

after add into crontab -e

COMPOSE_INTERACTIVE_NO_CLI=1
# Edit this file to introduce tasks to be run by cron.

https://stackoverflow.com/a/10657111/7670492

I try again with -T

/usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec -T php ./yii command

success