Supervisor: Supervisor 3.3.1: CRIT uncaptured python exception, closing channel

Created on 17 May 2017 · 4Comments · Source: Supervisor/supervisor

We're still experiencing the problem that supervisor logs a CRIT uncaptured on every log message produced when directing logs to /dev/stdout, similar to this:

2017-05-17 12:20:27,860 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140178721855392 for <Subprocess at 140178721855248 with name cause-error in state STARTING> (stdout)> (<type 'exceptions.IOError'>:[Errno 29] Illegal seek [/usr/local/lib/python2.7/dist-packages/supervisor/supervisord.py|runforever|227] [/usr/local/lib/python2.7/dist-packages/supervisor/dispatchers.py|handle_read_event|232] [/usr/local/lib/python2.7/dist-packages/supervisor/dispatchers.py|record_output|166] [/usr/local/lib/python2.7/dist-packages/supervisor/dispatchers.py|_log|142] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|info|275] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|log|293] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|emit|186] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|doRollover|211])

This looks similar to #638, but is still present with 3.3.1 and has a different stack trace.

I have created a minimal Dockerfile that will reproducably cause the problem on my side: https://gist.github.com/madmuffin1/33408001a402c2fd98bc0b227185b363

If you need any more details, let me know, I am happy to help.

logging

Source

madmuffin1

👍1

Most helpful comment

A late thanks to @mnaberez!

If anyone stumbles upon this error like me, be aware that the mentioned options apply to the [supervisord] section, within [program:x] sections, you need to use stdout_logfile_maxbytes = 0 and/or stderr_logfile_maxbytes = 0 respectively.

madmuffin1 on 19 Jun 2017

👍36 ❤5

All 4 comments

From further studying the documentation, I found that there is redirect_stdout and redirect_stderr, which works as expected. So this problem should problably be prevented by generating a more helpful error message.

madmuffin1 on 17 May 2017

The log message above shows that an illegal seek error occurred during log rotation.

Your config file is this:

RUN apt-get update && apt-get install -y -q locales python-pip && pip install supervisor && \
echo "[supervisord]nnodaemon=truenn[program:cause-error]ncommand=/bin/bash -c 'echo Test'nstdout_logfile=/dev/stdout" > /etc/supervisord.conf

That config uses log rotation (it's on by default), but you've specified a log file that is not seekable:

stdout_logfile=/dev/stdout

If you really want to use stdout, it may work, but minimally you'll need to disable log rotation (set stdout_maxbytes=0 (Configuration).

From further studying the documentation, I found that there is redirect_stdout and redirect_stderr, which works as expected.

There is no redirect_stdout option, and redirect_stderr will not affect this. The redirect_stderr option will cause supervisord to log stderr to the same file as stdout. You will still see the error if you don't either change the log file to a seekable file or disable rotation.