Gunicorn: OSError: [Errno 9] Bad file descriptor

@leond08 Thanks for the ticket!

In order to understand how it is happening can you provide a little more info?

• wich version of Gunicorn are you testing
• which worker are you using
• when does this happen?

I'm using gunicorn3 latest version
I'm using eventlet and gevent for this
i'm running my flask application - Flask-SocketIO

I start my background task after a user clicked the button
My background task function is to listen to an event,
after clicking the "DONE" button the background task must stop
then send an emit message to all the users

Had the same issue with aiohttp + gunicorn, observe the same message each time when ctrl + c.

[INFO] Error while closing socket [Errno 9] Bad file descriptor

I don't reproduce it. I suspect your application is closing some fds which create the issue above.

We are experiencing the same issue, only thing is it's only happening on 1 out of 8 containers running in docker swarm.

We have experienced the same issue with 1 out of 9 containers, it looks like related to docker and python3 and gevent.

gunicorn 20.0.4 + aiohttp 3.6.2

Gunicorn is running as dev server:

gunicorn --reload app:make_app --bind localhost:5000 --worker-class aiohttp.GunicornWebWorker --workers 2 --access-logfile -

Almost every Ctrl+C ends with

^C[2020-05-23 21:49:50 +0200] [38524] [INFO] Handling signal: int
Exception ignored when trying to write to the signal wakeup fd:
Exception ignored when trying to write to the signal wakeup fd:
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 42, in _sighandler_noop
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 42, in _sighandler_noop
    def _sighandler_noop(signum, frame):
    def _sighandler_noop(signum, frame):
OSError: [Errno 9] Bad file descriptor
OSError: [Errno 9] Bad file descriptor
[2020-05-23 21:49:50 +0200] [38526] [INFO] Worker exiting (pid: 38526)
[2020-05-23 21:49:50 +0200] [38528] [INFO] Worker exiting (pid: 38528)
[2020-05-23 21:49:50 +0200] [38524] [INFO] Shutting down: Master

It does not matter if application handled any request or not.

With Sanic 20.3.0:

^C[2020-05-26 13:24:55 +0200] [27706] [INFO] Handling signal: int
[2020-05-26 13:24:55 +0200] [27769] [INFO] Stopping server: 27769, connections: 0
[2020-05-26 13:24:55 +0200] [27769] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-05-26 13:24:55 +0200] [27769] [INFO] Worker exiting (pid: 27769)
[2020-05-26 13:24:55 +0200] [27771] [INFO] Stopping server: 27771, connections: 0
[2020-05-26 13:24:55 +0200] [27771] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-05-26 13:24:55 +0200] [27771] [INFO] Worker exiting (pid: 27771)
[2020-05-26 13:24:55 +0200] [27706] [INFO] Shutting down: Master

Same here with Gunicorn 20.0.4 + Uvicorn 0.11.5 worker class on every Ctrl+C

INFO:     [12621] [gunicorn.error] Handling signal: int
INFO:     [12635] [gunicorn.error] Error while closing socket [Errno 9] Bad file descriptor
INFO:     [12634] [gunicorn.error] Error while closing socket [Errno 9] Bad file descriptor
INFO:     [12635] [gunicorn.error] Worker exiting (pid: 12635)
INFO:     [12634] [gunicorn.error] Worker exiting (pid: 12634)
INFO:     [12621] [gunicorn.error] Shutting down: Master

any example of application? Also which version of Python are we talking about?

Ubuntu 20.04, system provided Python 3.8.2 in virtualenv

Example application: https://github.com/zgoda/newsloop-server/tree/d603a1c10c9e8be3d998f62ccc55dd73f4677115 (with aiohttp) or https://github.com/zgoda/newsloop-server/tree/b2a8a7f09fa9848d0384b51a3f6c0bb037fb459e (with Sanic). Exact gunicorn invocation in my earlier comment.

Differences in output between aiohttp and Sanic make me suspicious that there's something wrong involving workers.

Same issue, python 3.8.0
sanic 19.12.2
gunicorn 20.0.4

Edit: this happens when i run locally on my Mac, but not when running inside a Linux docker, might help you

Hello,
I guess this issue https://github.com/benoitc/gunicorn/issues/2064 has the same reasons.
We have almost the same errors like in issue, but we use gunicorn - 19.9.0

Im experiencing this too, FastAPI + the latest Gunicorn and uvicorn workers with Python 3.8.5

As soon as I stop using uvicorn (i.e. remove this line from my gunicorn config):

worker_class = "uvicorn.workers.UvicornWorker"

The errors dissappear.

As described above, this happens when stopping Gunicorn with Ctrl+C or sending a graceful kill signal to the PID.

[2020-09-12 11:56:37 +1000] [100390] [INFO] Starting gunicorn 20.0.4
[2020-09-12 11:56:37 +1000] [100390] [INFO] Listening at: http://0.0.0.0:6000 (100390)
[2020-09-12 11:56:37 +1000] [100390] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-09-12 11:56:37 +1000] [100392] [INFO] Booting worker with pid: 100392
[2020-09-12 11:56:38 +1000] [100392] [INFO] Started server process [100392]
[2020-09-12 11:56:38 +1000] [100392] [INFO] Waiting for application startup.
[2020-09-12 11:56:38 +1000] [100392] [INFO] Application startup complete.
[2020-09-12 11:56:48 +1000] [100390] [INFO] Handling signal: term
[2020-09-12 11:56:48 +1000] [100392] [INFO] Shutting down
[2020-09-12 11:56:48 +1000] [100392] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-09-12 11:56:48 +1000] [100392] [INFO] Waiting for application shutdown.
[2020-09-12 11:56:48 +1000] [100392] [INFO] Application shutdown complete.
[2020-09-12 11:56:48 +1000] [100392] [INFO] Finished server process [100392]
[2020-09-12 11:56:48 +1000] [100392] [INFO] Worker exiting (pid: 100392)
[2020-09-12 11:56:48 +1000] [100390] [INFO] Shutting down: Master

Here's an exact replication of the issue:

[fots@workstation testing]$ python3.8 -V
Python 3.8.5
[fots@workstation testing]$ python3.8 -m venv ~/.virtualenv/testing
[fots@workstation testing]$ source ~/.virtualenv/testing/bin/activate
(testing) [fots@workstation testing]$ pip install fastapi gunicorn uvicorn
Collecting fastapi
  Using cached fastapi-0.61.1-py3-none-any.whl (48 kB)
Collecting gunicorn
  Using cached gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
Collecting uvicorn
  Using cached uvicorn-0.11.8-py3-none-any.whl (43 kB)
Collecting pydantic<2.0.0,>=1.0.0
  Using cached pydantic-1.6.1-cp38-cp38-manylinux2014_x86_64.whl (11.5 MB)
Collecting starlette==0.13.6
  Using cached starlette-0.13.6-py3-none-any.whl (59 kB)
Requirement already satisfied: setuptools>=3.0 in /home/fots/.virtualenv/testing/lib/python3.8/site-packages (from gunicorn) (47.1.0)
Collecting h11<0.10,>=0.8
  Using cached h11-0.9.0-py2.py3-none-any.whl (53 kB)
Collecting websockets==8.*
  Using cached websockets-8.1-cp38-cp38-manylinux2010_x86_64.whl (78 kB)
Collecting httptools==0.1.*; sys_platform != "win32" and sys_platform != "cygwin" and platform_python_implementation != "PyPy"
  Using cached httptools-0.1.1-cp38-cp38-manylinux1_x86_64.whl (227 kB)
Collecting uvloop>=0.14.0; sys_platform != "win32" and sys_platform != "cygwin" and platform_python_implementation != "PyPy"
  Using cached uvloop-0.14.0-cp38-cp38-manylinux2010_x86_64.whl (4.7 MB)
Collecting click==7.*
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Installing collected packages: pydantic, starlette, fastapi, gunicorn, h11, websockets, httptools, uvloop, click, uvicorn
Successfully installed click-7.1.2 fastapi-0.61.1 gunicorn-20.0.4 h11-0.9.0 httptools-0.1.1 pydantic-1.6.1 starlette-0.13.6 uvicorn-0.11.8 uvloop-0.14.0 websockets-8.1
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/home/fots/.virtualenv/testing/bin/python3.8 -m pip install --upgrade pip' command.
(testing) [fots@workstation testing]$ ls -l
total 4
-rw-rw-r-- 1 fots fots 117 Sep 12 12:13 main.py
(testing) [fots@workstation testing]$ cat main.py
from fastapi import FastAPI

app = FastAPI()


@app.get("/")
async def root():
    return {"message": "Hello World"}
(testing) [fots@workstation testing]$ gunicorn -k uvicorn.workers.UvicornWorker main:app
[2020-09-12 12:19:05 +1000] [105788] [INFO] Starting gunicorn 20.0.4
[2020-09-12 12:19:05 +1000] [105788] [INFO] Listening at: http://127.0.0.1:8000 (105788)
[2020-09-12 12:19:05 +1000] [105788] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-09-12 12:19:05 +1000] [105790] [INFO] Booting worker with pid: 105790
[2020-09-12 12:19:05 +1000] [105790] [INFO] Started server process [105790]
[2020-09-12 12:19:05 +1000] [105790] [INFO] Waiting for application startup.
[2020-09-12 12:19:05 +1000] [105790] [INFO] Application startup complete.
^C[2020-09-12 12:19:06 +1000] [105788] [INFO] Handling signal: int
[2020-09-12 12:19:06 +1000] [105790] [INFO] Shutting down
[2020-09-12 12:19:06 +1000] [105790] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-09-12 12:19:06 +1000] [105790] [INFO] Waiting for application shutdown.
[2020-09-12 12:19:06 +1000] [105790] [INFO] Application shutdown complete.
[2020-09-12 12:19:06 +1000] [105790] [INFO] Finished server process [105790]
[2020-09-12 12:19:06 +1000] [105790] [INFO] Worker exiting (pid: 105790)
[2020-09-12 12:19:07 +1000] [105788] [INFO] Shutting down: Master

And here's the pip freeze output:

click==7.1.2
fastapi==0.61.1
gunicorn==20.0.4
h11==0.9.0
httptools==0.1.1
pydantic==1.6.1
starlette==0.13.6
uvicorn==0.11.8
uvloop==0.14.0
websockets==8.1

I attempted to install uvicorn and gunicorn from GitHub (master branch) to ensure I got the very latest fixes but the problem persisted.

Hope this helps
Fotis

1. I am able to reproduce this log error with a setup with just gunicorn and uvicorn. This error message starts occurring with uvicorn==0.11.4 and not the previous version 0.11.3 (both on OSx and in a Linux container). This is consistent with the reports above with uvicorn, where the versions reported are alwasy greater than 0.11.4. Evidence at the end
2. The commit responsible for this error is this one. The problem lies with these few lines of the commit just mentioned. The commit changes only the order of two blocks of code. If I revert that change of order, the log error go away, while still passing the test suite of uvicorn
3. the same log error happens if one either: uses starlette and fastapi on top of the stack gunicorn+uvicorn as reported above; - runs the latest uvicorn version 12.X instead of 0.11.4; - runs gunicorn with more than just one uvicorn worker

Evidence. In a new folder on osx run the attached script test.sh (tested on osx). For testing in the Linux container, save both the script and the Dockerfile and then read the header of the Dockerfile. I also attach the log of the script.

@benoitc, what do you think of this commit in uvicorn that seems to introduce the bug? The problem seems to be at the interface between gunicorn and uvicorn. Can you comment on the ordering of the 2 blocks of code changed in the commit mentioned above in uvicorn? This may help finding out why this happens with the other cases too. So far this has been reported also with aiohttp, gevent, Flask-SocketIO sanic. I attach also the log of the script for ease.

log_test.log

file _test.sh_

#!/bin/bash
python3 -m venv venv
source venv/bin/activate
pip install gunicorn==20.0.04 uvicorn==0.11.4
# create simple uvicorn app
printf "async def app(scope, receive, send):\n    await send()\n" > example.py
# spin up gunicorn with 1 uvicorn worker, and then send TERM signal to gunicorn
gunicorn example:app -w 1 -k uvicorn.workers.UvicornWorker &
sleep 5 && pkill -f gunicorn
sleep 1
# you will see 1 log entry like this one:
# [XX] [YY] [INFO] Error while closing socket [Errno 9] Bad file descriptor

printf "\n\n[INFO] if you instead bump down uvicorn's version from 11.4 to 11.3 [Errno 9] goes away:\n\n"
pip install uvicorn==0.11.3
gunicorn example:app -w 1 -k uvicorn.workers.UvicornWorker &
sleep 5 && pkill -f gunicorn

file _Dockerfile_

# run with:
# docker run -it $(docker build -q .)
FROM python:3.8
COPY test.sh .
RUN chmod +x /test.sh
CMD /test.sh

I had exactly same issue. Here is my case.

Brief : I'm trying to establish a test application for Django dwebsocket with gunicorn. When I trying to use websocket_client to test the result, after close the websocket this error happen everytime.

Environment :
Docker image : python:3.7
python version : python3.7.6
gunicorn : version = 20.0.4, work = gevent
Django version : Django==2.2
dwebsocket version : 0.5.12

Code:

view.py

from dwebsocket import accept_websocket

@accept_websocket
def my_ws(request):
    if request.is_websocket():
        ws = request.websocket
        while True:
            msg = ws.wait(timeout=15)
            if msg is None:
                print('get None message')
                break
            else:
                msg = b'echo :' + msg
                ws.send(msg)
                print('send ws seccess')
        print('websocket close')

urls.py

from websocketInfo.views import  my_ws
from django.conf.urls import url

urlpatterns = [
    url(r'my_ws/$', my_ws, name='my_ws')
]

websocket_client

from websocket import create_connection
ws = create_connection("ws://127.0.0.1:8080/my_ws/")
print("Sending 'Hello, World'...")
ws.send("Hello, World")
print("Receiving...")
result = ws.recv()
print(result)
ws.close()
print('ws close')

commad to run gunicorn server

gunicorn MyWebsocket.wsgi -b 0.0.0.0:8000 -w 3 -k gevent

error output

send ws seccess
get None message
websocket close
[2021-01-13 02:43:56 +0000] [101] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 65, in handle
    util.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 625, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
    super().handle_request(listener_name, req, sock, addr)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 128, in handle_request
    util.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 625, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 114, in handle_request
    resp.write(item)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 326, in write
    self.send_headers()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 322, in send_headers
    util.write(self.sock, util.to_bytestring(header_str, "latin-1"))
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 286, in write
    sock.sendall(data)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 523, in sendall
    return _socketcommon._sendall(self, data_memory, flags)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 367, in _sendall
    chunk_size = max(socket.getsockopt(SOL_SOCKET, SO_SNDBUF), 1024 * 1024)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 156, in __getattr__
    return getattr(self._sock, name)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 66, in _dummy
    raise OSError(EBADF, 'Bad file descriptor')
OSError: [Errno 9] Bad file descriptor

@ChrisXiaoShu The stack trace you posted tells us that this specific socket object has been explicitly closed at the Python level (that's when gevent uses its _dummy to generate the same exceptions that the operating system does). This means some part of your application stack is closing the socket before returning the response to let gunicorn handle it; at the point the error occurs, gunicorn hadn't even sent HTTP response headers yet.

Kind of the same problem in my case, with the difference that this error happens without doing anything. Sometimes after 5min, sometimes after 2hrs...