Gunicorn: OSError: [Errno 11] Resource temporarily unavailable on version 19.8.1 (and 19.8.0) on Heroku

Created on 29 May 2018  ·  14Comments  ·  Source: benoitc/gunicorn

I'm running a simple Flask server on Heroku:

web: gunicorn --worker-class eventlet -w 1 app:app --log-file=-

It's using Python 2.7.15 for compatibility with various other packages.

I seem to have run into a duplicate of this problem from ages ago since Heroku has moved to v. 19.8.1. Some images (anywhere from a few kb to a few mb in size) won't load; I have a site with lots of images (mostly sprite sheets for animation) and seemingly a random selection of them won't load each time, each throwing the following error (if the images are cached from an earlier version, it loads without issue):

2018-05-29T09:24:36.216949+00:00 app[web.1]: [2018-05-29 09:24:36 +0000] [10] [ERROR] Socket error processing request. 2018-05-29T09:24:36.216969+00:00 app[web.1]: Traceback (most recent call last): 2018-05-29T09:24:36.216971+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/async.py", line 66, in handle 2018-05-29T09:24:36.216972+00:00 app[web.1]: six.reraise(*sys.exc_info()) 2018-05-29T09:24:36.216974+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle 2018-05-29T09:24:36.216976+00:00 app[web.1]: self.handle_request(listener_name, req, client, addr) 2018-05-29T09:24:36.216978+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/async.py", line 129, in handle_request 2018-05-29T09:24:36.216980+00:00 app[web.1]: six.reraise(*sys.exc_info()) 2018-05-29T09:24:36.216981+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/async.py", line 112, in handle_request 2018-05-29T09:24:36.216983+00:00 app[web.1]: resp.write_file(respiter) 2018-05-29T09:24:36.216985+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/http/wsgi.py", line 403, in write_file 2018-05-29T09:24:36.216987+00:00 app[web.1]: if not self.sendfile(respiter): 2018-05-29T09:24:36.216989+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/http/wsgi.py", line 393, in sendfile 2018-05-29T09:24:36.216990+00:00 app[web.1]: sent += sendfile(sockno, fileno, offset + sent, count) 2018-05-29T09:24:36.216992+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/http/_sendfile.py", line 66, in sendfile 2018-05-29T09:24:36.216994+00:00 app[web.1]: raise OSError(e, os.strerror(e)) 2018-05-29T09:24:36.216996+00:00 app[web.1]: OSError: [Errno 11] Resource temporarily unavailable

these are the other versions in the requirements.txt:

Flask==0.12.2 gunicorn==19.8.1 pymongo==3.6.1 flask_socketio==2.9.6 flask_cors==3.0.3 eventlet==0.22.1 gevent==1.2.2

Changing gunicorn to 19.7.1 seems to resolve the problem; it persists with 19.8.0.
As with the similar problem from 2012, it's not a request timeout issue as the error it throws is pretty immediate. Rolling back to 19.7.1 has fixed it, so for now I'll stick with that, but it would be nice to use the latest version. Seems like this could be a Heroku-specific problem; I only noticed in the last month or so, but can't find any information about when they changed versions.

Most helpful comment

I fought with this same issue today, all day. And I _think_ I finally fixed it. I'm using nginx, Flask, gunicorn w/ eventlet, and docker.

My (relevant) pip freeze output:

eventlet==0.23.0
Flask==1.0.2
greenlet==0.4.14
gunicorn==19.9.0

My gunicorn command:
gunicorn -b 0.0.0.0:8000 --workers 1 --worker-class eventlet --log-level=DEBUG myapp.wsgi:app

The first symptom was large static file loads throwing ERR_CONTENT_LENGTH_MISMATCH in the browser. Obviously this broke the application, as large static JS libs weren't being loaded.

The second symptom was nginx logging the following to error.log: upstream prematurely closed connection while reading upstream

Finally, I traced it back to a gunicorn log item:

Socket error processing request. - [in /usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py:277]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 66, in handle
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 129, in handle_request
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 112, in handle_request
    resp.write_file(respiter)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 403, in write_file
    if not self.sendfile(respiter):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 393, in sendfile
    sent += sendfile(sockno, fileno, offset + sent, count)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/_sendfile.py", line 66, in sendfile
    raise OSError(e, os.strerror(e))
OSError: [Errno 11] Resource temporarily unavailable

My eventual solution was to start gunicorn with the --no-sendfile flag, and the problem went away. Why? Not sure... I'm just happy it's working.

Also worth mentioning, during my troubleshooting, I did my best to have my nginx.conf resemble the example found here: http://docs.gunicorn.org/en/stable/deploy.html

All 14 comments

I fought with this same issue today, all day. And I _think_ I finally fixed it. I'm using nginx, Flask, gunicorn w/ eventlet, and docker.

My (relevant) pip freeze output:

eventlet==0.23.0
Flask==1.0.2
greenlet==0.4.14
gunicorn==19.9.0

My gunicorn command:
gunicorn -b 0.0.0.0:8000 --workers 1 --worker-class eventlet --log-level=DEBUG myapp.wsgi:app

The first symptom was large static file loads throwing ERR_CONTENT_LENGTH_MISMATCH in the browser. Obviously this broke the application, as large static JS libs weren't being loaded.

The second symptom was nginx logging the following to error.log: upstream prematurely closed connection while reading upstream

Finally, I traced it back to a gunicorn log item:

Socket error processing request. - [in /usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py:277]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 66, in handle
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 129, in handle_request
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base_async.py", line 112, in handle_request
    resp.write_file(respiter)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 403, in write_file
    if not self.sendfile(respiter):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 393, in sendfile
    sent += sendfile(sockno, fileno, offset + sent, count)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/_sendfile.py", line 66, in sendfile
    raise OSError(e, os.strerror(e))
OSError: [Errno 11] Resource temporarily unavailable

My eventual solution was to start gunicorn with the --no-sendfile flag, and the problem went away. Why? Not sure... I'm just happy it's working.

Also worth mentioning, during my troubleshooting, I did my best to have my nginx.conf resemble the example found here: http://docs.gunicorn.org/en/stable/deploy.html

I meet this issue too, 19.7.0 work fine

Does this occur immediately or after a long response has been partially sent?

May be serve static file cause

Any update on this? I am facing the same issue in 19.9.0

Everything was working fine and all of a sudden it started happening.

@tilgovi let me know if you need some info regarding this. All of a sudden this issue started showing up

For me the problem was due to the eventlet worker. I removed eventlet and everything is fine now.

I meet the same problem. And the suggestion from @SaintSimmo works well for me. This problem occurs immediately when starting to download a large file. I am using flask and eventlet. And the downloading job is done by send_from_directory from Flask.
The gunicorn is started in following command:
gunicorn --worker-class eventlet -w 1 -b 0.0.0.0:4000 upload:app
which will give the error.
if "--no-sendfile" is added in the command, no error message is sent. If the downloading job can be done without "sendfile", so when this "sendfile" should be used?

gunicorn (version 19.9.0) same issue with eventlet

On 19.9.0, @SaintSimmo's recommendation of setting --no-sendfile also worked for me.

Yep, the same problem and after remove the eventlet worker it's working fine.

When i start the server with this command (gunicorn -w 1 -k eventlet -b 127.0.0.1:5000 wsgi:app) i receive the exceptions below and my image is truncated on the client response.
[ERROR] Socket error processing request.
...
BlockingIOError: [Errno 11] Resource temporarily unavailable

Removing the worker class definition, it work.
gunicorn -w 1 -b 127.0.0.1:5000 wsgi:app

@jacebrowning and @SaintSimmo, I confirm that --no-sendfile extra parameter into gunicorn command is effective.

This is usually due to the nproc is too few, you can increase the number of nproc for the user who run that program by editing the file '/etc/security/limits.conf'.

Are you suffering from this problem? If you are using Python 3.4 or above, update your Gunicorn version.

I had the same problem with --worker-class. I updated gunicorn to 20.0.4 and the problem was solved.

Changing worker class to gevent worked for me. --worker-class gevent

Was this page helpful?
0 / 5 - 0 ratings