Gunicorn: Old bug reproduced：'Response' object has no attribute 'status_code' in wsgi.py with websockets

do you have any simple example to reproduce it? Also please try with latest master if possible.

Previously I have tried several times in my local development environment，which is the same application code with production environment，but I can't reproduce it.

And I have checked version 19.9.0 release log，not found something related，I'll keep
watching this error log，if find something new，I would post here.

I too have this problem, specifically when I force all my connection from client to websocket protocol. My settings is the same as BoWuGit. If allow polling protocol before upgrade, this does not show up, but another error :
`
[ERROR] Error handling request /socket.io/?EIO=3&transport=polling&t=MPRHUoV&sid=cd64be7c940e474d8728b114c3fb9bbe

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)

File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)

File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1994, in __call__
return self.wsgi_app(environ, start_response)

File "/usr/local/lib/python3.6/site-packages/flask_socketio/__init__.py", line 43, in __call__
start_response)

File "/usr/local/lib/python3.6/site-packages/engineio/middleware.
py", line 47, in __call__
return self.engineio_app.handle_request(environ, start_response)

File "/usr/local/lib/python3.6/site-packages/socketio/server.py", line 360, in handle_request
return self.eio.handle_request(environ, start_response)

File "/usr/local/lib/python3.6/site-packages/engineio/server.py", line 279, in handle_request
socket = self._get_socket(sid)

File "/usr/local/lib/python3.6/site-packages/engineio/server.py", line 439, in _get_socket
raise KeyError('Session is disconnected')
`
But I suspect it might have something to do with each other, since I force the connection to be websocket, this error has not been seen again.

Having this issue as well with gunicorn 19.9.0 and Flask-socketIO 3.0.2, when using eventlet 0.24.1

AttributeError: 'Response' object has no attribute 'status_code'

Also experiencing this issue with the following requirements:

Flask==1.0.2
gunicorn==19.5.0
python-socketio==2.0.0
eventlet==0.24.1

Error message when closing web browser that has open socket connection:

 Error handling request /socket.io/?EIO=3&transport=websocket&sid=d43ec0ae0bb946debc51f1ca2e5b8a94
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 52, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 114, in handle_request
    resp.close()
  File "/usr/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 403, in close
    self.send_headers()
  File "/usr/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 319, in send_headers
    tosend = self.default_headers()
  File "/usr/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 300, in default_headers
    elif self.should_close():
  File "/usr/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 233, in should_close
    if self.status_code < 200 or self.status_code in (204, 304):
AttributeError: 'Response' object has no attribute 'status_code'

It looks like this issue has been fixed in the latest version of python-engineio..

Have tested with python-engineio latest version(2.3.2), still not work.

Any news on this issue? I get the same error when using sentry-python

I have the same issue

eventlet: 0.25.1
flask-socketio: 4.2.1
gunicorn: 19.9.0

how to reproduce it? cna you provide a simple example?

i'm also not sure how to reproduce it, but it happens OFTEN when i refresh a page on my gunicorn app

Encounter the same issue, and my environment is the same as @eazow while gunicorn == 20.0.4.
It seems that the issue happened after I installed sentry for bug tracking.
The issues can be reproduced by

1. refreshing the page(not opening a new page)
2. closing the page

Interestingly, opening a new page won't produce the issue. Not sure why. Thanks!

I have the same issue as @cowbonlin . Same gunicorn version, too.

After installing sentry, we're getting crazy amounts of this error. Though I find it hard to tell if this always happened or not - since we didn't track errors before sentry.

While it doesn't seem to influence actual functionality of our server, this is just a ton of spam.

We're experiencing the same. Sentry installed but disabled. Any ideas?

Same issue with sentry installed.

Do you hve any example that reproduce it without sentry (disabled or not) ?

In addition, instead of a namespace I manually hit /api.

In addition, instead of a namespace I manually hit /api.

what does it mean ? Is this sentry related?

In addition, instead of a namespace I manually hit /api.

what does it mean ? Is this sentry related?

No, this is related to socket.io namespaces. I tried removing them, and even after removing them, I get the error. I get this other error on local machine without gunicorn or nginx however, which may be related.

These are my requirements:

sentry_sdk == 0.14.3
Flask_SocketIO == 4.2.1
eventlet == 0.25.1

This is my flask-socketio code on the server side:

socketio = SocketIO(engineio_logger=True, logger=True, debug=True, cors_allowed_origins="*", path='/socket.io')
...
socketio.init_app(app, async_mode="eventlet")

And this is my React socket io code on the client side:

          this.socket = io.connect(`http://localhost:5000?info=${someInfo}`, {
            transports: ['websocket', 'polling'] // an attempt to keep polling as a fallback but start on websockets
          });

Let me know if this helps. On Ubuntu the error looks like the above one, and locally on Windows it looks like this:
```Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 599, in handle_one_response
write(b'')
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 491, in write
raise AssertionError("write() before start_response()")
AssertionError: write() before start_response()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 357, in __init__
self.handle()
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 390, in handle
self.handle_one_request()
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 466, in handle_one_request
self.handle_one_response()
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 609, in handle_one_response
write(err_body)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 538, in write
wfile.flush()
File "C:\ProgramData\Anaconda3\lib\socket.py", line 607, in write
return self._sock.send(b)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\greenio\base.py", line 397, in send
return self._send_loop(self.fd.send, data, flags)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\greenio\base.py", line 384, in _send_loop
return send_method(data, *args)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\hubs\hub.py", line 461, in fire_timers
timer()
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\hubs\timer.py", line 59, in __call__
cb(args, *kw)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\semaphore.py", line 147, in _do_acquire
waiter.switch()
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\greenthread.py", line 221, in main
result = function(args, *kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 818, in process_request
proto.__init__(conn_state, self)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 359, in __init__
self.finish()
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\wsgi.py", line 732, in finish
BaseHTTPServer.BaseHTTPRequestHandler.finish(self)
File "C:\ProgramData\Anaconda3\lib\socketserver.py", line 784, in finish
self.wfile.close()
File "C:\ProgramData\Anaconda3\lib\socket.py", line 607, in write
return self._sock.send(b)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\greenio\base.py", line 397, in send
return self._send_loop(self.fd.send, data, flags)
File "C:\ProgramData\Anaconda3\lib\site-packages\eventlet\greenio\base.py", line 384, in _send_loop
return send_method(data, *args)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine```

Can confirm this error disappears when sentry is fully disabled. Would be great if gunicorn was robust enough to deal with this.

Bump @benoitc

Can confirm this error disappears when sentry is fully disabled. Would be great if gunicorn was robust enough to deal with this.

I found that disabling the Sentry's FlaskIntegration also makes the error disappear.

Seeing similar behavior. Using New Relic in production causes this error with flask-socketio. In development, the werkzeug debugger middleware needs to be loaded before flask-socketio is initialized (so it's not applied to engineio's wsgi app). Problem is production is where I really don't want the errors tripping.

Can't replace the response in gunicorn config's post_request, but I tried forcing a status code onto resp.status_code. It didn't take though.

This error is reproducible by using Sentry's FlaskIntegration together with Gunicorn and Flask-SocketIO. Is it possible to solve it soon?

@Canicio we thought to try that to get rid of the error and even after disabling the integration, the error persists.

Does anyone have sharable code/a minimal example for @benoitc to go off of?

Sure:

import sentry_sdk
from flask import Flask
from flask_socketio import SocketIO
from sentry_sdk.integrations.flask import FlaskIntegration

sentry_sdk.init(
    dsn="https://[email protected]/0",
    integrations=[FlaskIntegration()]
)

app = Flask(__name__)
socketio = SocketIO(app)

@app.route('/')
def index():
    return '''
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.2.0/socket.io.js"></script>
<script>
    var socket = io()
</script>

requirements:

flask
sentry-sdk[flask]
flask-socketio
eventlet

example gunicorn config:

bind = '[::]:4444'
worker_class = 'eventlet'
accesslog = '-'

On loading / it'll connect to the websocket. On websocket disconnect (e.g. navigate away, refresh), will produce exception like so:

[2020-09-23 07:24:49 +0000] [16303] [ERROR] Error handling request /socket.io/?EIO=3&transport=websocket&sid=29f4c1adfac343d6bc6db56acf8fd0ee
Traceback (most recent call last):
  File "/home/ziddey/projects/sentry/venv_sentry/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/home/ziddey/projects/sentry/venv_sentry/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 115, in handle_request
    resp.close()
  File "/home/ziddey/projects/sentry/venv_sentry/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 402, in close
    self.send_headers()
  File "/home/ziddey/projects/sentry/venv_sentry/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 318, in send_headers
    tosend = self.default_headers()
  File "/home/ziddey/projects/sentry/venv_sentry/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 299, in default_headers
    elif self.should_close():
  File "/home/ziddey/projects/sentry/venv_sentry/lib/python3.8/site-packages/gunicorn/http/wsgi.py", line 219, in should_close
    if self.status_code < 200 or self.status_code in (204, 304):
AttributeError: 'Response' object has no attribute 'status_code'
2001:470:1f07:7eb:9dd4:254c:35d7:236c - - [23/Sep/2020:07:24:49 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=29f4c1adfac343d6bc6db56acf8fd0ee HTTP/1.1" 500 0 "-" "-"

Note: I've never actually used sentry myself. This is just from the sentry getting started page. The example dsn works fine for our test.

Commenting integrations=[FlaskIntegration()] will then eliminate the error (of course effectively disabling sentry).

For what it's worth, gevent-websocket can be used instead of eventlet without errors. However, it then seems to handle all requests..

Ok, did some playing around. Looks like sentry/newrelic wraps the response. Without sentry, we get <eventlet.wsgi._AlreadyHandled object at 0x7fd0f5b1c0d0> as expected and gunicorn's EventletWorker.is_already_handled() will stop iteration.

However, when using sentry, this becomes something like <sentry_sdk.integrations.wsgi._ScopedResponse object at 0x7f30155a5100> instead, failing the check

Instead, we could peek at respiter to see if it's empty. Will look further tomorrow.

Alright, here's the workaround I've come up with:

eventlet_fix.py:
see edit below

And in my gunicorn config.py: worker_class = 'eventlet_fix.EventletWorker.

The issue is that sentry/newrelic wraps the responses, so we can't simply check it against eventlet's ALREADY_HANDLED. Since the nature of an already handled request is that gunicorn's start_response doesn't get called, we can instead check for the presence of a response status.

So I've hijacked the wsgi call to then check for a response status, and hack response values as necessary. This allows the request to still be logged by gunicorn. If instead, it's desired to keep the original behavior, StopIteration can be raised instead.

Hacking the status to 101 is appropriate for our use case here (flask-socketio websocket), but otherwise, leaving it as None works as well since headers_sent and should_close are forced to True.

Again, this makes the assumption that if status isn't set, start_response wasn't called, and therefore the request must have been "already handled" externally.

edit: No good. Will need to reevaluate. If the request takes time to perform, start_response won't be called before resp.status is checked.

edit2: Here's a fixed version with a hacked response iterator:

from functools import wraps

from gunicorn.workers.geventlet import EventletWorker as _EventletWorker


class HackedResponse:
    def __init__(self, respiter, resp):
        self.respiter = iter(respiter)
        self.resp = resp
        if hasattr(respiter, "close"):
            self.close = respiter.close

    def __iter__(self):
        return self

    def __next__(self):
        try:
            return next(self.respiter)
        except StopIteration:
            if not self.resp.status:
                self.resp.status = "101"  # logger derives status code from status instead of using status_code
                self.resp.status_code = 101  # not actually needed since headers_sent/force_close result in status_code not being checked anymore
                self.resp.headers_sent = True
                self.resp.force_close()
            raise


def wsgi_decorator(wsgi):
    @wraps(wsgi)
    def wrapper(environ, start_response):
        respiter = wsgi(environ, start_response)
        resp = start_response.__self__
        return HackedResponse(respiter, resp)

    return wrapper


class EventletWorker(_EventletWorker):
    def load_wsgi(self):
        super().load_wsgi()
        self.wsgi = wsgi_decorator(self.wsgi)

Obviously this is just a monkey patch. The actual fix could potentially go in handle_request in base_async.py. The key may be to (indirectly) check if start_response was called after iterating through respiter, either by checking resp.status (just start_response called) or resp.headers_sent (confirmation that we've actually responded to the request).

@benoitc
@ziddey has found a way to solve the problem.

@ziddey quick questions for your example (as I am not using sentry).

• Is the error only impacting sentry or is the request also stopped, ie the worker is terminating (which I suspect it does if the response is wrapped)?
• do you expect to have something wrapped there or cleaning the request ven if the response is wrapped would be OK?

@benoitc not able to test currently, but looking at the traceback above https://github.com/benoitc/gunicorn/issues/1852#issuecomment-697189261 and https://github.com/benoitc/gunicorn/blob/4ae2a05c37b332773997f90ba7542713b9bf8274/gunicorn/workers/base_async.py#L107-L140

Normally, is_already_handled would return True and it would just end here.

However, because the response is wrapped, that method doesn't work. Instead, execution progresses, failing at line 115: resp.close() attempts to send headers, but start_response was never called, so there's no status code. Even if it did, it'd still ultimately fail obviously.

This results in an AttributeError that gets reraised and assumedly handled by handle_error. Since the request was already handled externally, there is no harm here other than log spam.

I can't say too much about Sentry-- I'm not using it either.

One detail though: the current already-handled mechanism results in no access logging. I suppose this technically makes sense since there's no way of knowing how it was handled externally. In my hacked response, I force the status code to 101, with headers_sent as True so the handler can proceed and the request still gets access logged.

Checking resp.status is a definitive test for determining if start_response was called.

@benoitc revisiting this. To more definitively conclude that the request was already handled, environ['gunicorn.socket'] could instead be some sort of proxy for the underlying object. That way, it can be recorded when the socket is accessed directly (e.g. wrapping get_socket() for eventlet), and used for something like is_already_handled

It would still require hacking a response status though if access logging is desired.