Werkzeug: Werkzeug crashes after writing to closed pipe

Use a production WSGI server such as Gunicorn or uWSGI, not the Werkzeug dev server.

I have a very similar problem except that I'm using the Werkzeug dev server for development (which is, AFAICT, its intended use), i.e with the browser directly connected to port 5000.

The error happens several times per hour, forcing a manual restart of the server to keep developing.

Here's a traceback:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/fermigier/envs/extranet-spr/lib/python2.7/site-packages/werkzeug/serving.py", line 659, in inner
    srv.serve_forever()
  File "/Users/fermigier/envs/extranet-spr/lib/python2.7/site-packages/werkzeug/serving.py", line 499, in serve_forever
    HTTPServer.serve_forever(self)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 238, in serve_forever
    self._handle_request_noblock()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 297, in _handle_request_noblock
    self.handle_error(request, client_address)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 655, in __init__
    self.handle()
  File "/Users/fermigier/envs/extranet-spr/lib/python2.7/site-packages/werkzeug/serving.py", line 216, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/Users/fermigier/envs/extranet-spr/lib/python2.7/site-packages/werkzeug/serving.py", line 251, in handle_one_request
    return self.run_wsgi()
  File "/Users/fermigier/envs/extranet-spr/lib/python2.7/site-packages/werkzeug/serving.py", line 193, in run_wsgi
    execute(self.server.app)
  File "/Users/fermigier/envs/extranet-spr/lib/python2.7/site-packages/werkzeug/serving.py", line 186, in execute
    write(b'')
  File "/Users/fermigier/envs/extranet-spr/lib/python2.7/site-packages/werkzeug/serving.py", line 152, in write
    self.send_header(key, value)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 401, in send_header
    self.wfile.write("%s: %s\r\n" % (keyword, value))
IOError: [Errno 32] Broken pipe

I experience the very same issue as @sfermigier with development server (debug=True), and have same traceback error.

This behavior happens usually in a very simple case: you're using some sort of auto-complete features. The browser starts a connection for query tokens and then stops and starts another request for more tokens. You'll end up with lots of Broken Pipes. And this wasn't a problem until last release which makes the dev server completely blocked.
So, suggesting to use a full featured application server is a good workaround but I still see a problem here. A development only problem of course but the fact it is so common to trigger makes it disturbing for lots of developers not used to _protocol internals_.
A broken pipe is very common (think about a long request by error and the developer hitting the browser stop button) and shouldn't break the development server.
Just my opinion. :)

@xcash

This behavior happens usually in a very simple case: you're using some sort of auto-complete features. The browser starts a connection for query tokens and then stops and starts another request for more tokens.

If it's related, I can confirm I experience this issue using browsersync with gulp.js.

Does anyone have solution that doesn't include running a WSGI server? I seem to be running into this problem with bots performing a SYN scan against my host.

@glennzw can you be more specific on your environment? You should not have your dev server public on the open web accessible by bots. :) In such a situation, like a demo host for clients, it's always better to use at least a real application server like gunicorn which has a really small footprint.

FWIW, I've haven't been keeping up with releases much and this (very annoying) problem started happening for me sometime between May - August 2016 best I can tell. I added this to my setup.py install_requires = ['Werkzeug<0.11', 'flask<0.11', ... - which seems to work around the issue (IME, just pinning Werkzeug didn't seem to do the trick?)

For me the duplication case was simple enough - load a page, but don't let it finish loading. That is, just trigger _any_ broken pipe error - and the webserver will crash and fail to service any other requests. IMHO, webservers can't _fall over_ when a client closes the connection prematurely - even development ones.

Could it be that you all have passthrough_errors set somewhere?

@untitaker in that case, pallets/flask#1674 pallets/flask#1679 pallets/flask#1928 possibly related?

I don't know, I would like one of the reporters to confirm.

On 26 August 2016 17:05:25 CEST, David Lord [email protected] wrote:

@untitaker in that case, pallets/flask#1674 pallets/flask#1679
pallets/flask#1928 possibly related?

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/werkzeug/issues/954#issuecomment-242761250

Sent from my Android device with K-9 Mail. Please excuse my brevity.

cc @miguelgrinberg

I think Werkzeug should handle broken pipe and connection reset errors. These are really not an indication of an error, the client just went away. Seems in this case a special exception should be raised, one that is recognized by the catch-all way above as one to ignore, even if error passthrough is set.

Here is how gunicorn does it: https://github.com/benoitc/gunicorn/blob/39f62ac66beaf83ceccefbfabd5e3af7735d2aff/gunicorn/workers/sync.py#L151-L154

That's what it is supposed to do. I'm trying to figure out how to reproduce
this behavior, but there is no clear testcase available yet. Hence the question
about passthrough_errors.

I suspect that this is not a bug in Werkzeug, and that the user's browser
simply has a connection still open that blocks other requests (instead of the
server crashing). If you close your browser and reopen, the server should
function again.

On Fri, Aug 26, 2016 at 11:54:16AM -0700, Miguel Grinberg wrote:

I think Werkzeug should handle broken pipe and connection reset errors. These are really not an indication of an error, the client just went away. Seems in this case a special exception should be raised, one that is recognized by the catch-all way above as one to ignore, even if error passthrough is set.

Here is how gunicorn does it: https://github.com/benoitc/gunicorn/blob/39f62ac66beaf83ceccefbfabd5e3af7735d2aff/gunicorn/workers/sync.py#L151-L154

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/werkzeug/issues/954#issuecomment-242821084

Oh, and broken pipe errors _are_ displayed, yes, but they should never hang up the server like described. See previous comment about the possible reason.

I tested again the latest bits and I'm still seeing the same behavior in my environment. But since you seemed to be having trouble reproducing I tried to pull out why I'm special.

import time
from flask import Flask
app = Flask(__name__)


@app.route('/')
def hello_world():
    time.sleep(5)
    return 'Hello, World!'


if __name__ == "__main__":
    app.run()

It work as expected with flask run but the web server would crash if you close your web browser before letting the response get rendered when started via python hello.py

It appears a part of your response is lost.

On Fri, Aug 26, 2016 at 12:29:39PM -0700, clayg wrote:

I tested again the latest bits and I'm still seeing the same behavior in my environment. But since you seemed to be having trouble reproducing I tried to pull out why I'm special.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/werkzeug/issues/954#issuecomment-242829536

yes, in slack triple-ticks is how you block quote and ctrl-return is how you new-line
on github, triple-ticks is how you block quote, but ctrl-return is how you submit
... anyway ... muscle memory

I edited my post immediately after submitting it to finish up - and I'm only responding because it appears you're replying from email and I'm not sure github will send you another notice after my edit.

I cannot reproduce with the sleep test by @clayg above. I actually don't get the broken pipe error at all. I close the browser before that request gets to return a response, but Werkzeug carries out the request until the end anyway, it prints the 200 log line to the console and does not show any error.

I also tried the same trick using my flask video streaming example which uses a streaming response to provide video frames in a never ending stream, and even for that one, I can close the browser and the request ends without any errors. This one is weird, because I am certain in the past this application would trigger a broken pipe error to the console before ending the request.

Actually, I spoke too soon. I can reproduce every time with my video streaming app when using Python 2.7. I cannot reproduce on 3.5. All the stack traces above are for 2.7, so keep that in mind if you are testing with Python 3.

One more interesting data point. If running with the reloader, when the child process exits, the master process running the reloader starts another one, so there is no interruption. But if you are running the server without the reloader, the broken pipe error brings you back to the console.

Edit: disregard the reloader starting another process part, that does not seem to be happening and instead I was likely seeing the effect of changing the passthrough error setting.

Okay, here is the analysis of what I think is happening:

• The client goes away mid-request
• The request keeps going. The socket connection appears to be buffered, so in most cases, writes to the socket will not cause any problems.
• Once the request ends, the socket server class will issue a flush() on the socket. This was the subject of an old bug in the Python library that is currently fixed: http://bugs.python.org/issue14574. The solution in this fix was to catch socket.error and ignore it.
• Then socket server tries to close the connection. This is the following stack frame, from the backtrace from the OP:

File "/usr/lib/python2.7/SocketServer.py", line 710, in finish self.wfile.close()

• Unfortunately, in Python 2.7, the first thing the socket.close() method does is flush again:

File "/usr/lib/python2.7/socket.py", line 279, in close self.flush()

• This second attempt at a flush is not protected with a try/except, so it raises the EPIPE exception.
• The socket server catches the exception, and then delivers it to the server's handle_error() method.
• The Werkzeug implementation of handle_error() looks at the passthrough_errors setting, and since we have this always set to True, re-raises the EPIPE error and lets it bubble up way to the top.

The socket code on Python 3 is completely different, and in particular, it does not appear to have any flush calls without try/excepts around them. The EPIPE error does not even bubble up to Werkzeug when using Python 3.

Do we even have passthrough_errors set to true? In Werkzeug it is false by default.

On 27 August 2016 02:10:13 CEST, Miguel Grinberg [email protected] wrote:

Okay, here is the analysis of what I think is happening:

• The client goes away mid-request
• The request keeps going. The socket connection appears to be
  buffered, so in most cases, writes to the socket will not cause any
  problems.
• Once the request ends, the socket server class will issue a flush()
  on the socket. This was the subject of an old bug in the Python library
  that is currently fixed: http://bugs.python.org/issue14574. The
  solution in this fix was to catch socket.error and ignore it.
• Then socket server tries to close the connection. This is the
  following stack frame, from the backtrace from the OP:

 File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
 self.wfile.close()

• Unfortunately, in Python 2.7, the first thing the socket.close()
  method does is flush again:

 File "/usr/lib/python2.7/socket.py", line 279, in close
 self.flush()

• This second attempt at a flush is not protected with a try/except, so
  it raises the EPIPE exception.
• The socket server catches the exception, and then delivers it to the
  server's handle_error() method.
• The Werkzeug implementation of handle_error() looks at the
  passthrough_errors setting, and since we have this always set to
  True, re-raises the EPIPE error and lets it bubble up way to the top.

The socket code on Python 3 is completely different, and in particular,
it does not appear to have any flush calls without try/excepts around
them. The EPIPE error does not even bubble up to Werkzeug when using
Python 3.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/werkzeug/issues/954#issuecomment-242881523

Sent from my Android device with K-9 Mail. Please excuse my brevity.

Oh, ahem: https://github.com/pallets/flask/pull/1679

I think passthrough_errors should be dependent on app.debug. NVM, is useless

I see no option other than reverting that PR actually. passthrough_errors=True just does what it's supposed to do, which is not a good default behavior if one hasn't attached a debugger to the program.

Nevermind, I found another way. Two PRs:

• https://github.com/pallets/flask/pull/1996 -- this would fix the issue on its own, except that the traceback is still visible
• https://github.com/pallets/werkzeug/pull/998 -- even if passthrough_errors=True, never reraise socket errors. I don't think that ever was the intention anyway

Since both are behavior changes in the broad sense, I'd rather not backport them.

I think https://github.com/pallets/flask/pull/1996 is an acceptable solution. The important thing is that it fixes the common case where you do not want exceptions propagated. If you do want to propagate, then you are debugging, and in that case getting the socket.error propagated when it shouldn't is not a huge deal.

The https://github.com/pallets/werkzeug/pull/998 fix is not great though. The application might raise these exceptions legitimately from something it is doing with sockets in its own handlers, and those would be silenced as well. The ideal solution would be that these are caught at the place they occur and then re-raised as some custom exception class that handle_error can recognize and ignore. Given that we probably don't want to change or overload SocketServer, I think my vote is to just leave this part as is. You will get the EPIPE dumped to the console, but only on Python 2, and at least it's not going to stop the server after the other fix goes in. It's harmless, and it's a behavior that existed in the past, before I made the passthrough_errors change.

The behavior you describe only happens with PASSTHROUGH_ERRORS enabled though. Otherwise the exception is caught from within Flask.

I guess this cosmetic improvement is not worth it though.

On 27 August 2016 18:29:30 CEST, Miguel Grinberg [email protected] wrote:

I think https://github.com/pallets/flask/pull/1996 is an acceptable
solution. The important thing is that it fixes the common case where
you do not want exceptions propagated. If you do want to propagate,
then you are debugging, and in that case getting the socket.error
propagated when it shouldn't is not a huge deal.

The https://github.com/pallets/werkzeug/pull/998 fix is not great
though. The application might raise these exceptions legitimately from
something it is doing with sockets in its own handlers, and those would
be silenced as well. The ideal solution would be that these are caught
at the place they occur and then re-raised as some custom exception
class that handle_error can recognize and ignore. Given that we
probably don't want to change or overload SocketServer, I think my
vote is to just leave this part as is. You will get the EPIPE dumped to
the console, but only on Python 2, and at least it's not going to stop
the server after the other fix goes in. It's harmless, and it's a
behavior that existed in the past, before I made the
passthrough_errors change.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/werkzeug/issues/954#issuecomment-242926832

Sent from my Android device with K-9 Mail. Please excuse my brevity.

Fixed in master.

The behavior you describe only happens with PASSTHROUGH_ERRORS enabled though

Yes, I omitted that detail. But this change would even affect Python 3, where none of this is an issue. On Py3, with passthrough errors enabled, a legitimate socket.error raised by the application would be silenced.

master seems to wfm, looking forward to the next release, thanks!

Hi , I use a Werkzeug dev server running behind NGINX, I am facing the same issue could anyone help me with this ,
11:13:11 web.1 | 127.0.0.1 - - [15/Sep/2016 11:13:11] "GET /api/method/frappe.utils.print_format.download_pdf?doctype=Purchase%20Order&name=PO-00001&format=PO&no_letterhead=0 HTTP/1.1" 200 - 11:13:11 web.1 | Error on request: 11:13:11 web.1 | Traceback (most recent call last): 11:13:11 web.1 | File "/home/ommi/frappe-bench/env/lib/python2.7/site-packages/werkzeug/serving.py", line 193, in run_wsgi 11:13:11 web.1 | execute(self.server.app) 11:13:11 web.1 | File "/home/ommi/frappe-bench/env/lib/python2.7/site-packages/werkzeug/serving.py", line 184, in execute 11:13:11 web.1 | write(data) 11:13:11 web.1 | File "/home/ommi/frappe-bench/env/lib/python2.7/site-packages/werkzeug/serving.py", line 152, in write 11:13:11 web.1 | self.send_header(key, value) 11:13:11 web.1 | File "/usr/lib/python2.7/BaseHTTPServer.py", line 401, in send_header 11:13:11 web.1 | self.wfile.write("%s: %s\r\n" % (keyword, value)) 11:13:11 web.1 | IOError: [Errno 32] Broken pipe

Please help

Ragav use another application server instead of werkzeug's built in dev
server, like gunicorn. It's the only solution right now.

2016-09-15 8:07 GMT+02:00 Ragav [email protected]:

Hi , I use a Werkzeug dev server running behind NGINX, I am facing the
same issue could anyone help me with this , ```
11:13:11 web.1 | 127.0.0.1 - - [15/Sep/2016 11:13:11] "GET
/api/method/frappe.utils.print_format.download_pdf?
doctype=Purchase%20Order&name=PO-00001&format=PO&no_letterhead=0
HTTP/1.1" 200 -
11:13:11 web.1 | Error on request:
11:13:11 web.1 | Traceback (most recent call last):
11:13:11 web.1 | File "/home/ommi/frappe-bench/env/
lib/python2.7/site-packages/werkzeug/serving.py", line 193, in run_wsgi
11:13:11 web.1 | execute(self.server.app)
11:13:11 web.1 | File "/home/ommi/frappe-bench/env/
lib/python2.7/site-packages/werkzeug/serving.py", line 184, in execute
11:13:11 web.1 | write(data)
11:13:11 web.1 | File "/home/ommi/frappe-bench/env/
lib/python2.7/site-packages/werkzeug/serving.py", line 152, in write
11:13:11 web.1 | self.send_header(key, value)
11:13:11 web.1 | File "/usr/lib/python2.7/BaseHTTPServer.py", line 401,
in send_header
11:13:11 web.1 | self.wfile.write("%s: %s\r\n" % (keyword, value))
11:13:11 web.1 | IOError: [Errno 32] Broken pipe

Please help

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pallets/werkzeug/issues/954#issuecomment-247243400,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6MZ6DNiRIfL91CLeYOoA70W9_nQQzGks5qqOCMgaJpZM4I58cy
.

Guided by the recent fix commit, I've been able to fix this problem by calling app.run with passthrough_errors=False. YMMV

The bug that caused the crash was fixed in Version 0.12, Released on December 21st 2016.

• Revert a behavior change that made the dev server crash instead of returning a Internal Server Error (pull request #2006).

Version 0.12 has only been released last week.

On Mon, Mar 20, 2017 at 09:05:00AM -0700, Alan Rotman wrote:

The bug that caused the crash was fixed in Version 0.12, Released on December 21st 2016.

• Revert a behavior change that made the dev server crash instead of returning a Internal Server Error (pull request #2006).

--
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/werkzeug/issues/954#issuecomment-287807602

I just saw ReleaseNotes today, and have been waiting for this fix for a long time.

Look at: http://flask.pocoo.org/docs/0.12/changelog/
Version 0.12
Released on December 21st 2016, codename Punsch.

https://pypi.python.org/pypi/Flask/0.12
File Type Py Version Uploaded on Size
Flask-0.12-py2.py3-none-any.whl (md5) Python Wheel 2.7 2016-12-21 80KB
Flask-0.12.tar.gz (md5) Source 2016-12-21 519KB

Ah, yes, you mean Flask. Sure.

On Mon, Mar 20, 2017 at 09:22:15AM -0700, Alan Rotman wrote:

I just saw ReleaseNotes today, and have been waiting for this fix for a long time.

Look at: http://flask.pocoo.org/docs/0.12/changelog/
Version 0.12
Released on December 21st 2016, codename Punsch.

https://pypi.python.org/pypi/Flask/0.12
File Type Py Version Uploaded on Size
Flask-0.12-py2.py3-none-any.whl (md5) Python Wheel 2.7 2016-12-21 80KB
Flask-0.12.tar.gz (md5) Source 2016-12-21 519KB

--
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/werkzeug/issues/954#issuecomment-287813405

Just a note for anyone who runs across this issue when running flask 0.12.2 on werkzeug in threaded=True mode:
In threaded mode, be default it seems like each werkzeug thread actually still has this problem, i.e. if you request a route that takes some time to return, and then close the connection from the client, that particular werkzeug logs an IOError Broken Pipe and then dies. The server overall continues to function, except that in my application I was finding that this was causing a memory leak somehow, with the flask process slowly growing after a broken pipe in any thread, using up all the RAM and then SWAP and then finally being killed by the OS.
Explicitly sending passthrough_errors=False in app.run seems to have resolved the issue - the threads no longer die when the client disconnects, they gracefully log the IOError and then also log this (which I never saw without explicitly setting passthrough_errors=False):

Exception happened during processing of request from ('127.0.0.1', 50652)
----------------------------------------

And then the server continues to run as normal. I still have to wait a few hours to see if the memory leak shows up again, but I'm hopeful that it won't.

Just in case that's helpful for anyone.

I saw this error too in an Ubuntu Docker container on Kubernetes on Ubuntu VM:

Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 270, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 261, in execute
    write(data)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 227, in write
    self.send_header(key, value)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 412, in send_header
    self.wfile.write("%s: %s\r\n" % (keyword, value))
IOError: [Errno 32] Broken pipe

I created a brand new Ubuntu xenial VM and ran the same code in Ubuntu Docker container on Kubernetes, and this error was not seen and Python Flask worked as expected. I think it was issue with my host (Ubuntu VM).

@vhosakot Can you let me know how you set up your app configuration? I encountered a similar issue in the same environment as yours.

In a route function, I used another function which was meant for routing.
I fetched the data from the response of that function.
Now, when I used _loads()_ on that data, I am getting the error.

...
response = get_contents().data
        if response:
            data = loads(response)
..

Error: IOError: [Errno 32] Broken pipe