Requests: socket.error: [Errno 54] Connection reset by peer

Created on 22 Sep 2011  ·  19Comments  ·  Source: psf/requests

I'm currently getting the error, socket.error: [Errno 54] Connection reset by peer when I use requests.get(url, params=kwargs) and the params contain large bodies of text. If I truncate the two large bodies of text to less than 2,900 characters each, it works. If I run the same get request from the command line using curl it works. I'm using requests version 0.6.1 that I installed using, pip install python-requests.

I'm not sure how to tell you to replicate the issue because I'm using the python-sendgrid library to add a newsletter to my sendgrid account and I don't want to post my api username and password in issue ticket. :)

To test from the command line using curl I created two files that each contained plain text and html text that was urlencoded using a urlencoder. Then I ran the following command.

export IDENTITY='<my identity number>'
export API_KEY='<my smtp password>'
export API_USER='<my smtp username>'
export NAME='<My Urlencoded Newsletter Name>'
export SUBJECT='<My Urlencoded Newsletter Subject>'
TEXT=`cat urlencoded.txt`; HTML=`cat urlencoded.html`; curl -d "api_user=$API_USER&api_key=$API_KEY&identity=$IDENTITY&name=$NAME&subject=$SUBJECT&text=$TEXT&html=$HTML" https://sendgrid.com/api/newsletter/newsletter/add.json
picture epicserve

Most helpful comment

Sweet action! Switching from requests.get(url, params=kwargs) to requests.post(url, data=kwargs) fixed it! Sorry for bother you with this and thanks for the help!

picture epicserve on 22 Sep 2011
🎉5

All 19 comments

Hmm, interesting. If that's the error that is being raised, I have little reason to believe that the server _isn't_ actually resetting the connection.

Do you have an HTTP proxy like Charles available? It might help shed some light onto what's actually going on.

kennethreitz42 picture kennethreitz42 on 22 Sep 2011
❤1 👍1

Sorry, I forgot to provide the full traceback. Here is the traceback I get when I get the error. Also, I'm not understanding how to use the HTTP proxy and why that would make a difference when it works from the command line with curl.

Traceback (most recent call last):
  File "/Users/oconnor/.virtualenvs/emails/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Users/oconnor/Sites/wenworld/emails/emails/management/commands/ww_daily_headlines_send.py", line 114, in handle
    self.add_newsletter_to_sendgrid(sendgrid_newsletter_name, NEWSLETTER_SLUG)
  File "/Users/oconnor/Sites/wenworld/emails/emails/management/commands/ww_daily_headlines_send.py", line 94, in add_newsletter_to_sendgrid
    html=email.html
  File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 67, in newsletter_add
    subject=subject, text=text, html=html)
  File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 157, in get
    return self.call(method, **kwargs)
  File "/Users/oconnor/.virtualenvs/emails/src/sendgrid/src/sendgrid/__init__.py", line 56, in call
    result_json = json.loads(response.content)
  File "/Users/oconnor/.virtualenvs/emails/lib/python2.7/site-packages/requests/models.py", line 429, in __getattr__
    self._content = self.read()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 351, in read
    data = self._sock.recv(rbufsize)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 553, in read
    s = self.fp.read(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1276, in read
    return s + self._file.read(amt - len(s))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 219, in recv
    return self.read(buflen)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 138, in read
    return self._sslobj.read(len)
socket.error: [Errno 54] Connection reset by peer
epicserve picture epicserve on 22 Sep 2011
❤1

The response I get back from github is...

<html>
<head><title>414 Request-URI Too Large</title></head>
<body bgcolor="white">
<center><h1>414 Request-URI Too Large</h1></center>
<hr><center>nginx/0.7.65</center>
</body>
</html>

Which is strange because, like I mentioned before it works with curl.

epicserve picture epicserve on 22 Sep 2011
❤2

If you take the time to take a look at the two requests through Charles (or similar), I'm sure you'll see the issue quite quickly.

As the error says, the Request-URI is too large for the server to process, so it's dropping the connection.

With Requests, you're sending all the data as a massive query string in the Request URI, known as a GET parameter. When you're using curl, you're uploading form-encoded data to the body of the request, known as POST data.

kennethreitz42 picture kennethreitz42 on 22 Sep 2011
👍3 ❤2

I've tried using both requests.get(url, params=kwargs) and requests.post(url, params=kwargs) and both return the same error. For some reason Charles isn't capturing requests from curl and it's just capturing requests from my browser and my python command lines script that uses python-requests.

epicserve picture epicserve on 22 Sep 2011
👎4

No worries.

params is for query strings, data is for POST data.

kennethreitz42 picture kennethreitz42 on 22 Sep 2011
🎉2

Sweet action! Switching from requests.get(url, params=kwargs) to requests.post(url, data=kwargs) fixed it! Sorry for bother you with this and thanks for the help!

epicserve picture epicserve on 22 Sep 2011
🎉5

No worries at all :)

kennethreitz42 picture kennethreitz42 on 22 Sep 2011
👎2 👍2

It's strange that I get the same error with similar data, the only difference is that in my case I do not have big list or size of parameters. The same request made via CURL works fine but via requests gives me this error.

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='sbarnea.com', port=443): Max retries exceeded with url: /jira/rest/api/2/group?groupname=jira-administrators&expand=users (Caused by : [Errno 54] Connection reset by peer)

Even more interesting, it seems that the request does not even reach the web server.

Any idea what it could be cause?

ssbarnea picture ssbarnea on 19 Dec 2013
👀1

@ssbarnea This issue is more than two years old: it has nothing to do with the problem you're currently experiencing. =) Can you open a new issue to track your problem please?

Lukasa picture Lukasa on 19 Dec 2013
🎉2

Even better, don't open an issue, go talk to StackOverflow, you're asking a question and you have no reason to believe it is an issue with how requests behaves. Hash out the details on StackOverflow and then we can file a bug report (if necessary).

sigmavirus24 picture sigmavirus24 on 19 Dec 2013
👎3

It's a very strange behavior and I haver reasons to believe it's not a bug in requests, that's why I did open a new bug. I'm still investigating what could cause this, it happens only from my machine, but the most interesting part is that CURL works (and browsers too). Thank you.

ssbarnea picture ssbarnea on 19 Dec 2013
❤2

Sorry to keep this here but I found what cause this:

#  resolver 127.0.0.1;
#  ssl_protocols TLSv1.2 TLSv1.1 ; # TLSv1
#  ssl_stapling on;
#  ssl_stapling_verify on;
#  ssl_session_cache builtin:1000;
#  ssl_session_timeout 30m;
#  ssl_ciphers HIGH:!RC4:!aNULL:!MD5;
#  ssl_prefer_server_ciphers on;

By disabling these tunnings on the nginx server it started to work with requests, clearly one of them is causing the malfunction.

ssbarnea picture ssbarnea on 19 Dec 2013
❤2

@ssbarnea
It most probably a buggy server doing wrong stuff when spoken to in specific SSL/TLS versions like described in #1567 (and several more).
Try to force a different version like it is explained here (some more kudos to @Lukasa)

t-8ch picture t-8ch on 19 Dec 2013
❤2

And the winner (losser) is ssl_protocols TLSv1.2 TLSv1.1;
It seems that removing TLSv1 from the supported protocols on the server prevent requests from working.

Now, I can call this a bug in requests or an underlying library?

Note: removal of TLSv1 is recommended due to security concerns, there are exploits with it.

ssbarnea picture ssbarnea on 19 Dec 2013
❤1

@ssbarnea You'll find that the limitation isn't in Requests but in Python. Currently no shipped versions of Python have support for any version of TLS more recent than V1.0 as you can see in the table here.

Python 3.4 (not yet shipped, no guarantees that Requests is compatible with it) allows support for TLS1.2 and TLS1.1, so you can try with that. Otherwise, you'll have to allow TLSv1 to allow Requests through.

Lukasa picture Lukasa on 19 Dec 2013
👎1

Still there is something strange: in both cases I used Python, when tested from OS X it failed, when tested from Ubuntu it worked.

Normally I wouldn't touch the default settings for SSL, but I run a SSL report and tried to solve most warnings:

FIPS requires: SSL v2 and SSL v3 are disabled (only TLS version protocols in use)
Also read about BEAST attacks.

Check it here:
https://sslcheck.globalsign.com/en_US/sslcheck/?host=sbarnea.com

So, I wondering if it's possible to fix this, clearly if it works with Python on Ubuntu, it cannot be specific to all python versions.

ssbarnea picture ssbarnea on 19 Dec 2013
😕1

Mm, sadly it can. The relative features of the ssl support you get depends on the version of OpenSSL that you have installed. Compare the two versions. =)

Lukasa picture Lukasa on 19 Dec 2013
🎉1

Just to give a bit of closure on the subject. If your webserver doesn't allow TLSv1 then you will get TLSv1.1 and 1.2 with python 2.7.9 (due out in december 2014). However you can also install PYOpenSSL and inject that into urrlib3 (http://urllib3.readthedocs.org/en/latest/security.html#openssl-pyopenssl) which seems to allow use of requests for a webserver that has disabled TLSv1 with python 2.7.6 (and probably others).

import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()

But really, the webserver should probably not be deprecating something which forces people to jump through this many weird hoops.

chrisdl picture chrisdl on 12 Nov 2014
😄1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

avinassh picture avinassh  ·  4Comments
eromoe picture eromoe  ·  3Comments
cnicodeme picture cnicodeme  ·  3Comments
xsren picture xsren  ·  3Comments
Gonzalliz picture Gonzalliz  ·  3Comments