Related to #1083, perhaps. Standard requests.get()
for this particular site/page https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html
results in:
>>> import requests
>>> requests.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
Using request-toolbelt
's SSLAdapter
to try various ssl versions, they all fail, it would seem... see following tracebacks.
TLSv1:
>>> adapter = SSLAdapter('TLSv1')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
SSLv3:
>>> adapter = SSLAdapter('SSLv3')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
SSLv2:
>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)
Note the last one gives a Connection reset by peer
error, which differs from the others, but I'm pretty sure SSLv2 isn't supported by the server anyhow.
For fun, I tried to pass through some more appropriate headers through on the last request as well:
>>> headers = {
... 'Accept': u"text/html,application/xhtml+xml,application/xml",
... 'User-Agent': u"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36",
... 'Accept-Encoding': u"gzip,deflate",
... 'Accept-Language': u"en-US,en;q=0.8"
... }
>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html', headers=headers)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)
No dice there either. Here's what the HTTPS connection info in Chrome on Mac looks like:
I'm not positive, but some googling indicates it's likely a cipher list issue, which is more urllib3, I think?
I tried to modify DEFAULT_CIPHER_LIST
in pyopenssl
, but started running into import errors. At this point it seemed like things were just broken, and there wasn't really a proper way to approach fixing this yet.
Version information:
OSX Mavericks
Python 2.7.5
OpenSSL 0.9.8y 5 Feb 2013 - (from python -c "import ssl; print ssl.OPENSSL_VERSION"
)
requests 2.2.1
requests-toolbelt 0.2.0
urllib3 1.8
Sadly, this is unrelated to the issue you identified, and entirely down to the crappy OpenSSL that OS X ships with by default. Version 0.9.8y has some real problems with performing SSL handshakes, and some servers don't tolerate it well. Using Python 3 on my OS X box (therefore using a newer OpenSSL) reveals that there's no problem.
You have two options:
env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install PyOpenSSL
.Ah, looks like I was following a red herring then - I don't plan on deploying anything on OSX anyhow. Looks like I'll move my testing to a linux virtualbox. Apologies for this long-winded issue!
No need to apologise, asking that question was the right thing to do: it's bizarrely specific knowledge to know that OS X has this problem. =)
Ok, this is a bummer. I created an Ubuntu 14.04 server 32bit Virtualbox image via Vagrant and this is all still happening except for the SSLv2 case, where it fails because the protocol isn't included in the OpenSSL version in Ubuntu 14.04 (by design, I believe - SSLv2 is old and outdated).
Versions:
Ubuntu 14.04 32bit (via Vagrant/Virtualbox combo)
Python 2.7.6
requests==2.2.1
requests-toolbelt==0.2.0
urllib3==1.8.2
EDIT: forgot the OpenSSL version...
python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 1.0.1f 6 Jan 2014
TLSv1:
>>> import requests
>>> from requests_toolbelt import SSLAdapter
>>> adapter = SSLAdapter('TLSv1')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
SSLError: [Errno 1] _ssl.c:510: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
SSLv2:
>>> import requests
>>> from requests_toolbelt import SSLAdapter
>>> adapter = SSLAdapter('SSLv3')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
SSLError: [Errno 1] _ssl.c:510: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
SSLv23:
>>> import requests
>>> from requests_toolbelt import SSLAdapter
>>> adapter = SSLAdapter('SSLv23')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
SSLError: [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
Perhaps this is a cipher list issue then? Or is the OpenSSL version used here still problematic?
I am absolutely willing to put in some time to help debug this if necessary... provided you guys give me some direction.
VM is downloading. I can't reproduce this on ArchLinux.
The stacktraces indicate this but I'd like to be sure: You are _not_ using PyOpenSSL but only the stdlib?
@t-8ch Thanks for taking a look at this, I'm a bit confused. OpenSSL makes my life really hard =(
@t-8ch I haven't installed PyOpenSSL if that's what you're asking?
I would have assumed (perhaps incorrectly) that pip install requests
should give me everything I need to successfully call requests.get('...')
on an HTTPS page. Which, of course, it works for the most part, just not for this site for some reason.
@jaddison It _mostly_ does. Unfortunately, Python 2.7s standard library sucks hard and doesn't support some features, such as SNI.
I wonder if this is SNI...
@jaddison There are two different codepaths behind the scenes. You shouldn't have to care about those, but it helps to know when debugging.
However I can now reproduce this on ubuntu. But only o Py2. On Py3 everything is fine.
I suspect @Lukasa is right and the server fails when the client is not using SNI.
It bothers me that an absence of SNI fails in multiple different ways depending on the server in question.
I did notice this change between OpenSSL 1.0.1f and 1.0.1g (https://www.openssl.org/news/openssl-1.0.1-notes.html):
Add TLS padding extension workaround for broken servers.
EDIT: Ahh, nevermind - the bug shouldn't vary between Py 2 and 3, I'd think.
@jaddison To test whether this is SNI, you'll need to install the SNI requirements for Python 2.
@Lukasa was right. Compare:
$ openssl s_client -connect docs.apitools.com:443
CONNECTED(00000003)
139846853338768:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:762:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 517 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
$ openssl s_client -connect docs.apitools.com:443 -servername docs.apitools.com
... happy handshake here
To elaborate: The second command enables the SNI functionality of openssl s_client
.
You can a) switch to python3 b) install extra dependencies.
The stdlib has at the moment no way to do SNI.
Thanks for the quick feedback. Seeing as there is no bug, I'll close this... again.
Hey, thank you guys !! I installed python3 on my mac and boom, it works.
Just want to chime in and say that I experienced this issue on OS X 10.9.5, Python 2.7.7 and OpenSSL 0.9.8zc.
I was able to fix my handshaking issue by:
brew install OpenSSL
cryptography
package linked against the new OpenSSL (env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography
) pip install requests[security]
Thanks, @Microserf. I'm pretty much running the same specs (10.9.5, Python 2.7.6 installed via Homebrew but compiled with system provided OpenSSL 0.9.8zg) and this was my entire process for getting requests
up and running for Django:
brew install openssl
Install requests
with a bunch of SNI stuff, compiled against our new install of OpenSSL. The [security]
option simply installs pyopenssl ndg-httpsclient pyasn1
env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install requests[security] urllib3
And we're good to go:
"""
This may or may not be needed. See:
https://urllib3.readthedocs.org/en/latest/security.html#openssl-pyopenssl
"""
# from urllib3.contrib import pyopenssl
# pyopenssl.inject_into_urllib3()
import requests
# r = requests.get(...)
Is there a definitive answer on how to get this working on ubuntu? I'm running into this issue, and it looks like the only answer here concerns how to get this working on a Mac. Upgrading our entire codebase to python 3 is not an option.
OK, I may have just answered my own question. What I did boils down to:
sudo apt-get install libffi-dev
pip install pyOpenSSL ndg-httpsclient pyasn1
@lsemel thank you, that just saved me a bunch of time
@lsemel Are your sure? I tried it on Ubuntu 15.10 and it still doesn't work with Python 2.7.10.
It works with Python 2.7 on Travis CI:
https://travis-ci.org/playing-se/swish-python
Got it to work now! I simply uninstalled pyOpenSSL:
pip uninstall pyOpenSSL
Maybe we should only pyopenssl.inject_into_urllib3() if Python version is less than 2.7.9? pyOpenSSL seems to break stuff on Ubuntu and Windows if Python version is 2.7.10.
PyOpenSSL should not be breaking anything. If it does, that's a bug that should be reported.
I will have to look into this, but is there any good reason to inject pyopenssl into urllib3 if Python version is 2.7.9 or newer?
I am thinking of something like this:
# Check if Modern SSL with SNI support
try:
from ssl import SSLContext
from ssl import HAS_SNI
except ImportError:
# Attempt to enable urllib3's SNI support, if possible
try:
from .packages.urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
except ImportError:
pass
Yeah, frequently there is. For example, on OS X most Pythons link against the system OpenSSL, which is version 0.9.8zg. PyOpenSSL, however, will link against a much newer OpenSSL (1.0.2). That makes using PyOpenSSL a substantial security improvement.
Additionally, PyOpenSSL gives us much better access to OpenSSL, allowing us to secure it more effectively.
OK, I have played around with this a little now.
It WORKS with pyopenssl BUT not if ndg-httpsclient is installed.
However, I can get it work with ndg-httpsclient if I uninstall pyasn1 giving me these warnings:
/usr/lib/python2.7/dist-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled. Import error is: No module named pyasn1.type
warnings.warn(import_error_msg)
/usr/lib/python2.7/dist-packages/ndg/httpsclient/ssl_peer_verification.py:25: UserWarning: SubjectAltName support is disabled - check pyasn1 package installation to enable
warnings.warn(SUBJ_ALT_NAME_SUPPORT_MSG)
/usr/lib/python2.7/dist-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled. Import error is: No module named pyasn1.type
warnings.warn(import_error_msg)
Same behavior on Ubuntu 15.10 and Windows 10 with Python 2.7.10 installed.
That's because without ndg-httpsclient the PyOpenSSL support isn't used.
Yes, I will have to dig into why it works if SubjectAltName is disabled. Any idea?
Almost certainly the problem is that you're using different OpenSSLs in each case.
I had the same issue on my Ubuntu 14.04 box and Python 2.7.11
It's from SNI
What worked for me was this:
I think there was an installation-time check on urllib3 or requests which kept things from working without the uninstall
@jvanasco what are you using to install those packages? I assume pip. Why are you installing urllib3 and requests separately?
Well I needed urllib3 in the virtualenv... but I installed it to try and get the requirements installed by pip and easy_install. (I used both)
I have a web indexer and a few urls broke. I wrote a quick script to try the broken ones, and kept reinstalling/delete+installing the packages in the urllib3 instructions on ssl issues until they worked.
On May 31, 2016, at 7:25 PM, Ian Cordasco [email protected] wrote:
@jvanasco what are you using to install those packages? I assume pip. Why are you installing urllib3 and requests separately?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
I'm still seeing this issue and i've tried the suggested work arounds.
I updated my python version to 2.7.11
I installed the 3 additional packages.
I tried the uninstall/install sequence @jvanasco suggested and still got the SSLError
Also using Ubuntu 14.04 unfortunately there's no OpenSSL update so i have to use the workarounds posted here and i'm having no luck.
Any extra steps you guys possibly took?
Thanks
@Lekinho I found that making a short test-script that tested the domain I was having problems with helped.
it was just:
import requests
r = requests.get(bad_url)
print r.__dict__
@Lekinho You can extract pyopenssl from requests in your code:
try:
from requests.packages.urllib3.contrib import pyopenssl
pyopenssl.extract_from_urllib3()
except ImportError:
pass
@Lekinho If you're still encountering this problem with Python 2.7.11 it's highly likely that the remote server doesn't support the TLS settings being used by requests. Is the server in question available on the public internet? If so, can you provide me with the URL?
i've tried the pyopenssl import as suggested.
Unfortunately this is not accessible publicly.
However I have the exact details of what openSSL version the server has.
Basically, we run on a redhat virtual machine, I had this openSSL when everything was working : openssl-1.0.1e-42.el6_7.4.x86_64
Then we did a redhat upgrade and there was an update for openssl : openssl-1.0.1e-48.el6_8.1.x86_64
This version always has the bad handshake issue when using openssl on ubuntu 14.04.
Do you guys have any public URLs i can try with, to see if the work arounds helped resolve the issue and its just this unique combination that I have that's the problem?
The same machine is fine when REST requests are sent through the browser(i.e. without the ubuntu openssl )
Thanks
Can you provide the output of rpm -q --changelog openssl
, please?
[admin@leke-2-2-8-11 ~]$ rpm -q --changelog openssl
It looks like @Lekinho deleted their github account? For the next person who has issues - it's possible that their upgrade of OpenSsl or Python broke some compiled c bindings. Whenever I have an upgrade like that, I trash my virtualenv or all packages and then build a new one.
@jvanasco i'm still here.
I was wondering, do you have a public URL i could test this with? I want to see if the workaround actually resolves the issue for confirmed cases ( this will mean I didnt screw something up while trying to do it)
@Lukasa
subset of changeset between working version and updated version :+1:
Mon May 02 2016 Tomáš Mráz [email protected] 1.0.1e-48.1
fix CVE-2016-2105 - possible overflow in base64 encoding
fix CVE-2016-2106 - possible overflow in EVP_EncryptUpdate()
fix CVE-2016-2107 - padding oracle in stitched AES-NI CBC-MAC
fix CVE-2016-2108 - memory corruption in ASN.1 encoder
fix CVE-2016-2109 - possible DoS when reading ASN.1 data from BIO
fix CVE-2016-0799 - memory issues in BIO_printf
Wed Feb 24 2016 Tomáš Mráz [email protected] 1.0.1e-48
fix CVE-2016-0702 - side channel attack on modular exponentiation
fix CVE-2016-0705 - double-free in DSA private key parsing
fix CVE-2016-0797 - heap corruption in BN_hex2bn and BN_dec2bn
Tue Feb 16 2016 Tomáš Mráz [email protected] 1.0.1e-47
fix CVE-2015-3197 - SSLv2 ciphersuite enforcement
disable SSLv2 in the generic TLS method
Fri Jan 15 2016 Tomáš Mráz [email protected] 1.0.1e-46
fix 1-byte memory leak in pkcs12 parse (#1229871)
document some options of the speed command (#1197095)
Thu Jan 14 2016 Tomáš Mráz [email protected] 1.0.1e-45
fix high-precision timestamps in timestamping authority
Mon Dec 21 2015 Tomáš Mráz [email protected] 1.0.1e-44
fix CVE-2015-7575 - disallow use of MD5 in TLS1.2
Fri Dec 04 2015 Tomáš Mráz [email protected] 1.0.1e-43
fix CVE-2015-3194 - certificate verify crash with missing PSS parameter
fix CVE-2015-3195 - X509_ATTRIBUTE memory leak
fix CVE-2015-3196 - race condition when handling PSK identity hint
Tue Jun 23 2015 Tomáš Mráz [email protected] 1.0.1e-42
Update :
So I found a work around for this.
Basically a colleague was reading up on the issue and saw some posts about RHEL openssl support for ECC/ECDH cipher not being 100% for whatever reason.
We tried out the request to the URL by explicitly disabling ECDH ciphers (adding the negation from openssl script itself i.e. openssl s_client -connect 10.85.103.218:8443 -cipher 'DEFAULT:!ECDH')
We were able to successfully connect.
Here's the default cipher list for the openssl on ubuntu 14.04
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:!eNULL:!MD5
So with that knowlege, I used pyopenssl to print out my default SSL ciphers and explicitly removed every ECDH cipher from the string. Did this right in the block to import urllib3 from requests package (i.e. before starting to make any actual requests) here's something similar :
https://github.com/kennethreitz/requests/issues/1308
I realize there may be security risks for this action but at least this gets us going and sheds more light on it.
Why those particular ciphers appear to be an issue for the RHEL, I have no idea.
I will try when i have more time to see what particular RHEL changes may have introduced this and read up on the purpose more.
Anyone know more about ciphers generally?
Have the same issue... ARG...
@lukas-gitl frustration will not help you solve the problem. Providing us with information about your environment (preferably some - if not all - of the information that we asked Lekinho above) will help.
@sigmavirus24 Apologies. I meant to provide more information and then got side tracked (since I had no time for this). I'm using Ubuntu 14.04, python 2.7.6 and the latest requests version on pip. This happens when I try to access as API Gateway endpoint (they might be quite restrictive).
I tried removing the virtualenv and regenerating it but unfortunately that didn't solve it.
Let me know what else you need. I switched to nodejs for the time but would be happy to help with a resolution.
@lukas-gitl It's highly likely that the server you're contacting requires ciphers you aren't offering, or TLS versions you aren't offering. This can be related to the OpenSSL you have installed. You should also try running pip install requests[security]
: you may be encountering problems with SNI.
Yeah, I already tried that too. Let me put a quick test script together here so we are on the same page.
virtualenv -p /usr/bin/python2.7 env
source env/bin/activate
pip install requests
pip install requests[security]
echo 'import requests' >> test.py
echo 'requests.get("https://API_ID.execute-api.us-west-2.amazonaws.com/ENV/ENPOINT")' >> test.py
python test.py
And what specific error are you seeing?
.../env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:318:
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
.../env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Traceback (most recent call last):
File "test.py", line 2, in <module>
requests.get("https://sbsz8eqowe.execute-api.us-west-2.amazonaws.com/dev/segment_to_s3_webhook")
File ".../env/local/lib/python2.7/site-packages/requests/api.py", line 71, in get
return request('get', url, params=params, **kwargs)
File ".../env/local/lib/python2.7/site-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File ".../env/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File ".../env/local/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File ".../env/local/lib/python2.7/site-packages/requests/adapters.py", line 477, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure```
So I'm basically required to update to a later version of python?
Ok, both of those warnings suggest that your requests is not actually using the extensions from requests[security]. It strongly suggests that whatever Python you are executing is _not_ the one you installed in your virtual environment: the requests[security] extension should remove those warnings.
@lukas-gitl please see my notes above.
Do you have access to the server? compare the default ciphers list for the server and the client.
It is highly likely 1 of them doesnt support the first set of ciphers in the other, hence the error.
You can check default ciphers with a simple script like what i used here :
import sys
import os
import ssl
print(ssl.OPENSSL_VERSION)
sys.path.insert(1, os.path.abspath(os.path.join(os.getcwd(), 'lib')))
sys.path.append('/usr/local/lib/python2.7/dist-packages')
import requests
from requests.packages.urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
print pyopenssl.DEFAULT_SSL_CIPHER_LIST
Ok, now I'm really confused. The error message are coming from the virtual environment. So how could those come from there while I execute from a different python environment?
So I tried pip install pyopenssl ndg-httpsclient pyasn1
instead of pip install requests[security]
and that worked...
Aha, I suspect your pip is too old to handle the extras.
Ah, damn. That explains a lot. Thank you very much for your help!
I encountered the same trouble here, I was to send a GET request by following code:
requests.get('https://mdskip.taobao.com/core/initItemDetail.htm?itemId=530444505608&showShopProm=false&queryMemberRight=true&isRegionLevel=false&tmallBuySupport=true&addressLevel=2&sellerPreview=false&isForbidBuyItem=false&cachedTimestamp=1466835924196&offlineShop=false&household=false&tryBeforeBuy=false&isSecKill=false&service3C=false&isApparel=true&isUseInventoryCenter=false&cartEnable=true&isAreaSell=false&callback=setMdskip×tamp=1466841669969&isg=Al9faN3XWRpIf6UEoQ88UH/1b7np0rNm&ref=https%3A%2F%2Fs.taobao.com%2Fsearch%3Fq%3D%25E6%258B%2589%25E5%25A4%258F%25E8%25B4%259D%25E5%25B0%2594%26imgfile%3D%26commend%3Dall%26ssid%3Ds5-e%26search_type%3Ditem%26sourceId%3Dtb.index%26spm%3Da21bo.50862.201856-taobao-item.1%26ie%3Dutf8%26initiative_id%3Dtbindexz_20160625')
unfortunately I was given the error info:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/requests/api.py", line 71, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 477, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')],)",)
I tried to brew install openssl, brew upgrade openssl, pip install --upgrade pip, pip install requests, pip install request[security], but they didn't work.
However when I type openssl version
I got OpenSSL 0.9.8zh 14 Jan 2016
, I don't know if it's all right.
Is there anyone who could help me with it?
@jschwinger23 Can you run pip install pyopenssl ndg-httpsclient pyasn1
as well please?
@Lukasa Thanks for your reply. I reconfirmed that I did install them:
$ pip install pyopenssl ndg-httpsclient pyasn1
Requirement already satisfied (use --upgrade to upgrade): pyopenssl in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
Requirement already satisfied (use --upgrade to upgrade): ndg-httpsclient in /Library/Python/2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages
but code still down.
Anyway, I figured out that everything goes well in Python3, and I am glad to be able to code in python3.
Thank you very much.
Followed above instructions but still running into this issue
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/requests/api.py", line 71, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 477, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')],)",)
any ideas?
``````
@rohanpai It is likely that you have either no cipher overlap, or that the remote server is unhappy with the versions you're offering, or that you're expected to provide a client cert and are not. It's hard to give more specific advice. Try this to investigate the issue.
On ubuntu 14.04LTS I needed to do this:
sudo pip install ndg-httpsclient pyasn1 --upgrade
Note that in Ubuntu it's not possible to upgrade/remove pyopenssl
as it's owned by the OS.
markstrefford's solution worked for me on mac os sierra too
@markstrefford 's solution also worked for me.
Just a heads up for anyone using OpenSSL 1.1:
You'll run into this issue as well, even when forcing TLS adapters, when the remote server offers Elliptic Curves as the first option.
The cause is: http://bugs.python.org/issue29697
Hey guys! I'm having the same issue with the following server https://34.200.105.231/SID/Service.svc?wsdl
. I've tried everything and I jump from and to the same 2 errors:
requests.exceptions.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:661)
Any ideas? @Lukasa, I see a few issues with the Certificate, but seems like it shouldn't be that bad: https://sslanalyzer.comodoca.com/?url=34.200.105.231
The certificate won't usually cause this problem: this problem is caused by the server hanging up on us, so usually it's the result of a cipher suite mismatch. In this case, that's exactly what's going on as you can see here.
This is a server that, frankly, should never be exposed to the open internet. There are no secure methods of communicating with this server: none, zero. This is why the handshake fails: Requests only accepts modern cipher suites, and there are no modern cipher suites available to this server. The best option is TLS_RSA_WITH_3DES_EDE_CBC_SHA
, an option we removed because it is vulnerable to practical attacks on large-scale data transfer.
If this server is yours, please upgrade it to a better TLS implementation or change the settings. Otherwise, my first bit of advice is to reconsider ever speaking to this server. If you must, then you can use the code here, but I strongly recommend that you put pressure on the server operator to fix this mess.
@Lukasa -- thanks for working through this with everyone! Ive read through and tried most of this
When running script on Windows it all works.
When running script on OSX receive:
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
Im not convinced it is not the server itself, but would appreciate any additional help to confirm and/or pop me out of this rabbit hole. Would be a huge win to get it to work.
env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install --force-reinstall --no-cache-dir {PACKAGE}
I am not 100% sure that installing against the openssl actually did anything, as it seemed to act the same as installing without (such as, speed and messaging all appeared the same)
As directed in another thread (above) connecting directly via openSSL appears
to be happy?
openssl s_client -connect XXX.102.7.147:443
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 198 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1493384325
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
Uh...OpenSSL is technically fine, but that OpenSSL negotiated no cipher (that is, it appears to have negotiated SSL_NULL_WITH_NULL_NULL
. Can you run ssllabs against your server and check what cipher suites it supports?
@Lukasa Its not exposed on the internet, is there some command line probe that I could fire off that could provide adequate insight for you?
You could try cipherscan.
@Lukasa got it installed ... its acting wonky (no output, watching it) ... will post back if I come up with anything that could be passed along. Thanks for the guidance!
@Lukasa thanks so much for your help - never actually got cipherscan working - but corrected our issues. It had nothing to do with any of this, and was a silly IP mismatch across our environments ... lessons learned! thank you ...
No problem at all, glad you got it sorted!
streamlink -l debug httpstream://https://www.arconaitv.us/stream.php?id=43 worst
[cli][info] streamlink is running as root! Be careful!
[cli][debug] OS: Linux-4.14.0-041400-generic-x86_64-with-Ubuntu-14.04-trusty
[cli][debug] Python: 2.7.6
[cli][debug] Streamlink: 0.13.0+27.g2ff314c
[cli][debug] Requests(2.19.1), Socks(1.6.7), Websocket(0.48.0)
[cli][info] Found matching plugin http for URL httpstream://https://www.arconaitv.us/stream.php?id=43
[plugin.http][debug] URL=https://www.arconaitv.us/stream.php?id=43; params={}
[cli][info] Available streams: live (worst, best)
[cli][info] Opening stream: live (http)
[cli][debug] Pre-buffering 8192 bytes
[cli][info] Starting player: /usr/bin/vlc
[cli][debug] Writing stream to output
[cli][info] Stream ended
[cli][info] Closing currently open stream..
tried but no luck
atlast got it working tvplayer on local pc . i installed tinyproxy in my local pc but in vps httpproxy xxxx not working .
is tinyproxy ok or i need some other proxy server to install in my local pc.
Hi @maanich, this doesn’t appear to be directly related to this issue, or to be a defect report for Requests which is what this issue tracker is reserved for. If you have questions about system configuration, those will be best addressed on a platform like StackOverflow. Thanks!
streamlink --https-proxy "http://8xxxx:8000/" --tvplayer-email [email protected] --tvplayer-password vcvdf3 --http-no-ssl-verify https://tvplayer.com/watch/itv best --player-no-close --stdout | /var/tmp/youtube/ffmpeg -y -i pipe:0 -vcodec copy -acodec copy -flags -global_header -hls_flags delete_segments -hls_time 10 -hls_list_size 6 /mnt/hls/arc.m3u8
ffmpeg version 4.0-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-libxml2 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg
libavutil 56. 14.100 / 56. 14.100
libavcodec 58. 18.100 / 58. 18.100
libavformat 58. 12.100 / 58. 12.100
libavdevice 58. 3.100 / 58. 3.100
libavfilter 7. 16.100 / 7. 16.100
libswscale 5. 1.100 / 5. 1.100
libswresample 3. 1.100 / 3. 1.100
libpostproc 55. 1.100 / 55. 1.100
[console][info] streamlink is running as root! Be careful!
[console][info] Found matching plugin tvplayer for URL https://tvplayer.com/watch/itv
error: Unable to open URL: https://live.tvplayer.com/stream.m3u8?id=204&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cCo6XC9cL2xpdmUudHZwbGF5ZXIuY29tXC9zdHJlYW0ubTN1OD9pZD0yMDQiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1MjkwNTc0OTR9LCJJcEFkZHJlc3MiOnsiQVdTOlNvdXJjZUlwIjoiNjIuMjEwLjE0Mi42NlwvMzIifX19XX0_&Signature=mHOteYcUu4QsbGDn0e~7meDUGT8VN7bVOBAHa-0Mk6ROA9XHYx3aIAZMAo3dFjOGuWk-3MszJzRFHdv~-CCsmX3D8XQa2zvzfuIWfMAT~yDshroXBN25iW6ZJ0-7lGla00jMTUpm5sW-uDy18OkiBWgGvDVas2Lz-EW~5-LTw2YWvEpqkvRB9OpcsHJj9RRQLuDVjwYKXwKvHTJmB1J~sGE3aigaL7AZyBaIAUMcpk-xYMpDuPV9BsBN9AT397lFfRPFt155u~yeBHZ4JlUN2GINUBt0-CzGuYVq3dsOkYYEZJo9cQTVhArpo7ek03VbDP5egtCM8obN63AEkA__&Key-Pair-Id=APKAJGWDVCU5SXAPJELQ (403 Client Error: Forbidden)
pipe:0: Invalid data found when processing input
advice please n what proxy server is good for streamlink if any
Most helpful comment
Sadly, this is unrelated to the issue you identified, and entirely down to the crappy OpenSSL that OS X ships with by default. Version 0.9.8y has some real problems with performing SSL handshakes, and some servers don't tolerate it well. Using Python 3 on my OS X box (therefore using a newer OpenSSL) reveals that there's no problem.
You have two options:
env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install PyOpenSSL
.