Requests: fails on older TLS stacks with OpenSSL 1.1.1pre9

Created on 5 Sep 2018  ·  12Comments  ·  Source: psf/requests

The upcoming adoption of OpenSSL 1.1 (or more specifically, 1.1.1) by Linux distributions might cause problems for requests when checking old websites.

This was first reported in the Debian bugtracker as bug #907807, originally against the linkchecker program (and forwarded upstream as https://github.com/linkchecker/linkchecker/issues/188), but it was found that the requests library directly suffers from this problem as well.

~This issue will probably block requests from being released with buster, the current "testing" release and upcoming (mid 2019?) stable release unless it is somewhat fixed.~ That assertion was incorrect: the bug is currently marked with a "normal" severity which is not blocking release.

The tested sites were:

All sites load correctly in Firefox 60.1.0 and although I haven't tested that with OpenSSL 1.1.1, I doubt it would be affected as Firefox (and Chromium) have their own TLS library (NSS). Also note that urllib3 seems to have no problem loading those sites itself:

>>> import urllib3
>>> http = urllib3.PoolManager()
>>> r = http.request('GET', 'https://get.adobe.com/')
>>>

Expected Result

All sites should load correctly, and do load correctly in Debian buster (which still has OpenSSL 1.1.0):

$ python
Python 2.7.15+ (default, Aug 31 2018, 11:56:52) 
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://get.adobe.com')
<Response [200]>

Actual Result

(unstable-amd64-sbuild)anarcat@curie:/$ python
Python 2.7.15+ (default, Aug 31 2018, 11:56:52) 
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
 >>> requests.get('https://get.adobe.com')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='get.adobe.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: WRONG_SIGNATURE_TYPE] wrong signature type (_ssl.c:726)'),))
 >>> requests.get('https://www.nada.kth.se')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='www.nada.kth.se', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:726)'),))
 >>> requests.get('https://caniuse.com')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='caniuse.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: VERSION_TOO_LOW] version too low (_ssl.c:726)'),))

Reproduction Steps

Run a Debian unstable machine to get the latest 1.1.1~~pre9 package. This can be done with Docker with:

docker run -it debian:unstable

If you do not have access to such an environment, the latest OpenSSL code can be found in their project page.

import requests

for url in ('https://get.adobe.com/', 'https://caniuse.com',
'https://www.nada.kth.se/~snilsson/publications/IP-address-lookup-using-LC-tries/'):
    requests.get(url)

System Information

$ python -m requests.help

(unstable-amd64-sbuild)anarcat@curie:/$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": ""
  }, 
  "idna": {
    "version": "2.6"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.15+"
  }, 
  "platform": {
    "release": "4.17.0-3-amd64", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "", 
    "version": null
  }, 
  "requests": {
    "version": "2.18.4"
  }, 
  "system_ssl": {
    "version": "10101009"
  }, 
  "urllib3": {
    "version": "1.22"
  }, 
  "using_pyopenssl": false
}

I am aware that we are one minor version behind upstream in Debian, and this is being worked on. But I have reviewed the requests changelog and there didn't seem to be any change relevant to this. I am therefore going under the assertion that current versions also suffer from the same problem.

Also note that this might be a regression in OpenSSL itself. I am filing this here to see what your opinion is regarding this issue and whether it is something that belongs in requests or the upstream cryptographic library(ies).

picture anarcat
👍4

Most helpful comment

The following works for me:

import requests
from requests import adapters
import ssl
from urllib3 import poolmanager


class TLSAdapter(adapters.HTTPAdapter):

    def init_poolmanager(self, connections, maxsize, block=False):
        """Create and initialize the urllib3 PoolManager."""
        ctx = ssl.create_default_context()
        ctx.set_ciphers('DEFAULT@SECLEVEL=1')
        self.poolmanager = poolmanager.PoolManager(
                num_pools=connections,
                maxsize=maxsize,
                block=block,
                ssl_version=ssl.PROTOCOL_TLS,
                ssl_context=ctx)

session = requests.session()
session.mount('https://', TLSAdapter())
session.get(TARGET)
picture Matir on 30 Mar 2019
👍19 🎉8

All 12 comments

For the adobe problem I've opened: https://github.com/openssl/openssl/issues/7126

kroeckx picture kroeckx on 5 Sep 2018

The caniuse problem is: https://github.com/Fyrd/caniuse/issues/4198

It's caused by Debian setting a minimum TLS version of 1.2. They really should add at least TLS 1.2 support.

kroeckx picture kroeckx on 5 Sep 2018

For the SNI problem, see: https://wiki.openssl.org/index.php/TLS1.3#Server_Name_Indication

You will get that problem for anything hosted by Google.

kroeckx picture kroeckx on 5 Sep 2018

So some of those issues are caused by Debian increasing the security level of openssl by default. The proper fix is to get the websites fixed. But it's possible to lower those security settings as a work around. Please don't override the defaults by default.

The SNI problem is something you really should fix. I don't know which part doesn't set it.

kroeckx picture kroeckx on 5 Sep 2018

Hi, could you provide the workarounds? I have tried to lower the security connection to avoid DH_KEY_TOO_SMALL, but I couldn't find any solution which would work for me (Debian testing, requests installed from debian package). Specifically for https://www.ceskatelevize.cz/. It's not very likely they will fix their security any time soon.

ziima picture ziima on 10 Nov 2018

See /usr/share/doc/libssl1.1/NEWS.Debian.gz

kroeckx picture kroeckx on 10 Nov 2018

I've seen that, but I don't want to lower the security for the whole system, just for specific script. I've also tried https://stackoverflow.com/q/38015537/2440346, but change of urllib3.util.ssl_.DEFAULT_CIPHERS doesn't work either.

ziima picture ziima on 11 Nov 2018

If you want to lower the security setting in an other file than
/etc/ssl/openssl.cfg you need to use DEFAULT@SECLEVEL=1

kroeckx picture kroeckx on 11 Nov 2018
👍1

I've tried number of approaches, but none work, see

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

# Attempt 1:
# requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = '@SECLEVEL=1'
# Attempt 2:
# requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'DEFAULT@SECLEVEL=1'
# Attempt 3:
# requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL'
requests.get('https://www.ceskatelevize.cz/')

# Attempt 4:
# CIPHERS = '@SECLEVEL=1'
# Attempt 5:
# CIPHERS = 'DEFAULT@SECLEVEL=1'
# Attempt 6:
CIPHERS = 'ALL'
class CustomAdapter(HTTPAdapter):
    """
    A TransportAdapter that re-enables 3DES support in Requests.
    """
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(CustomAdapter, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(CustomAdapter, self).proxy_manager_for(*args, **kwargs)

session = requests.Session()
session.mount('https://www.ceskatelevize.cz', CustomAdapter())
session.get('https://www.ceskatelevize.cz/')
ziima picture ziima on 11 Nov 2018

The following works for me:

import requests
from requests import adapters
import ssl
from urllib3 import poolmanager


class TLSAdapter(adapters.HTTPAdapter):

    def init_poolmanager(self, connections, maxsize, block=False):
        """Create and initialize the urllib3 PoolManager."""
        ctx = ssl.create_default_context()
        ctx.set_ciphers('DEFAULT@SECLEVEL=1')
        self.poolmanager = poolmanager.PoolManager(
                num_pools=connections,
                maxsize=maxsize,
                block=block,
                ssl_version=ssl.PROTOCOL_TLS,
                ssl_context=ctx)

session = requests.session()
session.mount('https://', TLSAdapter())
session.get(TARGET)
Matir picture Matir on 30 Mar 2019
👍19 🎉8

The following works for me:

import requests
from requests import adapters
import ssl
from urllib3 import poolmanager


class TLSAdapter(adapters.HTTPAdapter):

    def init_poolmanager(self, connections, maxsize, block=False):
        """Create and initialize the urllib3 PoolManager."""
        ctx = ssl.create_default_context()
        ctx.set_ciphers('DEFAULT@SECLEVEL=1')
        self.poolmanager = poolmanager.PoolManager(
                num_pools=connections,
                maxsize=maxsize,
                block=block,
                ssl_version=ssl.PROTOCOL_TLS,
                ssl_context=ctx)

session = requests.session()
session.mount('https://', TLSAdapter())
session.get(TARGET)

Thanks!
Its helpfuly for my SSLError [SSL: DH_KEY_TOO_SMALL]

codefather-labs picture codefather-labs on 6 Sep 2019

@codefather-labs thanks I had the same issue, very odd

riccardotacconi picture riccardotacconi on 10 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

brainwane picture brainwane  ·  3Comments
justlurking picture justlurking  ·  3Comments
cnicodeme picture cnicodeme  ·  3Comments
eromoe picture eromoe  ·  3Comments
remram44 picture remram44  ·  4Comments