Requests: SSL Error: bad handshake

Could you please tell us how you've installed requests and what version you've installed?

This is almost certainly an SNI problem, so depending on how the library was installed we may need to add some optional dependencies.

requests was installed as dependency for smartsheet-python-sdk via pip:

pip show requests
---
Metadata-Version: 2.0
Name: requests
Version: 2.10.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: [email protected]
Installer: pip
License: Apache 2.0
Location: /usr/lib/python3.4/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Natural Language :: English
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Programming Language :: Python :: Implementation :: CPython
  Programming Language :: Python :: Implementation :: PyPy

pip show smartsheet-python-sdk
---
Metadata-Version: 2.0
Name: smartsheet-python-sdk
Version: 1.0.1
Summary: Library that uses Python to connect to Smartsheet services (using API 2.0).
Home-page: http://smartsheet-platform.github.io/api-docs/
Author: Smartsheet
Author-email: [email protected]
Installer: pip
License: Apache-2.0
Location: /usr/lib/python3.4/site-packages
Requires: certifi, requests, six, python-dateutil, requests-toolbelt
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Natural Language :: English
  Operating System :: OS Independent
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Programming Language :: Python :: Implementation :: PyPy
  Programming Language :: Python :: Implementation :: CPython
  Topic :: Software Development :: Libraries :: Python Modules
  Topic :: Office/Business :: Financial :: Spreadsheet

Removed package in pip and installing from yum won't help...

@pensnarik Can you try running pip install -U requests[security] in your environment and then try again?

Thak you, Lukasa, for your advice, but it did not help...

[mutex@unica1 parser]$ sudo pip install -U requests[security]
[sudo] password for mutex: 
Collecting requests[security]
  Using cached requests-2.10.0-py2.py3-none-any.whl
Collecting pyOpenSSL>=0.13 (from requests[security])
  Using cached pyOpenSSL-16.0.0-py2.py3-none-any.whl
Collecting ndg-httpsclient (from requests[security])
Requirement already up-to-date: pyasn1 in /usr/lib/python2.7/site-packages (from requests[security])
Collecting cryptography>=1.3 (from pyOpenSSL>=0.13->requests[security])
  Using cached cryptography-1.3.2.tar.gz
Requirement already up-to-date: six>=1.5.2 in /usr/lib/python2.7/site-packages (from pyOpenSSL>=0.13->requests[security])
Requirement already up-to-date: idna>=2.0 in /usr/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL>=0.13->requests[security])
Requirement already up-to-date: setuptools>=11.3 in /usr/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL>=0.13->requests[security])
Requirement already up-to-date: enum34 in /usr/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL>=0.13->requests[security])
Requirement already up-to-date: ipaddress in /usr/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL>=0.13->requests[security])
Requirement already up-to-date: cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL>=0.13->requests[security])
Requirement already up-to-date: pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL>=0.13->requests[security])
Building wheels for collected packages: cryptography
  Running setup.py bdist_wheel for cryptography ... done
  Stored in directory: /root/.cache/pip/wheels/14/df/02/611097a49d7739151deb68d0172dff5ae7cba01b82769e56ef
Successfully built cryptography
Installing collected packages: cryptography, pyOpenSSL, ndg-httpsclient, requests
  Found existing installation: requests 2.6.0
    DEPRECATION: Uninstalling a distutils installed project (requests) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling requests-2.6.0:
      Successfully uninstalled requests-2.6.0
Successfully installed cryptography-1.3.2 ndg-httpsclient-0.4.0 pyOpenSSL-16.0.0 requests-2.10.0
You are using pip version 8.1.0, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[mutex@unica1 parser]$ ./update_region_price.py 
Traceback (most recent call last):
  File "./update_region_price.py", line 129, in <module>
    sys.exit(app.run(sys.argv))
  File "./update_region_price.py", line 122, in run
    self.update_basic()
  File "./update_region_price.py", line 65, in update_basic
    sheet = self.sm.Sheets.get_sheet(self.sheet_id, page_size=5000)
  File "/usr/lib/python2.7/site-packages/smartsheet/sheets.py", line 460, in get_sheet
    response = self._base.request(prepped_request, expected, _op)
  File "/usr/lib/python2.7/site-packages/smartsheet/smartsheet.py", line 178, in request
    res = self.request_with_retry(prepped_request, operation)
  File "/usr/lib/python2.7/site-packages/smartsheet/smartsheet.py", line 242, in request_with_retry
    return self._request(prepped_request, operation)
  File "/usr/lib/python2.7/site-packages/smartsheet/smartsheet.py", line 208, in _request
    res = self._session.send(prepped_request, stream=stream)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

Are you able to show me what host you're connecting to?

Lukasa, of course: https://api.smartsheet.com/2.0, I'm using python wrapper for smartsheet API (https://github.com/smartsheet-platform/smartsheet-python-sdk).

And do you happen to have certifi installed in your environment?

Yes, I have:

[mutex@unica1 parser]$ pip show certifi
---
Metadata-Version: 2.0
Name: certifi
Version: 2016.2.28
Summary: Python package for providing Mozilla's CA Bundle.
Home-page: http://certifi.io/
Author: Kenneth Reitz
Author-email: [email protected]
Installer: pip
License: ISC
Location: /usr/lib/python2.7/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Natural Language :: English
  Programming Language :: Python
  Programming Language :: Python :: 2.5
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3.0
  Programming Language :: Python :: 3.1
  Programming Language :: Python :: 3.2
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4

Aha, ok, we got there.

api.smartsheet.com serves its TLS using what's known as a "cross-signed certificate". This was used because Verisign, the CA for api.smartsheet.com, originally used a 1024-bit root certificate. These were deprecated and replaced by stronger root certificates, but some older browsers and systems may not have received updates, so sites like api.smartsheet.com serve a root certificate that is signed by the 1024-bit root.

That's not normally a problem, _except_:

• certifi removed the weak 1024-bit roots
• OpenSSL older than 1.0.2 sucks at building cert chains, and so fails to correctly validate the cross-signed root.

You can solve this in two ways. The first, better but more drastic way, is to upgrade your OpenSSL to 1.0.2 or later. This is hard to do on Centos, I'm afraid. The less good but more effective way is to get the output of running python -c "import certifi; print certifi.old_where()" and then set the REQUESTS_CA_BUNDLE environment variable to the printed path.

Argh, the smartsheet SDK overrides that environment variable by explicitly using certifi.

New plan. Right at the start after you import smartsheet, before you do _anything_ else, can you add the following lines?

import smartsheet.session
import certifi
smartsheet.session._TRUSTED_CERT_FILE = certifi.old_where()

That should hopefully help. You shouldn't need the environment variable with this approach.

Unfortunately, it does not work even if I change this line in /usr/lib/python2.7/site-packages/smartsheet/session.py:

_TRUSTED_CERT_FILE = certifi.where()

to

_TRUSTED_CERT_FILE = certifi.old_where()

When I change this var from my code, It does not helps too...

At this point, I'm pretty confident this is not a bug in requests and is in fact a problem with the smartsheet API wrapper. Have you tried reporting this to them?

Hrm. Are you confident that that code is being executed? Can you use print statements to confirm that it's being used?

Downgrading certifi from 2015.11.20.1 to 2015.4.28 fixes the problem!

ubuntu@ip-172-31-58-148:~/requests$ clear
ubuntu@ip-172-31-58-148:~/requests$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import requests
requests.get('https://logo.clearbit.com/beutifi.com')
requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Traceback (most recent call last):
File "", line 1, in
File "requests/api.py", line 71, in get
return request('get', url, params=params, *_kwargs)
File "requests/api.py", line 57, in request
return session.request(method=method, url=url, *_kwargs)
File "requests/sessions.py", line 477, in request
resp = self.send(prep, *_send_kwargs)
File "requests/sessions.py", line 587, in send
r = adapter.send(request, *_kwargs)
File "requests/adapters.py", line 491, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

This is my input to this problem. It doe snot happen with every https request.

@sProject You would need to resolve both of those warnings before we can determine if you have a new problem. If you got requests from pip, run pip install pyopenssl pyasn1 ndg-httpsclient. If you got Requests from your system provider, you should install those three packages from there.

This bug appears to have been resolved over two months ago. If I'm incorrect in this conclusion and there is still a bug to be found in requests, I'll be happy to reopen this.

Thanks everyone for collaborating on requests!

The example from pygodaddy raises the same exception

from pygodaddy import GoDaddyClient
client = GoDaddyClient()
if client.login(login[0],login[1]):
    print client.find_domains()
else: print 'falhou'


$ ./dynip.py 
Traceback (most recent call last):
  File "./dynip.py", line 8, in <module>
    if client.login(login[0],login[1]):
  File "/usr/lib/python2.7/site-packages/pygodaddy/client.py", line 99, in login
    r = self.session.get(self.default_url)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

Downgrading certifi as suggested by @pensnarik does not fix it. This is Fedora 24, Python 2.7.12.

# pip list
Babel (2.3.4)
beautifulsoup4 (4.5.1)
blinker (1.4)
certifi (2016.9.26)
cffi (1.9.1)
chardet (2.3.0)
click (6.6)
cryptography (1.5.3)
cssselect (0.9.2)
decorator (4.0.10)
django-htmlmin (0.9.1)
dnspython (1.14.0)
enum34 (1.1.6)
extras (1.0.0)
first (2.0.1)
fixtures (3.0.0)
Flask (0.11.1)
Flask-Babel (0.11.1)
Flask-Gravatar (0.4.2)
Flask-Login (0.3.2)
Flask-Mail (0.9.1)
Flask-Principal (0.4.0)
Flask-Security (1.7.5)
Flask-SQLAlchemy (2.1)
Flask-WTF (0.12)
gps (3.16)
gssapi (1.2.0)
html5lib (1.0b3)
idna (2.1)
importlib (1.0.4)
iniparse (0.4)
ipaclient (4.3.2)
ipaddress (1.0.17)
ipalib (4.3.2)
ipaplatform (4.3.2)
ipapython (4.3.2)
itsdangerous (0.24)
Jinja2 (2.8)
jwcrypto (0.3.2)
kitchen (1.2.4)
linecache2 (1.0.0)
lockfile (0.12.2)
lxml (3.6.4)
M2Crypto (0.25.1)
MarkupSafe (0.23)
munch (2.0.4)
ndg-httpsclient (0.4.2)
netaddr (0.7.18)
nose (1.3.7)
numpy (1.11.1)
passlib (1.6.5)
pbr (1.10.0)
pif (0.7.3)
Pillow (3.3.1)
pip (8.1.2)
pip-tools (1.7.0)
ply (3.9)
psutil (4.3.0)
psycopg2 (2.6.2)
pwquality (1.3.0)
pyasn1 (0.1.9)
pyasn1-modules (0.0.8)
pycparser (2.17)
pycrypto (2.6.1)
pycurl (7.43.0)
pygobject (3.20.1)
pygodaddy (0.2.2)
pygpgme (0.3)
pyliblzma (0.5.3)
pyOpenSSL (16.2.0)
pyrsistent (0.11.13)
PySocks (1.5.7)
python-dateutil (2.5.3)
python-fedora (0.8.0)
python-ldap (2.4.27)
python-mimeparse (1.5.2)
python-nss (1.0.0)
python-yubico (1.3.2)
pytz (2016.6.1)
pyusb (1.0.0)
pyxattr (0.5.5)
qrcode (5.3)
reportlab (3.3.0)
requests (2.12.1)
requests-file (1.4.1)
rpm-python (4.13.0)
scdate (1.10.9)
setuptools (28.8.0)
simplejson (3.8.2)
six (1.10.0)
slip (0.6.4)
speaklater (1.3)
SQLAlchemy (1.0.14)
sqlparse (0.2.1)
SSSDConfig (1.14.2)
Terminator (0.98)
testscenarios (0.5.0)
testtools (2.2.0)
tldextract (2.0.2)
traceback2 (1.4.0)
typing (3.5.2.2)
uniconvertor (2.0)
unittest2 (1.1.0)
urlgrabber (3.10.1)
urllib3 (1.16)
Werkzeug (0.11.11)
WTForms (2.1)
yum-metadata-parser (1.1.4)

GoDaddy is serving an incomplete certificate chain to you. That means we're missing one of the intermediate certificates and can't build up a trust chain. Either you'll need to add the missing intermediary certificate to the certifi trust store or you'll need to contact GoDaddy and tell them to sort their mess out.

@Lukasa how can one check if the problem is with requests or certificate issuer?
I have one cert issued by Entrust and my browsers are quite ok with it when I browse to the URL.
But when I try to get to that URL via requests I have [SSL: CERTIFICATE_VERIFY_FAILED]

Full traceback

$ python main.py
Traceback (most recent call last):
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 835, in _validate_conn
    conn.connect()
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
    _context=self)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
    self.do_handshake()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 624, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 64, in <module>
    r = s.get('https://infoproducts.alcatel-lucent.com/cgi-bin/get_doc_list.pl?entry_id=1-0000000000662&srch_how=&srch_str=&release=&model=&category=&contype=&format=&sortby=&how=all_prod',
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/Users/romandodin/venvs/nokdok/lib/python3.5/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

installed packages

$ pip list
certifi (2016.9.26)
pip (9.0.1)
requests (2.12.4)
setuptools (32.3.1)
wheel (0.29.0)

Cert details

Subject infoproducts.alcatel-lucent.com
SAN infoproducts.alcatel-lucent.com
documentation.alcatel-lucent.com
Valid From Thu, 01 Dec 2016 13:15:08 GMT
Valid Until Sat, 01 Dec 2018 13:45:07 GMT
Issuer Entrust Certification Authority - L1K

The problem is almost certainly not with the issuer, but with your server. Can you show me the output of running openssl s_client -connect "<your-host>:<your-port>" -showcerts -servername "<your-host>" against your server please?

I am not sure that I used the command right, but that's someting:

$ openssl s_client -connect "infoproducts.alcatel-lucent.com:443" -showcerts -servername "infoproducts.alcatel-lucent.com"
CONNECTED(00000003)
depth=0 /C=US/ST=Illinois/L=Naperville/O=Alcatel-Lucent USA Inc./CN=infoproducts.alcatel-lucent.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=US/ST=Illinois/L=Naperville/O=Alcatel-Lucent USA Inc./CN=infoproducts.alcatel-lucent.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=US/ST=Illinois/L=Naperville/O=Alcatel-Lucent USA Inc./CN=infoproducts.alcatel-lucent.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=Illinois/L=Naperville/O=Alcatel-Lucent USA Inc./CN=infoproducts.alcatel-lucent.com
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
-----BEGIN CERTIFICATE-----
MIIFeDCCBGCgAwIBAgIRAO3KTvH+nKgrAAAAAFDanDYwDQYJKoZIhvcNAQELBQAw
gboxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQL
Ex9TZWUgd3d3LmVudHJ1c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykg
MjAxMiBFbnRydXN0LCBJbmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxLjAs
BgNVBAMTJUVudHJ1c3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBMMUswHhcN
MTYxMjAxMTMxNTA4WhcNMTgxMjAxMTM0NTA3WjCBgTELMAkGA1UEBhMCVVMxETAP
BgNVBAgTCElsbGlub2lzMRMwEQYDVQQHEwpOYXBlcnZpbGxlMSAwHgYDVQQKExdB
bGNhdGVsLUx1Y2VudCBVU0EgSW5jLjEoMCYGA1UEAxMfaW5mb3Byb2R1Y3RzLmFs
Y2F0ZWwtbHVjZW50LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AK23xiJ/GCWNJa8Oa6SAYwT7QTmIJOXGrfLLZM9ZWDK81SVLm5xUAVRYaYmb3t/U
KpzeUoL+cJIa2xJGdbj50ehUTbB3SOXW7dxr15fuWahSChqaNkI/NClNAxy2Vho5
HxEtsjmuoJ0cNRcZLHZndLtWi27js3ivGxFxUcl7O3rGGj9yb+XXwJvEOsITPfZ/
gpURnHfAurZrw1+xpsArQVOF6+K6KkPnjGoCj0XCyU3LnWc6akcwwwV+HXcW8H5G
/CPvcm3DpI+45v/P2vVJ9+LiEJVHngVYK2QQ8fMDrvS2vs563I4PLnnRds0eOJph
LfHmn87oSW5jCP3RnsW0czECAwEAAaOCAa4wggGqMA4GA1UdDwEB/wQEAwIFoDAT
BgNVHSUEDDAKBggrBgEFBQcDATAzBgNVHR8ELDAqMCigJqAkhiJodHRwOi8vY3Js
LmVudHJ1c3QubmV0L2xldmVsMWsuY3JsMEsGA1UdIAREMEIwNgYKYIZIAYb6bAoB
BTAoMCYGCCsGAQUFBwIBFhpodHRwOi8vd3d3LmVudHJ1c3QubmV0L3JwYTAIBgZn
gQwBAgIwaAYIKwYBBQUHAQEEXDBaMCMGCCsGAQUFBzABhhdodHRwOi8vb2NzcC5l
bnRydXN0Lm5ldDAzBggrBgEFBQcwAoYnaHR0cDovL2FpYS5lbnRydXN0Lm5ldC9s
MWstY2hhaW4yNTYuY2VyMEwGA1UdEQRFMEOCH2luZm9wcm9kdWN0cy5hbGNhdGVs
LWx1Y2VudC5jb22CIGRvY3VtZW50YXRpb24uYWxjYXRlbC1sdWNlbnQuY29tMB8G
A1UdIwQYMBaAFIKicHTdvFM/z3vU981/p2DGCky/MB0GA1UdDgQWBBT9w+iH+fpG
UJ0IMw1vVpn+91hk6jAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IBAQAgOcDq
0YsihBWAxA4WFIGkdB/JdOKLRQ0Hvc7KStgtdkB1AqkUv7oBnDCHesazw4vyWbjv
w7dBTtB95jq1ZHX6eHyUDUo2xBmjnaeb2lSAdVV6j10sl7lMfRQh7ba1Im0ibhfe
7gvBn0tUlmdGqqvDqokzV4lVX74Z9nKKKr5D9e3vJsb5AvbDC/eYguBK9Oy8EDa2
ZcuPve3mB68lVy5UDg21RVZE072qC0FlYhNasZlMVUUg7tgDMlynQeeoxHe7Rcic
pHANQxJqtN8/bsE2mO/ryRZALyC7mWeDvG522ZXMaKslwTUr+jokpyUF7tS786Pi
n4zJ/KNZK2suVwcK
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=Illinois/L=Naperville/O=Alcatel-Lucent USA Inc./CN=infoproducts.alcatel-lucent.com
issuer=/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
---
No client certificate CA names sent
---
SSL handshake has read 1556 bytes and written 466 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 48D3C9611E37C997182C4F123E21BCFD909D8D4BE887B44355866B80D44E1163
    Session-ID-ctx:
    Master-Key: 0A4A9E2A10E33EAA248541C39AC71F2F879310BAE141A4EA26F425D56C27099FB6D6572C2CCFC763FC743ECF99DABB5B
    Key-Arg   : None
    Start Time: 1483448892
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
closed

Yup, so the server is wrong.

This is a really common problem with TLS. A TLS certificate validation is performed by first building a certificate chain that takes us from a certificate the server can prove it has the private key for (the "leaf") to a certificate we trust (the "root"). For OpenSSL there is an additional rule: the root must be self-signed.

In most cases, this chain is more than two steps long: that is, the leaf is not signed by the root, but is instead signed by a so-called intermediate certificate. That intermediate may itself be signed by the root, but may also be signed by other intermediates, until we eventually reach an intermediate that was signed by a root.

For example, for https://python-hyper.org, the chain is as follows:

1. python-hyper.org, the "leaf", which I applied for and had issued
2. Let's Encrypt Authority X3, an intermediate certificate used by the Let's Encrypt project
3. DST Root CA X3, the root certificate that my web browser (and Requests) trusts

The correct chain for the site you're accessing should be:

1. infoproducts.alcatel-lucent.com, the leaf
2. Entrust Certification Authority - L1K, the intermediate certificate which issued the leaf
3. Entrust Root Certification Authority - G2, the root certificate that issued the intermediate

For your use case, Requests ships with certificate number 3 in its trust database as a trusted root. However, your server is only sending certificate number 1. Requests cannot go from certificate 1 to certificate 3 without having certificate 2, and the server isn't sending it. Most browsers ship with commonly-used intermediate certs, or can maintain caches of them, but Requests cannot do that, so it has no way of getting hold of certificate 2. Without certificate 2, validation must fail.

Properly-configured TLS servers send the leaf certificate and all of the necessary intermediaries. This is to ensure that clients that have never seen the intermediaries and that cannot dynamically fetch them are still able to validate the chain. As a comparison, check out the output of a command like yours run against python-hyper.org:

% openssl s_client -connect python-hyper.org:443 -showcerts -servername "python-hyper.org"
CONNECTED(00000003)
depth=1 /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=python-hyper.org
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFBDCCA+ygAwIBAgISAwji03rFoFMPZnEC0rLc7jg3MA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzAxMDExMjA4MDBaFw0x
NzA0MDExMjA4MDBaMBsxGTAXBgNVBAMTEHB5dGhvbi1oeXBlci5vcmcwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCjF6VMEOe2qHsdbAnYstunDCW5/fBx
yhzNAxSqZKA5qfATdvhDmiPFnHJkQgUkUeGDwbBwemxuUFUaGZKTJDRhlrymLSkN
hkcouBzs/mxDjNlKacokJBm3hpu+oxYohhxPIZBs8NM4olUPDSG68r6sd1EwR+Ia
Lw//nRsMpcrGNmYy+howiBvV3CbuYsbgB59bJ+5y6G2ZqeHMwzFZ+No1oQmBck9T
KAvCh5TteQphzcM9s9NZiB6Z9C0s+vBPKOc1uLssCI3hfr29Af203CX2xgyBjXH7
M4WS17o1zLgs3Q2V03gQ7AmhtgjuJ+2NRf/d6Bsmk8ZhGnyyf+qbY+G/AgMBAAGj
ggIRMIICDTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFMD50AC8wul1eNKkGVxaAvXb
DSkDMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMHAGCCsGAQUFBwEB
BGQwYjAvBggrBgEFBQcwAYYjaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZy8wLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlw
dC5vcmcvMBsGA1UdEQQUMBKCEHB5dGhvbi1oeXBlci5vcmcwgf4GA1UdIASB9jCB
8zAIBgZngQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRw
Oi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENl
cnRpZmljYXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFy
dGllcyBhbmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRl
IFBvbGljeSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0
b3J5LzANBgkqhkiG9w0BAQsFAAOCAQEAgX00tBHyz9GiDQjw+Id7sbT1lrtHrmtR
DB+kofnq9pkwIExDXT0bAZ14EnU6atiVqhF3j3KxvxIfbNvmSr7emmhPwt+KqOf7
/1m+gxg3ode9LIg6oLtVOfulecxkS4/Wj990O40vuRNdy4XT4PSNze8iuJtGALoS
U9kP8G/V6VnrdbTYhSIIUW9nm0XQcOUbvupFWtiwE8vZw4t0pQloeECdMuALgVO/
1xSu0kqZgidLOkFwei/xqItx7foREzVvq3kUHD1OAuPI1azHQjErQC3N12OmxmBU
3KDOsaJC2Uu8fqI/y1YOkO97hpsgFZX4BiQNaNhtyy7sscD/teVhWQ==
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=python-hyper.org
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
---
SSL handshake has read 2638 bytes and written 451 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: BBDF2CA65B2142C1F0C416B28C4FDF8FD5118ED4D5FF8802789D0B54C6309469
    Session-ID-ctx: 
    Master-Key: BD9B8B97EC439248CC5849EBF4B2ED18E0A73472EB8D86FE6F038C4E6944AFD8A163D80D47A3A3B28A6237A44C1B31CE
    Key-Arg   : None
    Start Time: 1483447980
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Note that the server sent two certificates in my case, whereas in yours it only sent one.

This is the error. Please reconfigure your server to send all the relevant intermediaries as well as the leaf.

While I'm here, I should note that running Qualys SSL Labs against your server would also have told you about this problem. =)

Thanks @Lukasa for this educational reply you gave! Really appreciate and sorry for troubling with this, it was my TLS-knowledge-gap =) Mostly I went to issues because in was all good in browsers and didn't work in requests. But now its clear to me what is the root cause.

I believe there is no workaround (download intermediate cert upfront) for this case, but to reconfigure the server? For the time being I will skip verification checks

@hellt No need to apologise: this is a thing that is extremely non-obvious and it's a really common error to make, so you're in good company here. =)

There is a workaround, in fact. You can pass a certificate bundle to verify to override the use of certifi's bundle. This bundle can, in addition to root certs, contain intermediary certificates that become available to OpenSSL to use to build the chain. As a result, what you can do is take the certifi cert chain (which is in a file you can find by running python -c 'import certifi; print(certifi.where())', copy that out to somewhere else on your filesystem, and then add the intermediate certificate to the end of that file. The intermediate has to be in PEM format. If then pass the path to the new cert bundle to the verify kwarg, you'll find everything starts working.

Luka,
What if I get an this type of error;
pip install certifi
Collecting certifi
Could not fetch URL https://pypi.python.org/simple/certifi/: There was a problem confirming the ssl certificate: [Errno 2] No such file or directory - skipping
Could not find a version that satisfies the requirement certifi (from versions: )
No matching distribution found for certifi

Q

Seems like you are having trouble validating the TLS certificates from PyPI. However, that error message is a bit surprising. I don't entirely know where it is coming from, but it suggests a problem either with the way your pip is configured or with your Python. Do you encounter similar errors with other python packages?

Hello All,
I am trying to do the kerberos authentication using Python within the intranet of my organisation but getting this error.

import requests
from requests_kerberos import HTTPKerberosAuth
r = requests.get("https:*****", auth=HTTPKerberosAuth())

SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

Does any one know how to add certificates for Python ? Will that help to get this resolved?

@ajitmuley5 while you're using Requests, questions such as yours belong on StackOverflow. Please post this there with the required amount of detail. There are people who will happily help you without starting discussions on old and closed issues.

@Lukasa Thanks ! sudo pip install -U requests[security] worked for me.

Thanks all, I got some ideas!