Requests: Specify password for SSL client side certificate

Created on 3 Sep 2013  ·  121Comments  ·  Source: psf/requests

As far as I know currently it's not possible to specify the password for the client side certificate you're using for authentication.
This is a bit of a problem because you typically always want to password protect your .pem file which contains the private key. openssl won't even let you create one without a password.

Documentation Planned

Most helpful comment

@botondus I think I found a simpler way to achieve this with request library. I am documenting this for other people who are facing the issue.

I assume that you have a .p12 certificate and a passphrase for the key.

Generate certificate and private key.

// Generate the certificate file.
openssl pkcs12 -in /path/to/p12cert -nokeys -out certificate.pem
// Generate private key with passpharse, First enter the password provided with the key and then an arbitrary PEM password //(say: 1234) 
openssl pkcs12 -in /path/to/p12cert -nocerts -out privkey.pem

Well, we are not done yet and we need to generate the key that doesn't require the PEM password every time it needs to talk to the server.

Generate key without passphrase.

// Running this command will prompt for the pem password(1234), on providing which we will obtain the plainkey.pem
openssl rsa -in privkey.pem -out plainkey.pem

Now, you will have certificate.pem and plainkey.pem, both of the files required to talk to the API using requests.

Here is an example request using these cert and keys.

import requests
url = 'https://exampleurl.com'
headers = {
            'header1': '1214141414',
            'header2': 'adad-1223-122'
          }
response = requests.get(url, headers=headers, cert=('~/certificate.pem', '~/plainkey.pem'), verify=True)
print response.json()

Hope this helps:

cc @kennethreitz @Lukasa @sigmavirus24

All 121 comments

Something like:

requests.get('https://kennethreitz.com', cert='server.pem', cert_pw='my_password')

Pretty sure you're supposed to use the cert param for that: cert=('server.pem', 'my_password')

@sigmavirus24
The tuple is for (certificate, key). Currently there is no support for encrypted keyfiles.
The stdlib only got support for those in version 3.3.

Heh, @t-8ch, you accidentally linked to a file on your local FS. ;) Correct link.

Quite right @t-8ch. This is why I should never answer issues from the bus. :/

So the current consensus is we don't support this. How much work is it likely to be to add support in non-3.3 versions of Python?

How hard would it be to throw an error on this condition? I just ran into this silly problem and it took two hours to figure out, it would be nice if it would throw an error, it currently just sits there looping. Thanks for the awesome library!

Wait, it sits where looping? Where in execution do we fail? Can you print the traceback from where we loop?

It seems to hang right here:

r = requests.get(url,
auth=headeroauth,
cert=self.cert_tuple,
headers=headers,
timeout=10,
verify=True)

I tried turning the timeout out up or down to no avail, but I imagine it knows well before the timeout it can't use the cert. Thanks!

Ah, sorry, I wasn't clear. I meant to let it hang and then kill it with Ctrl + C so that python throws a KeyboardInterrupt exception, then to see where we are in the traceback. I want to know where in Requests the execution halts.

What's happening (or at least what I've seen in many cases) is that OpenSSL, upon being given a password-protected certificate, will prompt the user for a password. It shows up in no logs (because the prompt is directly printed), and it doesn't time out because it's waiting for a user to press enter.

Needless to say, it's cubmersome, dangerous behavior when the code is running on a server (because it'll hang your worker with no option for recovery other than killing the process).

Is there a way to make requests raise an exception in that case instead of prompting for a password, or is that completely out of your control and in OpenSSL's hands?

@maxnoel I'm pretty sure this is in OpenSSL's hands but if you can answer @Lukasa's question (the last comment on this issue) it would be very helpful in giving a definite answer regarding if there was anything we can do to help.

You can confirm OpenSSL is blocking on stdin for the passphrase from the interactive python prompt:

>>> r = requests.get("https://foo.example.com/api/user/bill", cert=("client.crt", "client.key"))
Enter PEM pass phrase:
>>>

If you're running from a backgrounded process, I assume OpenSSL will block waiting on that input.

That's correct. Is there anything requests can do to prevent that from happening? Raising an exception when no password is given would be far more useful than prompting for stuff on stdin (especially in a non-interactive program).

I'm afraid that I don't know of any way. @reaperhulk?

There are ways to stop OpenSSL from doing this, but I'm not sure if they're exposed by pyOpenSSL. Where does requests call pyopenssl to load the client cert? I can dig a bit.

@reaperhulk It's done from in urllib3, here.

We also do something very similar for the stdlib, which will be a whole separate problem.

So we can do this with PyOpenSSL using a patch like this. In the stdlib version, we need to use load_cert_chain with a password.

Has this problem been solved? I'm currently running into this while trying to connect to an Apache server.

It has not.

What about PKCS#12 formatted (and encrypted) containers which could contain a client cert/key? Would this fall under the same feature request?

@mikelupo Yup.

@telam @mikelupo
I have the same problem and Googled a lot, finally, I solved it by using pycurl.
In my situation, I use openssl to convert my .pfx file to .pem file which contains both cert & key(encrypted with pass phrase), then invoke the following code.

import pycurl
import StringIO

b = StringIO.StringIO()
c = pycurl.Curl()
url = "https://example.com"
c.setopt(pycurl.URL, url)
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.setopt(pycurl.CAINFO, "/path/cacert.pem")
c.setopt(pycurl.SSLKEY, "/path/key_file.pem")
c.setopt(pycurl.SSLCERT, "/path/cert_file.pem")
c.setopt(pycurl.SSLKEYPASSWD, "your pass phrase")
c.perform()
c.close()
response_body = b.getvalue()

BTW, for security, it's better to not do hardcode for pass phrase

Of course. That said, the problem isn't really that a pass phrase is required -- it's that OpenSSL makes your program hang while waiting for someone to type a passphrase in stdin, even in the case of a non-interactive, GUI or remote program.

When a passphrase is required and none is provided, an exception should be raised instead.

if you use a default passphrase of '' for the key, openssl won't hang.
it'll return a bad password text. you can immediately alter your py flow
to then notify the user without that apparant stall

any plan to add this feature

We want to add it, but we have no schedule to add it at this time.

@botondus I think I found a simpler way to achieve this with request library. I am documenting this for other people who are facing the issue.

I assume that you have a .p12 certificate and a passphrase for the key.

Generate certificate and private key.

// Generate the certificate file.
openssl pkcs12 -in /path/to/p12cert -nokeys -out certificate.pem
// Generate private key with passpharse, First enter the password provided with the key and then an arbitrary PEM password //(say: 1234) 
openssl pkcs12 -in /path/to/p12cert -nocerts -out privkey.pem

Well, we are not done yet and we need to generate the key that doesn't require the PEM password every time it needs to talk to the server.

Generate key without passphrase.

// Running this command will prompt for the pem password(1234), on providing which we will obtain the plainkey.pem
openssl rsa -in privkey.pem -out plainkey.pem

Now, you will have certificate.pem and plainkey.pem, both of the files required to talk to the API using requests.

Here is an example request using these cert and keys.

import requests
url = 'https://exampleurl.com'
headers = {
            'header1': '1214141414',
            'header2': 'adad-1223-122'
          }
response = requests.get(url, headers=headers, cert=('~/certificate.pem', '~/plainkey.pem'), verify=True)
print response.json()

Hope this helps:

cc @kennethreitz @Lukasa @sigmavirus24

I have heard through the grapevine that Amazon does exactly this, internally.

I'm facing this issue too. My concern is that I don't want to store the plain private key to the file system (May have risk of stolen by others). So in my opinion, the more extensible way to implement this is to support using something like PEM encoded string of private key instead of the file path to specify the private key. Just left the encryption / decryption of the private key / certificate to the developers in their favor way.
After reading the source code of requests, it seems that it's not quite easy to implement since requests depends on the ssl lib of python which only support certificate / private key file. I'm just wondering if we could use pyopenssl instead of python stdlib? pyopenssl has a wrapper of openssl connection, see: https://pyopenssl.readthedocs.io/en/latest/api/ssl.html#connection-objects . Thus we can use 'pkey' object as private key instead of file path.

Requests already supports PyOpenSSL, as long as it and a few other required dependencies are installed. However, that will never become mandatory: it's important to us that we work well with the standard library.

In a future release we will have support for passing SSLContext objects to urllib3 in order to handle TLS: that will enable this function.

For those facing this problem, until requests adds the ability to pass a ssl.SSLContext/OpenSSL.SSL.Context to urllib3, here's a workaround that actually supports using encrypted certificate/keyfile (requires that PyOpenSSL is installed and used instead of the standard library ssl, which it should be if it's installed)

import requests

 # Get the password from the user/configfile/whatever
password = ...

# Subclass OpenSSL.SSL.Context to use a password callback that gives your password
class PasswordContext(requests.packages.urllib3.contrib.pyopenssl.OpenSSL.SSL.Context):
    def __init__(self, method):
        super(PasswordContext, self).__init__(method)
        def passwd_cb(maxlen, prompt_twice, userdata):
            return password if len(password) < maxlen else ''
        self.set_passwd_cb(passwd_cb)

# Monkey-patch the subclass into OpenSSL.SSL so it is used in place of the stock version
requests.packages.urllib3.contrib.pyopenssl.OpenSSL.SSL.Context = PasswordContext

# Use requests as normal, e.g.
endpoint = 'https://example.com/authenticated'
ca_certs = '/path/to/ca/certs/bundle'
certfile = '/path/to/certificate'
keyfile = '/path/to/encrypted/keyfile'
requests.get(endpoint, verify=ca_certs, cert=(certfile, keyfile))

@ahnolds: Does this also work for for PKCS#12 files, or is this PEM only?

@Lukasa: Is the PKCS#12 case really supposed to be handled here, or should I open a separate issue for that?

PKCS#12 is a trickier issue, but basically you'll need to do whatever is required to customise your SSLContext.

@Lukasa: I was thinking more of providing a good high-level API in requests. For example, simply providing the client_cert.p12 filename and the password through the cert=... keyword parameter.

@vog What code do you believe would be required to make that work?

@Lukasa I'm not sure about the internals of requests, so maybe I underestimate what is already there, but I think one of the following things needs to be done:

  • Either we have a way to provide a PKCS#12 filename directly to the lower layers (urllib3, etc.). And maybe the password, too. (Because I know nobody who wants a URL library to ask the admin interactively to enter their PKCS#12 password on a tool that runs server-side.)
  • If that is impossible, we would need to convert PKCS#12 (+password) to PEM, then providing these to the lower levels. This is done with a few calls directly to the OpenSSL binding. However, the result is the PEM certificate as a string, and I haven't yet found a way to provide the (unencrypted) PEM as string to the lower layers (except maybe by using OpenSSL / python "ssl" "buffer" wrapper, e.g. wrap_bio, but this is only available in latest Python 3 versions, not Python 2).
  • So if that is impossible, too, we would not only need to convert PKCS#12 to PEM, but also have to create a temporary file containing the (unencrypted) PEM data.

Note that the last point is what I'm essentially doing at the moment, but I don't like this at all. Why can't I provide a simple string to OpenSSL containing the certificate? Moreover, why can't I simply pass the PKCS#12 filename and password to the lower layers?

I'm going to tag in @reaperhulk as an OpenSSL expert, but my understanding is that there are no APIs for OpenSSL to load PKCS#12 format certs for client certs. This means we'd need to absolutely convert to PEM. Doing it in memory is certainly possible, but at a certain point I wonder if we don't just want to consider this expert enough that we will delegate to whatever SSLContext you pass us.

@Lukasa Thanks for taking this issue seriously. Sorry if this sounds too technical, but essentially it is just this:

You want to access a service through a client certificate. Almost everywhere you get this as a file and a password (where the file is PKCS#12 encoded). In most APIs, such as the Java standard library, you simply give it the filename and password, and are done with it.

However, in Python this is complicated as hell.

That's why almost nobody does it. Instead, they convert their file and password to a PEM file by hand, through OpenSSL, and use that file. This is administrative overhead for every user of such an application. Because they can't simply name the (PKCS#12) file and password.

I think the requests library should make it at least as simple as in Java.

requests already does a great job of simplifying stupid complex APIs, and the PKCS#12 use case is just another example of a stupid complex API.

the PKCS#12 use case is just another example of a stupid complex API.

Yeah, I don't disagree with that at all: I'd be totally happy to have some sort of solution for PKCS#12 support somewhere in the stack.

What I'm trying to get a feel for is what code is required to make that work, and as a result where this should be placed. My reasoning is like this:

  1. Generally speaking, Requests only adds to its API surface if there is substantial utility in doing so (that is, if it's used by lots of people or used very heavily by some), and if the thing we're doing is difficult to get right or has subtle edge cases.
  2. Normally supporting PKCS#12 would count as an addition to the API surface, but if it doesn't change the syntax of cert= at all (just widens the things it will support) and doesn't regress the behaviour (that is, we can reliably tell the difference between PKCS#12 files and PEM files, or we can easily just process through both chains of logic), I'd say it counts as a sufficiently minor change to the surface that it's probably worth it.
  3. However, there are other places this may go. For example, at the Transport Adapter level, or as a helper in the Requests Toolbelt, or something else.

That means I want to weigh up how subtle this is, how complex the code is, whether it requires extra dependencies, and then use that information to work out where best to put the code. For example, I have a _suspicion_ right now that the standard library cannot handle PKCS#12, which would mean that _at best_ Requests would only be able to use PKCS#12 with the [security] extra installed. In an even worse case we may not have the functions available in any OpenSSL binding at all, in which case we'll have to do some real bonkers stuff to get it to work. That's why I wanted @reaperhulk to weigh in: he'll likely be able to clarify this for us faster than I could do the research.

I would like to see this support added: I just need to get some people who know what the scope of the work is to comment here and let me know how big the mountain we need to move actually is.

One more detail for a PKCS#12 implementation: Older versions of the Python OpenSSL bindings fail if the password is given as unicode object instead of byte string. So it should be converted before passing it to load_pkcs12() like this:

if isinstance(password, unicode):
    password_bytes = password.encode('utf8')
else:
    password_bytes = password
pkcs12 = OpenSSL.crypto.load_pkcs12(pkcs12_data, password_bytes)

A full converter might look like this, where pkcs12_data is expected to be a byte string with binary data, while password may be a byte string or a unicode string:

def pkcs12_to_pem(pkcs12_data, password):
    # Old versions of OpenSSL.crypto.load_pkcs12() fail if the password is a unicode object
    if isinstance(password, unicode):
        password_bytes = password.encode('utf8')
    else:
        password_bytes = password
    p12 = OpenSSL.crypto.load_pkcs12(pkcs12_data, password_bytes)
    p12_cert = p12.get_certificate()
    p12_key = p12.get_privatekey()
    pem_cert = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12_cert)
    pem_key = OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12_key)
    pem = pem_cert + pem_key
    return pem

The PKCS#12 discussion seems to me to be beyond the scope of the initial issue, since the issue in question is whether requests should support PKCS#12 natively. I'd vote it gets its own issue, but obviously that's up to the folks in charge.

That said, as a workaround that doesn't require unencrypted temp files, the OpenSSL.crypto.dump_privatekey method has an optional passphrase parameter, so you could get a copy of the encrypted private key in PEM format that way. That would reduce this to the encrypted PEM problem we started with.

Alternatively, you could also probably write a hack similar to the one I suggested before using the use_privatekey method of OpenSSL.SSL.Context. Off the top of my head (untested) something like

# From somewhere
pkcs12_data = ...
password_bytes = ...

class Pkcs12Context(requests.packages.urllib3.contrib.pyopenssl.OpenSSL.SSL.Context):
    def __init__(self, method):
        super(PasswordContext, self).__init__(method)
        p12 = OpenSSL.crypto.load_pkcs12(pkcs12_data, password_bytes)
        self.use_certificate(p12.get_certificate())
        self.use_privatekey(p12.get_privatekey())
# Monkey-patch the subclass into OpenSSL.SSL so it is used in place of the stock version
requests.packages.urllib3.contrib.pyopenssl.OpenSSL.SSL.Context = Pkcs12Context

Then just use requests.get etc without specifying a cert at all, since it's already handled in the constructor.

Reviewing this thread now. Rephrase of the original:

given an encrypted PEM formatted client cert, can requests handle providing a password?

Since this is in current standard libs, it'd be great to integrate this option. This would be extremely valuable for enterprise security considerations (where our certs are encrypted, and are meant to stay so).

At this point, this can be done by passing a custom ssl context to urllib3 using a transport adapter. This can do whatever the standard library ssl context allows. You can see an example of passing a custom context here.

I was able to use .pfx and .p12 with requests by converting it to a .pem using a temporary file. See https://gist.github.com/erikbern/756b1d8df2d1487497d29b90e81f8068

If there's any interest, I can submit a PR. It would be great to avoid the temporary file and the context manager. Let me know.

This is not likely to be merged, I'm afraid, but note that you can now pass a PyOpenSSL context directly to Requests via transport adapters, so you may find you can circumvent that issue.

This is not likely to be merged, I'm afraid, but note that you can now pass a PyOpenSSL context directly to Requests via transport adapters, so you may find you can circumvent that issue.

sorry for being confused but are you saying pfx/p12 support in general is likely not to be merged? (assuming done the right way, through the context etc). Happy to give it a shot but obviously not worth my time if it's not going to be merged.

I believe the "not likely to be merged" was about the temporary file solution.

@erikbern To be clear, I am happy to approach and merge any solution that works somewhat consistently. For example, a solution to using PKCS#12 through the PyOpenSSL contrib module in urllib3 would be acceptable.

However, a temporary file solution is not acceptable (as noted by @vog). This means support for PKCS#12 is unlikely to work with the standard library, as the standard library ssl module does not expose any support for it, and so it will not be supported in all Requests configurations.

Sounds good. I also agree that the temp file bad since there's a security risk storing decrypted keys on disk. Might take a look at it next week. Thanks for the heads up about the ssl module – if the limitation is outside requests then obviously it gets trickier

I looked into it and the ssl module added a cadata argument where you can pass pem data as a raw string: https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations

We would have to patch urllib3 in a bunch of places to make this work, so I might start there

@erikbern To be clear, almost any solution like that will work better just by passing an appropriately-configured SSLContext object to urllib3 using a TransportAdapter.

https://github.com/kennethreitz/requests/issues/2519 seems to be identical to this issue, so they probably should be merged

Any Update on this issue I am trying to use a Client certificate which is password encrypted and an unable to get it to work. Should I look for other options other than requests? Could you please respond ASAP.

Do we document this? I feel like this is our most requested feature.

I believe this thread started in 2013 and I did not find any resolution explained till the end. Did you guys provide an option for providing the password? or is this still an in-progress?

I am trying to use requests in a App security product that I am building. So any pointers will be helpfull

@AnoopPillai Did you check this comment? https://github.com/requests/requests/issues/1573#issuecomment-188125157

Yes I did read this comment this is a work around but In my case I do not want to convert it into 2 cert files as that will have to done outside of my application. More over we use a vault like thing for storing the password for the encrypted .pem file.

This password is retrieved dynamically by the app at run time so no hard coding.

@AnoopPillai Alright.

@kennethreitz No, we didn't document it.

@AnoopPillai Yeah, this works fine. You just need to use some lower-level hooks. In this case, we allow you to pass an SSLContext to urllib3 directly at the Transport Adapter level. This allows you access to the underlying functions which will allow you to provide a passphrase or passphrase function. This is how we recommend you support this.

@AnoopPillai workaround using a temp file that i've found useful: https://gist.github.com/erikbern/756b1d8df2d1487497d29b90e81f8068

Thanks Lukasa for letting me know that there is a way to do it.
I am very new to python and am using 3.6 version. Could you guide me where I can find option like ciphers for passing the password of the client certificate.
@Erikbern I have not gone through the temp file solution yet will take a look at the same today. Thanks for the response.

@AnoopPillai You'll want load_cert_chain.

@Lukasa would you mind documenting it? should only take a few minutes (I'm thinking in the advanced section, or perhaps in a new advanced advanced section)

Sorry guys my lack of experience with python could be the reason but I am unable to modify the code Lukasa explained above. My code is:

class DESAdapter(HTTPAdapter):
    """
    A TransportAdapter that re-enables 3DES support in Requests.
    """
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(load_cert_chain='rtmqa-clientid.pem',password='weblogic')
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(load_cert_chain='rtmqa-clientid.pem', password='weblogic')
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)
s = requests.Session()
s.mount(url, DESAdapter())
r = s.get(url, headers=request_header).json()

and I get an error
TypeError: create_urllib3_context() got an unexpected keyword argument 'load_cert_chain'

That is a mistake, yes. You want to call create_urllib3_context and get its return value, and then call load_cert_chain on the returned object. Try playing about with those functions in the interactive interpreter to see how they work.

The urllib3..util.ssl_.py that is installed on my mac does not have the latest option for password.
this is the code

    if certfile:
        context.load_cert_chain(certfile, keyfile)
    if HAS_SNI:  # Platform-specific: OpenSSL with enabled SNI
        return context.wrap_socket(sock, server_hostname=server_hostname)

The password option is missing. How do I update ssl_.py to get the latest version?

@AnoopPillai You don't. Call the function without arguments, then call load_cert_chain on the returned object. You do not need changes to urllib3.

To be clear, like this:

ctx = create_urllib3_context()
ctx.load_cert_chain(your_arguments_here)

Let's document this :)

@ erikbern I tried your tempfile solution But I got the follow error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 441, in wrap_socket
    cnx.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1716, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1456, in _raise_ssl_error
    _raise_current_error()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 595, in urlopen
    self._prepare_proxy(conn)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 816, in _prepare_proxy
    conn.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 448, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='credit-cards-accounts-qa.kdc.capitalone.com', port=443): Max retries exceeded with url: /credit-cards-accounts/credit-cards/accounts/XqLuxBTABbIDvpw56ba34p2WV9JoWUSkPJ09hrBlWD8= (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tsu892/Desktop/Office/Pythone-work/ASR-pythone/ASR-python3.6/test-request.py", line 48, in <module>
    r = requests.get(url, headers=request_header, cert=cert).json()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='credit-cards-accounts-qa.kdc.capitalone.com', port=443): Max retries exceeded with url: /credit-cards-accounts/credit-cards/accounts/XqLuxBTABbIDvpw56ba34p2WV9JoWUSkPJ09hrBlWD8= (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

Below is my code:

import requests
import json
import OpenSSL.crypto
import tempfile
import os
import contextlib
import ssl

json_file='apiInput.json'
hdr_key=[]
hdr_value=[]
json_data=open(json_file)
data = json.load(json_data)
request_body={}
#pprint(data)
json_data.close()
request_data = data['request1']
request_header=request_data['header-data']
url=request_header['url']

@contextlib.contextmanager
def pfx_to_pem():
    print('inside pfx tp pem')
    with tempfile.NamedTemporaryFile(suffix='.pem') as t_pem:
        f_pem = open(t_pem.name, 'wb')
        fr_pfx = open('rtmqa-clientid.pfx', 'rb').read()
        p12 = OpenSSL.crypto.load_pkcs12(fr_pfx,'xxxxxxxxx')
        f_pem.write(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey()))
        f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate()))
        ca = p12.get_ca_certificates()
        if ca is not None:
            for cert in ca:
                f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert))
        f_pem.close()
        yield t_pem.name

with pfx_to_pem() as cert:
    print(cert)
    r = requests.get(url, headers=request_header, cert=cert).json()
print(r.status_code)
print(r.json())

sorry, hard to know why it's breaking looking from your comment. i've used it for bunch of applications and haven't had any issues

@Lukasa I did try with that code change (code pasted below) and ended up with the same error that i got with the tempfile method.

import requests
import json
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

json_file='apiInput.json'
hdr_key=[]
hdr_value=[]
json_data=open(json_file)
data = json.load(json_data)
request_body={}
#pprint(data)
json_data.close()
request_data = data['request1']
request_header=request_data['header-data']
url=request_header['url']

class DESAdapter(HTTPAdapter):
    """
    A TransportAdapter that re-enables 3DES support in Requests.
    """
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context()
        context.load_cert_chain('rtmqa-clientid.pem',password='weblogic')
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context()
        context.load_cert_chain('rtmqa-clientid.pem',password='weblogic')
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)

s = requests.Session()
s.headers=request_header
s.mount(url, DESAdapter())
r = s.get(url)
/Users/tsu892/Python3.6/bin/python /Users/tsu892/Desktop/Office/Pythone-work/ASR-pythone/ASR-python3.6/Test-ASRreq.py
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 441, in wrap_socket
    cnx.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1716, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1456, in _raise_ssl_error
    _raise_current_error()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 595, in urlopen
    self._prepare_proxy(conn)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 816, in _prepare_proxy
    conn.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 448, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='credit-cards-accounts-qa.kdc.capitalone.com', port=443): Max retries exceeded with url: /credit-cards-accounts/credit-cards/accounts/XqLuxBTABbIDvpw56ba34p2WV9JoWUSkPJ09hrBlWD8= (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tsu892/Desktop/Office/Pythone-work/ASR-pythone/ASR-python3.6/Test-ASRreq.py", line 37, in <module>
    r = s.get(url)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='credit-cards-accounts-qa.kdc.capitalone.com', port=443): Max retries exceeded with url: /credit-cards-accounts/credit-cards/accounts/XqLuxBTABbIDvpw56ba34p2WV9JoWUSkPJ09hrBlWD8= (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

@erikbern could it be a setup issue on my laptop. I use a mac, Pythone3.6

6c40089ea258:~ tsu892$ pip3 show requests
Name: requests
Version: 2.18.4
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Requires: idna, certifi, chardet, urllib3
6c40089ea258:~ tsu892$ pip3 show certifi
Name: certifi
Version: 2017.7.27.1
Summary: Python package for providing Mozilla's CA Bundle.
Home-page: http://certifi.io/
Author: Kenneth Reitz
Author-email: [email protected]
License: MPL-2.0
Location: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Requires: 

Do you think anything is wrong with the certifi?

What's the output of python -m requests.help?

@Lukasa The output is:

6c40089ea258:~ tsu892$ python3 -m requests.help?
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3: No module named requests.help?

Please remove the question mark from your command line.

6c40089ea258:~ tsu892$ python3 -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.0.3"
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "16.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "1010006f",
    "version": "17.2.0"
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}

So the error you're getting is caused by us being unable to validate the server TLS certificate. certifi and OpenSSL seem right, so I assume the server is misbehaving. What server are you trying to reach?

The application is deployed on Cloud AWS. But when we call the API it first goes to OSB which authenticates the certificate and then route the request to AWS.
I use the same certificate and using postman or my ruby code the API works fine

Do you need a specific root cert? Can you provide the host name being reached for me?

The Host URL without the path is https://credit-cards-accounts-qa.kdc.capitalone.com
This is a internal endpoint

Yeah, so I can't see what's going on there. Can you run openssl s_client -showcerts -connect credit-cards-accounts-qa.kdc.capitalone.com:443 and provide the complete output?

Deleted

This looks very much like you aren't using a globally trusted root cert. Where is the root cert for that service?

I am not using any other certificates. Not sure if behind the scene any root certificate is use is there a way for me to find out?

Yeah, Chrome's developer Tools will tell you the full certificate chain it is using.

you probably don't want to be posting some internal certificate online for anyone to see...

@erikbern This is public information. You can achieve the same result by running the same command.

@SethMichaelLarson From @erikbern's GitHub profile "Chief Troll Officer". Perhaps they were just trolling?

@erikbern @sigmavirus24 Ah! I didn't know who I was speaking to. Proceed! 🙇

I cant see anything except the sha-1 certificate when I run from postman
May be i need to add this somehow to Pycharm

If you literally browse to the website in Chrome that should be sufficient.

@SethMichaelLarson running what command? FYI the comment has been deleted now but there was an entire BEGIN CERTIFICATE blob here earlier... afaik you don't want to share that online

@erikbern That was just the public key for the cert...

Certificates are public data; they are transmitted in plaintext over the network on each connection attempt.

I went to the cert chain and only found a Sha-1 cert and the .Pem cert that I am using to hit the API

@AnoopPillai I got your example code from Sep 1 working without issue using a client-side pem file with password. It seems the host is using a regular cert. With @Lukasa thanks very much !

I'm unfortunately still having issues, even with the Temp File method. I can use the .pfx in Google Postman and have no issues authenticating (so I know my credentials work), but I'm still getting 401s with Python. Unfortunately the support guy from the company I'm dealing with hasn't been much help - does anyone have any suggestions for troubleshooting?

At this stage I'm genuinely unsure of where to even look for the problem since other people are reporting success with the Temp File method and I still haven't heard anything back from their Cert Management team.

Any advice would be much appreciated - please let me know if I can provide any additional information to make this easier.

Thanks :)

Just a suggestion, did you try converting PFX to PEM? Also, if the server is also using a username/password, you'll need to add that the get/post request using auth=(). I've been using the class DESAdapter(HTTPAdapter) approach above for several weeks now without issue, using a password protected PEM file.

@ideasean Getting invalid credentials still. I should be pointing the load_cert_chain at a .pem file generated by the pfx_to_pem function written for the Temp File method, correct? It has the private key and the cert in it.

Since the .pfx works with Postman but it won't authenticate here, could that mean that something's going wrong in the conversion process?

I did not use the temp file method. I used the DESAdapter approach pretty much as written in AnoopPillai's post on Sep1 above starting with -

I did try with that code change (code pasted below) and ended up with the same error that i got with the tempfile method.

I can't speak to the conversion process, but perhaps a good test is to try using the converted pem file with Postman?

Also note that I used the approach above because my pem file was encrypted / password protected, and Python requests currently does not support that. If your pem ends up being not password protected, then you should be able to use native requests per link (but then you will have an unprotected cert on your file system).

@ideasean I broke down the .pfx as per this method and got a .pem file with Bag Attributes and Certificate as well as a .pem file with Bag Attributes and an Encrypted Private Key.

Still getting invalid credentials, I guess I'll try putting the certs through on Postman and seeing if they work but I can't figure out why I'm apparently unable to unpack this .pfx properly

I also tried the openssl command openssl pkcs12 -in <my_pfx>.pfx -out certificate.cer -nodes, and it's still giving me a 401 error when I change to it like so: context.load_cert_chain('certificate.cer')

I installed the above-mentioned .cer and Postman doesn't even ask to use it when I make the API call (unlike the popup when it asks to use the .pfx), not sure how else I can make it use that specific cert since there's no "Certificates" panel in the settings like the docs say there is.

You may be using the browser version of Postman, which doesn't include the cert panel, ssl validation disable etc. Try the full client to change certificate settings. You may want to continue this discussion on a different thread then, as we are a bit off topic.

@mkane848 saw your original comment where you were getting a ValueError: String expected. You might want to check https://github.com/pyca/pyopenssl/issues/701 and https://github.com/shazow/urllib3/issues/1275.

I use my private pem with a password using this:

from requests.adapters import HTTPAdapter

from urllib3.util.ssl_ import create_urllib3_context

class SSLAdapter(HTTPAdapter):
    def __init__(self, certfile, keyfile, password=None, *args, **kwargs):
        self._certfile = certfile
        self._keyfile = keyfile
        self._password = password
        return super(self.__class__, self).__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).proxy_manager_for(*args, **kwargs)

    def _add_ssl_context(self, kwargs):
        context = create_urllib3_context()
        context.load_cert_chain(certfile=self._certfile,
                                keyfile=self._keyfile,
                                password=str(self._password))
        kwargs['ssl_context'] = context

For your information, I just implemented PKCS#12 support for requests as a separate library:

The code is a clean implementation: it uses neither monkey patching nor temporary files. Instead, a custom TransportAdapter is used, which provides a custom SSLContext.

Any feedback and improvements are welcome!

Of course, I wish requests would provide this functionality directly, but until we are there, this library will alleviate the pain.

It would be very nice if we could simply do this:

~~~
cert=("cert.pem", "key.pem", "somepassphrase") # separate cert/key

cert=("keycert.pem", None, "somepassphrase")    # combined cert/key

~~~

...even if it only worked on python 3.3+. This would only be a minor addition to the API surface.

AFAICS, this would mean a small change to urllib3 so that HTTPSConnection accepts an optional password argument; this is passed down through ssl_wrap_socket, ending up with:

~
if certfile:
if password is not None:
context.load_cert_chain(certfile, keyfile, password)
else:
context.load_cert_chain(certfile, keyfile)
~

Then it would be backwards-compatible, raising an exception only if you try to use a private key passphrase on an older platform that doesn't support it.

Note that the contrib/pyopenssl.py adapter already supports this extra argument to load_cert_chain, and so does python 2.7.


Aside: I am using AWS KMS to manage "secret" data, so I would load the key password at runtime from KMS, not hard-code it into the application.

I personally wouldn’t be against this change, as I think it would greatly improve our user interface for many users across the board.

@sigmavirus24 any thoughts?

@candlerb @kennethreitz Would it be acceptable to include the PKCS#12 case into that API as well?

cert=('keycert.p12', None, 'somepassphrase')

The distinction could be either by file extension (*.p12 versus *.pem), or by looking at the first bytes of that file.

I don't have a problem with allowing requests to take a pkcs#12, as long as it can be done safely - and in my opinion that precludes writing the extracted private key to a temporary file.

Googling for Python pkcs#12, I find:

  • Someone's code which writes out the private key
  • Some other code which I think depends on pyOpenSSL to read in the pkcs#12. It returns the certificate and key as data items.

So doing this, I think it would be necessary to hook things up in such a way that the key/cert themselves are passed to OpenSSL, not the filenames containing those things. That sounds like a much bigger change.

If that's too hard, then it just means that the user has to convert pkcs#12 to PEM off-line, which is pretty straightforward (and can be documented).

@candlerb As I wrote in my previous comment (https://github.com/requests/requests/issues/1573#issuecomment-348968658), I already created a clean implementation that integrates well with requests.

So the problems you are describing are already solved.

Right now my implementation adds new pkcs12_* keywords arguments, to stay out of the way as much as possible.

But I think it should be integrated into the cert keyword argument instead, and my question is:

  • Would that be acceptable in general?
  • Would my concrete proposal cert=('keycert.p12', None, 'somepassphrase') be acceptable?
  • How should we distinguish between PKCS#12 and PEM? (By file name suffix, or by file contents?)

(Moreover, I'd prefer to see that into requests rather than my separate requests_pkcs12 library. But given the age of this issue, I have little hope that this will go upstream anytime soon. However, if there was a concrete statement about which kind of implementation exactly is wanted, maybe I could adjust my implementation accordingly and propose a pull request.)

So, a few things:

  1. I don't think we should take the cert keyword and expand it like this. It's implicitly structured data and people are already confused by the tuples in the files keyword. I think continuing a known-bad pattern is foolish.

  2. I think that if anything, the pkcs12 adapter should be modified and upstreamed into the requests-toolbelt. I think it would be better to modify it to create the ssl_context once instead of storing the pkcs12 password in memory on that object.

I think there's still other work that needs doing before we can handle this in the more general case no matter what and that includes determining the right API for this for Requests 3.0.

@sigmavirus24 Thanks for the feedback.

  1. Okay, so let's keep the separate pkcs12_* keywords.
  2. Yes, that's definitely worth improving. I created an issue tracker entry for that: https://github.com/m-click/requests_pkcs12/issues/2

How would the PKCS#12 TransportAdapter class be included into requests? Would that class simply be added to requests, or is there another way to include it on a "deeper" level, so it can be used without any request()/get()/... wrappers and without having to explicitly load that adapter?

My organization has a need to use PKCS12 certificates and is willing to make the necessary enhancements to your library in order to do so. Decrypting the .p12 files to .pem files is considered too much of a risk and it adds an extra step to deal with. We'd like to add functionality to generate and provide an appropriate ssl_context for a given session. Is this still functionality your team would be willing to accept assuming it is implemented properly?

Just a quick reminder: A clean implementation has already been provided by our company, but as a separate adapter: https://github.com/m-click/requests_pkcs12

Feel free to reformat it into a pull request for requests itself.

Along the way, you might want to fix a minor issue: The ssl_context should not be held in memory for a whole session, but as shortly as possible, just for a single given connection. See also:

In case you fix it along the way, it would be nice if you could provide it as a small pull request to https://github.com/m-click/requests_pkcs12 in addition to requests itself.

That way, all people who are using the requests_pkcs12 library right now would automatically benefit from that improvement as well, without having to switch to the (then improved) new API for requests itself.

Yeah, https://github.com/m-click/requests_pkcs12 worked for me and did exactly what I wanted it to do. Thanks so much @vog ! I hope requests is able to support that eventually.

I am also going to thank @vog for his implementation, works just as expected, and solves the problem of keeping cert/key in the non-secure storages like S3 in my case. Hopefully, this can make its way to requests.

Was this page helpful?
0 / 5 - 0 ratings