httpie refuses to trust a self-signed certificate

Created on 25 Apr 2017 · 9Comments · Source: httpie/httpie

I don't entirely understand the discussion for issue #480 but I think the root cause may be the same for this issue. I need httpie to trust a self-signed certificate. Technically it is a self-signed intermediate certificate authority, but I can use one of the actual server certificates from that authority and get the same result. This certificate satisfies every other application on my Mac (OS 10.12), including curl. Httpie simply won't trust it. Sample output:

$ http --debug -j --verify=/usr/local/etc/openssl/certs/intermediate-authority.pem https://www.testdomain.com/robots.txt
HTTPie 0.9.9
Requests 2.12.3
Pygments 2.1.3
Python 3.6.1 (default, Mar 24 2017, 17:14:46)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
/usr/local/Cellar/httpie/0.9.9/libexec/bin/python3.6
Darwin 16.5.0

<Environment {
    "colors": 256,
    "config": {
        "__meta__": {
            "about": "HTTPie configuration file",
            "help": "https://github.com/jkbrzt/httpie#config",
            "httpie": "0.9.4"
        },
        "default_options": "[]"
    },
    "config_dir": "/Users/username/.httpie",
    "is_windows": false,
    "stderr": "<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>",
    "stderr_isatty": true,
    "stdin": "<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>",
    "stdin_encoding": "UTF-8",
    "stdin_isatty": true,
    "stdout": "<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>",
    "stdout_encoding": "UTF-8",
    "stdout_isatty": true
}>

>>> requests.request(**{
    "allow_redirects": false,
    "auth": "None",
    "cert": "None",
    "data": "",
    "files": {},
    "headers": {
        "Accept": "application/json, */*",
        "Content-Type": "application/json",
        "User-Agent": "HTTPie/0.9.9"
    },
    "method": "get",
    "params": {},
    "proxies": {},
    "stream": true,
    "timeout": 30,
    "url": "https://www.testdomain.com/robots.txt",
    "verify": "/usr/local/etc/openssl/certs/intermediate-authority.pem"
})


http: error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) while doing GET request to URL: https://www.testdomain.com/robots.txt
Traceback (most recent call last):
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 801, in _prepare_proxy
    conn.connect()
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 624, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/http", line 11, in <module>
    load_entry_point('httpie==0.9.9', 'console_scripts', 'http')()
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/__main__.py", line 11, in main
    sys.exit(main())
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/core.py", line 227, in main
    log_error=log_error,
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/core.py", line 99, in program
    final_response = get_response(args, config_dir=env.config.directory)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/client.py", line 70, in get_response
    response = requests_session.request(**kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

If I use --verify=no, the connection is accepted. I've tried converting the certificate file to DER format instead of PEM with no change.

Httpie was installed via Homebrew, which installed its own Python3 as a dependency.

Source

sensibleish

Most helpful comment

Trusting a self-signed certificate is never encouraged. It is dangerous if a program accepts it by default. A hacker can simply inject a self-signed one to make the users feeling secure while it is not the case.

However, I would suggest httpie to response the issue of using a self-signed certificate with a clearer message rather than simply fail it. A suggestion to use --verify=no to disable the verification for testing purpose would be great.

alvis on 19 May 2017

👎13 👍3

All 9 comments

alvis on 19 May 2017

👎13 👍3

@giskard22

I don't entirely understand the discussion for issue #480 but I think the root cause may be the same for this issue. I need httpie to trust a self-signed certificate. Technically it is a self-signed intermediate certificate authority, but I can use one of the actual server certificates from that authority and get the same result.

Looking at #480 this is not the same root cause. HTTPie is, in this case, using the bundle you provide it.

What I suspect is actually the cause is the following:

Your "test domain" is not returning the full certificate chain but only its leaf (which works fine for browsers which can dynamically retrieve the rest of the chain) but not for httpie and other non-browser clients. In this case, you're providing part of the chain via --verify but still not all of the intermediaries. If you place them all in one bundle and provide that instead, I suspect you'll find that this works.

Trusting a self-signed certificate is never encouraged. It is dangerous if a program accepts it by default. A hacker can simply inject a self-signed one to make the users feeling secure while it is not the case.

@alvis blanket advice is usually wrong and in this case it's quite alarmingly wrong. Self-signed certificates are a solution to a problem. I don't know @giskard22's use case for using a self-signed certificate here and neither do you. In that case your advice is not only most likely useless to @giskard22, it's also noise for the rest of the people receiving email notifications from issues on this repository.

Self-signed certificates can be dangerous to trust without verification but in this case it seems @giskard22 knows the trust chain and trusts it making this usage perfectly safe. Yes, if you happen upon a site using HTTPS that you might need to enter sensitive information into, it is not advisable to use a self-signed certificate but this is a wildly different situation as I understand it.

sigmavirus24 on 28 May 2017

Yes, the use case here is using httpie through a corporate web proxy that does HTTPS man-in-the-middle so it can inspect traffic. It generates a domain-specific certificate on the fly. Blah.

I don't understand how the proxy's failure to provide a full certificate chain causes this issue. Perhaps OpenSSL's behavior (or is it Python's behavior?) is different than MacOS's built-in mechanism.

The trust chain for connections through the proxy looks like this:

On-the-fly certificate issued to www.testdomain.com
Intermediate CA certificate called proxy-name
Self-signed root CA certificate called company-name

I have been unable to obtain certificate 3. However, certificate 2 is available from the proxy. I imported into my Keychain and marked it trusted for SSL use. That was enough to satisfy everything on my Mac except httpie, including other command-line tools like curl. The observed behavior is the validation mechanism follows the chain until it finds a trusted certificate, then stops. The root certificate's absence is irrelevant.

In the httpie debug output I posted in my initial report, I had stored certificate 2 in intermediate-authority.pem and specified that file with --verify. Are you saying that for httpie, that's not enough and I must also provide certificate 3?

sensibleish on 28 May 2017

Perhaps OpenSSL's behavior (or is it Python's behavior?) is different than MacOS's built-in mechanism.

So OpenSSL will look in the bundle you provide it for whatever it's missing. If it cannot find it there it fails the verification. This is transitively also Python's behaviour. SecureTransport (MacOS's built-in mechanism) works slightly differently but only if the certificates are in MacOS's Keychain. Also, on MacOS the default Python has a hacked-up (for lack of a better term) version of OpenSSL that uses Keychain and thus can resolve these even if you provide it your own certificate bundle (i.e., if you tell it only to trust what's in a bundle file, it'll say "LOL, but I got Keychain access and that trusts it so YOLO" ... or YOVO? (You Only Verify Once... Idk)).

If the entire trust chain for these on-the-fly certificates is what you described, I think you should be able to get away with having a bundle that only includes links 2 and 3 in the chain.

Also, if you installed httpie from homebrew on MacOS then it was installed with the homebrew versions of OpenSSL, Python, and virtualenv which is why it cannot take advantage of a Mac's hacked-up OpenSSL.

sigmavirus24 on 29 May 2017

I was finally able to identify and obtain the self-signed root certificate. Upon putting both that cert and the intermediate cert in a .pem file and using that file with --verify, I no longer got the SSL error. This is good! As suggested, OpenSSL requires the entire certificate chain. Unlike SecureTransport, it won't accept a partial chain even if manually supplied.

Any thoughts on the following additional complication? The proxy lets traffic go to certain domains without inserting its custom certificate. It looks like you cannot use multiple --verify arguments. The last one wins. Is there any way to use multiple certificate bundles?

I have a workaround that involves concatenating OpenSSL's provided root CA bundle and the custom bundle upon demand, but hoping to avoid that.

sensibleish on 1 Jun 2017

So OpenSSL supports providing a directory full of .pem files for verification. That said, support for that in Python is spotty. I don't honestly think anything older than Python 3.5 or 3.6 supports directories in the standard library and I don't recall if pyOpenSSL will support it. The short of it is that I don't think you can accomplish your ends without having one all-serving .pem file. The other catch here is that the HTTP transport library that HTTPie uses doesn't quite support directories either given the limited support across versions and TLS library implementations.

sigmavirus24 on 1 Jun 2017

ol' Postman to the rescue. You can turn off detection of self signed cert.

PostImpatica on 27 Aug 2017

SSL is good, probably devs know that. Disabling SSL because of corner cases and language limitations or whatever stinks. I know people just want to get work done. I just wonder when we'll say SSL is good and have the same gut reaction at saying "disable SSL verification". If you disable SSL verification, SSL is off. But SSL is good. We all know that right? Is it confusing or common knowledge that SSL verification is really what the whole SSL thing is? If postman has an option to disable SSL verification is that just a fancy way of saying something that is potentially insecure? curl has that same option as a flag called --insecure. Is it just a fancy way that hides exactly what the problem is (it turns off SSL)? I think it's fine to test in Postman. TIL that Postman has that option. Am I going to turn it back on? Or did I just disable SSL for every current and future project? Am I sending API keys from a coffee shop? :|

I hope https dev machines, self signed certs and corporate CAs are supplanted by scriptable automatic letsencrypt certs one day. It's a shame that python makes http less organic at trusting the system keychain / ssl store. It's a curl replacement except in this area. I guess golang has operating system detection for all this stuff. So a modern curl replacement (like bat) that's written in go would have pretty output, a nice CLI experience and handle custom CAs or ssl termination (load balancers).

squarism on 11 May 2018

😕1 🎉1

SSL is good, probably devs know that. Disabling SSL because of corner cases and language limitations or whatever stinks. I know people just want to get work done. I just wonder when we'll say SSL is good and have the same gut reaction at saying "disable SSL verification". If you disable SSL verification, SSL is off. But SSL is good. We all know that right? Is it confusing or common knowledge that SSL verification is really what the whole SSL thing is? If postman has an option to disable SSL verification is that just a fancy way of saying something that is potentially insecure? curl has that same option as a flag called --insecure. Is it just a fancy way that hides exactly what the problem is (it turns off SSL)? I think it's fine to test in Postman. TIL that Postman has that option. Am I going to turn it back on? Or did I just disable SSL for every current and future project? Am I sending API keys from a coffee shop? :|

I hope https dev machines, self signed certs and corporate CAs are supplanted by scriptable automatic letsencrypt certs one day. It's a shame that python makes http less organic at trusting the system keychain / ssl store. It's a curl replacement except in this area. I guess golang has operating system detection for all this stuff. So a modern curl replacement (like bat) that's written in go would have pretty output, a nice CLI experience and handle custom CAs or ssl termination (load balancers).

Nope, not good. Need to be able to curl even if adhoc developer certificate.