Requests: TLS SNI Support

Not posisble in Python 2.x on top of the stdlib's ssl module.

Twisted has it, right? Perhaps time to depend on pyopenssl or some other ssl module?

There's a pull request for urllib3 adding SNI support for Py32+, but still needs test coverage. https://github.com/shazow/urllib3/pull/89

shzow: Thanks for the pointer. I'm unfortunately using Twisted/Flask, which means I'm stuck to 2.7. Twisted it seems can do something like this, but i've not seen examples, and it would need pyopenssl + twisted decoding to do this. Am not an expert in either to pull this off. At this point, just thinking of wrapping some command line utility like wget... :(

PyOpenSSL should be able to do it on 2.x. So there is hope.

I could make it work with pyopenssl. But I still am having a hard time getting it done in twisted correctly. http://pbin.be/show/719/ -- kind of works. But then one has to build an API on it for it to be any useful. I still cant figure out how to pass the correct context to Agent.

Lukasa: was this resolved?

No. We're blocked on shazow/urllib3#89.

Merged. Carry on. :)

Note that it's only supported on Py32+ for now. Should be enough to get started though.

Another fair warning: Some tests on Py32 are failing on urllib3 master right now. Some help resolving this would be greatly appreciated (and might affect this functionality): https://github.com/shazow/urllib3/issues/128

works for me on archlinux x64 on py26, py27, py32, py33

Ah I should have qualified that this is on OSX. Scumbag OSX.

t-8ch: Are the sni tests passing on py26/27/32 and 33 for you?

The SNI tests aren't even run on py2, but on py32 and py33 it works.
There is no way to do SNI with the ssl module of the standard library before py32.

@shazow: Presumably you have no interest in making urllib3 depend on PyOpenSSL?

I'd be happy to have a separate library which adds SSL via PyOpenSSL support to urllib3.

It looks like @t-8ch is knocking it out of the park with the work on urllib3. I'll let him tell us when this can be closed.

urllib3 has optional support for SNI with python2, though with a few optional dependencies. (See urllib3#156).

I belive this should work using:

from requests.packages.urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3

After adding this to packages in setup.py

requests.packages.urllib3.contrib

Note that the contrib package is not imported by default.

There are basically two alternatives to enable SNI in requests:

• Create function to manually enable SNI that basically runs the above code. (the downside is that users will need to explicitly enable SNI support).
• Try to import the pre-requisites. If they exists, enable SNI, else don't.

I've no problem implementing any of both alternatives, though I'd prefer to know which would be the prefer one for requests. I personally vote for the second alternative.

BTW: can we reopen the issue, now that we have support from urllib3?

I actually prefer the former because the people who need it are the ones who know they'll need it and they're few enough that it won't cause too many complaints. However, if we're going to be consistent with the existing API, the latter would be the way to implement it. Currently, urllib3 (and requests) will provide the API to send HTTPS requests without the ssl module present but will fail in the case where the ssl module is not available. In other words, the API is there and you can use it but it just won't work and that is indicated by an exception.

As to re-opening this, that's up to @kennethreitz. The issue of course (with all of this) is that we're currently under a feature freeze (#1165, #1168) so I'm not sure if the work is even worth doing because it might not be accepted.

Let me be a bit clearer: what I meant by the second alternative was "Try to use SNI if the optional deps are available, but still use SSL as we always have it they aren't.". This shouldn't break anything existing up to now.

As for reopening (or not) the issue, IMHO, this issue shouldn't be taken lightly. requests can become useless without SNI support for many scenarios. In particular, multiple domains on a single IPv4 host without SNI become impossible. And additional IPv4 addresses are out of the questions for many users (due to their cost).

In any case, this is half feature, half bug, though I'll await @kennethreitz's reply regarding reopening the issue.

Hugo: I just did pip install -U requests, and when I can't import contrib.
How do I fix that? (Importerror : No module named contrib).

On Fri, May 3, 2013 at 1:12 PM, Hugo Osvaldo Barrera <
[email protected]> wrote:

urllib3 has optional support for SNI with python2, though with a few
optional dependencies. (See urllib3#156https://github.com/shazow/urllib3/pull/156
).

I belive this should work using:

from requests.packages.urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3

Note that the contrib package is not imported by default.

There are basically two alternatives to enable SNI in requests:

• Create function to manually enable SNI that basically runs the above
  code. (the downside is that users will need to explicitly enable SNI
  support).
• Try to import the pre-requisites. If they exists, enable SNI, else
  don't.

I've no problem implementing any of both alternatives, though I'd prefer
to know which would be the prefer one for requests. I personally vote for
the second alternative.

—
Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/749#issuecomment-17406518
.

request's setup.py doesn't include that package (the source it's there, but it's excluded when installing/packaging), so it's not installed, that's why I mentioned:

After adding this to packages in setup.py
requests.packages.urllib3.contrib

You'll basically need to install from source.
Check out my pull request if you're interested, making this work is just a couple of lines. :)

"Try to use SNI if the optional deps are available, but still use SSL as we always have it they aren't."

SSL is not always there though. That was my point. Currently we try and use it but fail hard and fast if we don't have it and a user tries to make an https request. The way you had described it though, I was under the impression SNI would require the user to pass some extra notion to urllib3 and you were focusing on just the set-up for it. If all that is necessary is #1347 then I'm 100% behind this.

It would be really nice if this was the default (SNI Support) and requests
just works without any extra code or setup from source. For urllib3 its 2
lines of code. Why not just add that to requests. Am I day dreaming? :)

On Fri, May 3, 2013 at 10:43 PM, Ian Cordasco [email protected]:

"Try to use SNI if the optional deps are available, but still use SSL as
we always have it they aren't."

SSL is not always there though. That was my point. Currently we try and
use it but fail hard and fast if we don't have it and a user tries to make
an https request. The way you had described it though, I was under the
impression SNI would require the user to pass some extra notion to urllib3
and you were focusing on just the set-up for it. If all that is necessary
is #1347 https://github.com/kennethreitz/requests/issues/1347 then I'm
100% behind this.

—
Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/749#issuecomment-17426626
.

@pythonmobile spin a top. If it doesn't stop spinning, you're dreaming.

Leaving a comment for the sake of others who may run into the same issue as me:
If you are having trouble making secure requests using the requests library, in particular when trying to hit an app hosted on Google App Engine, this may be why. The error I was seeing was "EOF occurred in violation of protocol". To diagnose this, try hitting the same host using s_client (replace example.com with the host in question):

openssl s_client -connect example.com:443
openssl s_client -connect example.com:443 -servername example.com

If the first command fails with "handshake failure" and the second succeeds, then you are trying to hit a server that uses SNI. The present thread has some good info on how to get requests working in this situation. What I did was try to run from requests.packages.urllib3.contrib import pyopenssl in ipython, and I kept on pip installing any import errors. YMMV.

@mshang Version 1.2.3 should do this for you already; are you sure you're running that version?

@hobarrera Yes, I'm running 1.2.3. requests.packages.urllib3.contrib was there, but that import statements still threw a bunch of import errors. I had to install ndg-httpsclient and I forget what else.

Correct. Requests just attempts to import the needed modules, and if it can't gives up and goes on with its life. It might be a good idea to document what you need to use SNI with Requests, though.

This is specified in requests/packages/urllib3/contrib/pyopenssl.py:

• pyOpenSSL (tested with 0.13)
• ndg-httpsclient (tested with 0.3.2)
• pyasn1 (tested with 0.1.6)

(Or python3.2 or higher, which always works)

Seems like both requests and urllib3 are broken for SNI currently. One
problem is that if one adds timeout=5.0, the code behaves differently than
without timeout. The second problem is with domains that are non existant.
The tests need to add more SNI domains compared to only depending on
vertex/httpbin/. They both dont reflect SNI usage at all.

On Mon, Jun 10, 2013 at 4:57 AM, Thomas Weißschuh
[email protected]:

This is specified in requests/packages/urllib3/contrib/pyopenssl.pyhttps://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/contrib/pyopenssl.py
:

• pyOpenSSL (tested with 0.13)
• ndg-httpsclient (tested with 0.3.2)
• pyasn1 (tested with 0.1.6)

(Or python3.2 or higher, which always works)

—
Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/749#issuecomment-19187417
.

@pythonmobile If that's the case, and you have a good reproducible test case, please open an issue highlighting it on urllib3. Requests has no SNI-specific code (as far as I know), so a fix there will fix us as well.

Unless urllib3 is fixed already, which seems to happen a lot these days. Shazow _et. al._ are pretty awesome. @t-8ch, have you guys seen/fixed this already?

I think if we write the tests in requests, they are easier to write and tests, even though, it only means that it will test mostly urllib3 code. That way, it will be easier to check at every import of urllib3 in requests, if sni is broken or not and raise an alarm. I'll write the requests tests soon.

Sorry @pythonmobile, I don't agree. =)

SNI is explicitly urllib3 functionality. If SNI is broken in urllib3 you absolutely should provide that project with a fix, a test, or at the very least a bug report. When that project is fixed, Requests will magically become fixed as well (when we next update urllib3, which we do fairly frequently). =)

While more tests for requests certainly wouldn't hurt, as @Lukasa said, you'll need to write a test for urllib3 if you'd like to see this actually fixed. :)

We already have many tests in a similar vein, ping me if you need any help.

It seems the timeout behaviour of pyOpenSSL is different from that of the
standard ssl module:

from OpenSSL import SSL
import socket
import ssl

sock = socket.create_connection(('httpbin.org', 443), timeout=4.0)
ssl_sock = ssl.wrap_socket(sock)
ssl_sock.send('GET /ip HTTP/1.1\r\nHost: httpbin.org\r\n\r\n')
print(ssl_sock.recv(10000))

ctx = SSL.Context(SSL.TLSv1_METHOD)
sock = socket.create_connection(('httpbin.org', 443), timeout=4.0)
ctn = SSL.Connection(ctx, sock)
ctn.set_connect_state()

ctn.send('GET /ip HTTP/1.1\r\nHost: httpbin.org\r\n\r\n')
print(sock.read(10000))

This throws a WantReadError, the same that is also thrown out from urllib3
when specifying a timeout.

The documentation has
this to say about the error:

The operation did not complete; the same I/O method should be called again
later, with the same arguments. Any I/O method can lead to this since new
handshakes can occur at any time.

The wanted read is for dirty data sent over the network, not the clean data
inside the tunnel. For a socket based SSL connection, read means data coming at
us over the network. Until that read succeeds, the attempted
OpenSSL.SSL.Connection.recv, OpenSSL.SSL.Connection.send, or
OpenSSL.SSL.Connection.do_handshake is prevented or incomplete. You probably
want to select() on the socket before trying again. 

I don't know how where the timeout should be handled. Does the socket magically
remember the timeout over invocations of select() or is this the task of
urllib3?
Keeping track of the timeout by hand doesn't sound really funny.

@pythonmobile What do you mean with nonexistent domains. I can't follow you.

@pythonmobile Could you try the changes from shazow/urllib3#233 and look if the code works with pyopenssl and timeouts?

@t-8ch @pythonmobile : Can you guys look at this thread as well: https://github.com/kennethreitz/requests/issues/1522#issuecomment-22443282

It would be nice to get these requests in a unit test inside urllib3 or requests.

The problem with getting these as a unit test is that they only appear in very specific configurations. For instance, I have never been able to reproduce either the second half of this issue or #1522 on any of my systems, whether it's Windows, OS X or Ubuntu. That's why I asked @pythonmobile for a reproducible unit test. If we can find one, we can develop against it, otherwise we're stuck hoping that @t-8ch continues to be a genius and magically fix all our problems up at urllib3.

NB: @t-8ch, I don't think I thank you enough for your work, both here and at urllib3. You're awesome. =D :cake: :pineapple: :banana: :cookie: :beers:

@lukasa Thanks :smile: I guess I could add myself to AUTHORS.txt some time in the future.

@t-8ch Do it. :+1:

The changes in shazow/urllib3#233 (now pulled into urllib3 master) fix the issue for me (SSLError('bad handshake', WantReadError())) which occurred for some urls, some times, in some configurations :rage4:. Can we get this pulled into requests master?

@rcoup We'll pull in an up-to-date version into Requests when we release 2.0. =)

I am running Python 2.7.6 and can not install pyOpenSSL. I have tried the work arounds in the htis post -> https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484

I can not also upgrade by Python. Any other solutions?

I'm requesting we reconsider the unconditional use (if present) of "inject_into_urllib3()".

Added 7 years ago, the goal was to "add SNI support for Python 2"

"urllib3.contrib.pyopenssl.inject_into_urllib3()" is described to "Monkey-patch urllib3 with PyOpenSSL-backed SSL-support.":
(https://github.com/urllib3/urllib3/blob/master/src/urllib3/contrib/pyopenssl.py)

Justification:
1: Security/stability: Monkey patching to swap out standard library use for a 3rd party library should arguably be more surgical, especially for ssl. If the patch is for support in Python2, then at least check for a major version. Or even better, check if SNI support is already present ssl.HAS_SNI.

2: Unnecessary: As of Dec. 10, 2014, the entirety of Python 3.4's ssl module has been backported for Python 2.7.9. See PEP 466 for justification. https://www.python.org/downloads/release/python-279/. This enables SNI support in the standard library for > v2.7.9

3: Inflexible: As implemented, there is no way to disable this behavior. The only option to prevent requests use of pyopenssl context is to uninstall pyopenssl for my entire python environment.