Requests: Streaming gzipped responses

No it should not default to setting that for a wide variety of reasons. What you should do is use functools.partial to replace the read method on the response (or just wrap it another way) so that you do something like:

response.raw.read = functools.partial(response.raw.read, decode_content=True)

and then pass response.raw to your parser.

@sigmavirus24 Thanks, that's definitely an elegant solution to the problem I outlined above!

I would recommend adding that to requests' documentation, e.g. in the FAQ: http://docs.python-requests.org/en/latest/community/faq/#encoded-data
Currently, the statement "Requests automatically decompresses gzip-encoded responses" is not correct for the stream=True case and can lead to surprises.

As for my problem, as you've read on the urllib3 issue, the urllib3 implementation of the gzip decompression has its own little quirks I have to work around in my code, but that is no longer a problem for requests.

but that is no longer a problem for requests.

As in you feel this can be closed?

@sigmavirus24 I believe it should be documented, as the current documentation is incorrect.

But if you disagree with that, yes, close away!

The documentation could be clearer. To me (and this is entirely because I'm a core developer) the first paragraph speaks to the 90% of users who will never touch the raw response, while the second paragraph contradicts the first in saying "but if you need to access the raw data, it's there for you". Like I said, that's apparent to me, but I can see how that could be made clearer. I'll work on that tonight.

For me, it's more that I would have interpreted "raw data" as "raw payload", i.e. a decompressed stream. I just have to read it in whatever chunks I need. As opposed to .content, which is a decompressed blob (also the payload, but in a different form).

The actual decompression feels like a concern of the HTTP library to me—an implementation detail of HTTP if you will, one that I would expect requests to abstract away. Whether I read the payload from requests as a stream or as a prefetched blob of data wouldn't make a difference. Either way, requests would abstract the implementation detail 'compression'.

(This assumption was also at the core of my original request to default decode_content to True. Of course now that I see what a leaky abstraction this is, I'm no longer suggesting that.)

But yeah, I absolutely agree that 99% of your users will never be affected by this detail.

Feel free to close this issue.

So this actually leads to something that's been rattling around in my head for a while and which I haven't proposed yet because it would be a significant API change.

I don't like the fact that we suggest people use r.raw because it's an object which we don't document and it's an object provided by urllib3 (which we've claimed in the past is more of an implementation detail). With that in mind, I've been toying with the idea of providing methods on a Response object which just proxy to the urllib3 methods (read would just proxy to raw.read, etc.). This gives us extra flexibility around urllib3 and allows us to handle (on behalf of the users) an API change in urllib3 (which historically has almost never been a problem, so there isn't any urgency in that).

With that said, we already have enough methods on a Response object in my opinion and growing our API isn't ideal. The best API is the API from which there's nothing left to remove. So I'm continuously on the fence about this.

This assumption was also at the core of my original request to default decode_content to True. Of course now that I see what a leaky abstraction this is, I'm no longer suggesting that.

For others who find this and may not be certain why this is true, allow me to explain.

There are several users of requests who turn off automatic decompression to validate the length of a response, or to do other important things with it. One consumer of the former kind is OpenStack. Many of the OpenStack clients validate the Content-Length header sent to the client and the actual length of the body received. To them, handling decompression is a fair trade-off to be certain they're receiving and handling a valid response.

Another consumer is Betamax (or really any tool that (re)constructs Response objects) because when it is handling the full process of making a totally valid response, it needs the content to be in the compressed format.

I'm sure there are others that neither @Lukasa or I know about that also rely heavily on this behaviour.

Hit the same issue today, and ended up making the same assumption as there is no other way to stream responses at the moment.

Rather than multiple new methods on Response why not a single new attribute e.g. response.stream which would play the same role of proxy to .raw? It would also nicely mirror the stream=True setting/parameter, and would not affect users needing the current .raw behavior.

I've done this in the past

r = requests.get('url', stream=True)
r.raw.decode_content = True
...

Note that the workaround by @sigmavirus24 breaks the semantics of the tell method, which will return incorrect offsets.

I ran into this when streaming a response as a resumable upload into the Google Cloud Storage API, which uses tell() to figure out the number of bytes that were just read (here).