requests force override query string encoding

Created on 11 Jul 2013 · 3Comments · Source: psf/requests

I am dealing with a legacy platform that does not understand URL encoded parameters.

I have the following parameters

params = {'fiz': 'foo:bar', 'biz': 'barfoo'}

url generated by requests

http://localhost/?fiz=foo%3Abar&biz=barfoo

required url for the legacy platform

http://localhost/?fiz=foo:bar&biz=barfoo

I tried creating a custom query string and adding that to the url, however requests will decode this query, and re-encode it.

improt urllib
qry = urllib.urlencode(params).replace('%3A', ':')
requests.get(url+'?'+qry) #: requests encodes the colon back to %3A

Is there any quick work around that doesn't look as verbose as using straight urllib2?

is it possible to get a "safe" or "replace" list hooked into the API, this would need to be done at the requests layer as python 2.7 is required.

Source

thadeusb

Most helpful comment

This is a bit awkward to do in Requests. As I see it, you have two options. The first is to prepare the Request objects yourself:

import requests
import urllib

url = 'http://test.com/'
qry = urllib.urlencode(params).replace('%3A', ':')

s = requests.Session()
req = requests.Request(method='GET', url=url)
prep = req.prepare()
prep.url = url + qry
r = s.send(prep)

Your other option would be to build a Transport Adapter that mutates the URLs before sending them on, like so:

import requests
import urllib

class MutatingAdapter(requests.adapters.HTTPAdapter):
    def send(self, request, **kwargs):
        main, querystr = request.url.split('?') # Nasty hack, not always valid.
        querystr = urllib.unquote(querystr)
        querystr = urllib.quote(querystr, '/&:=')
        request.url = '?'.join([main, querystr])

        super(MutatingAdapter, self).send(request, **kwargs)

Lukasa on 11 Jul 2013

👍12 🎉2

All 3 comments

This is a bit awkward to do in Requests. As I see it, you have two options. The first is to prepare the Request objects yourself:

import requests
import urllib

url = 'http://test.com/'
qry = urllib.urlencode(params).replace('%3A', ':')

s = requests.Session()
req = requests.Request(method='GET', url=url)
prep = req.prepare()
prep.url = url + qry
r = s.send(prep)

Your other option would be to build a Transport Adapter that mutates the URLs before sending them on, like so:

import requests
import urllib

class MutatingAdapter(requests.adapters.HTTPAdapter):
    def send(self, request, **kwargs):
        main, querystr = request.url.split('?') # Nasty hack, not always valid.
        querystr = urllib.unquote(querystr)
        querystr = urllib.quote(querystr, '/&:=')
        request.url = '?'.join([main, querystr])

        super(MutatingAdapter, self).send(request, **kwargs)

Lukasa on 11 Jul 2013

👍12 🎉2

I recommend the first option.