Requests: UnicodeEncodeError: 'latin-1' ์ฝ”๋ฑ์€ ๋ฌธ์ž๋ฅผ ์ธ์ฝ”๋”ฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค

์— ๋งŒ๋“  2013๋…„ 12์›” 20์ผ  ยท  7์ฝ”๋ฉ˜ํŠธ  ยท  ์ถœ์ฒ˜: psf/requests

์š”์ฒญ์€ ์ตœ์‹  ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
ํ•œ์ž๊ฐ€ ํฌํ•จ๋œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฒŒ์‹œํ•˜๋ ค๊ณ  ํ•˜๋ฉด ์ด ์˜ˆ์™ธ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

Traceback (most recent call last):
  File "X/threading.py", line 639, in _bootstrap_inner
  File "X/threading.py", line 596, in run
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\salesforce\api.py", line 546, in execute_anonymous
    headers=headers)
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\requests\api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\requests\sessions.py", line 338, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\requests\sessions.py", line 441, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\requests\adapters.py", line 292, in send
    timeout=timeout
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\requests\packages\urllib3\connectionpool.py", line 428, in urlopen
    body=body, headers=headers)
  File "C:\Users\Administrator\Dropbox\Sublime3056\Data\Packages\SublimeApex\requests\packages\urllib3\connectionpool.py", line 280, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "X/http/client.py", line 1049, in request
  File "X/http/client.py", line 1086, in _send_request
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 1632-1633: ordinal not in range(256)

๊ฐ€์žฅ ์œ ์šฉํ•œ ๋Œ“๊ธ€

๋”ฐ๋ผ์„œ ST 3์ด์ง€๋งŒ ๊ฐ€์žฅ ์ตœ๊ทผ์˜ ๊ฐœ์ •ํŒ์€ ์•„๋‹™๋‹ˆ๋‹ค. ์ข‹์•„, ๊ทธ๊ฒƒ์€ ์šฐ๋ฆฌ์—๊ฒŒ ๋ญ”๊ฐ€๋ฅผ ์ค€๋‹ค. ํŠนํžˆ Sublime Text 3๋Š” Python 2.7(Sublime Text 2์—์„œ ์‚ฌ์šฉ)์ด ์•„๋‹Œ Python 3.3์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ฆ‰, Sublime Apex์˜ ๋ชจ๋“  ๊ธฐ๋ณธ ๋ฌธ์ž์—ด์€ ์œ ๋‹ˆ์ฝ”๋“œ ๋ฌธ์ž์—ด์ž…๋‹ˆ๋‹ค.

Python 3.3 http.client ํŒŒ์ผ์„ ์—ด๋ฉด _send_request() ํ•จ์ˆ˜๊ฐ€ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

# Honor explicitly requested Host: and Accept-Encoding: headers.
header_names = dict.fromkeys([k.lower() for k in headers])
skips = {}
if 'host' in header_names:
    skips['skip_host'] = 1
if 'accept-encoding' in header_names:
    skips['skip_accept_encoding'] = 1

self.putrequest(method, url, **skips)

if body is not None and ('content-length' not in header_names):
    self._set_content_length(body)
for hdr, value in headers.items():
    self.putheader(hdr, value)
if isinstance(body, str):
    # RFC 2616 Section 3.7.1 says that text default has a
    # default charset of iso-8859-1.
    body = body.encode('iso-8859-1')
self.endheaders(body)

์ด์ œ ISO-8859-1์€ ์šฐ๋ฆฌ๊ฐ€ ๋ฌธ์ œ๋ฅผ ๊ฒช๊ณ  ์žˆ๋Š” ์ฝ”๋ฑ์ธ Latin-1์˜ ๋ณ„์นญ์ž…๋‹ˆ๋‹ค. ์šฐ๋ฆฌ๊ฐ€ ๊ฐ€์ง„ ๋ฌธ์ œ๋Š” Sublime Apex๊ฐ€ ์š”์ฒญ์— ์œ ๋‹ˆ์ฝ”๋“œ ๋ฌธ์ž์—ด ๋ณธ๋ฌธ์„ ์ œ๊ณตํ•˜๊ณ  ์žˆ๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. ์ด ๋ณธ๋ฌธ์€ httplib๊ฐ€ ๋ฐ”์ดํŠธ๋กœ ์ธ์ฝ”๋”ฉํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. RFC 2616์˜ ๊ธฐ๋ณธ๊ฐ’์„ ์‚ฌ์šฉํ•˜๋ฉด ํ•œ์ž๋ฅผ ํฌํ•จํ•˜์ง€ ์•Š๋Š” Latin-1์„ ์›ํ•œ๋‹ค๋Š” ๊ฒฐ๋ก ์ด ๋‚˜์˜ต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋ฉด ๋ถ„๋ช…ํžˆ ์ธ์ฝ”๋”ฉ์ด ์‹คํŒจํ•˜๊ณ  ๋ฌธ์ œ์˜ ์˜ˆ์™ธ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

Sublime Apex๊ฐ€ ๋ณด๋‚ด๋Š” ํ—ค๋”์—์„œ UTF-8 ์ธ์ฝ”๋”ฉ ๋ฐ์ดํ„ฐ(ํ˜„์žฌ๋Š” ๊ฑฐ์ง“๋ง์ž„)๋ฅผ ์ „์†กํ•œ๋‹ค๊ณ  ์ฃผ์žฅํ•œ๋‹ค๋Š” ์ ์„ ๊ณ ๋ คํ•˜๋ฉด Sublime Apex๋Š” ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด๋‚ด๊ธฐ ์ „์— ๋ฐ์ดํ„ฐ๋ฅผ UTF-8๋กœ ์ธ์ฝ”๋”ฉํ•˜๊ธฐ๋ฅผ ์›ํ•ฉ๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด๋‚ด๋Š” ๋ชจ๋“  ๋ผ์ธ(์ด ๊ฒฝ์šฐ salesforce/api.py 545๋ฒˆ ๋ผ์ธ)์ด ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ฝ์–ด์•ผ ํ•จ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค.

response = requests.post(self.apex_url, soap_body.encode('utf-8'), verify=False, headers=headers)

๋‚ด ์ง„๋‹จ์„ ํ™•์ธํ•˜๋ ค๋Š” ๋‹ค๋ฅธ ์‚ฌ๋žŒ์„ ์œ„ํ•ด ๋‹ค์Œ์€ ๋ฌธ์ œ๋ฅผ ํ™•์ธํ•˜๋Š” ๊ฐ„๋‹จํ•œ ์ƒ˜ํ”Œ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค.

a = "\u13E0\u19E0\u1320"
a.encode('latin1')  # Throws UnicodeEncodeError, proves that this can't be expressed in ISO-8859-1.
a.encode('utf-8')  # Totally fine.
r = requests.post('http://httpbin.org/post', data=a)  # Using unicode string, throws UnicodeEncodeError blaming Latin1.
r = requests.post('http://httpbin.org/post', data=a.encode('utf-8'))  # Works fine.

์ด ๋ฌธ์ œ๋ฅผ ์ œ๊ธฐํ•ด ์ฃผ์…”์„œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ ์ด๊ฒƒ์€ ์š”์ฒญ ๋ฒ„๊ทธ๊ฐ€ ์•„๋‹™๋‹ˆ๋‹ค. =)

๋ชจ๋“  7 ๋Œ“๊ธ€

ํŒŒ์ผ "X/http/client.py"

๋กœ์ปฌ ํŒŒ์ผ์˜ ๊ฒฝ๋กœ์ด๊ธฐ ๋•Œ๋ฌธ์— X ๋ผ๊ณ  ์ผ์Šต๋‹ˆ๊นŒ? ๊ทธ๋ ‡๋‹ค๋ฉด ๋””๋ ‰ํ† ๋ฆฌ ๊ตฌ์กฐ๊ฐ€ urllib3๋ฅผ ํ˜ผ๋™ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ ‡์ง€ ์•Š์€ ๊ฒฝ์šฐ ์š”์ฒญ์„ ์ฒ˜๋ฆฌํ•ด์•ผ ํ•œ๋‹ค๊ณ  ์ƒ๊ฐํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๊ธฐ ๋•Œ๋ฌธ์— bugs.python.org์—์„œ ์ด ๋ฌธ์ œ๋ฅผ ์ œ๊ธฐํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ httplib (๋˜๋Š” Python 3์˜ http ์—์„œ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋Š” ๊ฒƒ์œผ๋กœ ์ถ”์ธก๋จ)์—์„œ ์ƒ์Šนํ•˜๋Š” ๊ฒƒ์ฒ˜๋Ÿผ ๋ณด์ž…๋‹ˆ๋‹ค.

@sigmavirus24 ,

๋‚˜๋Š” sublime ํ”Œ๋Ÿฌ๊ทธ์ธ์—์„œ ์š”์ฒญ์„ ์‚ฌ์šฉํ–ˆ๋Š”๋ฐ, ์•„๋ž˜ ๊ตฌ๋ฌธ์˜ soap_body์— ํ•œ์ž๊ฐ€ ํฌํ•จ๋˜์–ด ์žˆ์ง€ ์•Š์œผ๋ฉด ์˜ˆ์™ธ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

response = requests.post(self.apex_url, soap_body, verify=False, headers=headers)

์ฒซ์งธ, ๊ณต๊ฐœ ์ €์žฅ์†Œ์— ์žˆ๋Š” ๊ฒƒ๊ณผ ๋‹ค๋ฅธ ๋ฒ„์ „์˜ Sublime Apex๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š๋Š” ํ•œ Requests๋Š” ์ตœ์‹  ๋ฒ„์ „์ด ์•„๋‹ˆ๋ผ 1.2.3 ๋ฒ„์ „์ž…๋‹ˆ๋‹ค. ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋Š” Sublime Text์˜ ๋ฒ„์ „์„ ์•Œ๋ ค์ฃผ์‹ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

์ˆญ๊ณ ํ•œ ํ…์ŠคํŠธ 3056์ž…๋‹ˆ๋‹ค.

๋”ฐ๋ผ์„œ ST 3์ด์ง€๋งŒ ๊ฐ€์žฅ ์ตœ๊ทผ์˜ ๊ฐœ์ •ํŒ์€ ์•„๋‹™๋‹ˆ๋‹ค. ์ข‹์•„, ๊ทธ๊ฒƒ์€ ์šฐ๋ฆฌ์—๊ฒŒ ๋ญ”๊ฐ€๋ฅผ ์ค€๋‹ค. ํŠนํžˆ Sublime Text 3๋Š” Python 2.7(Sublime Text 2์—์„œ ์‚ฌ์šฉ)์ด ์•„๋‹Œ Python 3.3์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ฆ‰, Sublime Apex์˜ ๋ชจ๋“  ๊ธฐ๋ณธ ๋ฌธ์ž์—ด์€ ์œ ๋‹ˆ์ฝ”๋“œ ๋ฌธ์ž์—ด์ž…๋‹ˆ๋‹ค.

Python 3.3 http.client ํŒŒ์ผ์„ ์—ด๋ฉด _send_request() ํ•จ์ˆ˜๊ฐ€ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค.

# Honor explicitly requested Host: and Accept-Encoding: headers.
header_names = dict.fromkeys([k.lower() for k in headers])
skips = {}
if 'host' in header_names:
    skips['skip_host'] = 1
if 'accept-encoding' in header_names:
    skips['skip_accept_encoding'] = 1

self.putrequest(method, url, **skips)

if body is not None and ('content-length' not in header_names):
    self._set_content_length(body)
for hdr, value in headers.items():
    self.putheader(hdr, value)
if isinstance(body, str):
    # RFC 2616 Section 3.7.1 says that text default has a
    # default charset of iso-8859-1.
    body = body.encode('iso-8859-1')
self.endheaders(body)

์ด์ œ ISO-8859-1์€ ์šฐ๋ฆฌ๊ฐ€ ๋ฌธ์ œ๋ฅผ ๊ฒช๊ณ  ์žˆ๋Š” ์ฝ”๋ฑ์ธ Latin-1์˜ ๋ณ„์นญ์ž…๋‹ˆ๋‹ค. ์šฐ๋ฆฌ๊ฐ€ ๊ฐ€์ง„ ๋ฌธ์ œ๋Š” Sublime Apex๊ฐ€ ์š”์ฒญ์— ์œ ๋‹ˆ์ฝ”๋“œ ๋ฌธ์ž์—ด ๋ณธ๋ฌธ์„ ์ œ๊ณตํ•˜๊ณ  ์žˆ๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. ์ด ๋ณธ๋ฌธ์€ httplib๊ฐ€ ๋ฐ”์ดํŠธ๋กœ ์ธ์ฝ”๋”ฉํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. RFC 2616์˜ ๊ธฐ๋ณธ๊ฐ’์„ ์‚ฌ์šฉํ•˜๋ฉด ํ•œ์ž๋ฅผ ํฌํ•จํ•˜์ง€ ์•Š๋Š” Latin-1์„ ์›ํ•œ๋‹ค๋Š” ๊ฒฐ๋ก ์ด ๋‚˜์˜ต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋ฉด ๋ถ„๋ช…ํžˆ ์ธ์ฝ”๋”ฉ์ด ์‹คํŒจํ•˜๊ณ  ๋ฌธ์ œ์˜ ์˜ˆ์™ธ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

Sublime Apex๊ฐ€ ๋ณด๋‚ด๋Š” ํ—ค๋”์—์„œ UTF-8 ์ธ์ฝ”๋”ฉ ๋ฐ์ดํ„ฐ(ํ˜„์žฌ๋Š” ๊ฑฐ์ง“๋ง์ž„)๋ฅผ ์ „์†กํ•œ๋‹ค๊ณ  ์ฃผ์žฅํ•œ๋‹ค๋Š” ์ ์„ ๊ณ ๋ คํ•˜๋ฉด Sublime Apex๋Š” ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด๋‚ด๊ธฐ ์ „์— ๋ฐ์ดํ„ฐ๋ฅผ UTF-8๋กœ ์ธ์ฝ”๋”ฉํ•˜๊ธฐ๋ฅผ ์›ํ•ฉ๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด๋‚ด๋Š” ๋ชจ๋“  ๋ผ์ธ(์ด ๊ฒฝ์šฐ salesforce/api.py 545๋ฒˆ ๋ผ์ธ)์ด ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ฝ์–ด์•ผ ํ•จ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค.

response = requests.post(self.apex_url, soap_body.encode('utf-8'), verify=False, headers=headers)

๋‚ด ์ง„๋‹จ์„ ํ™•์ธํ•˜๋ ค๋Š” ๋‹ค๋ฅธ ์‚ฌ๋žŒ์„ ์œ„ํ•ด ๋‹ค์Œ์€ ๋ฌธ์ œ๋ฅผ ํ™•์ธํ•˜๋Š” ๊ฐ„๋‹จํ•œ ์ƒ˜ํ”Œ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค.

a = "\u13E0\u19E0\u1320"
a.encode('latin1')  # Throws UnicodeEncodeError, proves that this can't be expressed in ISO-8859-1.
a.encode('utf-8')  # Totally fine.
r = requests.post('http://httpbin.org/post', data=a)  # Using unicode string, throws UnicodeEncodeError blaming Latin1.
r = requests.post('http://httpbin.org/post', data=a.encode('utf-8'))  # Works fine.

์ด ๋ฌธ์ œ๋ฅผ ์ œ๊ธฐํ•ด ์ฃผ์…”์„œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ ์ด๊ฒƒ์€ ์š”์ฒญ ๋ฒ„๊ทธ๊ฐ€ ์•„๋‹™๋‹ˆ๋‹ค. =)

๊ฐ์‚ฌ ํ•ด์š”.

r = requests.post(' http://httpbin.org/post ', data=a.encode('utf-8'))
๋งค์šฐ ์œ ์šฉํ•œ,
๊ฐ์‚ฌ ํ•ด์š”!

์ด ํŽ˜์ด์ง€๊ฐ€ ๋„์›€์ด ๋˜์—ˆ๋‚˜์š”?
0 / 5 - 0 ๋“ฑ๊ธ‰