Requests: `PUT` to Amazon S3 is failing

Created on 10 Jul 2015 · 4Comments · Source: psf/requests

I am trying to upload a file to Amazon S3 with Python Requests (Python is v2.7.9 and requests is v2.7). Following the curl command which works perfectly:

curl --request PUT --upload-file img.png https://mybucket-dev.s3.amazonaws.com/6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ

However when I tried to do same with requests, it fails with 403 with following error:

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

Here's what I tried:

url = https://mybucket-dev.s3.amazonaws.com/6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ
headers = {'Content-Length': '52369', 'Host': 'mybucket-dev.s3.amazonaws.com', 'Expect': '100-continue', 'Accept': '*/*', 'User-Agent': 'curl/7.37.1'}
payload={'Expires': '1436595966', 'AWSAccessKeyId': 'AQ', 'Signature': 'Ow%3D'}

requests.put(url, files={'file': base64_encoded_image})
requests.put(url, files={'upload_file': base64_encoded_image})
requests.put(url, files={'file': base64_encoded_image}, headers=headers)
requests.put(url, files={'file': base64_encoded_image}, headers=headers, data=payload)

They all fail, with same error. Here's curl in verbose mode:

* Hostname was NOT found in DNS cache
*   Trying 54.231.168.134...
* Connected to mybucket-dev.s3.amazonaws.com (54.231.168.134) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: *.s3.amazonaws.com
* Server certificate: VeriSign Class 3 Secure Server CA - G3
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> PUT /6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ HTTP/1.1
> User-Agent: curl/7.37.1
> Host: mybucket-dev.s3.amazonaws.com
> Accept: */*
> Content-Length: 52369
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< x-amz-id-2: 5lLCQ3FVrTBg2vkyk44E+MecQJb2OGiloO0+2pKePtxPgZptKECNlUyYN43sl4LBNe9f8idh/cc=
< x-amz-request-id: 636A24D53DEB5215
< Date: Fri, 10 Jul 2015 12:04:44 GMT
< ETag: "5802130d4320b56a72afe720e2c323a7"
< Content-Length: 0
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
<
* Connection #0 to host mybucket-dev.s3.amazonaws.com left intact

Source

avinassh

Most helpful comment

So, I made a random image and did

curl --request PUT --upload-file img.png https://httpbin.org/put

To verify that curl was doing exactly what I expected (which it does). It uploads the file as raw data.

The files= parameter in requests does multipart/form-data uploads, not raw data uploads. To replicate the same behaviour in requests all you need to do is the following:

with open('img.png', 'rb') as data:
    requests.put(url, data=data)

Requests will handle setting the content-length and everything else.

sigmavirus24 on 10 Jul 2015

👍3

All 4 comments

Well, step 1 is to avoid sending Expect: 100 Continue. We don't handle them properly. Please also don't send your own Content-Length header: requests will do that for you.

The real problem is that I think your upload is going to be multipart/form-encoded, but curl is uploading the file directly. Try: requests.put(url, data=open('img.png', 'rb'))

Lukasa on 10 Jul 2015

So, I made a random image and did

curl --request PUT --upload-file img.png https://httpbin.org/put

To verify that curl was doing exactly what I expected (which it does). It uploads the file as raw data.

The files= parameter in requests does multipart/form-data uploads, not raw data uploads. To replicate the same behaviour in requests all you need to do is the following:

with open('img.png', 'rb') as data:
    requests.put(url, data=data)

Requests will handle setting the content-length and everything else.

sigmavirus24 on 10 Jul 2015

👍3

works! Thank you very much guys :dancer: :smile:

avinassh on 10 Jul 2015

With using trying an zip file I get the following, any clues: (requests-2.13.0)

with open('default.zip', 'rb') as data:
    requests.put(url, data=data)

Output:

C:\svn\libraries\cpp>python req_put.py default.zip
MD5: 5dc0658d93e942fa7d1fa443e04bba83
SHA1: 18e7980d9d9ac544ac3e684a4d051932c3a3a336
Traceback (most recent call last):
  File "req_put.py", line 102, in <module>
    resp = requests.put(uri,  headers=headers, data=data) # data=data , params=payload)
  File "C:\Python27\lib\site-packages\requests\api.py", line 124, in put
    return request('put', url, data=data, **kwargs)
  File "C:\Python27\lib\site-packages\requests\api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 473, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(10054, 'An existing connection was forcibly closed by the remote host'))