Requests: Generate multipart posts without a file

This has been discussed before. It would represent a significant change to the API which I'm not sure @kennethreitz would like.

Personally, I would be more in favor of exposing a function to generate multipart data from dictioniaries (lists of tuples, etc) in the API so users can use that and just pass the generated data to requests. Ostensibly, if they're not using files, there shouldn't be a huge memory hit, but even so, they already have one huge string in memory, the second won't really kill them and it would be their fault, not ours.

Perhaps @kennethreitz would be more amenable to the second solution. I don't think it fits in with requests' design philosophy either and would be extraordinarily bizarre given the rest of the API, but _shrug_ who knows.

Indeed, current API does not support sending multipart data other than files and this is a bad thing. See issue #935 and shazow/urllib3/issues/120

Maybe I'm missing something, but the changes seem rather minor to me. I forked the repository and have a proposed change in my version here: https://github.com/spacecase/requests/commit/45b0b3ce1e76b241b323570a5fc88ae2089c3c3d
(if there's a better way to do this let me know? I'm rather new to github).

It might need a couple unittests and would need a change in a docstring in api.py, but is something like this reasonable?

The issue with the change isn't that it's complicated to implement, it's that it's a divergence in the API. I'm kind of on the fence in this ongoing discussion: I think it would probably be useful to have a good way to upload multipart form-data, but I also think the current files API is a very good one. I _don't_ think that adding multipart to the Request API is the way to go though.

Honestly, I would personally rather control it through the content type header but that would likely be 100x more error prone and confusing to new users than what we currently do. I'm on the fence about this as well, but I would still err on the side of not doing this.

Why? If we accept this feature request, then it becomes more likely that someone will complain about not having a json parameter. And I'm sure other people could come up with even more parameters they'd love to see added. As it is now, the API does exactly what it should and has little to no kruft. One way of looking at this is KISS. This makes the requests and returns a great object that makes using the response easy and natural. It does what is advertised and you do the rest. It does advertise multipart encoding and does it through its documented design. It may look awkward but it is documented and it works.

_(...) someone will complain about not having a json parameter_

There would be no ground for such a complain because json is a subtype of application media type (rfc4627) not multipart media type (httpbis draft 21)

_It may look awkward (...)_

It is awkward whereas it should not be.

After reading previous comments I'd like to reiterate my position: multipart/form-data is the most common multipart MIME type currently used (citation needed :)) and not supporting it is a gross omission.

@piotr-dobrogost: Despite what I said above, I don't find the argument you just made even slightly compelling.

Requests does not support MIME types, it supports use cases. This viewpoint makes both your comments above seem weird. For instance, the complaint about not having a JSON parameter will be because uploading JSON-formatted data is very common - probably more common amongst users of Requests than uploading non-file multipart data. Arguing that we won't provide it because 'we only special-case subtypes of multipart' just seems like a bizarre thing to say.

Regardless, the core of the issue is this: the API is the whole point of this library. If you can't come up with a _beautiful_ way to implement this functionality, it will not happen.

There would be no ground for such a complain [sic]

@piotr-dobrogost you have more hope for the future of developers than I do. Beyond that, the requested parameter is inexact. It would need to be called something more like form_data than multipart. As you note, (although indirectly) multipart could refer to plenty of different media types.

It is awkward whereas it should not be.

Not all things can be elegant (even in python).

and not supporting it

But it is supported. But that aside, we have data for application/x-www-form-urlencoded, files (or files+data) for multipart/form-data, why do we need yet another parameter for just multipart/form-data when we have it?

You don't need to use a combination of data and files to get the intended result. You can do something akin to: requests.post('http://example.com/', files=[('key1', 'param1'), ('key2', 'param2')]) without having an actual file there.

And if we were to be exact, form_data might not be entirely obvious, so why not use the parameter multipart_form_data, but now that's clumsy as well. It's explicit, yes, and PEP 8 calls for explicity, but the existing behaviour is well documented. If @kennethreitz does decide to accept this feature request, all the parameter will need to do is act as an alias to the files parameter. But considering the behaviour is already supported I don't think it is necessary.

@sigmavirus24 and @Lukasa summed this up perfectly.

@piotr-dobrogost your contributions are appreciated, but not your tone. Please stop making firm assertions against our project and it's goals.

From my perspective it is quite frustrating that requests already has support for multi-part posts, but does not give the user access to that without using a file. @sigmavirus24 's suggestion to just stick a file in there is not adequate. The web applications I work with will return error codes if something like this is tried.

I suspect this will continue to be a problem for users in the future, and I am a little confused why there doesn't seem to be an effort to address this.

See my response to #935

Oh, thanks. I look forward to it!

@Lukasa

_Requests does not support MIME types, it supports use cases._

Sending multipart/form-data data is a common use case.

_(...) uploading JSON-formatted data is very common (...)_

We can't compare sending json with sending multipart/form-data. Sending json is easy; you set Content-type, encode data in one line using built-in module and that's it. Sending multipart/form-data is more complex because the request's body has to have specific structure. In other words sending json is kind of transparent as much as HTTP is concerned but sending multipart/form-data is not. As Requests is the HTTP library it should take care of creating such a structure.

_If you can't come up with a beautiful way to implement this functionality, it will not happen._

Having current files param to send multipart/form-data which has NOTHING to do with files is not beautiful in any way (it's ugly), yet it somehow managed to get into codebase :). Coming up with something less ugly is really easy :)

@sigmavirus24

_(...) but the existing behaviour is well documented._

No amount of documentation makes good API out of bad API. The better the API the less it needs documentation.

_You can do something akin to (...)_

Right, but this is very misleading and unintuitive. I described what's wrong with using files param for this in my previous comment.

Bottom line: to come up with something better we need to admit the current api is bad in respect to sending multipart/form-data data.

@spacecase

_From my perspective it is quite frustrating that requests already has support for multi-part posts, but does not give the user access to that without using a file_

I agree and that's why I created issue #935.

This issue is closed.

Forgive me @kennethreitz but @spacecase I did not use a file anywhere in that example. I used the files parameter to do exactly what you want. Had I used a file you would have seen open('filename').

@sigmavirus24, Unfortunately that is not what I want. It's not a matter of whether the client is reading from a file or not. It's what is being told to the server. In your example, the post body is

Content-Disposition: form-data; name="key1"; filename="key1"
Content-Type: application/octet-stream

param1
--2f8732ee35564115a6c6e0c1032773e8
Content-Disposition: form-data; name="key2"; filename="key2"
Content-Type: application/octet-stream

param2
--2f8732ee35564115a6c6e0c1032773e8--

Note the use of filename=. It's telling the server a file is being sent. In the web apps I work with this incorrect behavior results in an error.

I'm sorry, but for you to be so presumptuous to tell me that is exactly what I want offends me. I came to offer ideas and help, and it sounds like you don't want either in this case. That is fine, but please don't make me feel talked down to.

Hm, seems I remembered the behavior incorrectly. Sorry about that. It wasn't my intention to offend you at all or make you feel talked down to. That's definitely enough to make me sway to needing a better way to handle multipart/form-data, but for now I still disagree that the ideas for the API are not at all elegant.

@spacecase please see my response to #935. Things will be better. Your feedback is important and much appreciated. :)

@spacecase as a gesture to show that your opinion does matter, check out sigmavirus24/requests-data-schemes as a stop-gap measure. You'll have to set your own Content-Type header, but that may be a minor annoyance.

Thanks @sigmavirus24 , I'll try it out. It looks like it will do the trick.
I appreciate that you didn't mean to offend me. I feel better about that and I don't hold anything against you or the project.

Yeah, it seems I come off as brash to some people, so I guess I need to refine what I write on the internet. I don't understand it and others have agreed with me, but I'm working on it. I guess I just need a sample size large enough to realize what's offensive to people without my intending it to be (besides my making an ass of myself by remembering something working a way in which it doesn't).

I have a file and some key-values to post. So what is the correct way to send such a mutilpart-form request? I hope you can help me.

Content-Disposition: form-data; name="up"; filename="aa.PNG"
Content-Type: image/png

file data
---------------------------7dee5302248e
Content-Disposition: form-data; name="exp"


-----------------------------7dee5302248e
Content-Disposition: form-data; name="ptext"

text
-----------------------------7dee5302248e
Content-Disposition: form-data; name="board"

DV_Studio
-----------------------------7dee5302248e--

I tried this way but only got a 504 error.
myfile=[('file',open('bb.jpg')),('exp','python'),('ptext',''),('board','DV_Studio')]
r = requests.post(url,files=myfile)

@deerstalker for questions please use StackOverflow. To answer your question though, there is a project underway to address this problem sigmavirus24/requests-toolbelt

Also note that I've answered this question before on StackOverflow, as you can see here.

I have the same issue of generating multipart posts without a file. At the moment, it makes no difference if you do requests.post(url, data=data_dict) or requests.post(url, data=data_dict, files={}). But, since the files keyword defaults to None, we should be able to have two distinct behaviours. A multipart/form-data when files={} is specified, and application/x-www-form-urlencoded when not.

Did I miss something?

@jwoillez Did you follow the stack overflow link I posted?

I think so, but your answer over there deals with Multipart POST with one file. I went back to the original issue: Multipart POST with no file.

The file object is allowed to be a string. =) That should provide you with the information you wanted.

But if I follow your suggestion, won't I end-up with something like this:

Content-Disposition: form-data; name="file"; filename="filename.txt"
Content-Type: text/plain

content
--3eeaadbfda0441b8be821bbed2962e4d--

where content is the string that you invite me to use instead of the file?

I'm really after this only:

Content-Disposition: form-data; name="key1"

value1
--3eeaadbfda0441b8be821bbed2962e4d

The fields in the tuple you don't want can be left as the defaults:

files = {'name': ('', 'content')}

In the future, please direct your questions to StackOverflow. All the maintainers regularly keep track of it, and it's the more appropriate place to ask these questions.

That's the answer I was looking for, thanks. Sorry for the noise.

Maybe one last question, is the following possible (specified empty filename, empty content)?

Content-Disposition: form-data; name="file"; filename=""
Content-Type: text/plain


--3eeaadbfda0441b8be821bbed2962e4d--

You can provide an empty content by using an empty string in the content section of the tuple. You can't provide a literal empty filename, but a non-present filename should be treated in exactly the same way.

I've occasionally needed to do this for various odd reasons.
I'd suggest this approach for anyone wanting to strong-arm the API into this use case:

class ForceMultipartDict(dict):
    def __bool__(self):
        return True


FORCE_MULTIPART = ForceMultipartDict()  # An empty dict that boolean-evaluates as `True`.


client.post("/", data={"some": "data"}, files=FORCE_MULTIPART)

Or you can use the toolbelt and not resort to hacks.