Request: File upload works only with standard read streams

I experience the same problem when I try to wrap a Buffer in a read stream to upload it as a valid multipart request.

I have also been trying to find a way around this issue for some time and have experienced the same issues mentioned above. Has anyone found a workaround for this? Perhaps this is actually a bug with the form-data package?

Same problem here, I have a base64 string and I need to send it as a file. I convert my base64 to a buffer and then, just like @philipp-spiess I wrap it in a read stream, but it doesn't work. Did you guys find a work around ?

update: for now my workaround is using bhttp (https://www.npmjs.com/package/bhttp) which uses streams2

I've managed to upload my file using the buffer directly. You just need to specify file-related information (file name, type, ...)

var formData = {
  'file': {
    value: buffer,
    options: {
      filename: file.filename,
      contentType: file.mimetype
    }
  }
}

https://github.com/form-data/form-data#alternative-submission-methods

@PierreCavalet, I tried this way but unfortunately it didn't work.

My "workaround" was simply not to use request library and implement multipart POST using nodejs http service.

@PierreCavalet Is your buffer already encoded text? For me the buffer happens to be a binary blob which broke the encoding of the request (since it was apparently not properly encoded).

Edit: Apparently no special encoding is necessary, maybe just a wrong bit offset?! Anyway I recall that the error was the server complaining about an invalid byte.

@philipp-spiess my buffer is created with:

Buffer.from(base64string, 'base64')

So yes it is a binary blob. The documentation says that it creates a new Buffer containing the given JavaScript string. If provided, the encoding parameter identifies the character encoding of string.

@PierreCavalet The Buffer I was using was straight from multer, a file upload middleware for express.

@philipp-spiess This is weird. My base64 string is coming from a base64 encoding of a buffer, and this buffer is straight from my file upload. (The encoding is done by an other service, thats why we "encode and decode" for "nothing"). So it should work for you too.

@samkelly thank you so much for your recommendation of using bhttp. It works perfectly!

@bchr02 np!

Tracked this error down: turns out to be an issue with form-data's _length_retriever function. If you have a stream that isn't one of the 3 fully understood streams (basically files with an fd field, HTTP responses or request output streams) you basically have to lie to form-data to get it to do the right thing (and you need to have the precise byte length of the stream). To get it to work, I ended up doing this:

var buffer = Buffer.from(base64string, 'base64');
buffer['httpVersion'] = true;
buffer.headers = { 'content-length': 64 }
var formData = {
  'file': {
    value: buffer,
    options: {
      filename: file.filename,
      contentType: file.mimetype
    }
  }
}

Very important: you MUST specify the filename field in the options, otherwise you get another exception.

If you can specify the file options, you should be able to specify options.knownLength with the byte length of your file stream and it will get picked up by FormData.

It's mentioned in FormData's readme though not really publicized much. Request does not mention it anywhere right now so far I can tell.

Ah, that worked: teach me to do any debugging at 2am. I could have sworn I tried knownLength and got an exception, but that may have been the lack of the filename field causing chaos.

this doesn't work for my situation, because I am uploading something that is part of a gzip stream, therefore I do not know the length.

@sam0x17 Indeed it won't as, so far as I can tell, request doesn't currently support transfer-encoding: chunked, thus your use of bhttp.

See also

• https://github.com/request/request/issues/2671
• https://github.com/request/request/issues/2380
• https://github.com/request/request/issues/2488

mark

I faced the same problem, here is my workaround, using directly http module.

const streamSample = fs.createReadStream('data.csv').pipe(passThrough);

const formHTTP = new FormData();
formHTTP.append('sampleFieldText', 'text-sample');
formHTTP.append('myFile', streamSample);

const requestToSend = http.request({
    method: 'post',
    host: 'localhost',
    path: '/http',
    headers: formHTTP.getHeaders()
});

formHTTP.pipe(requestToSend);

requestToSend.on('response', (res) => {
    console.log('HTTP response', res.statusCode);
});

requestToSend.on('error', (e) => {
    console.log('ERROR HTTP', e);
});

@BenjD90
I used your workaround, but it seems that the stream doesn't get sent at all. I am using multer on the receiving end, if it is of any relevance.

@Ncifra I'm sure my workaround works, because I use it everyday on my current project.
You can test your stream on https://beeceptor.com/ ;)

I actually solved this by following: https://github.com/form-data/form-data/issues/356#issue-234978025
Here is my original issue related to the problem: https://github.com/form-data/form-data/issues/409

@BenjD90 Currently I reverted the code and can't test it since it was a very delayed feature on the project I am working. Maybe I was missing something since all that testing had created too much sparsed code. I did log the requests though, and the form-data object seemed to have the file in it, but multer didn't seem to receive it, while the other non binary data were being sent correctly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale This bug is steel present and should be fixed on day ^^

Yes agreed, it's as important now as it was back then. I've basically been waiting for this to be fixed before I use request again rather than bhttp. This is a basic feature of the streams and streams2 API and should be fully supported by request.

See https://github.com/form-data/form-data/pull/397#issuecomment-471976669 and https://github.com/bitinn/node-fetch/pull/707#issuecomment-569850948