Request: Can't pipe from request that has a POST body

Try with this one:

var express = require('express')
var request = require('request')

var app = express()

// put it before any other middleware
app.use(function(req, res) {
  req.pipe(
    request[req.method.toLowerCase()]('http://example.com'))
  .pipe(res)
})

app.listen(3000, function () {
  console.log('Express server listening on port 3000')
})

In case you want to modify the body, just construct a new request and pipe to the response.

@simov I don't want to modify the body, just the URL. So I want to keep the headers and other properties of the original request but pipe into a new request with a new URL. The problem is that the body is not copied over when the original request is piped into the new one (not sure if this is a bug), and if I try to copy the body over myself by specify the form property in the request constructor, I get the error mentioned above.

When you say "construct a new request" are you suggesting that I simply create a new request from scratch with the body I want and copy over the headers from the original request myself, rather than piping them in? I can do that but it would be less repetitive and fragile if I could use the built-in pipe support to copy the original request in the new one.

I still think one or both of these two issues is probably a bug (i.e. 1) the request body is not copied when piping and 2) the exception mentioned in my originak post).

I don't know too much about express or request (just using these for a single-page app proxy without really understanding how they work), but this fixed the write after end error for me:

request.post({ url: 'http://www.example.com', form: { foo: 'bar' }}).pipe(res);

Instead of:

req.pipe(request.post({ url: 'http://www.example.com', form: { foo: 'bar' }})).pipe(res);

@matthewgertner just try out my code example, it works.

@simov It definitely works, but your example doesn't cover the case where req has a body that I want to copy over. I was actually using code almost identical to your example, but I got stuck when I had to deal with POST requests since the body doesn't get copied to the new request object.

@matthewgertner can you give me your _exact_ code example that doesn't work?

@simov It's the sample in my original post. If I use this one then the body is not sent:

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post('http://www.example.com'));
});

app.listen(process.env.PORT || 3000);

If I use this then I get that exception:

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post({ url: 'http://www.example.com', form: { foo: 'bar' }}));
});

app.listen(process.env.PORT || 3000);

Can you replace http://www.example.com with some generated URL from http://requestb.in/ and paste your link here? It should look something like this http://requestb.in/1mv5l8h1?inspect

Or you can use my example code from above as well, because that's what I'm using for server. Then I'm executing this script:

var request = require('request')
request.post('http://localhost:3000', {form:{some:'data'}})

As you can see in my link I have the form:{some:'data'} sent.

Let me explain my use case. I implemented a simple web API proxy (see https://github.com/salsita/web-api-proxy). All it does is receive HTTP requests and redirect them to another host, replacing URL parameters if necessary with environment variables on the proxy server. The point is to avoid having to embed secret keys into client code.

It works great for GET requests, but I recently had a requirement to use it for POST as well. So I'm not initializing the body of my request as in your previous comment since the body is coming through the Express request. Your most recent example definitely works but that doesn't really help me. What I need to do is:

req.pipe(request.post('http://some/new/url'));

It seems to me that if req has a body, it doesn't get copied over in this case. And anyway, the truth is that I _do_ need to modify the body (since it might have placeholders that I need to replace using environment variables) so I'm going to have to do:

req.pipe(request.post({ url: 'http://some/new/url', form: { some: 'form', data: 'here' }));

That causes an exception to be thrown due to what I suspect is a bug in request.

Anyway, I can build a completely new request object myself, but I wanted to make sure that I wasn't missing something and also to report this issue in case it is a bug.

It doesn't work because you are using a GET request always. Take a look at my code example again.

request[req.method.toLowerCase()]('http://example.com')) sets the request method based on the incoming request.

Besides that if you need to modify the body, again from my first comment:

In case you want to modify the body, just construct a new request and pipe to the response.

It doesn't work because you are using a GET request always. Take a look at my code example again.

But I'm using request.post. That's the same thing as request['post'].

In case you want to modify the body, just construct a new request and pipe to the response.

It's a bummer that I basically need to rewrite the code that's already in request to pipe requests into new requests (copying headers, etc.). There may be a bunch of edge cases that I need to deal with. But yeah, I guess this is what I'll end up doing.

IMO it should be possible to pipe a request into another request that has a form body without an exception being thrown, or do you think there is some reason for this behavior?

This is making GET request always because you're not specifying anything else explicitly.

The constructing of new object applies only when you want to modify the body.

Sure, the current codebase does not support POST. I'm trying to fix that, which is when I ran into the problems outlined here. I only referenced the existing codebase to explain the use case better.

I'm pretty sure I need to create a new object always, even if I don't want to modify the body, since the body is not copied over by the pipe operation. Did you try the example I provided? You need an Express request with a body that you pipe into a request object. You'll see that the body is undefined in the new object.

GET requests does not have a body, that's by design.

This one works (your example):

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post('http://www.example.com'));
});

app.listen(process.env.PORT || 3000);

Using this to make the request:

var request = require('request')
request.post('http://localhost:3000', {form:{some:'data'}})

some=data got sent.

You'll see that the body is undefined in the new object.

Where is that, I'm not following you.

Hmmm, ok. When I use requestb.in (nice tip!) I can see that the body actually _is_ sent. I actually had (and still have) two problems that led me to believe it was not being sent:

1) When I use cURL to send a request to my server (using the code in your previous comment), it hangs after sending the request (i.e. cURL does not terminate).
2) When I look at request.body it is undefined whereas this is not the case if I create the request using request.post with an explicit form property.

I guess that 2) is irrelevant and the body is simply stored somewhere else. I'd be interested to know if you can reproduce 1) though. If you send the request to localhost:3000 via cURL (curl --data "some=data" localhost:3000), does it terminate or hang?

1) The request hangs because in your example you are not returning anything back, I used it just to make sure we're on the same page. Just .pipe(res) to receive the response.
2) request.body is undefined when using the stream API

I actually used .pipe(res) in my real example and I verified that it doesn't work. It doesn't appear to send the request and it hangs. Since your example works, I quickly found the cause of the problem:

app.use(bodyParser.urlencoded({ extended: true }));

It looks to me like when I use body-parser, the piping no longer works. Do you agree and, if so, do you have any idea what the cause might be?

Yep, if you take a look at my first comment again:

// put it before any other middleware

body-parser processes the request before entering your middleware, but you need the raw request to pipe it successfully.

Phew! Okay, thanks _a ton_ for your help and patience. Took a while to get to the bottom of that. I still think it's kind of a bummer that I can't pipe in all the headers and stuff and then modify the body, but I guess there's no way around that.

@simov

app.use(function(req, res) {
  req.pipe(
    request[req.method.toLowerCase()]('http://example.com'))
  .pipe(res)
})

will fail for DELETE. as request.del is the method and not req.delete

I know this is old, but I had a hard time finding a suitable solution to this, and thought it might be useful to others.

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post({ url: 'http://www.example.com', form: { foo: 'bar' }}), {end: false}).pipe(res);
});

app.listen(process.env.PORT || 3000);

As shown in the https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options, adding this makes it so the new stream isn't automatically terminated when req is terminated.

While this caused this example to work, I've noticed some issues making a proxy for other methods using this. The body somehow causes it to close everything properly, but without a body it still hangs. If using a json body parser, this shouldn't cause a problem since it causes the body to be read as {} even when there is no body, but without one it could still have issues. My current solution is to check for a content-length header on the request, and use the options object if one exists.

@JeffreyAngell just try your resolution, it works. thx

This works for me:
`router.put('/', function (client_req, client_res, next) {
var json = JSON.stringify(client_req.body);
var options = createOptions('PUT', client_req.baseUrl, json);

var proxy = https.request(options, function (res) {
    client_res.statusCode = res.statusCode;
    res.pipe(client_res, {
        end: true
    });
});

client_req.pipe(proxy, {
    end: true
});

proxy.write(json);
proxy.end();

});

function createOptions(method, targetUrl, json) {
targetUrl = targetUrl.replace('jira/', '');
console.log("serving: " + targetUrl);

const auth = 'Basic ' + new Buffer('usr' + ':' + 'pass').toString('base64');
var headers = {
    'Authorization': auth,
    'Content-Type': 'application/json'
};

return {
    hostname: 'jira.acme.com',
    port: 443,
    path: targetUrl,
    method: method,
    headers: headers
};

}`

@simov

Yep, if you take a look at my first comment again:

// put it before any other middleware

body-parser processes the request before entering your middleware, but you need the raw request to pipe it successfully.

I experienced the same issue. Could you explain why it has to be before certain middleware in my case only the bodyParser middleware causes the error? It seems I can use other middleware e.g. user roles, sanitizing without any problems.

EDIT:
I just want to load my routes after all the middleware is loaded, so I went for this solution which excludes bodyParser from all routes with proxy in the beginning:

pathToRegexp = require('path-to-regexp')
exports.excludeMiddleware = (path, middleware) ->
  (req, res, next) ->
    if pathToRegexp(path).test req.path
      next()
    else
      middleware req, res, next

  app.use(routesService.excludeMiddleware '/proxy/(.*)', bodyParser.json({
    extended: false,
    parameterLimit: 10000,
    limit: 1024 * 1024 * 10
  }))

https://stackoverflow.com/questions/27117337/exclude-route-from-express-middleware