Request: JSON.parse failing because of BOM char

Created on 5 Jan 2018  ·  3Comments  ·  Source: request/request

Summary

BOM not stripped from response body when body filled with buffer

Simplest Example to Reproduce

    request({
        url: url,
        json: true,
        strictSSL: false,
    }, (error, response, data) => {
        if (error) {
            throw error;
        }

        if (response.statusCode === 200) {
            // data is string instead of json deserialized object 
        }
    })

Expected Behavior

BOM char stripped and json response deserialized

Current Behavior

JSON.parse fails and a string is returned

Possible Solution

    if (bufferLength) {
      debug('has body', self.uri.href, bufferLength)
      response.body = Buffer.concat(buffers, bufferLength)
      if (self.encoding !== null) {
        response.body = response.body.toString(self.encoding)
      }
      // `buffer` is defined in the parent scope and used in a closure it exists for the life of the Request.
      // This can lead to leaky behavior if the user retains a reference to the request object.
      buffers = []
      bufferLength = 0
    } else if (strings.length) {
      // The UTF8 BOM [0xEF,0xBB,0xBF] is converted to [0xFE,0xFF] in the JS UTC16/UCS2 representation.
      // Strip this value out when the encoding is set to 'utf8', as upstream consumers won't expect it and it breaks JSON.parse().
      if (self.encoding === 'utf8' && strings[0].length > 0 && strings[0][0] === '\uFEFF') {
        strings[0] = strings[0].substring(1)
      }
      response.body = strings.join('')
    }

As we can see the BOM char is stripped when strings are used instead of buffer
Extract the code outside of the else if (strings.length) to always clean the response body.

Context

Reading json response from a company custom swagger based on swashbuckle

Your Environment

| software | version
| ---------------- | -------
| request | 2.83.0
| node | 9.3.0
| npm | 5.5.1
| Operating System | Windows 10

stale

Most helpful comment

If we set encoding: 'utf8' in the request's options, the response is parsed

All 3 comments

If we set encoding: 'utf8' in the request's options, the response is parsed

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I am seeing this issue too - thank you for posting the workaround - would be nice if the library handled this though.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

mlegenhausen picture mlegenhausen  ·  4Comments

ghost picture ghost  ·  3Comments

ghost picture ghost  ·  3Comments

svlungade picture svlungade  ·  3Comments

pixarfilmz112 picture pixarfilmz112  ·  3Comments