Node-redis: Serialization of null values in a hash

I realized that node-redis does not do any data type conversion for string values retrieved from redis. I had implicitly assumed that booleans, numbers etc. would be auto converted into respective javascript type. If you do not consider this to be within the scope of the project, please close the issue.

But auto conversion will be nice to have, nevertheless. I just spent a significant amount of time debugging a bug arising due to a false value being retrieved as "false" and passing through javascript's truth checking conditions.

Right, the true/false one is the one that gets me, too.

I actually would like to address this in the library, though there are definitely performance concerns.

@brycebaril we could introduce a flag to JSON.stringify/parse the input. That should be a good solution, don't you think so?

Maybe there can be an option in the createClient constructor. Something like...
_parse_bool_: converts "true" to true, "false" to false
_parse_null_: converts "null" to null

But then how would you deal with the actual string value of "true" and "null"? There's no way to detect what the original input type was.

@Azmisov as @dirkbonhomme pointed out this would not be a good solution. Therefor only using JSON.stringify/parse might be a option.

I'd like to solve this not on a specific command base but for all commands. That way there's also no performance penalty, since we check every argument anyway and more importantly: it's solved for any command no matter how you pass the arguments (array, object, arguments).

I prefer to return an error if any command contains a undefined or null value. This is almost always a sign that something is wrong with the passed arguments.
Some people might want to have a way to handle undefined and or null values different though, so I'd add a option that you can pass a function to and this function is triggered if any command contains a null / undefined value.
That function always get's the command name and the arguments passed to it and the return value is going to replace the original argument(s). That way any user might handle these values as they want.
The same logic could be applied for true / false.

@NodeRedis/contributors @dirkbonhomme what do you think? :)

@BridgeAR I'd disagree that null is 'a sign that something is wrong'. Take something like this:

var myIndexes = ['one','two','three'], myMulti = client.multi();

myIndexes.forEach(function(aKey) { 
  myMulti.hget(aKey,'username'); 
  myMulti.hget(aKey,'address1');  
  myMulti.hget(aKey,'address2');
});

myMulti.exec(cb);

Say if 'address2' is optional in your data, then the null return is fine.

That being said, I think it would be a breaking change for many people so something like you are proposing should be optional, not default.

@stockholmux I'm not sure we speak about the same. The return value won't be changed by any means.

What I want to change is that using null will result in the string 'null' instead:

client.set('foo', null);
client.get('foo', function (err, res) {
    console.log(res); // 'null'
});

I won't break anything either, as I'm planing on deprecating the old behavior (some people might rely on the broken behavior by building around the issue) and adding the option to handle this on your own right now and return an error from v.3.0.0 for such values.

Does the 'null'value issue gets solved or not?

Would u define a special token in your program that represent the real null value. When user set something to null, you capture that command and replace the null value with your special token.When the user get that thing, you capture that command and replace the special token with real null value to the user?

Just a suggestion. this issue is really bothering. Because any user in any sites could have a name called 'null' , you wouldn't want to parse the username to the real null value;

Has any progress, or decision, been made on the "JSON.{parse,stringify} on input" flag? It would be nice to have this done automatically instead of in any given get/set logic.

Hey @taylorzane I have little time at the moment but it's definitely planned to implement that.

That's good to hear. If I have some time I may take a stab at it, it would definitely be nice to have, and I enjoy hacking at the source of modules I use. :+1:

Is this working now?

I ended up using a different method for serializing data into Redis, so I never got around to implementing this.

If you dont mind sharing @taylorzane how are you storing JSON objects with fields that may have null values inside using Redis

@slidenerd I'm flattening the JSON object, and then serializing each flattened node's value before storing it in Redis. So all values are strings. It's plenty performant even with thousands of values being de/serialized.

Example:

const a = {
  a: {
    b: {
      c: null,
      d: true,
      e: "hello"
    }
  }
}

const b = flatten(a)
console.log(b) /* =>
{
  '/a/b/c': null,
  '/a/b/d': true,
  '/a/b/e': 'hello'
}
*/

Object.entries(b).forEach(([key, value]) => {
  redis.set(key, JSON.stringify(value)) // or
  redis.hset('my_hash', key, JSON.stringify(value)) /*
  => "/a/b/c" -> "null"
  => "/a/b/d" -> "true"
  => "/a/b/e" -> "\"hello\""
  */
})

Feel free to reach out to me on Gitter or via the email in my profile, if you'd like more guidance.

@slidenerd The other option is to use ReJSON to store your JSON objects - its usually a better option that serializing everything manually and it can handle nulls and undefined. It's Redis Module, so you'll have to have a version that supports it (4.0+) and install the module into the server. It's beyond the scope of this Node.js module (and this issue), but it solves the problem.

            items.push(JSON.stringify(object, (key, value) => {
                if (value === null || value === undefined) {
                    return undefined
                }
                else {
                    return isNaN(parseFloat(value)) ? value : parseFloat(value)
                }
            }))

Thanks @stockholmux I came up with a better solution without using an external dependency, the idea is to use the replacer function in the JSON.stringify method which happens to be the second argument