Libelektra: yajl: Elektra's boolean not supported

@kodebach is this still possible that the type plugin can be reconfigured to normalize to "true" instead of "1".

No this is not supported by the type plugin, I didn't know there was a use case for this.

IMO it also doesn't make sense. The Elektra-way of representing a boolean is 0 and 1. If a storage format supports types, the conversion from the Elektra-representation to the representation of the storage format, should be done by the storage plugin. In the end the storage plugin for format X should be the bridge between Elektra and format X.

Also isn't the bigger problem the restoring of values? Restoring is done in presetstorage, and you explicitly requested, that values are always restored to the representation chosen by the user, when setting the value. This means, if you used kdb set user/tests/yajl on in your example, the yajl plugin will receive the value on.

What we actually need here, is a way to set the representation to which values are restored, regardless of how the user set the value. This could be added very easily. However I am not sure there is currently a way to specify the config for a plugin, from another plugin. Meaning the user would still have to configure type correctly when using yajl.

that values are always restored to the representation chosen by the user

This would not be a problem as in JSON the only available presentation is true/false, so the user cannot choose.

Meaning the user would still have to configure type correctly when using yajl.

This would also no be no problem because yajl can add a config/needs to reconfigure the type plugin.

The Elektra-way of representing a boolean is 0 and 1. If a storage format supports types, the conversion from the Elektra-representation to the representation of the storage format, should be done by the storage plugin.

Yes, I fully agree. The advantage of your approach is that it will also allow intermediate plugins (between type and storage) to see the correct representation of booleans.

I updated the "Implementation Hint" above to reflect this.

@kodebach another question: JSON only supports double/boolean/string. Is it possible to say to the type plugin that only these 3 types are allowed?

This would also fix the JSON types mentioned in #1092.

This would not be a problem as in JSON the only available presentation is true/false, so the user cannot choose.

They cannot choose in JSON, but they can choose when using kdb set

between type and storage

The ordering should be like this: getstorage, type, [other], type, setstorage, Meaning there shouldn't ever be any plugins between type and storage.

I updated the "Implementation Hint" above to reflect this.

There is still the problem that e.g. kdb set user/tests/yajl on will not work, without changes to type, because in this case type will pass the value on to the setstorage plugin.

JSON only supports double/boolean/string. Is it possible to say to the type plugin that only these 3 types are allowed?

It shouldn't be hard to restrict the allowed types via the config.

They cannot choose in JSON, but they can choose when using kdb set

What is chosen in kdb set cannot be remembered anyway if the file format does not support it.

There is still the problem that e.g. kdb set user/tests/yajl on will not work, without changes to type, because in this case type will pass the value on to the setstorage plugin.

Why does it not transform to "1" in this case?

It shouldn't be hard to restrict the allowed types via the config.

Maybe it does not even matter. Does it make a difference if the storage plugin or the type plugin says that a type is not allowed? It would be good if the storage tutorial would also say something about types (@sanssecours ?)

Does it make a difference if the storage plugin or the type plugin says that a type is not allowed?

Not that I know of. It would probably be even better to raise the error in the storage plugin, because then the user sees that the problem is the use of yajl not the use of type.

Why does it not transform to "1" in this case?

The normalization/restore procedure can be described by the following cases:

• Case 1: The Key existed in kdbGet and is unchanged between kdbGet and kdbSet
  This is the easiest and most obvious case. The value is normalized in kdbGet and the original value is restored in kdbSet, so that the underlying storage file remains unchanged (wrt the key in question)
• Case 2: The Key didn't exist in kdbGet
  Here we normalize the value to verify the type and then restore it immediately.
• Case 3: The Key existed in kdbGet, but its value was changed between kdbGet and kdbSet
  This is essential the same as Case 2. keySetString removes the origvalue metadata, so for the type plugin this kind of key didn't exist in kdbGet.

There is one special case. I already added functionality that might be used here (I forgot to add it to infos/metadata):

https://github.com/ElektraInitiative/libelektra/blob/6e609f7e78039db188e32d32d2ea13908c0abe38/src/plugins/type/README.md#L84-L88

If yajl injects the metadata check/boolean/true = true and check/boolean/false = false for all boolean keys, all the normalization and restoring should work as intended. The type plugin will then accept the values true, 1, false and 0 for boolean keys (in get and set), but it will always pass true or false to the setstorage plugin. The yajl plugin should still return 0/1 in get and it should accept true/false as well as 0/1 in set, so that it works with or without the type plugin.

If we choose to go down this way however, we have to document it very well, because mounting the type plugin will no longer allow to use different values for boolean keys in kdb set, because yajl secretly overrides any configuration given by the user.

Not that I know of. It would probably be even better to raise the error in the storage plugin, because then the user sees that the problem is the use of yajl not the use of type.

Exactly.

The normalization/restore procedure can be described by the following cases
[...] (I forgot to add it to infos/metadata)

Thank you for the detailed explanation. Can you add this to our documentation please?

will no longer allow to use different values for boolean keys in kdb set

With "config/needs" yajl can ensure that type is mounted using "check/boolean/true = true and check/boolean/false = false".

So should I change the implementation hint?

With "config/needs" yajl can ensure that type is mounted using "check/boolean/true = true and check/boolean/false = false".

You misunderstood, right now check/boolean/true and check/boolean/false has to be set as metadata on an individual key, not in the config of type. General support within the config would have to be added (easy enough).

Ok. No, then leave it as is. It makes more sense to implement everything in the yajl plugin.
I added as implementation hint:

The yajl plugin needs to:

• [ ] render Elektra's "0" and "1" values to JSON's false and true.
• [ ] fail with type errors if non-supported types are found

@sanssecours can you add this info to the storage plugin tutorial?

yajl also has to add the metadata check/boolean/true = true and check/boolean/false = false to each key with type = boolean. Otherwise the problem for kdb set /some/key on mentioned above will occur.

yajl also has to add the metadata

Is this also needed if yajl would always give you only "0" and "1" for booleans?

Yes, because the problem occurs, when the user changes the key value after kdbGet. yajl has no influence on that.

But nevermind that, we need changes to the type plugin anyway, because if the user adds a new key, the metadata will not be present and the solution won't work.

So we need the type plugin to be available twice in the kdbSet path so that the normalization will work properly? Can you create an issue?

No. With #2582 yajl (or the user) just needs to make sure that the config for type either contains

• no booleans array and boolean/restore = #1
  or
• has a booleans array which contains "true" and "false" at position #X and boolean/restore = #X

I'll try to fix this, but have a question.

Shouldn't it be enough to set type=boolean and type/boolean/restoreas=none on a key that I parse as boolean in elektraYajlGet? Then in elektraYajlSet I should receive this key with a value of either 1 or 0, right?

But I'm still receiving the user-supplied value

# kdb mount conf.json user/tests/yajl yajl type
# kdb set user/tests/yajl 1
Set string to "1"
# kdb setmeta user/tests/yajl type boolean
# kdb set user/tests/yajl false
Sorry, 1 warning was issued ;(
    Sorry, module yajl issued the warning C03200:
    Validation Semantic: Got boolean which is neither true nor false
Set string to "false"

Case 2: The Key didn't exist in `kdbGet`
  Here we normalize the value to verify the type and then restore it immediately.

@kodebach
Is it possible that in this case the value is always restored even if type/boolean/restoreas=none?

/boolean/restoreas is not a metakey for individual keys. It is part of the config for the instance of the type plugin used for the whole mountpoint.

I think it should be enough to add config/needs = type/boolean/restoreas=none to the header of src/pluigns/yajl/README.md. (At least for mounting via kdb mount)

Adding - infos/config/needs = type/boolean/restoreas=none seems to be a compiler error.

/elektra/build/src/plugins/yajl/readme_yajl.c:11:55: error: expected ‘)’ before ‘keyNew’
 "- infos/config/needs = type/boolean/restoreas=none\n"
                                                       ^
                                                       )

Can I programmatically set the key too? I can't find documentation on how to configure another plugin.

I think it should just be config/need not infos/config/needs. I can't verify it right now though.

The header of the README is turned into lines of keyNew, which you then include into the plugins get method. You could manually add stuff there, using the README is preferred, since it automatically provides documentation.

Can I programmatically set the key too?

Yep, just add it to the contract key set. For example, the following line shows how YAML CPP configures the type plugin:

https://github.com/ElektraInitiative/libelektra/blob/5519cb8066a096215a3701ca3d8c02fcebe54914/src/plugins/yamlcpp/yamlcpp.cpp#L44
.

Thank you both!

With #3012, the yajl plugin has the following behavior.

With the type plugin

kdb mount conf.json user/tests/yajl yajl type
kdb set user/tests/yajl 1
kdb get user/tests/yajl
#> 1
kdb setmeta user/tests/yajl type boolean
kdb set user/tests/yajl on
kdb get user/tests/yajl
#> 1
kdb set user/tests/yajl/subkey disable
kdb setmeta user/tests/yajl/subkey type boolean
kdb get user/tests/yajl/subkey
#> 0

cat `kdb file user/tests/yajl`
{
    "___dirdata": true,
    "subkey": true
}

Without the type plugin

The yajl plugin should still return 0/1 in get and it should accept true/false as well as 0/1 in set, so that it works with or without the type plugin.

kdb mount conf.json user/tests/yajl yajl
kdb set user/tests/yajl 1
kdb getmeta user/tests/yajl type
#> boolean
kdb set user/tests/yajl false
kdb getmeta user/tests/yajl type
#> boolean
kdb get user/tests/yajl
#> 0

# Without the type plugin, 'on' is mapped to a string and a warning is emitted.
kdb set user/tests/yajl on
#> RET: 2

* fail with type errors if non-supported types are found

Does that refer to when the type plugin is mounted or without?

In this last case, the behavior until now was that a warning is emitted.

 Sorry, 1 warning was issued ;(
    Sorry, module yajl issued the warning C03200:
    Validation Semantic: Got boolean which is neither 1 or true nor 0 or false

I think this is still fine, since without the type plugin, there shouldn't be type checking.

Actually, it should warn (or even fail) about anything except "1" or "0". Also "true", "false" are not Elektra's booleans.

Imho, the yajl plugin should have a "require" dependency to the type plugin. (but first we also need to fix the "number", as it is not one of Elektra's types).

Actually, it should warn (or even fail) about anything except "1" or "0". Also "true", "false" are not Elektra's booleans.

@kodebach wrote

The yajl plugin should still return 0/1 in get and it should accept true/false as well as 0/1 in set, so that it works with or without the type plugin.

That's what I also allow "true" and "false".

Imho, the yajl plugin should have a "require" dependency to the type plugin. (but first we also need to fix the "number", as it is not one of Elektra's types).

Yes, I totally agree about the dependency. Then we can remove support for "true" and "false".

Regarding the number problem: Yajl maps the Number type to double, which is fine, I think. And from quickly checking, the type plugin's double type also support json's E-notation (i.e. 3.4e2).
What do you see as the problem?

What do you see as the problem?

Ahh, I now see that src/plugins/yajl/testmod_yajl.c 240-251 is commented out. This should be removed. (There is still an occurrence of number.)