Libelektra: Library for types

I think it should be the exceptional way to form libraries based on proposals. It makes much more sense to extract common functionality between plugins and to make a library out of that. @bauhaus93's TOML plugin has tons of code that would make much of sense to be put into a library (e.g. the comment handling code).

Don't get me wrong, it is good input to @bauhaus93. And if there is a second plugin that needs the same functionality, we definitely should put it into a library. But making libraries is much more effort than having plugin-specific code and the effort should only be done if there are actually at least two customers.

@bauhaus93 if you do not see immediate benefit in having a type library (e.g. reuse with another plugin), please close the issue.

if there is a second plugin that needs the same functionality, we definitely should put it into a library

As long as the developer of the second plugin knows where to find the first plugin and how to extract the code from there without breaking everything that is a good strategy yes. Sadly, this is often not the case. Especially, since often the first plugin has to be changed extensively so that it can use a library.

Also having the library might actually encourage someone to write a new storage plugin, since some of the work is already done.

But I tagged this as low priority, exactly because I knew its a lot of work and there is probably nobody that wants to do it.

For types, I'm not sure about reusability, except for maybe the validation/conversion of non-decimal integers (binary/octal/hexadecimal) or datetimes (TOML uses RFC 3339 datetimes).
Unrelated to types, I see some reusability in the preparation of KeySets to-be-written (eg. functions that update/add array metakeys, remove array metakeys of invalid arrays or complete missing comment/#X metakey data).

You might not have noticed it yet, but there can also be problems with the normalization of booleans the type plugin does. This is probably the most reused code, since most formats will use true/false but Elektra uses 1/0. I think yamlcpp had to add some special code, so that booleans won't be converted into integers or vice-versa (or something along those lines).

Ah yeah, forgot about that, the TOML plugin also has to normalize these values.

The problem is not just, that you have to normalize the values. You also have to know exactly what the type plugin does, since it runs before the toml plugin in the kdbSet phase. I'm not sure, if we ever implemented it, but you should also ensure that the type plugin only ever produces 1/0 or true/false in the kdbSet phase, regardless of the user provided configuration. Otherwise toml would have to be able to parse the type plugin config, since the user can set custom true and false values (see test case below)

https://github.com/ElektraInitiative/libelektra/blob/4b0e9f8bdd6d890a1a3abaf04c543b9c7d33e984/src/plugins/type/testmod_type.c#L734-L771

The interesting part is:

https://github.com/ElektraInitiative/libelektra/blob/4b0e9f8bdd6d890a1a3abaf04c543b9c7d33e984/src/plugins/type/testmod_type.c#L749-L753

The type plugin receives a 1 in kdbGet (could be from toml), but returns a t in kdbSet. This could go as far as a user defining that true = 0 and false = 1 which would totally confuse the toml plugin. (I think all booleans would flip on every kdbSet)

This hugely complex interaction is why I think the type plugin should be for storage formats that don't have types and those that do have them should explicitly forbid the usage of type. But to make that feasible we need an easy way to support types within storage plugins, i.e. a type library.

Since the toml plugin also handles hexadecimal numbers, it should probably also explicitly disallow using the hexnumber plugin. (although in that case there is less potential for problems I think)

Also having the library might actually encourage someone to write a new storage plugin, since some of the work is already done.

Most users probably only want to write the grammar and the rest should magically happen by itself. I wonder how far @bauhaus93's TOML plugin actually achieved this goal. Would be interesting to make a non-TOML plugin with the code-base of the TOML plugin. A type library alone, however, seems to me much to specialized to be overly useful for the big task of "writing a storage plugin".

For types, I'm not sure about reusability, except for maybe the validation/conversion of non-decimal integers (binary/octal/hexadecimal) or datetimes (TOML uses RFC 3339 datetimes).

Date/times existed in Elektra only for validation but there only RFC2822 was supported. The date plugin could reuse your parser to validate RFC 3339 but I would consider this as very low priority.

What would be a little bit more exciting is something like keyGetDateTime(Key *k, struct tm *tm) from a key with a (TOML) date.

Unrelated to types, I see some reusability in the preparation of KeySets to-be-written (eg. functions that update/add array metakeys, remove array metakeys of invalid arrays or complete missing comment/#X metakey data).

Yes, that could be interesting. But even more fascinating would be the possibility to directly modify the grammar and touch code as little as possible :wink:

You might not have noticed it yet, but there can also be problems with the normalization of booleans the type plugin does.

What is missing is some tutorial/explanation in doc/tutorials/storage-plugins.md which gives an up-to-date view of which plugins a storage plugin actually should rely on. #2330 will show if "binary" can be needed and still being suitable as default storage.

You also have to know exactly what the type plugin does, since it runs before the toml plugin in the kdbSet phase.

Probably, storage plugins should stop using the type plugin and simply do their normalization simply themselves (from whatever is true/false in their format to 1/0 in Elektra and so on). @bauhaus93 which other functionality of the type plugin do you use?

This could go as far as a user defining that true = 0 and false = 1 which would totally confuse the toml plugin.

I would be in favor of reducing the functionality of the type plugin. If anything, some additional true/false values should be definable by the user (which obviously were not true/false values before).

Since the toml plugin also handles hexadecimal numbers, it should probably also explicitly disallow using the hexnumber plugin. (although in that case there is less potential for problems I think)

There is no way to express that plugins do not work together (Having such a functionality would make the mounting much more complicated. It would also be too much work for us to maintain a table of which plugin works with which plugin.). But probably nobody will have the idea to use them together, as the TOML plugin already has this functionality and more (e.g. also binary numbers).

Most users probably only want to write the grammar and the rest should magically happen by itself.

That seems a bit too magical to be possible. It might be possible with a custom code generator based on a modified Yacc or ANTLR grammar. But I don't think there is a way to have code that creates KeySets simply from a grammar for vastly different formats like JSON, XML, TOML and edn. As @'sanssecours found there also formats like YAML that are very hard to express in a standard grammar format.

A type library alone, however, seems to me much to specialized to be overly useful for the big task of "writing a storage plugin".

We could of course extend the library into a general storage library that provides more helper functions for storage plugins (e.g. dealing with stdin/stdout and pipes for import/export). The type stuff would then be a part of that.

I also think you underestimate how complicated the type stuff can be. If we had a standard type library, extending the type system would also be easier e.g. to store pre-parsed binary versions of integers in addition (to the string versions to reduce parsing overhead). A storage library could also use internal APIs (if needed), since it is maintained by the Elektra developers.

Another a standard type library would also guarantee standard error messages for type problems. So there would even be an advantage for end-users.

What would be a little bit more exciting is something like keyGetDateTime(Key *k, struct tm *tm)

Sounds like a job for the conversion part of libease (as elektraKeyToDateTime (const Key * key, struct tm * dateTime)).

@bauhaus93 the functions there might be useful for toml as well. They are designed such that e.g. elektraKeyToFloat and elektraFloatToString perfectly and losslessly roundtrip (no matter if you start with a float of a string) and they are also used in the highlevel API. So if toml creates the key with elektra*ToString it can definitely be read correctly by the highlevel API and toml can definitely correctly read whatever the highlevel API produced with elektraKeyTo*.

an up-to-date view of which plugins a storage plugin actually should rely on.

IMO, a storage plugin should never _rely_ on any other plugin. Storage plugins can be enhanced by other plugins, but they should ideally work standalone.

Even dealing with binary keys, would be better solved by a storage/type library. Again it guarantees some baseline standard for error messages etc. It also avoids use of the binary plugin for formats that don't need it. While binary combined with e.g. quickdump would work, it is bad for both speed and storage size.

Probably, storage plugins should stop using the type plugin and simply do their normalization simply themselves

Which is why I proposed a library. Its much better to provide a library that does the work than to just given an informal description (or even a formal specification) of what the plugin has to do. Especially, if the specification might change over time.

I would be in favor of reducing the functionality of the type plugin.

Why take away functionality that is already implemented? Originally, the idea was to have a separate boolean plugin, but that also caused problems, because of plugin ordering and such things.

If anything, some additional true/false values should be definable by the user (which obviously were not true/false values before).

There is really no need for that. There is only a problem, if more than one plugin defines what a boolean value is.

Even true = 0 and false = 1 are totally fine. Elektra itself fines booleans as "1 is the only true value and 0 is the only false value".

There is no way to express that plugins do not work together (Having such a functionality would make the mounting much more complicated. It would also be too much work for us to maintain a table of which plugin works with which plugin.).

I disagree. An exhaustive list would of course be impossible (after all there are also 3rd party plugins), but the solution is simple. Let the plugins do the check themselves. Either in elektra<Plugin>Get or in another function that is called during kdb mount.

Besides boolean, the TOML plugin also sets the string, double and (unsigned_)long_long types on reading.

I agree with @kodebach, that the type plugin should only be used by plugins without a built-in type system, because of difficulties in interaction.
Eg. the TOML plugin sets the type metakey for integers on reading also on non-decimal values (while converting it to decimal, storing the non-decimal representation in origvalue). However, if you want to change such a typed value with kdb set (and do that by binary/octal/hex value) you could not do that directly (by setting the key value), because the type plugin type checks would not be successful. You'd have to change the origvalue metakey instead. (Deleting the type metakey before setting a new value would not work, since the metakey will be re-set on reading.)

The problem is not just, that you have to normalize the values. You also have to know exactly what the type plugin does, since it runs before the toml plugin in the kdbSet phase. I'm not sure, if we ever implemented it, but you should also ensure that the type plugin only ever produces 1/0 or true/false in the kdbSet phase, regardless of the user provided configuration. Otherwise toml would have to be able to parse the type plugin config, since the user can set custom true and false values (see test case below)

Yeah I already wondered, if/how I actually can check for the user defined boolean values. Currently, on writing, the TOML plugin only considers values of "1" and "true" as true, the rest would be seen as false.

@kodebach wrote:

That seems a bit too magical to be possible.

With Augeas it is possible (but the trees you get are not so desirable). It would be interesting how far off we are with the current TOML solution. Of course you need to write some emitter code but this is usually not so dramatic.

JSON, XML

It is the advantage of Elektra that such vastly different formats can be implemented in different technologies. Trying to implement XML from scratch is probably not the best idea. If INI-like formats could be covered, it would already be amazing.

But of course the first and most important thing is to get a good TOML plugin :1st_place_medal:

Another a standard type library would also guarantee standard error messages for type problems.

The checking can be easily done in other plugins (once the #2963 is done). If a plugin only does checking and failing with a beautiful error message, there are little/no interactions.

Sounds like a job for the conversion part of libease

Yes, probably some of the stuff from TOML could go to libease or libmeta.

IMO, a storage plugin should never rely on any other plugin.

For binary I am not yet sure (as its probably something that is not needed for default backends), for all the other functionality, this is also my conclusion. Iirc this conclusion is not yet broadly accepted (@sanssecours?) nor documented, though.

Which is why I proposed a library. Its much better to provide a library that does the work than to just given an informal description (or even a formal specification) of what the plugin has to do. Especially, if the specification might change over time.

It depends: if there is a huge number of valid possibilities, a tutorial/description might be better than a complicated library that somehow tries to give this flexibility. And for serializations there is an immense range of valid possibilities and many of them have trivial implementations (e.g. simply calling elektraFormat with some parameters) up to complicated implementations (e.g. if they support many convenience representations).

Why take away functionality that is already implemented?

Elektra is currently collapsing on the weight of maintaining too much code. Every useless (in the meaning that nobody uses it) line of code we get rid of makes Elektra better.

Unfortunately, even code separated in plugins creates problems: e.g. on the build server or once a user tries to use it there are unwanted interactions/missing docu/...

We cannot release everything we currently have as 1.0, Elektra would be a disappointment then. We need to get rid of everything which is not needed. We appreciate every cleanup you can do.

That is also the reason why TOML will replace INI. #3491

Let the plugins do the check themselves.

This was rarely a good solution. It gets very inconsistent and the order of adding plugins might produce different results. Does such code already exist somewhere?

@bauhaus93 wrote:

Besides boolean, the TOML plugin also sets the string, double and (unsigned_)long_long types on reading.

But it all does it by itself without the type plugin? What is the type plugin used for?

You'd have to change the origvalue metakey instead.

Yes, here we do not have a nice solution yet (#3056). keySetString removes origvalue but somehow the behavior is still not as a user would expect.

Currently, on writing, the TOML plugin only considers values of "1" and "true" as true, the rest would be seen as false.

Probably we should be more strict and fail on everything not "0" or "1". In the TOML file itself you also only allow "true" and "false" and nothing else?

But it all does it by itself without the type plugin? What is the type plugin used for?

Yes, it does it by itself, the different types will get matched during lexing. However, it doesn't check if the decimal/double values over-/underflow long_long/double, so I think that is done by the typeplugin.

Probably we should be more strict and fail on everything not "0" or "1". In the TOML file itself you also only allow "true" and "false" and nothing else?

Yes, I can make that more strict. In the file itself only true or false is written/read as boolean.

for all the other functionality, this is also my conclusion.

In that case we need a type library. Otherwise any storage plugin for a format with types would have reimplement the type stuff (since relying on another plugin is out of the question).

if there is a huge number of valid possibilities, a tutorial/description might be better

But in that case a separate plugin also isn't the solution, because then there is also a huge number of possible interactions that need to be considered in both plugins.

Does such code already exist somewhere?

AFAIK it doesn't I'm not even sure a plugin could detect which other plugins are mounted.

But it all does it by itself without the type plugin? What is the type plugin used for?

Yes, it does it by itself, the different types will get matched during lexing. However, it doesn't check if the decimal/double values over-/underflow long_long/double, so I think that is done by the typeplugin.

Yes type also does range checking, validates float/double and some other stuff like enums.

Probably we should be more strict and fail on everything not "0" or "1". In the TOML file itself you also only allow "true" and "false" and nothing else?

And that is exactly one of these hugely complex and complicated cases for types or more specifically conversion/normalization.

Let's assume toml only accepts true/false as input and only produces 0/1 in kdbGet and it only accepts 0/1 and only produces true/false in kdbSet. That would comply with both the Elektra spec and the TOML spec for booleans. But a user would probably expect that kdb set /some/key/mounted/with/toml true would work. However, it does not. With the correct config for type it might work, but it quickly becomes awkward. For example, what if the key did exist before. Then there is no type metakey and the type plugin just ignores it, toml receives true and complains that true is not a valid boolean...

This just shows that a storage plugin MUST be able to handle all types that the storage format supports and the associated conversions WITHOUT using any other plugins.

Elektra is currently collapsing on the weight of maintaining too much code. Every useless (in the meaning that nobody uses it) line of code we get rid of makes Elektra better.

That is a fair point, although I don't think simply deleting code is the right solution. At least not when it comes to plugins. Within the core, I agree, the less LOC the better. For plugins, we can easily say that the plugin or some part of it is experimental.

IMO, a storage plugin should never rely on any other plugin.

Iirc this conclusion is not yet broadly accepted (@sanssecours?) nor documented, though.

As far as I know, there is no place in the documentation that states that a storage plugin should not rely on other plugins. I also do not think that is great from a separation of concerns point of view. Writing a good storage plugin is already quite a lot of work in my opinion. Requiring that a storage plugin takes care of type conversion, directory keys and binary data (Base64 encoding) does not make that job easier.

I also do not think that is great from a separation of concerns point of view. Writing a good storage plugin is already quite a lot of work in my opinion.

That's the point of the helper library. It splits out the common code and thereby provides (some) separation of concerns as well as making the development of the plugin easier.

Regarding @markus2330's concern that developing a general purpose library is harder than a similar plugin: That is factually wrong, because you could simple provide the a slightly modified version of the current elektraTypeGet function as a library. The modifications are only required because we need to replace the Plugin * handle with a KeySet * config. While this might not solve all problems, it at least lets the storage plugin control exactly how and when the type stuff is done.

As an example, toml could have an INI fallback mode where it doesn't use the TOML type system but instead defers to type and a normal TOML mode where it provides elektraTypeGet with a very precise config that ensures everything conforms to the TOML spec.

In short, a library is simple a lot more flexible than a plugin from the storage plugin's view. At least with the current plugin configuration possibilities.

Requiring that a storage plugin takes care of type conversion, directory keys and binary data (Base64 encoding) does not make that job easier.

Lets break this down:

• Binary data: There is not really any effort involved doing something like:
  if (keyIsBinary (key)) { writeValue (elektraBase64Encode (keyValue (key), keyGetValueSize (key))); } else { writeValue (keyString (value)); }
  A separate plugin would also work, as long as the storage plugin can enforce that the other plugin is mounted under all circumstances and can simply assume there are no binary keys. If the plugin still has to call keyIsBinary and throw an error, there is zero benefit to this solution. I also doesn't like that a separate binary plugin has to convert the whole KeySet at once, because then all the Base64 keys waste quite a bit of memory.
• Directory keys: First these non-leaf keys with values are weird under almost all circumstances and we should actually recommend that they be avoided. When it comes to dealing with these keys, as stated in #3256 I think a library is much better suited for this case. There are many reasons, including memory and processing overhead, unnecessarily complicating mountpoint configs and flexibility. It seems @markus2330 (somewhat) agrees with me on this point.
• Types: Any storage plugin for a format that has a native type system will have to do at least some work for types. This could range from doing the full conversion from scratch through calling some library to configuring another plugin. It will never be possible for everything to just work. Even if a have a very detailed formal specification for types so that no direct configuration of the type plugin is needed, the storage plugin will have to implement this spec in some way or another.

However, it doesn't check if the decimal/double values over-/underflow long_long/double, so I think that is done by the typeplugin.

Ok, so you basically only use it for checking.

But in that case a separate plugin also isn't the solution

No, the separate plugins unfortunately are not the solution. We failed there. The current solution is that every storage plugin implements everything (except of binary handling) based on doc/tutorials/storage-plugins.md

But a user would probably expect that kdb set /some/key/mounted/with/toml true would work.

No, a user should not expect that. In Elektra 1/0 are true/false. Only storage plugins might map this to something else.

Rationale: As at least on the specification level only 1/0 are working, it would be only confusing to make true/false working anywhere else (with the exception of storage plugins).

This just shows that a storage plugin MUST be able to handle all types that the storage format supports and the associated conversions WITHOUT using any other plugins.

Yes, I agree.

That is a fair point, although I don't think simply deleting code is the right solution.

Of course we need to delete the "right" code: the one that gives you expectations without fulfilling it or introducing problems with other parts of Elektra. And some functionality of the type plugin seems to create problems together with other parts of Elektra.

At least not when it comes to plugins. Within the core, I agree, the less LOC the better. For plugins, we can easily say that the plugin or some part of it is experimental.

Unfortunately, this does not work (yet*). People do not judge the status of plugins correctly, e.g. recently see: #3472 where the unmaintained and bug-riven INI plugin, and even worse the discouraged legacy xmltool plugin, were believed to work perfectly.

• It might work better if I rework #666 and we output warnings during mounting.

Requiring that a storage plugin takes care of type conversion, directory keys and binary data (Base64 encoding) does not make that job easier.

@sanssecours how do the YAML plugins handle type conversion?

That is factually wrong

We will see when it is done :stuck_out_tongue_winking_eye:

In short, a library is simple a lot more flexible than a plugin from the storage plugin's view. At least with the current plugin configuration possibilities.

Nobody argued about that. But flexibility sometimes comes with a huge price.

There is not really any effort involved doing something like

base64 is not the only way to encode binary data. For standards where base64 is required it could be hard-coded. (@sanssecours Is it the case for YAML?)

For TOML however, it simply says "Base64 or another suitable ASCII or UTF-8 encoding", so other binary plugins could be used. So the current solution has advantages, as users could mount other binary plugins as wanted or needed.

@bauhaus93 - infos/needs = base64 should be changed to - infos/needs = binary.

First these non-leaf keys with values are weird under almost all circumstances and we should actually recommend that they be avoided.

There are very common in many formats. And they can be very useful when you extend a specifcation (create subkeys from keys that had values).

[Directory keys] It seems @markus2330 (somewhat) agrees with me on this point.

I agree on storage plugins need to handle it (maybe with a library but not with the "directoryvalue" plugin as escaping does not work and correct escaping is one of the main jobs of a storage plugin).

Ok, so you basically only use it for checking.

I disagree with the phrase "the toml plugin uses the type plugin". In the current version it only recommends type.

https://github.com/ElektraInitiative/libelektra/blob/c61e388c4aa950cf84aa2f00fba7cdd34a47640e/src/plugins/toml/README.md#L5-L6

Even if type was declared as needs, I wouldn't call it "using". toml doesn't really have any direct control over type. That is why I said toml relies on type. It expects type do certain things in a certain way, but I can't do anything, if that is not the case.

It might work better if I rework #666 and we output warnings during mounting.

I agree. Maybe even on every kdbGet too, unless a special "I acknowledge this mountpoint uses experimental plugins" key has been set.

base64 is not the only way to encode binary data.

That is actually an argument for a binary plugin. Although I would still like an interface through which plugin can work a single key at a time, this could achieve some memory savings and maybe performance savings as well. (But that's another issue)

There are very common in many formats.

Do you have an example?

And they can be very useful when you extend a specifcation (create subkeys from keys that had values).

True, but in that case I would recommend an "either-or" solution (from the end-user POV). Either you use the old key with the single value, or you use the new version with subkeys. Otherwise the config will probably, be confusing.

To be clear, I don't think non-leaf keys with values should never be used, but they should be avoided if possible and rarely (if ever) are the preferred solution.

but not with the "directoryvalue" plugin as escaping does not work

In #3256 I suggested the use of metadata to solve escaping problems. Also there are my ideas from #3223 that would solve the escaping problem (for every use-case not just directoryvalue) as well as some other things. See also the (very formal) proposal. I will add plain English explainations today or tomorrow to that document, as I am still very much in favour of these changes.

@sanssecours how do the YAML plugins handle type conversion?

Yan LR and YAML CPP use the type plugin for booleans. YAML CPP also uses the base64 plugin for binary data.

base64 is not the only way to encode binary data. For standards where base64 is required it could be hard-coded. (@sanssecours Is it the case for YAML?)

Yes, as far as I know binary data in YAML always uses the Base64 encoding.

[non-leaf keys with values] Do you have an example?

XML and most INI dialects (as they allow key=value even when [key] exists)

True, but in that case I would recommend an "either-or" solution (from the end-user POV). Either you use the old key with the single value, or you use the new version with subkeys. Otherwise the config will probably, be confusing.

Yes, that sounds reasonable. Once you know something is a section, you usually move the value somewhere within the section.

E.g. deluser.conf has BACKUP = 0 and BACKUP_TO = ".". With sections, most applications would use BACKUP_ENABLE instead of simply BACKUP.

Also there are my ideas from #3223 that would solve the escaping problem (for every use-case not just directoryvalue) as well as some other things.

I have now updated the proposal in #3223. I hope it is now easier to understand (it is still very long).

https://github.com/kodebach/libelektra/blob/dcec6e865bba32f6d83c40c2f711c2e70dde6d62/doc/proposals/keynames.md

The big question is who will implement all this. It is quite far away from status quo, i.e., a lot of work (alone the changes of the terminology is a huge amount of work). If you want to implement it, you can create a PR with the proposal and I comment on it. Otherwise, we should take a step back and think about feasible solutions within our reach.

Btw. the first part of the proposal is actually a wonderful tutorial about how key names work. It would be amazing to have this as tutorial. :sparkling_heart:

The big question is who will implement all this.

That is always the question...

It is quite far away from status quo, i.e., a lot of work (alone the changes of the terminology is a huge amount of work).

The changes in terminology are definitely the biggest part, especially changing all documentation. However, this doesn't have to be done by one person. These changes are quite simple, they are just tedious, so anybody can help, even without extensive knowledge about the code. For the most part it just amounts to reading all the documentation and replacing occurrences of things like "base name".

If you want to implement it, you can create a PR with the proposal and I comment on it. Otherwise, we should take a step back and think about feasible solutions within our reach.

I will start a PR, but whether I have time to finish it (alone) in the foreseeable, I can't say.
I already have a local branch in which all call to keyBaseName, keySetBaseName and keyAddBaseName have been replace by keyLastUnescapedPart, keySetLastUnescapedPart/keySetLastLiteralPart and keyAddUnescapedPart/keyAddLiteralPart. I created that using a semi-automatic regex-replace after I wrote the first proposal.

But in that branch the new functions are just #defines for the old ones, so the actual implementation still has to be written and then tests and probably storage plugins have to be updated.

The biggest problem for me would be the storage plugins. I can implement the updates to the core and the tests and maybe one storage plugin. But I would prefer, if I don't have to study the code for all storage plugins and figure out all the intricacies of all the formats.

Btw. the first part of the proposal is actually a wonderful tutorial about how key names work. It would be amazing to have this as tutorial.

That's why I wrote it this way. My plan was to reuse parts of this proposal for the documentation in #3447. I can't use it 1:1 because I left some very important parts out (e.g. canonical vs non-canonical).

I need to be be plain-spoken and not give you any wrong hope: As you saw in LCDproc, it does not happen that others are jumping forward completing tasks, especially not when they are tedious. A PR that destroys all storage plugins except one is unmergable. And storage plugins do not get much benefit from this PR, actually they get "worse" in the sense that they will suddenly produce syntax errors on files that they could parse now. This does not mean it is overall a bad idea. With some refinements it might be better than the status-quo but I simply do not see how we can manage to do such a big task with so many other urgent important open tasks we have right now.

So please let us concentrate on #3447 (docu), finally getting #2969 done, improving TOML plugin and other important topics we need for 1.0 (https://github.com/ElektraInitiative/libelektra/milestone/12 and https://github.com/ElektraInitiative/libelektra/milestone/14). Once this looks good, we can see which other ideas we can accomplish.

For me the conclusion of this discussion is that there is definitely need for libraries to make it easier to write storage plugins as the plugin-approach did not work (except for binary is to be seen). This conclusion should be in the storage plugin tutorial.
@bauhaus93 can you take the task to continue the storage plugin tutorial? You have a lot of insight now, which cannot be seen by looking at the source code of the toml plugin.

A PR that destroys all storage plugins except one is unmergable.

I didn't expect anybody to merge such a PR... I just said it would be nice if other people could help with updates to storage plugins. Ideally the original author of the plugin. After #3491 some of the more complex storage plugins will be removed anyway, so the whole task will be easier.

And storage plugins do not get much benefit from this PR, actually they get "worse" in the sense that they will suddenly produce syntax errors on files that they could parse now.

This should not be the case. There is even a partial explanation at the end of the proposal. AFAIK there should not be a case, where a storage plugin must reject a file, because it cannot be translated into a KeySet.

The one exception would be formats, that directly use Elektra's Key Names (escaped or unescaped). These might accept less files after this proposal than before. But since the syntax for these files always depended on the syntax for Key Names, this is entirely to be expected.

With some refinements it might be better than the status-quo but I simply do not see how we can manage to do such a big task with so many other urgent important open tasks we have right now.

I never expected this proposal to be adopted immediately. But I think we should definitely consider implementing it before 1.0. Once 1.0 is release the proposal would be a huge breaking change. I don't think releasing a version 2.0 even 1 or 2 years after 1.0 (nevermind earlier than that) would be a good idea, considering the target audience of Elektra.

So please let us concentrate on #3447 (docu), finally getting #2969 done, improving TOML plugin and other important topics we need for 1.0

I totally, agree. But again, we should still consider it, because after 1.0 is released, adding breaking changes will be very hard for quite a while.

For me the conclusion of this discussion is that there is definitely need for libraries to make it easier to write storage plugins

👍

except for binary is to be seen

IMO, the case of binary values is very different. It is a 1:1 key value translation, like many others (rgbcolor, macaddr, ipaddr, etc), so a plugin is actually well suited here. However, like I said, a new plugin API that deals with just one key at time could be an improvement. But that can easily be added post-1.0 since it wouldn't be a breaking change if done correctly.

most -> must?

Yes, I see it the same way: it is unrealistic that this change will happen once 1.0 is out (the whole purpose of 1.0 is to freeze such decisions so that others can rely on it) and it would be nice to have such improvements before. But as said: we really need to concentrate on getting the critical stuff done and not lose our energy in nice-to-have battles we cannot win with our current man power.