Libelektra: mmapstorage doesn't work with kdb export

Created on 15 Feb 2019  ·  22Comments  ·  Source: ElektraInitiative/libelektra

Steps to Reproduce the Problem

kdb set user/tests/hello world
#> Create a new key user/tests/hello with string "world"

kdb export user/tests/hello mmapstorage > test.mmap

Expected Result

A file called test.mmap is created, which can be re-imported with kdb import.

Actual Result

An empty file called test.mmap is created and the following message is printed (includes logs from ENABLE_LOGGER):

src/plugins/mmapstorage/mmapstorage.c:944:libelektra_mmapstorage_LTX_elektraPluginset: could not unlink
src/plugins/mmapstorage/mmapstorage.c:1003:libelektra_mmapstorage_LTX_elektraPluginset: strerror: Permission denied
Sorry, the error (#9) occurred ;(
Description: Insufficient permissions to open configuration file for writing. You might want to retry as root.
Reason: Permission denied
Ingroup: kdb
Module: 
At: /home/klemens/data/bsc/libelektra/src/plugins/mmapstorage/mmapstorage.c:1004
Mountpoint: user
Configfile: /dev/stdout

System Information

  • Elektra Version: master

Further Information

~Works when kdb export is called as root.~ (EDIT: see comment below) kdb export user/tests/hello mmapstorage test.mmap also works, but is a completely undocumented feature of kdb export. So a better error message and updated documentation might be all we need.

bug

Most helpful comment

Thank you for your reply!

The mmap error messages might be misleading.

Please improve the error messages.

To my knowledge it will not work on stdout. It also does not work with pipes

Please add that info to the error messages.

The solutions above do not change much of the mmapstorage logic.

As your plugin is currently the only affected one, it makes sense that you do the stat and copy everything if needed. Then we could close this issue without larger changes in the framework.

All 22 comments

Thank you for trying this and for this detailed report!

So a better error message and updated documentation might be all we need.

Yes, I fully agree.

Maybe we even should have a serializable in infos/status #666 and fail explicitly if the file is /dev/stdout? (@mpranj Or is there a way to detect if a file is not mmapable? Is it really identically to a permission denied?)

Looking at the code (and the error message) I think the current problem is that we cannot call unlink on stdout without root access. Also open will probably fail aswell, because stdout is already opened. And whether or not stdout is mmapable depends on what stdout is connected to I think.
Using fstat(fileno(stdout), &stat) to check whether it is a regular file might work. Maybe isatty(3) also works.

In any case we could simply check whether the output file is /dev/stdout (or CON for _WIN32) and create a temporary file for use with mmap and afterwards simply copy that to stdout. It would of course be slower but we would retain the possibility of piping the output of kdb export.

Maybe we even should have a serializable in infos/status #666 and fail explicitly if the file is /dev/stdout?

If anything, I would have a status humanreadable and refuse to export to a tty stdout, if the format is not human readable.

Works when kdb export is called as root.

I take that back... DON'T TRY THAT! Calling kdb export <something> mmapstorage as root user (even if stdout is redirected) destroys /dev/stdout. Using /dev/stdout (e.g. via fopen) won't work on your system until you recreate the default symlink with sudo rm /dev/stdout && sudo ln -s /dev/stdout /proc/self/fd/1.

I think the best solution is that kdb export and kdb import always works on temporary files. Then kdb export will be more similar to what KDB does. This would solve the unlinking problem (which is necessary for mmaped files.) The downside is only loss of performance (copy of the whole content needed) but then import/export will work with all storage plugins, which is imho much more important.

I think the best solution is that kdb export and kdb import always works on temporary files.

I think plugins should create the temporary file themselves, if needed. That way we avoid the performance penalties for plugins that can directly output to TTYs. AFAIK mmapstorage is currently the only plugin which doesn't work with TTYs. We could also add a simple shell test that ensures all storage plugins can be called via kdb export /some/key plugin > export.file. The logic is also more self-contained, because it really is the plugins responsibility to know how to produce the expected output.

Also for kdb import temporary files are unnecessary because reading a file should always work, even for TTYs (if you have the correct permissions).

I think plugins should create the temporary file themselves, if needed.

Then the plugins (or more specifically mmap) would need to know if it is used within KDB or kdb export.

@mpranj What is your opinion? Can this be fixed within the mmap plugin?

That way we avoid the performance penalties for plugins that can directly output to TTYs.

In which cases is this performance penalty a problem? Maybe in the backup/restore for every test case?

AFAIK mmapstorage is currently the only plugin which doesn't work with TTYs.

For export yes. But for import we had several plugins that do not work. If a plugin uses fseek or similar it obviously cannot work, e.g. csvstorage or mozprefs. (dump should now be fixed)

We could also add a simple shell test that ensures all storage plugins can be called via kdb export /some/key plugin > export.file.

You mean tests/shell/check_export.sh line 46?

In which cases is this performance penalty a problem?

Actually, probably never. As long as you use the 3 argument versions of export/import instead of piping. See proposal below.

You mean tests/shell/check_export.sh line 46?

Yes, but that is broken, because is_not_rw_storage doesn't work. Not even dump is recognized as a storage plugin.


I think to solve this quickly and easily we should do the following:

  1. Fix is_not_rw_storage in include_common.sh.in tests/shell/check_export.sh so we detect problems like this in future.
  2. In kdb export create a temporary file, only if no third argument is given. Use this file in the kdbSet calls and afterwards copy print its contents to stdout (shouldn't be more than one or two lines in C++).
  3. Similarly for kdb import create a temporary file if stdin is used. Copy all of stdin into the temporary file and use this file to call kdbGet.
  4. Update the documentation of kdb import and kdb export to state that the third argument exists. Also state that if the two argument version is used we create a temporary file.

PS. This might be good first issue

Thank you for reporting that is_not_rw_storage is broken, I opened #2423

Sorry, I was away for a week so I could not look into it.

The mmap error messages might be misleading. mmap() fails with EACCES when trying to map non-regular files. To my knowledge it will not work on stdout. It also does not work with pipes, as discussed in #2209.

From POSIX:

The mmap() function shall be supported for the following memory objects:

  • Regular files
  • [SHM] Shared memory objects
  • [TYM] Typed memory objects

As for solutions within mmapstorage, we can check if it is a regular file with stat and then:

  • write to a temporary regular file, as suggested above
  • write to a temporary memory location (simply malloc instead of mmap) and then copy to the non regular file (pipe, stdout,..)

I'm open to other solutions too. The solutions above do not change much of the mmapstorage logic. Reworking mmapstorage to work with pipes or stdout directly does not make too much sense to me.

Thank you for your reply!

The mmap error messages might be misleading.

Please improve the error messages.

To my knowledge it will not work on stdout. It also does not work with pipes

Please add that info to the error messages.

The solutions above do not change much of the mmapstorage logic.

As your plugin is currently the only affected one, it makes sense that you do the stat and copy everything if needed. Then we could close this issue without larger changes in the framework.

Then we could close this issue without larger changes in the framework.

We should also update the man pages for kdb import and kdb export. Currently they do not mention the 3 argument versions that do not rely on stdin/stdout.

Thank you both for reporting the issue and for the input!

Unfortunately, strerror prints these misleading error messages. I can add a hint in that case.

I will make the requested changes in a PR this week.

We should also update the man pages for kdb import and kdb export. Currently they do not mention the 3 argument versions that do not rely on stdin/stdout.

Maybe we do not need the argument to specify the file? The argument would conflict once kdb import/export supports multiple plugins.

I will make the requested changes in a PR this week.

Thank you!

Maybe we do not need the argument to specify the file?

Then we have to support stdin and stdout (redirected to a regular file) in all plugins. Otherwise they could never be used with kdb import/export. That makes sens for mmapstorage because it isn't portable, but other plugins which are portable (maybe even human-readable) might need a regular file too (e.g. if they use fseek).

The argument would conflict once kdb import/export supports multiple plugins.

We could easily move to using the options -i FILE, --input=FILE, -o FILE, --ouput=FILE when we switch to using elektraGetOpts.

Yes, we can add -i, -o options. But fixing the plugins (or the import/export framework) so that also stdin/stdout works would be nice in any case.

To fix this problem with kdb export it is sufficient to implement the workaround in the plugin->kdbSet() function. I've already implemented this, PR soon to come.

This works now: kdb import user/tests/ mmapstorage < test.mmap,
but this does not: cat test.mmap | kdb import user/tests/ mmapstorage, since again mmap can not handle it.

To make the solution consistent (thus completely compatible with non-regular files) it would be needed to solve it for the kdbGet() function too. There are roughly three solutions for this:

  1. write to a tempfile and unlink it (somewhat ugly, but simple solution)
  2. rework half of my mmap work (ugly & lots of work)
  3. make lots of copies and reimplement ksDeepDup to make copies of meta-keysets too (really ugly, semi-simple code)

Is any of this desirable or do we ignore it?

I think we can ignore it for now, as we already have quickdump. Simply document it as not being suitable for serialization. We will now rework the import/export framework anyway, then we will find a proper solution. @mpranj I unassigned you for now.

Actually, it is maybe better if #2639 closes this issue and @mpranj you make a new one for the cat ... problem.

2639 closes all problems here. It works fine on Linux, I just need to fix a bug so it works on the BSDs too.

I had a really nice solution (using realpath and stat) which unfortunately will not work on BSDs. It does not make sense to invest much more time into it.

I have decided to throw it away, thus not making mmapstorage completely compatible with non-regular files. It will only work with kdb import/export. The new solution will simply check if kdb import/export is used, by checking if the file is "/dev/stdin" or "/dev/stdout". It's done similarly in quickdump.

I had a really nice solution (using realpath and stat) which unfortunately will not work on BSDs.

This is the commented out solution?

It does not make sense to invest much more time into it.

You are right, the framework should handle this. I created #2640.

will only work with kdb import/export

Also with the cat | kdb import variant?

It's done similarly in quickdump.

Where are the differences?

Can this code be moved to the import/export framework?

This is the commented out solution?

I left it in the history but removed the lines later. The imho better solution was up until https://github.com/ElektraInitiative/libelektra/pull/2639/commits/a523f9b38b56687d532f5101c7ef44c078e2308d. Note that it worked well on Linux but not BSD.

One problem I encountered is that stdin/stdout can not be open()ed on BSDs. The other is that you really have to use the realpath to stat() the file and determine whether it is a regular file. Otherwise stat only resolved one level of symlinks for me. This approach failed on BSDs for me, because realpath resolved to a nonexistent file somehow.

Also with the cat | kdb import variant?

Yes!

Where are the differences?

The relevant part is the same, sorry for the confusion. What I meant is that we strcmp for /dev/stdin instead of using stat to determine whether it is a regular file. That means that it will still fail if we use /dev/fd/. My other solution used stat instead of strcmp so it worked with any path.

Edit:

Can this code be moved to the import/export framework?

Yes I think the code was almost completely there, but I had no time to fix the BSD problems properly.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

mpranj picture mpranj  ·  3Comments

sanssecours picture sanssecours  ·  4Comments

markus2330 picture markus2330  ·  4Comments

mpranj picture mpranj  ·  3Comments

sanssecours picture sanssecours  ·  4Comments