kdb set user/tests/hello world
#> Create a new key user/tests/hello with string "world"
kdb export user/tests/hello mmapstorage > test.mmap
A file called test.mmap
is created, which can be re-imported with kdb import
.
An empty file called test.mmap
is created and the following message is printed (includes logs from ENABLE_LOGGER
):
src/plugins/mmapstorage/mmapstorage.c:944:libelektra_mmapstorage_LTX_elektraPluginset: could not unlink
src/plugins/mmapstorage/mmapstorage.c:1003:libelektra_mmapstorage_LTX_elektraPluginset: strerror: Permission denied
Sorry, the error (#9) occurred ;(
Description: Insufficient permissions to open configuration file for writing. You might want to retry as root.
Reason: Permission denied
Ingroup: kdb
Module:
At: /home/klemens/data/bsc/libelektra/src/plugins/mmapstorage/mmapstorage.c:1004
Mountpoint: user
Configfile: /dev/stdout
~Works when kdb export
is called as root.~ (EDIT: see comment below) kdb export user/tests/hello mmapstorage test.mmap
also works, but is a completely undocumented feature of kdb export
. So a better error message and updated documentation might be all we need.
Thank you for trying this and for this detailed report!
So a better error message and updated documentation might be all we need.
Yes, I fully agree.
Maybe we even should have a serializable
in infos/status #666 and fail explicitly if the file is /dev/stdout? (@mpranj Or is there a way to detect if a file is not mmapable? Is it really identically to a permission denied?)
Looking at the code (and the error message) I think the current problem is that we cannot call unlink
on stdout
without root access. Also open
will probably fail aswell, because stdout
is already opened. And whether or not stdout
is mmapable depends on what stdout
is connected to I think.
Using fstat(fileno(stdout), &stat)
to check whether it is a regular file might work. Maybe isatty(3)
also works.
In any case we could simply check whether the output file is /dev/stdout
(or CON
for _WIN32
) and create a temporary file for use with mmap and afterwards simply copy that to stdout
. It would of course be slower but we would retain the possibility of piping the output of kdb export
.
Maybe we even should have a
serializable
in infos/status #666 and fail explicitly if the file is /dev/stdout?
If anything, I would have a status humanreadable
and refuse to export to a tty stdout, if the format is not human readable.
Works when
kdb export
is called as root.
I take that back... DON'T TRY THAT! Calling kdb export <something> mmapstorage
as root user (even if stdout is redirected) destroys /dev/stdout
. Using /dev/stdout
(e.g. via fopen
) won't work on your system until you recreate the default symlink with sudo rm /dev/stdout && sudo ln -s /dev/stdout /proc/self/fd/1
.
I think the best solution is that kdb export
and kdb import
always works on temporary files. Then kdb export
will be more similar to what KDB does. This would solve the unlinking problem (which is necessary for mmaped files.) The downside is only loss of performance (copy of the whole content needed) but then import/export will work with all storage plugins, which is imho much more important.
I think the best solution is that
kdb export
andkdb import
always works on temporary files.
I think plugins should create the temporary file themselves, if needed. That way we avoid the performance penalties for plugins that can directly output to TTYs. AFAIK mmapstorage
is currently the only plugin which doesn't work with TTYs. We could also add a simple shell test that ensures all storage plugins can be called via kdb export /some/key plugin > export.file
. The logic is also more self-contained, because it really is the plugins responsibility to know how to produce the expected output.
Also for kdb import
temporary files are unnecessary because reading a file should always work, even for TTYs (if you have the correct permissions).
I think plugins should create the temporary file themselves, if needed.
Then the plugins (or more specifically mmap) would need to know if it is used within KDB or kdb export
.
@mpranj What is your opinion? Can this be fixed within the mmap plugin?
That way we avoid the performance penalties for plugins that can directly output to TTYs.
In which cases is this performance penalty a problem? Maybe in the backup/restore for every test case?
AFAIK mmapstorage is currently the only plugin which doesn't work with TTYs.
For export yes. But for import we had several plugins that do not work. If a plugin uses fseek or similar it obviously cannot work, e.g. csvstorage or mozprefs. (dump should now be fixed)
We could also add a simple shell test that ensures all storage plugins can be called via kdb export /some/key plugin > export.file.
You mean tests/shell/check_export.sh line 46?
In which cases is this performance penalty a problem?
Actually, probably never. As long as you use the 3 argument versions of export/import instead of piping. See proposal below.
You mean tests/shell/check_export.sh line 46?
Yes, but that is broken, because is_not_rw_storage
doesn't work. Not even dump
is recognized as a storage plugin.
I think to solve this quickly and easily we should do the following:
is_not_rw_storage
in include_common.sh.in
tests/shell/check_export.sh
so we detect problems like this in future.kdb export
create a temporary file, only if no third argument is given. Use this file in the kdbSet
calls and afterwards copy print its contents to stdout (shouldn't be more than one or two lines in C++).kdb import
create a temporary file if stdin is used. Copy all of stdin into the temporary file and use this file to call kdbGet
.kdb import
and kdb export
to state that the third argument exists. Also state that if the two argument version is used we create a temporary file.PS. This might be good first issue
Thank you for reporting that is_not_rw_storage is broken, I opened #2423
Sorry, I was away for a week so I could not look into it.
The mmap error messages might be misleading. mmap()
fails with EACCES
when trying to map non-regular files. To my knowledge it will not work on stdout. It also does not work with pipes, as discussed in #2209.
From POSIX:
The mmap() function shall be supported for the following memory objects:
- Regular files
- [SHM] Shared memory objects
- [TYM] Typed memory objects
As for solutions within mmapstorage, we can check if it is a regular file with stat
and then:
I'm open to other solutions too. The solutions above do not change much of the mmapstorage logic. Reworking mmapstorage to work with pipes or stdout directly does not make too much sense to me.
Thank you for your reply!
The mmap error messages might be misleading.
Please improve the error messages.
To my knowledge it will not work on stdout. It also does not work with pipes
Please add that info to the error messages.
The solutions above do not change much of the mmapstorage logic.
As your plugin is currently the only affected one, it makes sense that you do the stat and copy everything if needed. Then we could close this issue without larger changes in the framework.
Then we could close this issue without larger changes in the framework.
We should also update the man pages for kdb import
and kdb export
. Currently they do not mention the 3 argument versions that do not rely on stdin/stdout.
Thank you both for reporting the issue and for the input!
Unfortunately, strerror
prints these misleading error messages. I can add a hint in that case.
I will make the requested changes in a PR this week.
We should also update the man pages for kdb import and kdb export. Currently they do not mention the 3 argument versions that do not rely on stdin/stdout.
Maybe we do not need the argument to specify the file? The argument would conflict once kdb import/export
supports multiple plugins.
I will make the requested changes in a PR this week.
Thank you!
Maybe we do not need the argument to specify the file?
Then we have to support stdin
and stdout
(redirected to a regular file) in all plugins. Otherwise they could never be used with kdb import/export
. That makes sens for mmapstorage
because it isn't portable, but other plugins which are portable (maybe even human-readable) might need a regular file too (e.g. if they use fseek
).
The argument would conflict once
kdb import/export
supports multiple plugins.
We could easily move to using the options -i FILE, --input=FILE
, -o FILE, --ouput=FILE
when we switch to using elektraGetOpts
.
Yes, we can add -i, -o options. But fixing the plugins (or the import/export framework) so that also stdin/stdout works would be nice in any case.
To fix this problem with kdb export
it is sufficient to implement the workaround in the plugin->kdbSet() function. I've already implemented this, PR soon to come.
This works now: kdb import user/tests/ mmapstorage < test.mmap
,
but this does not: cat test.mmap | kdb import user/tests/ mmapstorage
, since again mmap can not handle it.
To make the solution consistent (thus completely compatible with non-regular files) it would be needed to solve it for the kdbGet() function too. There are roughly three solutions for this:
Is any of this desirable or do we ignore it?
I think we can ignore it for now, as we already have quickdump. Simply document it as not being suitable for serialization. We will now rework the import/export framework anyway, then we will find a proper solution. @mpranj I unassigned you for now.
Actually, it is maybe better if #2639 closes this issue and @mpranj you make a new one for the cat ...
problem.
I had a really nice solution (using realpath
and stat
) which unfortunately will not work on BSDs. It does not make sense to invest much more time into it.
I have decided to throw it away, thus not making mmapstorage completely compatible with non-regular files. It will only work with kdb import/export. The new solution will simply check if kdb import/export is used, by checking if the file is "/dev/stdin
" or "/dev/stdout
". It's done similarly in quickdump
.
I had a really nice solution (using realpath and stat) which unfortunately will not work on BSDs.
This is the commented out solution?
It does not make sense to invest much more time into it.
You are right, the framework should handle this. I created #2640.
will only work with kdb import/export
Also with the cat | kdb import
variant?
It's done similarly in quickdump.
Where are the differences?
Can this code be moved to the import/export framework?
This is the commented out solution?
I left it in the history but removed the lines later. The imho better solution was up until https://github.com/ElektraInitiative/libelektra/pull/2639/commits/a523f9b38b56687d532f5101c7ef44c078e2308d. Note that it worked well on Linux but not BSD.
One problem I encountered is that stdin/stdout can not be open()ed on BSDs. The other is that you really have to use the realpath to stat() the file and determine whether it is a regular file. Otherwise stat only resolved one level of symlinks for me. This approach failed on BSDs for me, because realpath resolved to a nonexistent file somehow.
Also with the cat | kdb import variant?
Yes!
Where are the differences?
The relevant part is the same, sorry for the confusion. What I meant is that we strcmp for /dev/stdin instead of using stat to determine whether it is a regular file. That means that it will still fail if we use /dev/fd/
Edit:
Can this code be moved to the import/export framework?
Yes I think the code was almost completely there, but I had no time to fix the BSD problems properly.
Most helpful comment
Thank you for your reply!
Please improve the error messages.
Please add that info to the error messages.
As your plugin is currently the only affected one, it makes sense that you do the stat and copy everything if needed. Then we could close this issue without larger changes in the framework.