The specload test failure is pretty explicit:

reason: : App '/home/jenkins/workspace/libelektra_master-Q2SIBK3KE2NBEMJ4WVGJXAXCSCB77DUBUULVLZDKHQEV3WNDXBMA/libelektra/obj-x86_64-linux-gnu/bin/elektra-specload-testapp' doesn't exist or is not executable

For quickdump I can't say exactly what is wrong, but it fails somewhere in elektraQuickdumpSet. There should be an error set in setKey, but it is not logged I think (that should probably be changed).

No idea, why gopts fails. Since no error message was printed, I suspect the testapp elektra-gopts-testapp couldn't be executed (its the only case where no error message would be printed). It would also be in line with the specload error.

The specload test failure is pretty explicit:

Yes, but it is wrong to expect binaries in the build directory after the application was installed. The binaries should be installed and searched in either build or installation directory.

The specload test failure is pretty explicit:

Yes, but it is wrong to expect binaries in the build directory after the application was installed. The binaries should be installed and searched in either build or installation directory.

Oh, I missed that you installed Elektra. However, that doesn't explain the quickdump failure, its test data is installed correctly.

I am also not sure how to solve this problem. Of course we can install the executables of the test apps, but the actual test executables would have to be modified during install time to change where they look for the test app.

It is too much work to search for the binaries (in either build or installed dir), we can also exclude the tests from being installed.

We could simply use a relative path and ensure that it remains the same during install, e.g. have both binaries in the same directory.

Yes, good idea.

A more general solution would be if we have kdb <command> available also in the build directory (currently only works if Elektra is installed). It would be quite some work as KDB_EXEC_PATH only supports one path but the binaries are scattered in many different parts of the build directory.

we have kdb available also in the build directory

Tests can easily be executed via ctest and/or make. As for the other commands, most of them only make sense for an installed version of Elektra anyway.

We could simply use a relative path and ensure that it remains the same during install, e.g. have both binaries in the same directory.

Turns out this is harder than expected. The stdlib would treat the relative path as relative to the current working directory, which is not helpful and resolving the path relative to the current executable's path is not possible in a platform independent way.

Thus the kdb <command> way would be quite attractive.

For the installation it is enough to install it to TARGET_TOOL_EXEC_FOLDER

Only during build time we need some way to combine build_dir/bin build_dir/scripts source_dir/scripts and the current directories (maybe also both build+source).

@kodebach is this still open? Can we many detect this situation and let the tests not run in this situation?

Or we fix it: KDB_EXEC_PATH now allows multiple paths, so we could add whatever folders we need from the build/source directory. Then you simply let kdb do the job of finding the executable.

AFAIK this is still open, yes.

We already have the function srcdir_file for tests. We could introduce a similar bindir_file. bindir would default to ${CMAKE_INSTALL_PREFIX}/${TARGET_TOOL_EXEC_FOLDER}, but their would be some way to override it. CTest would be set up (via CMake's add_test) such that bindir is set to ${CMAKE_BINARY_DIR}/bin when using ctest.

And simply adding the bindir to KDB_EXEC_PATH and using kdb <bin> does not work?

No, for a few reasons:

1. These are testmod_ tests, they shouldn't rely on kdb.
2. How would we find kdb then? Executing tests with make run_all should not use the installed kdb, but the one in ${CMAKE_BINARY_DIR}/bin.
3. KDB_EXEC_PATH would still contain the ${CMAKE_BINARY_DIR}/bin folder, after installing kdb, which could cause different problems.

These are testmod_ tests, they shouldn't rely on kdb.

Yes, I agree. It is good if they work standalone.

How would we find kdb then? Executing tests with make run_all should not use the installed kdb, but the one in ${CMAKE_BINARY_DIR}/bin.

Shellrecorder tests already use kdb, and they work for both installed Elektra and from the build directory (even if Elektra is installed).

KDB_EXEC_PATH would still contain the ${CMAKE_BINARY_DIR}/bin folder, after installing kdb, which could cause different problems.

No, KDB_EXEC_PATH is not set for an installed kdb (unless the user sets it)

Shellrecorder tests already use kdb, and they work for both installed Elektra and from the build directory (even if Elektra is installed).

That works because shell tests always execute "$KDB" and never kdb. And also because make run_all and ctest use e.g. testscr_check_meta.sh scripts, while kdb uses e.g. check_meta.sh. In the first we set $KDB to ${CMAKE_BINARY_DIR}/bin/kdb in the second one, it is simply set to kdb (and therefore resolved via $PATH).

No, KDB_EXEC_PATH is not set for an installed kdb (unless the user sets it)

I just took the master version of Elektra and installed it. The file /usr/local/lib/elektra/tool_exec/check_meta contains the line:

export KDB_EXEC_PATH="/home/klemens/libelektra/build/bin:$KDB_EXEC_PATH"

Of course this doesn't affect testmod_* tests, but it is wrong nonetheless. I will create a new issue.

@petermax2 @kodebach what is the status of this issue? Is as #3246 is fixed now (via #3409), the rest of this issue only packaging problems or is there something left to be implemented?

AFAIK the original issue with specload still exists. TBH I'm not sure why we even have to option to install testmod_ tests. It makes no sense. These are standalone tests that test a single plugin in isolation. If they actually run, their results will be the same not matter if executed in the build directory or the install directory.

TBH I'm not sure why we even have to option to install testmod_ tests.

At the moment we do not have an option to disable installing tests but we could make a local override to make add_plugintest not installing tests (like INSTALL_TESTING but for individual add_plugintest).

It makes no sense. These are standalone tests that test a single plugin in isolation. If they actually run, their results will be the same not matter if executed in the build directory or the install directory.

:wink:, there are plenty of differences, but most of them were related to the plugin system itself (They are hopefully all fixed by now). But still, there is a large range of possible dependency problems and installation problems so running the testmod tests in the installed state is much better than running nothing. But you are right that if there is a shellrecorder test, the testmod test is useless.

• [X] So for quickdump it is clear, the testmod test does not need to be installed.
• [ ] For specload there is a test but it seems quite minimal, maybe okay but better you check again.
• [ ] For gopts we could let example programs run in doc/tutorials/command-line-options.md?

@kodebach can you check if these tests can be safely excluded?

@robaerd can you exclude them in one of your CI PRs? (Before or where you add testing of packages.)

@markus2330 Does the quickdump problem still occur for you? AFAIK we never figured out what was wrong there. I also couldn't reproduce it.

For gopts and specload see #3618

Yes, still occurs:

kdb testmod_quickdump                                                                                                                                                                                                                    
QUICKDUMP     TESTS
==================

test varint
test basics
/home/jenkins/workspace/libelektra_master/libelektra/src/plugins/quickdump/testmod_quickdump.c:111: error in test_basics: call to kdbSet was not successful
zsh: segmentation fault (core dumped)  noglob kdb testmod_quickdump

Or when I call it from source:

```QUICKDUMP TESTS

test varint
test basics
/home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:92: error in test_basics: call to kdbGet was not successful
/home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:93: error in test_basics: real size of mmks2 was 0
/home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:93: error in test_basics: Compare keyset failed, size of keysets are not equal with size(mmks1): 8, size(mmks2): 0
mmks1:
0x55a9efd438d0 key: dir:/tests/bench/__112, string: gQHLlzB36CqIFlf, meta:/meta/_35: O6xNya6srhNhMFC, meta:/meta/_39: ublVuvyh1DgfOKU, meta:/meta/_58: 5Nyde2MHJODCBAT, meta:/meta/_79: ZK2xlaRMfobquxp, meta:/meta/_90: 0kCcc1pK7hOgY3F
0x55a9efd44250 key: dir:/tests/bench/__114, string: , meta:/binary:
0x55a9efd441a0 key: dir:/tests/bench/__333, string: SxTUAjM6OIpUV6s
0x55a9efd440f0 key: dir:/tests/bench/__506, string: cGqEvmXxUayNCf8
0x55a9efd44040 key: dir:/tests/bench/__859, string: rOI5aVFGlnjPLYJ
0x55a9efd43f90 key: dir:/tests/bench/__863, string: 8IBjbd5pzYBehrs
0x55a9efd43f20 key: dir:/tests/bench/__868, string: UVM0OPTf68yNXij
0x55a9efd43d90 key: dir:/tests/bench/__911, string: PgNbwPxfeqD30pH, meta:/meta/_35: O6xNya6srhNhMFC
mmks2:
/home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:111: error in test_basics: call to kdbSet was not successful
zsh: segmentation fault (core dumped) LD_LIBRARY_PATH=lib bin/testmod_quickdump

Stacktrace:

0 0x00007fa7c770ed74 in _IO_getc (fp=0x0) at getc.c:37

1 0x000055a9ede54c58 in compare_binary_files (filename2=0x55a9efd42a30 "/usr/local/share/elektra/test_data/quickdump/test.quickdump.out", filename1=0x55a9efd429e0 "/usr/local/share/elektra/test_data/quickdump/test.quickdump")

at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:31

2 test_basics () at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:113

3 0x000055a9ede545d7 in main (argc=1, argv=0x7ffff28085f8) at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:278

0 0x00007fa7c770ed74 in _IO_getc (fp=0x0) at getc.c:37

37 getc.c: Datei oder Verzeichnis nicht gefunden.
(gdb) bt

0 0x00007fa7c770ed74 in _IO_getc (fp=0x0) at getc.c:37

1 0x000055a9ede54c58 in compare_binary_files (filename2=0x55a9efd42a30 "/usr/local/share/elektra/test_data/quickdump/test.quickdump.out", filename1=0x55a9efd429e0 "/usr/local/share/elektra/test_data/quickdump/test.quickdump")

at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:31

2 test_basics () at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:113

3 0x000055a9ede545d7 in main (argc=1, argv=0x7ffff28085f8) at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:278

(gdb) bt full

0 0x00007fa7c770ed74 in _IO_getc (fp=0x0) at getc.c:37

    result = <optimized out>

1 0x000055a9ede54c58 in compare_binary_files (filename2=0x55a9efd42a30 "/usr/local/share/elektra/test_data/quickdump/test.quickdump.out", filename1=0x55a9efd429e0 "/usr/local/share/elektra/test_data/quickdump/test.quickdump")

at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:31
    f2 = 0x0
    c1 = <optimized out>
    f1 = 0x0
    result = 0
    c2 = <optimized out>
    f1 = <optimized out>
    f2 = <optimized out>
    result = <optimized out>
    c1 = <optimized out>
    c2 = <optimized out>
    end1 = <optimized out>
    end2 = <optimized out>

2 test_basics () at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:113

    conf = <optimized out>
    modules = 0x55a9efd43a00
    setKey = 0x55a9efd42bb0
    errorKey = <optimized out>
    plugin = 0x55a9efd43b00
    ks = <optimized out>
    infile = 0x55a9efd429e0 "/usr/local/share/elektra/test_data/quickdump/test.quickdump"
    outfile = 0x55a9efd42a30 "/usr/local/share/elektra/test_data/quickdump/test.quickdump.out"
    __func__ = "test_basics"

3 0x000055a9ede545d7 in main (argc=1, argv=0x7ffff28085f8) at /home/markus/Projekte/Elektra/current/src/plugins/quickdump/testmod_quickdump.c:27

```

Or when I call it from source:

When running tests directly please use ctest (e.g. ctest -R testmod_quickdump --ouptut-on-failure) to make sure, environment, args, etc. are set correctly.

In this case, I think LD_LIBRARY_PATH=lib bin/testmod_quickdump ../src/plugins/quickdump is what you need (if you want to debug with e.g. gdb).

Does /usr/local/share/elektra/test_data/quickdump/test.quickdump exist on your system and does the process have the rights to write to /usr/local/share/elektra/test_data/quickdump/test.quickdump.out? (Maybe you need sudo kdb testmod_quickdump)

If so, please add

output_error (setKey);
output_warnings (setKey);

in testmod_quickdump.c:112 and post the output.

elektraQuickdumpSet failed, so the fact that compare_binary_files fails is completely expected. (The error message could of course be better and the segfault could be avoided, but it is a _failed_ test so it doesn't really matter that much)

Yes, the problem is that the test cases try to create temporary files into a folder not suitable for that. sudo kdb testmod_quickdump runs without problems. So it should be enough to use our functions to create temporary files instead of using "test_data/quickdump/test.quickdump.out"

We can change that yes, but many tests don't run without sudo anyway so I don't really see this as a problem

Which tests are you talking about? I always run the whole suite without sudo (but with write permissions to Elektra's path). testmod_quickdump is afaik the only test with that problem.

but with write permissions to Elektra's path

Okay well that works too, but on a standard system those paths are root access only.

I still think the whole concept of running unit tests in an installed version of Elektra is weird and the quickdump issue is purely a user rights issue not a bug, but I'll change the path in #3618.

I have now updated the code in #3618. I also found similar code in testmod_dump and changed that was well. No idea, why that didn't fail for you. testmod_xmltool might have a similar problem (although the code is different so I didn't change it).

testmod_quickdump was to only one of those that used binary files and therefore didn't use compare_line_files. Therefore the others wouldn't segfault, but the tests should still fail.

nd the quickdump issue is purely a user rights issue not a bug,

It is a violation of FHS, applications are not supposed to write to /usr, it even might be read-only.

For writing we should only use tempfiles and we should also clean them up after the tests finished.