Libelektra: Some Tests Fail on OS X

I'm sorry but I don't have OS X. However ctest -V is your friend.

btw calling lua $file isn't the same as running the tests. You're using the system installed kdb library while the tests are obviously using the library located in the build directory. You should at least set LUA_CPATH. see https://github.com/ElektraInitiative/libelektra/blob/master/src/bindings/swig/lua/tests/CMakeLists.txt#L12

90% of swig related crashes were related to build environment faults. So first try to regenerate(!) + recompile the swig bindings.

This issue might be related to ad537b3 (at least on Linux kdb*lua|python were crashing, but fix for Linux is part of ad537b3). Without output of the test one cannot really tell.

You can use make run_all which suppresses the output of successful test cases but shows the output for the failed ones.

@sanssecours Any update on this?

This issue might be related to ad537b3 (at least on Linux kdb*lua|python were crashing, but fix for Linux is part of ad537b3). Without output of the test one cannot really tell.

@sanssecours Any update on this?

Sorry for the late response. I thought I already wrote a comment containing extended information about this issue. Let me start by saying that testpy2_kdb.py also fails on my machine. So it might be a problem with my specific setup. I installed SWIG via Homebrew and currently use the following cmake command to generate the build files for Elektra:

cmake .. -GNinja -DENABLE_TESTING=ON -DCMAKE_PREFIX_PATH=/usr/local/opt/qt5 -DTOOLS='qt-gui;kdb' -DBUILD_PDF=ON -DBINDINGS=SWIG

Here is the output of ctest -V -R test_kdb.lua:

test 65
    Start 65: test_kdb.lua

65: Test command: /usr/local/bin/lua "/Users/rene/Dropbox/Studium/Master Thesis/Configuration Parsing/Source/Elektra/src/bindings/swig/lua/tests/test_kdb.lua"
65: Environment variables:
65:  LUA_CPATH=/Users/rene/Dropbox/Studium/Master Thesis/Configuration Parsing/Source/Elektra/build/src/bindings/swig/lua/?.so
65: Test timeout computed to be: 1500
65: /usr/local/bin/lua: kdb:1 Warning was issued:
65:  Warning number: 1
65:     Description: could not load module, dlopen failed
65:     Ingroup: modules
65:     Module: dl
65:     At: ../src/libs/loader/dl.c:82
65:     Reason: of module: libelektra-resolver.so, because: dlopen(libelektra-resolver.so, 2): image not found
65:     Mountpoint:
65:     Configfile:
65: Error (#40) occurred!
65: Description: Failed to open default backend (see warnings for more information)
65: Ingroup: kdb
65: Module:
65: At: ../src/libs/elektra/kdb.c:282
65: Reason: could not open default backend
65: Mountpoint:
65: Configfile:
65:
65: stack traceback:
65:     [C]: in ?
65:     [C]: in function 'KDB'
65:     .../Source/Elektra/src/bindings/swig/lua/tests/test_kdb.lua:20: in main chunk
65:     [C]: in ?
1/1 Test #65: test_kdb.lua .....................***Failed    0.00 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
bindings    =   0.00 sec (1 test)
kdbtests    =   0.00 sec (1 test)
memleak     =   0.00 sec (1 test)

Total Test time (real) =   0.01 sec

The following tests FAILED:
     65 - test_kdb.lua (Failed)
Errors while running CTest

Looks like Lua is not able to find libelektra-resolver.so, which is located in build/lib and /usr/local/lib/elektra on my machine. Do I need to set some library path for this to work? By the way, it looks like testpy2_kdb.py also fails because of the same exact problem.

The bindings don't load any elektra library on runtime. They are just bindings. You can see this by looking at the exact error location, which is ../src/libs/loader/dl.c:82.

So this is either an issue with your environment, a general elektra issue on OSX or a build system issue on OSX. I suspect the former. ping @markus2330

dlopen(libelektra-resolver.so, 2): image not found actually looks like that it has nothing to do with lua.

libelektra-resolver.so should be a symlink to the resolver you selected with KDB_DEFAULT_RESOLVER, maybe your setup/installation is broken? Can you check if the symlinks are correct?

Strange is: libelektra-resolver.so is used in every kdb-related test case, why is it only _not_ working in this test case? Maybe this test case (and the python one) use the installed library and not the library in the build directory. Can you check with strace which libelektra-resolver.so the test case tries to load?

image not found is quite generic, it could be that within the library the image for your architecture is not found? Do you use fat binaries?

libelektra-resolver.so should be a symlink to the resolver you selected with KDB_DEFAULT_RESOLVER, maybe your setup/installation is broken?

Is there an easy way to check if the local Elektra installation is broken? The command kdb seems to work.

Can you check if the symlinks are correct?

Both of the libelektra-resolver.so aliases link to a file libelektra-resolver_fm_hpu_b.so located in the same directory as the aliases. So it looks like they are correct.

Can you check with strace which libelektra-resolver.so the test case tries to load?

I tried sudo dtruss ctest -V -R test_kdb.lua. Here is the output:
test_kdb.lua - dtruss Output.txt. Looks like the test uses the Elektra version in the build directory. I uninstalled Elektra via sudo ninja uninstall. I then ran the test again. The output is still the same.

Do you use fat binaries?

Not that I am aware of.

Do you use OSX El Capitan? It might be a weird System Integrity Protection problem .

It's kind of strange that kdb works though.

If it has to do with a System Integrity Protection because of the python/lua installation it might make sense.

Do you use OSX El Capitan? It might be a weird System Integrity Protection problem .

Yes, I use OS X 10.11.4. I disabled SIP temporarily and ran the test again. Still the same problem. Although, after I disabled SIP, I encountered a new test that fails: testscr_check_kdb_internal_suite. And now testscr_check_merge also does not work anymore 😢.

I went back to 0.8.16, cleaned my build directory and ran ninja test again. Both testscr_check_kdb_internal_suite and testscr_check_merge still fail, although I remembered, that at least testscr_check_kdb_internal_suite worked when Elektra 0.8.16 was released. Here is the output of the additional test that fail now too:
testscr_check_kdb_internal_suite.txt
testscr_check_merge.txt

I've changed the title and removed myself. I don't have direct access to an OSX machine and lacking in-depth knowledge. Without a machine to poke around I can't tell a thing from the text output.

Did you also look at the other hints of the links? E.g. reinstalling python/lua?

@petermax2 @mpranj Can you reproduce these issues?

Did you also look at the other hints of the links? E.g. reinstalling python/lua?

Sorry, not really. The Python 2 installation is the one that came with the system. To reinstall it, I would have to reinstall the whole operating system.

I only installed Lua just to test the Lua Elektra plugin. So I do not think that reinstalling makes much sense. However, since it only takes seconds, I executed brew reinstall lua. After that I started with a clean build directory, ran the build commands and ctest -VV -R test_kdb.lua. The test still shows the same output.

By the way: The tests testscr_check_kdb_internal_suite and testscr_check_merge now also work correctly again, after I removed all files from /etc/kdb.

I currently do not have another idea. (except time-consuming isolation of the problem: to reduce the dlopen call to a minimal example where it won't work) So it might be best to wait if someone can reproduce the issue, maybe it shines new light on the problem.

I can confirm the same behaviour on my machine.

Solution: set the library path correctly: e.g.
LD_LIBRARY_PATH="/Users/mpranj/workspace/libelektra/build/lib" ctest -V -R test_kdb.lua works fine for me.

dlopen on OS X searches $LD_LIBRARY_PATH, $DYLD_LIBRARY_PATH, current working directory, $DYLD_FALLBACK_LIBRARY_PATH, I think in that order.

The PR #710 fixes it, however I'm unsure if the fix is very clean.

No, the fix is definitely not clean.

As far as I remember our tests rely on DT_RPATH being set in libelektra-kdb.so. If this isn't available on OSX we should define this in some global build environment place.

Edit: But thanks for finding out!

RPATH is set for all core libraries, but maybe libelektra-core should suffice: at this place the dlopen happens.

Does image not found simply mean library not found? If so, any idea why the RPATH works for all but the lua and python tests?

Just for your information: Elektra also supports systems without RPATH (e.g. openwrt with musl as libc) by simply setting TARGET_PLUGIN_FOLDER to an empty string.

It appears that similar things happen on FreeBSD btw.

 66/118 Test  #66: test_kdb.py ........................***Failed    0.09 sec
..EEEE
======================================================================
ERROR: test_ctor (__main__.KDB)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mpranj/workspace/libelektra/src/bindings/swig/python/tests/test_kdb.py", line 25, in test_ctor
    self.assertIsInstance(kdb.KDB(), kdb.KDB)
  File "/home/mpranj/workspace/libelektra/build/src/bindings/swig/python/kdb.py", line 1742, in __init__
    _kdb.KDB_swiginit(self, _kdb.new_KDB(*args))
kdb.KDBException: 1 Warning was issued:
 Warning number: 1
    Description: could not load module, dlopen failed
    Ingroup: modules
    Module: dl
    At: /home/mpranj/workspace/libelektra/src/libs/loader/dl.c:82
    Reason: of module: libelektra-resolver.so, because: Shared object "libelektra-resolver.so" not found, required by "python3"
    Mountpoint:
    Configfile:
Error (#40) occurred!
Description: Failed to open default backend (see warnings for more information)
Ingroup: kdb
Module:
At: /home/mpranj/workspace/libelektra/src/libs/elektra/kdb.c:282
Reason: could not open default backend
Mountpoint:
Configfile:

Haven't been able to check lua on FreeBSD yet.

@mpranj good to know, so FreeBSD seems to be a good hint if it works with MacOSX, too (next to OpenBSD). Do the other kdb testcases and the kdb command-line tool work?

Can you open issues or append to OpenBSD issues about the test cases that do not work on FreeBSD?

If you mean these:

        Start 115: testkdb_allplugins
115/118 Test #115: testkdb_allplugins .................   Passed    0.02 sec
        Start 116: testkdb_nested
116/118 Test #116: testkdb_nested .....................   Passed    0.27 sec
        Start 117: testkdb_conflict
117/118 Test #117: testkdb_conflict ...................   Passed    0.17 sec
        Start 118: testkdb_simple
118/118 Test #118: testkdb_simple .....................   Passed    0.42 sec

... then yes. The kdb tool appears to work (just a quick check with ls).

Otherwise lots of things are failing but I won't be able to check/report all today. But sure I'll report everything when I get to test it properly.

My guess is the kdb tests work because they're linked statically, right?
If I disable BUILD_STATIC the tests don't work either.

@markus2330 I don't think we can avoid setting LD_LIBRARY_PATH on some platforms. How is TARGET_PLUGIN_FOLDER working? Or more specific how does this solve dynamic library searching?

Thanks that is a good idea: @sanssecours can you disable BUILD_STATIC and BUILD_FULL and rerun all (testkdb) tests?

If TARGET_PLUGIN_FOLDER is empty the plugins are installed where the libraries are and no RPATH or LD_LIBRARY_PATH is needed. But it would only helps when the installed Elektra is used (which might be the case for the python/lua tests).

https://cmake.org/Wiki/CMake_RPATH_handling says ".. RPATH on Mac OS X. It will be enabled for both the build tree and install tree", so actually also the build tree binaries should have RPATH set. @sanssecours Can you check this? E.g. using readelf -a lib/libelektra-core.so.0.8.16 | grep RPATH.

The issue is not about RPATH itself, but about dlopen on linux reading DT_RPATH vs. other platforms.

man dlopen from glibc:

• (ELF only) If the executable file for the calling program contains a DT_RPATH tag, and does not contain a DT_RUNPATH tag, then the directories listed in the DT_RPATH tag are searched.
• If, at the time that the program was started, the environment variable LD_LIBRARY_PATH was defined to contain a colon-separated list of directories, then these are searched. (As a security measure this variable is ignored for set-user-ID and set-group-ID programs.)
• (ELF only) If the executable file for the calling program contains a DT_RUNPATH tag, then the directories listed in that tag are searched.
• The cache file /etc/ld.so.cache (maintained by ldconfig(8)) is checked to see whether it contains an entry for filename.
• The directories /lib and /usr/lib are searched (in that order).

man dlopen on OSX:

When path doesn't contain a slash character (i.e. it is just a leaf name), dlopen() searches the following the following until it finds a compatible Mach-O file: $LD_LIBRARY_PATH, $DYLD_LIBRARY_PATH, current working directory, $DYLD_FALLBACK_LIBRARY_PATH.

dlopen should work with absolute paths here, or it needs to be in one of the paths mentioned above

But we don't do absolute paths. So no idea how this information helps

Absolute RPATH or absolute pathes to plugins? CMake claims it has support for build-tree RPATH, so it should use an absolute RPATH if required.

Absolute pathes to plugins is what many do and it would be an easy change, but I do not see how it would not help for the in-build test cases.

First it would be interesting what the actual problem is here. Does the installed version really fully work? E.g. kdb run_all would be interesting too (next to what I wrote above already).

My paste from dlopen manpages clearly states what the problem is about.
The search path of dlopen(libelektra-resolver.so) is different on different platforms. Linux works because dlopen honors DT_RPATH of the libelektra-kdb.so

It seems that setting LD_LIBRARY_PATH is what valve is also doing.
https://github.com/ValveSoftware/steam-runtime/blob/master/runtime/run.sh

You mean the problem is that they do not write documentation? @rpath is not even mentioned there.

I think until we do not know what actually works when installed/with BUILD_SHARED we should not further speculate.

@markus2330
EDIT: disabling BUILD_STATIC and BUILD_FULL:
the testkdb* tests run fine.

About documentation, they mention @rpath here.

Just verified my statement by a short example:

• runtimelib ...a library providing some random functions. (like elektra-resolver)
• lib ...a library which loads runtimelib at runtime using dlopen. It has DT_RPATH set to the directory of runtimelib. (like kdb.so of the bindings)
• runner ...an executable which loads lib at runtime using dlopen (like lua/python)

See https://gist.github.com/manuelm/43a4fa9dd424b4dcf03bd1d773a0e122

This scenario works on Linux, but fails on FreeBSD.

I know that RPATH set in a library is not portable (it also did not work for musl). But it seemed to work for Mac OS X (@mpranj also reported that testkdb* tests work fine and @omnidan reported kdb works with Mac OS X). Thus I asked to investigate why only the python/lua tests fail.

The portable way to install Elektra is not setting TARGET_PLUGIN_FOLDER and for tests we could use (as suggested) LD_LIBRARY_PATH. (We would only need to set it within run_memcheck and run_all).

@mpranj also reported that testkdb* tests work fine and @omnidan reported kdb works with Mac OS X

testkdb* and kdb have DT_RPATH set them self. Python and lua interpreter have not. That's fundamentally different.

[...] and for tests we could use (as suggested) LD_LIBRARY_PATH.

Then please add it. Thanks

Yes, you are right for the build system: CMake seems to set RPATH on every executable, but obviously not for python/lua.

When installed, however, it would not be set. So please someone confirm if kdb works (with TARGET_PLUGIN_FOLDER set).

@sanssecours can you disable BUILD_STATIC and BUILD_FULL and rerun all (testkdb) tests?

Looks like this does not change that much, except that ninja test runs less tests (54 instead of 114). The tests test_kdb.lua and testpy2_kdb.py still fail. All other tests (seem) to work.

https://cmake.org/Wiki/CMake_RPATH_handling says ".. RPATH on Mac OS X. It will be enabled for both the build tree and install tree", so actually also the build tree binaries should have RPATH set. E.g. using readelf -a lib/libelektra-core.so.0.8.16 | grep RPATH.

I used the command otool -l lib/libelektra-resolver.so — libelektra-core.so.0.8.16 does not exist on my machine. According to the output:

…
Load command 12
          cmd LC_RPATH
      cmdsize 104
         path /Users/rene/Dropbox/Studium/Master Thesis/Configuration Parsing/Source/Elektra/build/lib (offset 12)
…

RPATH is set.

E.g. kdb run_all would be interesting too (next to what I wrote above already).

On my machine 5 out of 655 test fail. 4 of the tests (testmod_crypto, testmod_iterate, testmod_semlock and testmod_template) fail because of the same basic error:

dyld: Library not loaded: @rpath/libelektra-full.4.dylib
  Referenced from: /usr/local/lib/elektra/tool_exec/testmod_crypto
  Reason: Incompatible library version: testmod_crypto requires version 4.0.0 or later, but libelektra-full.4.dylib provides version 0.0.0
/usr/local/lib/elektra/tool_exec/run_all: line 1070: 76960 Trace/BPT trap: 5       "$KDB" $t
error: testmod_crypto

dyld: Library not loaded: @rpath/libelektra-full.4.dylib
  Referenced from: /usr/local/lib/elektra/tool_exec/testmod_iterate
  Reason: Incompatible library version: testmod_iterate requires version 4.0.0 or later, but libelektra-full.4.dylib provides version 0.0.0
/usr/local/lib/elektra/tool_exec/run_all: line 1070: 77093 Trace/BPT trap: 5       "$KDB" $t
error: testmod_iterate

dyld: Library not loaded: @rpath/libelektra-full.4.dylib
  Referenced from: /usr/local/lib/elektra/tool_exec/testmod_semlock
  Reason: Incompatible library version: testmod_semlock requires version 4.0.0 or later, but libelektra-full.4.dylib provides version 0.0.0
/usr/local/lib/elektra/tool_exec/run_all: line 1070: 77239 Trace/BPT trap: 5       "$KDB" $t
error: testmod_semlock

dyld: Library not loaded: @rpath/libelektra-full.4.dylib
  Referenced from: /usr/local/lib/elektra/tool_exec/testmod_template
  Reason: Incompatible library version: testmod_template requires version 4.0.0 or later, but libelektra-full.4.dylib provides version 0.0.0
/usr/local/lib/elektra/tool_exec/run_all: line 1070: 77272 Trace/BPT trap: 5       "$KDB" $t
error: testmod_template

The test testmod_python2 shows the following error output:

PYTHON      TESTS
==================

Testing simple variable passing...
There are 1 warnings
buffer is: warnings/#00
number: 11
description: open of plugin returned unsuccessfully (reason contains plugin, see other warnings for details)
ingroup: kdb
module:
file: ../src/libs/elektra/plugin.c
line: 297
reason: python2
reason:
reason:
../src/plugins/python2/../python/testmod_python.c:53: error in test_variable_passing: warnings in kdbOpen for plugin python2
number: 111
description: : python error
ingroup: : plugin
module: : python
at: ../src/plugins/python2/../python/python.cpp:245
reason: : Unable to import kdb module
mountpoint: :
configfile: :
../src/plugins/python2/../python/testmod_python.c:53: error in test_variable_passing: error in kdbOpen for plugin python2
../src/plugins/python2/../python/testmod_python.c:53: fatal in test_variable_passing: could not open python2 plugin
error: testmod_python2

Below is the output of testmod_lua, which reports no errors.

--- running testmod_lua ---


LUA         TESTS
==================

Testing simple variable passing...
[LUA-1] open -->
[LUA-1] get
[LUA-1] <-- close
Testing loading of two active lua plugins...
[LUA-1] open -->
[LUA-2] open -->
[LUA-2] <-- close
[LUA-1] <-- close

========================================================================
NOTE: The following errors are intended. We're testing error conditions!
========================================================================
Testing return values from lua functions...
Testing lua script with syntax error...
There are 1 warnings
buffer is: warnings/#00
number: 11
description: open of plugin returned unsuccessfully (reason contains plugin, see other warnings for details)
ingroup: kdb
module:
file: ../src/libs/elektra/plugin.c
line: 297
reason: lua
reason:
reason:
number: 131
description: : lua error
ingroup: : plugin
module: : lua
at: ../src/plugins/lua/lua.cpp:80
reason: : /usr/local/share/elektra/test_data/lua/lua_plugin_wrong.lua:2: attempt to call global 'wrong' (a nil value)
mountpoint: :
configfile: :

test_lua RESULTS: 29 test(s) done. 0 error(s).

On OS X dylibs are created, not .so. Thereis definitely a libelektra-core.0.8.16.dylib or similar.

Thereis definitely a libelektra-core.0.8.16.dylib or similar.

You are right. Looks like RPATH is not set for libelektra-core.0.8.16.dylib, since otool -l lib/libelektra-core.0.8.16.dylib | grep RPATH displays no output.

Don't grep output is short, or grep lowercase rpath

Don't grep output is short, or grep lowercase rpath

Do you mean “Don't grep output, that is short, or grep lowercase rpath”? If so, why not? I also searched for RPATH in the output of otool -l lib/libelektra-core.0.8.16.dylib via the built-in search of my Terminal application. This also shows no occurrence of LC_RPATH.

If I search for rpath, then I see one occurrence of @rpath:

...
Load command 3
          cmd LC_ID_DYLIB
      cmdsize 56
         name @rpath/libelektra-core.4.dylib (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 0.0.0
compatibility version 0.0.0
...

As far as I know @rpath is only a placeholder for (the values stored in) RPATH. I do not think that this occurrence of rpath tells us if RPATH is set or not.

According to the CMake wiki the command otool -l <file> | grep LC_RPATH -A2 is one possible way to show the RPATHs of some file. I think the less fancy version that I used – otool -l lib/libelektra-core.0.8.16.dylib | grep RPATH – should be more or less fine too.

Guys, why are you checking this at all?

@sanssecours sorry, I sent the message quickly from my phone, so I didn't see that it made no sense.

I was referring to otool -L which shows the dependencies and only displays @rpath. But yes you're right otool -l is the right command.

Btw I haven't found a single library in my system which is using RPATH, except elektra.

/usr/local/lib/libelektra.0.8.13.dylib
          cmd LC_RPATH
      cmdsize 40
         path /usr/local/lib/elektra (offset 12)

As @manuelm states though, these checks might be rather pointless...

I do not think we fully understand all issues above, but I agree that the kind of remote debugging we are doing is not efficient. Somebody should investigate and document what is the best way to support non-glibc systems. Let us discuss in the meeting how to proceed.

@markus2330 I fully understand the issue. Set LD_LIBRARY_PATH before tests start and I promise the issue is gone.

All other mentioned issues are red herrings or just wrong.

@manuelm I wanted to be nice and said "we". Obviously the LD_LIBRARY_PATH would fix the issues of not finding libraries in the build directory (unless it is reset somewhere, which seems to be the case at least for python/lua GI tests). But if your theory is correct that RPATH must be set in the binary, also the installed version would be broken, which is currently not really confirmed. This should be investigated.

My theory isn't just a theory anymore, because I've done the proof.

Once again: On linux the bindings work because the library implementing the bindings (kdb.so for lua/python and libgelektra-4.0.so for glib) have DT_RPATH set _AND_ dlopen honors this.

Of course the bindings will work after elektra has been installed into the global system library paths. I absolutely see no reason why this should suddenly stop working.

The tests of the bindings don't work for !=linux. That's all.

PS: I know the GI bindings tests require LD_LIBRARY_PATH. That's why I wanted you to add some sort of build system macro which integrates/appends the GI case.

It seems like the binding tests are the only place where the LD_LIBRARY_PATH is needed, so I added it just there.

@mpranj Hopefully we soon get a travis that can confirm if it works?

@markus2330 no it does not work. (neither on travis nor on my machine)

@mpranj thanks for testing. Do you have the travis file ready to test it again without bothering someone? Both test cases still failing? Did I forget LD_LIBRARY_PATH somewhere or is the approach simply not working?

@markus2330 yes I am currently testing on travis, but it has mostly the same behaviour as on my machine. It seems like da243f9e25d8fa14f8286c48b4338a73c1e7242d made no difference.

You can see it here: https://travis-ci.org/mpranj/libelektra
And btw it seems like @manuelm was pretty much done with travis+OS X. The tests are failing but travis is actually almost set up completely I guess.

Yes, he said that. Mainly the matrix setup to build Mac OS X and others with one travis file was missing.

Thanks for all your help, issue should be fixed now.