Libelektra: Tests behave oddly in certain environments (debian packages)

Thank you for reporting it separately!

As a sideeffect https://build.libelektra.org/jenkins/job/elektra-git-buildpackage-jessie/ seems to be working again since yesterday...

GPG seems to have a problem with the cleared environment. I will experiment with setting the GNUPGHOME variable to /tmp. GPG relies on this directory (default is ~/.gnupg/) for looking up its key-chain. I think the gpg-agent info is also stored in this directory. Setting it to the same directory for all tests might also fix the error of multiple gpg-agents spawning up for every test run (see #1928 ).

Yes, I also thought that the failing crypto test cases are related to #1928. It seems that gpg uses .gnupg below the TMPDIR. (At least you see a .gnupg leftover after running TMPDIR=`pwd`/x ctest --output-on-failure -R fcrypt).

The failing fcrypt testcases might show a bug in the handling of temporary files, also related to TMPDIR.

it seems that gpg uses .gnupg below the TMPDIR

this is a wrong assumption. gpg uses ~/.gnupg as a default so it is not related to TMPDIR but to HOME.

Please also take note that setting GNUPGHOME to the same directory for all tests might influence parallel test execution.

this is a wrong assumption

Which assumption do you mean? I was only referring that I get a .gnupg folder within TMPDIR.

mkdir x && env TMPDIR=`pwd`/x ctest --output-on-failure -R fcrypt; find x
Test project /home/markus/Projekte/Elektra/build
    Start 64: testmod_fcrypt
1/1 Test #64: testmod_fcrypt ...................***Failed    0.02 sec
FCRYPT       TESTS
==================

/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:187: error in test_file_crypto_operations: kdb set failed
/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:188: error in test_file_crypto_operations: file content did not change during encryption
/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:191: error in test_file_crypto_operations: kdb get (pregetstorage) failed
/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:195: error in test_file_crypto_operations: kdb get (postgetstorage) failed
/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:196: error in test_file_crypto_operations: postgetstorage did not encrypt the file again
/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:231: error in test_file_signature_operations: kdb set failed
/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:232: error in test_file_signature_operations: file content did not change during encryption
/home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:235: error in test_file_signature_operations: kdb get failed

fcrypt Results: 39 Tests done — 8 errors.


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.03 sec

The following tests FAILED:
         64 - testmod_fcrypt (Failed)
Errors while running CTest
x
x/elektra-test.qPoz5P
x/elektra-test.qPoz5P/.gnupg
x/elektra-test.qPoz5P/.gnupg/random_seed
x/elektra-test.qPoz5P/.gnupg/trustdb.gpg
x/elektra-test.qPoz5P/.gnupg/pubring.kbx~
x/elektra-test.qPoz5P/.gnupg/pubring.kbx

that is because something in the tests(ctest?) sets the home directory for each test suite to a tmpdir in TMPDIR.
GPG creates the files in HOME, but HOME is set to TMPDIR/randomdir/ by the testsuite.
Hence my point is that it is not tmpdir's sole fault and debugging effort should not be restricted to it.

I think it is a good idea to have seperate home dirs for each test so they do not influence each other (parallelisation) and hence do not think that overriding gpg directories to share a single directory is a good idea.

Yes, I agree.

Does #1977 make sense? Or should it be done after ctest finished completely?

Or asked differently: Does the gpg-agent survives a run of a single test suite or are they respawned for every test suite?

@markus2330 As mentioned in #2040 . I followed along the instructions on building the Docker images. Inside the docker container I ran:

# cmake -DBUILD_DOCUMENTATION=OFF -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
# make -j16
# mkdir x
# TMPDIR=`pwd`/x ctest -R crypt .
Test project /elektra/build
    Start 46: testmod_crypto_openssl
1/3 Test #46: testmod_crypto_openssl ...........   Passed    1.32 sec
    Start 47: testmod_crypto_gcrypt
2/3 Test #47: testmod_crypto_gcrypt ............   Passed    1.16 sec
    Start 48: testmod_crypto_botan
3/3 Test #48: testmod_crypto_botan .............   Passed    1.52 sec

100% tests passed, 0 tests failed out of 3

Label Time Summary:
memleak    =   1.16 sec (1 test)

Total Test time (real) =   4.01 sec

So (a part of) the problem is that the test cases assumes the TMPDIR to be present. And you could not reproduce it because you always created the directory before running the tests :smile:

But, for me fcrypt nevertheless fails, even when x is present (you did not run fcrypt). But with other messages:

64: FCRYPT       TESTS
64: ==================
64: 
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:187: error in test_file_crypto_operations: kdb set failed
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:188: error in test_file_crypto_operations: file content did not change during encryption
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:191: error in test_file_crypto_operations: kdb get (pregetstorage) failed
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:195: error in test_file_crypto_operations: kdb get (postgetstorage) failed
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:196: error in test_file_crypto_operations: postgetstorage did not encrypt the file again
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:231: error in test_file_signature_operations: kdb set failed
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:232: error in test_file_signature_operations: file content did not change during encryption
64: /home/markus/Projekte/Elektra/current/src/plugins/fcrypt/testmod_fcrypt.c:235: error in test_file_signature_operations: kdb get failed
64: 
64: fcrypt Results: 39 Tests done — 8 errors.
1/1 Test #64: testmod_fcrypt ...................***Failed    0.02 sec

But it works in my container (I've been waiting a long time to drop that line :laughing: )...

jenkins@924c627f07b8:/elektra/build$ ctest --output-on-failure -R fcrypt
Test project /elektra/build
    Start 59: testmod_fcrypt
1/1 Test #59: testmod_fcrypt ...................   Passed    1.07 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   1.08 sec
jenkins@924c627f07b8:/elektra/build$ TMPDIR=`pwd`/x ctest --output-on-failure -R fcrypt
Test project /elektra/build
    Start 59: testmod_fcrypt
1/1 Test #59: testmod_fcrypt ...................***Failed    0.01 sec
FCRYPT       TESTS
==================

/elektra/tests/cframework/tests.c:80: error in init: mkdtemp failed
/elektra/tests/cframework/tests.c:90: error in init: mkstemp failed
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:159: error in test_gpg: failed to install the GPG test key
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:71: error in writeTestFile: can not write to temporary file
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:187: error in test_file_crypto_operations: kdb set failed
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:99: error in isTestFileCorrect: can not read from temporary file
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:191: error in test_file_crypto_operations: kdb get (pregetstorage) failed
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:99: error in isTestFileCorrect: can not read from temporary file
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:192: error in test_file_crypto_operations: file content could not be restored during decryption
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:195: error in test_file_crypto_operations: kdb get (postgetstorage) failed
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:99: error in isTestFileCorrect: can not read from temporary file
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:71: error in writeTestFile: can not write to temporary file
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:231: error in test_file_signature_operations: kdb set failed
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:99: error in isTestFileCorrect: can not read from temporary file
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:235: error in test_file_signature_operations: kdb get failed
/elektra/src/plugins/fcrypt/testmod_fcrypt.c:80: error in writeFaultySignatureFile: can not write to temporary file

fcrypt Results: 32 Tests done — 16 errors.


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.02 sec

The following tests FAILED:
        59 - testmod_fcrypt (Failed)
Errors while running CTest
jenkins@924c627f07b8:/elektra/build$ mkdir x
jenkins@924c627f07b8:/elektra/build$ TMPDIR=`pwd`/x ctest --output-on-failure -R fcrypt
Test project /elektra/build
    Start 59: testmod_fcrypt
1/1 Test #59: testmod_fcrypt ...................   Passed    1.06 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   1.07 sec
jenkins@924c627f07b8:/elektra/build$

some more interesting stuff:

• why does mkdtemp fail?
• clean_temp_home apparently has been written to cleanup gnupg stuff already. Why isn't it working?
  
  
  
  • seems like the names changed
  
  
• even when the dir in TMPDIR exists i have fcrypt failing (the other tests pass)
  
  
  
  • when not specifying TMPDIR fcrypt passes
  
  

well... check the warnings for tests.c and everything becomes much more clearer...

I am prepping a PR for it though which should resolve some of the issues.

If somebody could find out if fcrypt has some hard paths specified somewhere that could lead to the tests failing that would be lovely.

Ìn #2056 I cleanup the generated TMPDIR properly. This resolves the lingering gpg-agent issue.
It also fixes an issue with newer glibc versions that did not activate mkdtemp.

However the fcrypt test still fails as soon as TMPDIR is set manually and I think there might be a real bug in the code somewhere.

jenkins@0585a86e6f5f:~/workspace/build$ pidof gpg-agent
jenkins@0585a86e6f5f:~/workspace/build$ TMPDIR=`pwd`/tmp ctest --output-on-failure -R crypt
Test project /home/jenkins/workspace/build
    Start 51: testmod_crypto_openssl
1/4 Test #51: testmod_crypto_openssl ...........   Passed    1.58 sec
    Start 52: testmod_crypto_gcrypt
2/4 Test #52: testmod_crypto_gcrypt ............   Passed    1.34 sec
    Start 53: testmod_crypto_botan
3/4 Test #53: testmod_crypto_botan .............   Passed    1.83 sec
    Start 64: testmod_fcrypt
4/4 Test #64: testmod_fcrypt ...................***Failed    1.17 sec
FCRYPT       TESTS
==================

/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:187: error in test_file_crypto_operations: kdb set failed
/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:188: error in test_file_crypto_operations: file content did not change during encryption
/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:191: error in test_file_crypto_operations: kdb get (pregetstorage) failed
/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:195: error in test_file_crypto_operations: kdb get (postgetstorage) failed
/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:196: error in test_file_crypto_operations: postgetstorage did not encrypt the file again
/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:231: error in test_file_signature_operations: kdb set failed
/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:232: error in test_file_signature_operations: file content did not change during encryption
/home/jenkins/workspace/src/plugins/fcrypt/testmod_fcrypt.c:235: error in test_file_signature_operations: kdb get failed

fcrypt Results: 39 Tests done — 8 errors.


75% tests passed, 1 tests failed out of 4

Label Time Summary:
memleak    =   1.34 sec (1 test)

Total Test time (real) =   5.98 sec

The following tests FAILED:
     64 - testmod_fcrypt (Failed)
Errors while running CTest
jenkins@0585a86e6f5f:~/workspace/build$ pidof gpg-agent
jenkins@0585a86e6f5f:~/workspace/build$ ls -l tmp
total 0

However the fcrypt test still fails as soon as TMPDIR is set manually and I think there might be a real bug in the code somewhere.

You are right!

getTemporaryFileName in fcrypt.c does not check the environment. So if no tmpdir is specified in the plugin configuration, it uses /tmp as default fallback.

I suppose the correct prioritization would be:

1. plugin conf
2. environment $TMPDIR
3. default fallback /tmp

Do you agree?

ISO/IEC 9945 (POSIX): The path supplied by the first environment variable found in the list TMPDIR, TMP, TEMP, TEMPDIR. If none of these are found, "/tmp".

Since we are using mkdtemp (POSIX 2008) anyway you could probably use that as well and get rid of some code.

Thank you for the info! Have you been able to verify that the proposed fix in #2090 fixes the fcrypt test problem on the build server?

if it works on your system with TMPDIR=dir/that/exists ctest -R fcrypt it should work on the build server as well.

The issue should be fixed now.

i concur

There still seems to be some kind of oddity with the tests. I run some builds of master mostly to finetune executor and parallel settings since we don't do hyperthreading anymore. And in one case the tests suddenly failed with the same 'could not install key' problem again:
https://build.libelektra.org/jenkins/blue/organizations/jenkins/libelektra/detail/master/122/pipeline/410

Sorry to pile on here, but I ran into the failed to install the GPG test key error too. I use a vagrant artful32 vm and can consistently reproduce these errors (master branch):

 55/198 Test  #55: testmod_crypto_gcrypt ....................***Failed    0.60 sec
CYPTO        TESTS
==================

/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/crypto/test_internals.h:273: error in test_gpg: failed to install the GPG test key

crypto_gcrypt Results: 65 Tests done — 1 error.

 66/198 Test  #66: testmod_fcrypt ...........................***Failed    0.03 sec
FCRYPT       TESTS
==================

/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:160: error in test_gpg: failed to install the GPG test key
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:188: error in test_file_crypto_operations: kdb set failed
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:189: error in test_file_crypto_operations: file content did not change during encryption
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:192: error in test_file_crypto_operations: kdb get (pregetstorage) failed
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:196: error in test_file_crypto_operations: kdb get (postgetstorage) failed
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:197: error in test_file_crypto_operations: postgetstorage did not encrypt the file again
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:232: error in test_file_signature_operations: kdb set failed
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:233: error in test_file_signature_operations: file content did not change during encryption
/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/fcrypt/testmod_fcrypt.c:236: error in test_file_signature_operations: kdb get failed

fcrypt Results: 39 Tests done — 9 errors.

I can provide the vagrantfile via PR if it helps to reproduce the errors.

Thank you for reporting! Can you tell us your TMPDIR and HOME? Try to reproduce it with env -i.

@petermax2 Do you have time to look at it?

For the errors above:
TMPDIR: empty
HOME: /home/jenkins/workspace/elektra-master-full-artful32/user

As for env -i, one more test fails then:

 54/198 Test  #54: testmod_crypto_openssl ...................***Failed    0.31 sec
CYPTO        TESTS
==================

/home/jenkins/workspace/elektra-master-full-artful32/src/plugins/crypto/test_internals.h:273: error in test_gpg: failed to install the GPG test key

crypto_openssl Results: 65 Tests done — 1 error.

Not sure if this is a coincidence, but this was the first time that this test failed.

I can investigate over the weekend. @mpranj could you send me the vagrant file please?

@petermax2 the Vagrantfile & steps to reproduce are in #2211. I hope this helps do debug the problem more easily. Maybe it is really just an ENV problem.

$HOMEDIR is not being deleted after the test case is done, thus the gpg-agent keeps runnig, which causes the problem. If GNUPGHOME is set to a different directory, the test case succeeds again.

EDIT: $HOMEDIR is being deleted properly, yet gpgagent does not quit as expected.

EDIT v2:

vagrant@ubuntu-artful:~/libelektra/build$ gpg --list-key
gpg: directory '/home/vagrant/.gnupg' created
gpg: new configuration file '/home/vagrant/.gnupg/dirmngr.conf' created
gpg: new configuration file '/home/vagrant/.gnupg/gpg.conf' created
gpg: keybox '/home/vagrant/.gnupg/pubring.kbx' created
gpg: /home/vagrant/.gnupg/trustdb.gpg: trustdb created
vagrant@ubuntu-artful:~/libelektra/build$ pgrep agent
3630
vagrant@ubuntu-artful:~/libelektra/build$ rm -rf /home/vagrant/.gnupg
vagrant@ubuntu-artful:~/libelektra/build$ pgrep agent
3630
vagrant@ubuntu-artful:~/libelektra/build$

Thank you for the investigation!

Is there some overview in which version which env vars do what and since when the gpg agent closes when the files are removed?

What is the best way to fix this?

Sorry for the delay, I had no time to further investigate so far.

Is there some overview in which version which env vars do what and since when the gpg agent closes when the files are removed?

All the documentation I could find was

_(GNUPGHOME) If set directory used instead of "~/.gnupg"._ [1]

What is the best way to fix this?

I am not sure. It does not seem like there is a single solution that solves all the problems we are facing with GnuPG. I once tried to save the PIDs of startetd GnuPG agents but it did not turn out well.

Do you have any ideas?

Sources

• [1] [https://www.gnupg.org/documentation/manuals/gnupg/GPG-Configuration.html](https://www.gnupg.org/documentation/manuals/gnupg/GPG-Configuration.html)

Maybe we should not try to solve every possible problem and only support some versions of gnupg?

Which versions of gnupg do not clean themselves up after removing their files? Are they relevant enough to still support them? Maybe we simply disable the tests for them?

Which versions of gnupg do not clean themselves up after removing their files?

Ubuntu Artful (17.10)'s version of gpg is the only version so far where the agent does not terminate after deleting the GNUPGHOME. I have not seen this behavior in other versions of gpg.

The GPG in Ubuntu Artful is 2.1.15. In Ubuntu 18.04.1 gpg version 2.2.4 is used. The latest version is 2.2.10.

We might add a dependency note that gpg version >= 2.2 is required.

If if understand the release notes correctly the termination is implemented in gpg version 2.1.16. See Release Notes.

EDIT: the release note states that the termination is only implemented for Linux. Meaning that on other OS we should get the same problem. I'll test it on FreeBSD.

I'll test it on FreeBSD.

Yep, gpg-agent is still running after GNUPGHOME is being removed.

ATM I'm testing another solution. Executing gpg-connect-agent KILLAGENT shuts down the agent in gpg version 2.2.4. If we execv this command after the unit test is complete, the gpg-agent should shut down. I will test this solution under Linux and FreeBSD. Please wait for a comment or a hopefully a PR!

Please re-open this issue if the problem occurs again.