* Makefile.am (TESTS_ENVIRONMENT): New variable.
(check-all): New phony target to run all tests.
* tests/gpgme/gpgme-defs.scm (have-gpgme?): New function that tests
whether the GPGME test suite is available instead of exiting the
process.
* tests/gpgscm/init.scm (export): New macro.
* tests/gpgscm/tests.scm (run-tests): New function.
(load-tests): Likewise.
* tests/gpgme/run-tests.scm: Simplify and move the parsing of the list
of tests to 'all-tests.scm'.
* tests/gpgsm/run-tests.scm: Likewise.
* tests/migrations/run-tests.scm: Likewise.
* tests/openpgp/run-tests.scm: Likewise.
* tests/gpgme/Makefile.am: To select the tests to run, use the
variable 'TESTS'. This harmonizes the interface with the automake
test suite.
* tests/gpgsm/Makefile.am: Likewise.
* tests/migrations/Makefile.am: Likewise.
* tests/openpgp/Makefile.am: Likewise.
* tests/openpgp/README: Likewise.
* agent/all-tests.scm: New file.
* common/all-tests.scm: Likewise.
* g10/all-tests.scm: Likewise.
* g13/all-tests.scm: Likewise.
* tests/gpgme/all-tests.scm: Likewise.
* tests/gpgsm/all-tests.scm: Likewise.
* tests/migrations/all-tests.scm: Likewise.
* tests/openpgp/all-tests.scm: Likewise.
* tests/run-tests.scm: Likewise.
--
This change allows us to run all tests in parallel and write one XML
report capturing the results of every test. It also lays the
foundation to parametrize test suites.
Signed-off-by: Justus Winter <justus@g10code.com>
--
Locate every resource and every script used in the tests using a path
relative to the top of the source tree.
This is a purely mechanical change, mostly done using regular
expressions, with a few manual fixups here and there.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (test::scm): Add a setup argument.
(test::binary): Likewise.
(run-tests-parallel): Remove setup parameter.
(run-tests-sequential): Likewise.
(make-environment-cache): New function that handles the cache
protocol.
* tests/gpgme/run-tests.scm: Adapt accordingly.
* tests/gpgsm/run-tests.scm: Likewise.
* tests/migrations/run-tests.scm: Likewise.
* tests/openpgp/run-tests.scm: Likewise.
--
This change allows us to have different environments for tests. This
is needed to run more GPGME tests, and to increase concurrency while
running all tests.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/openpgp/run-tests.scm: Use sequential test runner if only one
test is given.
--
This allows one to set the environment variable TESTFLAGS to
'--parallel' and enjoy faster test execution times without interfering
with stdio when one works on a single test.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/openpgp/run-tests.scm (locate-test): New function.
(test): Factor-out the code starting the child process.
(test::binary): New function.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/openpgp/run-tests.scm (test): Add field 'logfd'.
(test::new, test::set-*): Adapt accordingly.
(test::set-logfd): New function.
(test::open-log-file): Likewise.
(test::run-sync): Use the new function.
(test::run-async): Likewise.
(test::report): Replay the log.
(run-tests-parallel): Reverse the results to restore the original
order.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/openpgp/run-tests.scm (run-tests-parallel): Catch errors when
removing the working directory. On Windows this can fail if there is
still a process using one of the files there.
(run-tests-sequential): Likewise.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (log): New function.
* tests/openpgp/run-tests.scm (run-tests-parallel): Do not run the
startup and teardown scripts.
(run-tests-sequential): Likewise.
* tests/openpgp/setup.scm: Move all functions...
* tests/openpgp/defs.scm: ... here and make them less verbose.
(setup-environment): New function.
(setup-legacy-environment): Likewise.
(start-agent): Make less verbose, run 'stop-agent' at interpreter
exit.
(stop-agent): Make less verbose.
* tests/openpgp/finish.scm: Drop file.
* tests/openpgp/Makefile.am (EXTRA_DIST): Drop removed file.
* tests/openpgp/4gb-packet.scm: Use 'setup-environment' or
'setup-legacy-environment' as appropriate.
* tests/openpgp/armdetach.scm: Likewise.
* tests/openpgp/armdetachm.scm: Likewise.
* tests/openpgp/armencrypt.scm: Likewise.
* tests/openpgp/armencryptp.scm: Likewise.
* tests/openpgp/armor.scm: Likewise.
* tests/openpgp/armsignencrypt.scm: Likewise.
* tests/openpgp/armsigs.scm: Likewise.
* tests/openpgp/clearsig.scm: Likewise.
* tests/openpgp/conventional-mdc.scm: Likewise.
* tests/openpgp/conventional.scm: Likewise.
* tests/openpgp/decrypt-dsa.scm: Likewise.
* tests/openpgp/decrypt.scm: Likewise.
* tests/openpgp/default-key.scm: Likewise.
* tests/openpgp/detach.scm: Likewise.
* tests/openpgp/detachm.scm: Likewise.
* tests/openpgp/ecc.scm: Likewise.
* tests/openpgp/encrypt-dsa.scm: Likewise.
* tests/openpgp/encrypt.scm: Likewise.
* tests/openpgp/encryptp.scm: Likewise.
* tests/openpgp/export.scm: Likewise.
* tests/openpgp/finish.scm: Likewise.
* tests/openpgp/genkey1024.scm: Likewise.
* tests/openpgp/gpgtar.scm: Likewise.
* tests/openpgp/gpgv-forged-keyring.scm: Likewise.
* tests/openpgp/import.scm: Likewise.
* tests/openpgp/issue2015.scm: Likewise.
* tests/openpgp/issue2417.scm: Likewise.
* tests/openpgp/issue2419.scm: Likewise.
* tests/openpgp/key-selection.scm: Likewise.
* tests/openpgp/mds.scm: Likewise.
* tests/openpgp/multisig.scm: Likewise.
* tests/openpgp/quick-key-manipulation.scm: Likewise.
* tests/openpgp/seat.scm: Likewise.
* tests/openpgp/shell.scm: Likewise.
* tests/openpgp/signencrypt-dsa.scm: Likewise.
* tests/openpgp/signencrypt.scm: Likewise.
* tests/openpgp/sigs-dsa.scm: Likewise.
* tests/openpgp/sigs.scm: Likewise.
* tests/openpgp/ssh.scm: Likewise.
* tests/openpgp/tofu.scm: Likewise.
* tests/openpgp/use-exact-key.scm: Likewise.
* tests/openpgp/verify.scm: Likewise.
* tests/openpgp/version.scm: Likewise.
* tests/openpgp/issue2346.scm: Likewise and simplify.
--
The previous Bourne Shell-based test suite created the environment
before running all tests, and tore it down after executing them. When
we created the Scheme-based test suite, we kept this design at first,
but introduced a way to run each test in its own environment to
prevent tests from interfering with each other. Nevertheless, every
test started out with the same environment.
Move the creation of the test environment into each test. This gives
us finer control over the environment each test is run in. It also
makes it possible to run each test by simply executing it using gpgscm
without the use of the runner. Furthermore, it has the neat
side-effect of speeding up the test suite if run in parallel.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/openpgp/README: Add documentation about debugging and
interfacing with GnuPG.
* tests/openpgp/run-tests.scm (test::run-sync): Hand stdin to the
child so that we can use a repl in the tests.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/openpgp/run-tests.scm (test::run-sync): Pass additional
arguments to the test.
(test::run-sync-quiet): Likewise.
(test::run-async): Likewise.
(run-tests-{parallel,sequential}-isolated): Create a tarball of the
gnupghome, then extract it for each test.
* tests/openpgp/setup.scm: Refactor into functions, add an interface
to tar-up the created environment, and untar it multiple times.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/ffi.c (ffi_init): Rename 'mkdtemp'.
* tests/gpgscm/tests.scm (mkdtemp): New function that uses a sensible
location and template if no arguments are given.
(with-temporary-working-directory): Simplify accordingly.
(make-temporary-file): Likewise.
* tests/openpgp/run-tests.scm (run-tests-parallel-isolated): Likewise.
(run-tests-sequential-isolated): Likewise.
Signed-off-by: Justus Winter <justus@g10code.com>
