* tests/gpgscm/tests.scm (test-pool): Add delimiter lines.
--
This is to make those summaries a bit more simlar to those from
automake.
Signed-off-by: Werner Koch <wk@gnupg.org>
* tests/gpgscm/repl.scm (prompt-yes-no?): New function.
* tests/gpgscm/tests.scm (pathsep-split): Likewise.
(pathsep-join): Likewise.
(with-path): Use the new function.
Signed-off-by: Justus Winter <justus@g10code.com>
* ffi.c (do_wait_processes): Suppress the timeout error.
* tests.scm (semaphore): New definition.
(test-pool): Only run a bounded number of tests in parallel.
(test::started?): New function.
(run-tests-parallel): Do not report results, do not start the tests.
(run-tests-sequential): Adapt.
(run-tests): Parse the number of parallel jobs.
--
This change limits the number of tests that are run in parallel. This
way we do not overwhelm the operating systems' scheduler. As a
side-effect, we also get more accurate runtime information, and it
will be easy to implement timeouts on top of this.
Use TESTFLAGS to limit the number of jobs:
$ make check-all TESTFLAGS=--parallel=16
Signed-off-by: Justus Winter <justus@g10code.com>
* Makefile.am (TESTS_ENVIRONMENT): New variable.
(check-all): New phony target to run all tests.
* tests/gpgme/gpgme-defs.scm (have-gpgme?): New function that tests
whether the GPGME test suite is available instead of exiting the
process.
* tests/gpgscm/init.scm (export): New macro.
* tests/gpgscm/tests.scm (run-tests): New function.
(load-tests): Likewise.
* tests/gpgme/run-tests.scm: Simplify and move the parsing of the list
of tests to 'all-tests.scm'.
* tests/gpgsm/run-tests.scm: Likewise.
* tests/migrations/run-tests.scm: Likewise.
* tests/openpgp/run-tests.scm: Likewise.
* tests/gpgme/Makefile.am: To select the tests to run, use the
variable 'TESTS'. This harmonizes the interface with the automake
test suite.
* tests/gpgsm/Makefile.am: Likewise.
* tests/migrations/Makefile.am: Likewise.
* tests/openpgp/Makefile.am: Likewise.
* tests/openpgp/README: Likewise.
* agent/all-tests.scm: New file.
* common/all-tests.scm: Likewise.
* g10/all-tests.scm: Likewise.
* g13/all-tests.scm: Likewise.
* tests/gpgme/all-tests.scm: Likewise.
* tests/gpgsm/all-tests.scm: Likewise.
* tests/migrations/all-tests.scm: Likewise.
* tests/openpgp/all-tests.scm: Likewise.
* tests/run-tests.scm: Likewise.
--
This change allows us to run all tests in parallel and write one XML
report capturing the results of every test. It also lays the
foundation to parametrize test suites.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (test-pool): Rework reporting. Filter using
the computed test status instead of the return value. Also print the
new categories 'failed expectedly' and 'passed unexpectedly'.
(test): If a test ends with a bang (!), it is expected to fail. Adapt
status, status-string, and xml accordingly.
--
Allow tests to be marked as being expected to fail by appending a bang
(!) to the tests name. If such a test fails, it will not be counted
as failure, but will still be prominently displayed in the report. If
it succeeds unexpectedly, this is counted as a failure.
Fixes T3134.
GnuPG-bug-id: 3134
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (call-with-io): When being verbose, include
the pid in the output, and avoid duplicating the command arguments.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (test-pool, tests): Previously, these methods
updated objects by creating new updated copies of the object being
manipulated. This made the code awkward without any benefit,
therefore I change it to just update the object.
Signed-off-by: Justus Winter <justus@g10code.com>
--
Locate every resource and every script used in the tests using a path
relative to the top of the source tree.
This is a purely mechanical change, mostly done using regular
expressions, with a few manual fixups here and there.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (mkdtemp-autoremove): New function that
cleans up at interpreter shutdown.
(run-tests-parallel): Use the new function.
(run-tests-sequential): Likewise.
(make-environment-cache): Execute setup with an temporary working
directory.
--
Make sure to remove all resources created in the filesystem even if
the test runner is interrupted. Make sure to remove anything that the
setup script creates.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/ffi.c (do_get_temp_path): New function.
(ffi_init): Make function available.
* tests/gpgscm/tests.scm (mkdtemp): Use the new function.
Fixes-commit: 06f1f163e96f1039304fd3cf565cf9de1ca45849
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (mkdtemp): Create temporary directories in
'/tmp' on UNIX, or in '%Temp' on Windows.
* tests/migrations/common.scm (run-test): Turn error into a warning.
* tests/openpgp/defs.scm (start-agent): Likewise.
--
This fixes the problem of GnuPG components being unable to communicate
because of too long GnuPG home directories in important build
environments like the Debian build servers despite the use of socket
directories.
This reverts d75d20909d9f60d33ffd210def92278c0f383aad.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (test::scm): Add a setup argument.
(test::binary): Likewise.
(run-tests-parallel): Remove setup parameter.
(run-tests-sequential): Likewise.
(make-environment-cache): New function that handles the cache
protocol.
* tests/gpgme/run-tests.scm: Adapt accordingly.
* tests/gpgsm/run-tests.scm: Likewise.
* tests/migrations/run-tests.scm: Likewise.
* tests/openpgp/run-tests.scm: Likewise.
--
This change allows us to have different environments for tests. This
is needed to run more GPGME tests, and to increase concurrency while
running all tests.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (mkdtemp): Do not magically obey the
environment variable 'TMP', make sure to always return an absolute
path.
* tests/gpgme/Makefile.am (TMP): Drop variable.
(TESTS_ENVIRONMENT): Drop 'TMP'.
* tests/gpgme/gpgme-defs.scm (create-gpgmehome): Start the agent. Do
not create private key store, the agent does that for us.
* tests/gpgsm/Makefile.am (TMP): Drop variable.
(TESTS_ENVIRONMENT): Drop 'TMP'.
* tests/gpgme/gpgme-defs.scm (create-gpgsmhome): Start the agent. Do
not create private key store, the agent does that for us.
* tests/migrations/Makefile.am (TMP): Drop variable.
(TESTS_ENVIRONMENT): Drop 'TMP'.
* tests/migrations/common.scm (gpgconf): New variable.
(run-test): Create and remove socket directory.
* tests/migrations/extended-pkf.scm (src-tarball): Remove variable.
(setup): Remove function.
(trigger-migration): Likewise.
Use 'run-test' to execute the test.
* tests/migrations/from-classic.scm (src-tarball): Remove variable.
(setup): Remove function.
Use 'run-test' to execute the tests.
* tests/openpgp/Makefile.am (TMP): Drop variable.
(TESTS_ENVIRONMENT): Drop 'TMP'.
* tests/openpgp/README: Do not mention 'TMP'.
* tests/openpgp/defs.scm (with-home-directory): New macro.
(create-legacy-gpghome): Do not create private key store, the agent
does that for us.
(start-agent): Make sure to terminate the right agent with 'atexit'.
--
Previously, the test suite relied upon creating home directories in
'/tmp'. This has been problematic in some build environments,
although POSIX mandates that '/tmp' must be available.
We now rely on 'gpgconf --create-socketdir' to create a suitable
socket directory for us. This allows us to get rid of some cruft. It
also aligns the environment the tests are run in closer with the
environment that we intend that GnuPG runs in.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/init.scm (finally): New macro.
* tests/gpgscm/tests.scm (letfd): Rewrite.
(with-working-directory): Likewise.
(with-temporary-working-directory): Likewise.
(lettmp): Likewise.
--
Rewrite all our macros using 'define-macro'. Use the new control flow
mechanism 'finally', or 'dynamic-wind' where appropriate. Make sure
the macros are hygienic. Reduce code duplication.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (absolute-path?): New function.
(canonical-path): Use the new function.
* tests/gpgscm/lib.scm (string-split-pln): New function.
(string-indexp, string-splitp): Likewise.
(string-splitn): Express using the above function.
(string-ltrim, string-rtrim): Fix corner case.
(list->string-reversed): New function.
(read-line): Fix performance.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/ffi.scm: Move the customized exception handling and
atexit logic...
* tests/gpgscm/init.scm: ... here.
(throw): Record the current history.
(throw'): New function that is history-aware.
(rethrow): New function.
(*error-hook*): Use the new throw'.
* tests/gpgscm/main.c (load): Fix error handling.
(main): Save and use the 'sc->retcode' as exit code.
* tests/gpgscm/repl.scm (repl): Print call history.
* tests/gpgscm/scheme.c (_Error_1): Make a snapshot of the history,
use it to provide a accurate location of the expression causing the
error at runtime, and hand the history trace to the '*error-hook*'.
(opexe_5): Tag all lists at parse time with the current location.
* tests/gpgscm/tests.scm: Update calls to 'throw', use 'rethrow'.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (log): New function.
* tests/openpgp/run-tests.scm (run-tests-parallel): Do not run the
startup and teardown scripts.
(run-tests-sequential): Likewise.
* tests/openpgp/setup.scm: Move all functions...
* tests/openpgp/defs.scm: ... here and make them less verbose.
(setup-environment): New function.
(setup-legacy-environment): Likewise.
(start-agent): Make less verbose, run 'stop-agent' at interpreter
exit.
(stop-agent): Make less verbose.
* tests/openpgp/finish.scm: Drop file.
* tests/openpgp/Makefile.am (EXTRA_DIST): Drop removed file.
* tests/openpgp/4gb-packet.scm: Use 'setup-environment' or
'setup-legacy-environment' as appropriate.
* tests/openpgp/armdetach.scm: Likewise.
* tests/openpgp/armdetachm.scm: Likewise.
* tests/openpgp/armencrypt.scm: Likewise.
* tests/openpgp/armencryptp.scm: Likewise.
* tests/openpgp/armor.scm: Likewise.
* tests/openpgp/armsignencrypt.scm: Likewise.
* tests/openpgp/armsigs.scm: Likewise.
* tests/openpgp/clearsig.scm: Likewise.
* tests/openpgp/conventional-mdc.scm: Likewise.
* tests/openpgp/conventional.scm: Likewise.
* tests/openpgp/decrypt-dsa.scm: Likewise.
* tests/openpgp/decrypt.scm: Likewise.
* tests/openpgp/default-key.scm: Likewise.
* tests/openpgp/detach.scm: Likewise.
* tests/openpgp/detachm.scm: Likewise.
* tests/openpgp/ecc.scm: Likewise.
* tests/openpgp/encrypt-dsa.scm: Likewise.
* tests/openpgp/encrypt.scm: Likewise.
* tests/openpgp/encryptp.scm: Likewise.
* tests/openpgp/export.scm: Likewise.
* tests/openpgp/finish.scm: Likewise.
* tests/openpgp/genkey1024.scm: Likewise.
* tests/openpgp/gpgtar.scm: Likewise.
* tests/openpgp/gpgv-forged-keyring.scm: Likewise.
* tests/openpgp/import.scm: Likewise.
* tests/openpgp/issue2015.scm: Likewise.
* tests/openpgp/issue2417.scm: Likewise.
* tests/openpgp/issue2419.scm: Likewise.
* tests/openpgp/key-selection.scm: Likewise.
* tests/openpgp/mds.scm: Likewise.
* tests/openpgp/multisig.scm: Likewise.
* tests/openpgp/quick-key-manipulation.scm: Likewise.
* tests/openpgp/seat.scm: Likewise.
* tests/openpgp/shell.scm: Likewise.
* tests/openpgp/signencrypt-dsa.scm: Likewise.
* tests/openpgp/signencrypt.scm: Likewise.
* tests/openpgp/sigs-dsa.scm: Likewise.
* tests/openpgp/sigs.scm: Likewise.
* tests/openpgp/ssh.scm: Likewise.
* tests/openpgp/tofu.scm: Likewise.
* tests/openpgp/use-exact-key.scm: Likewise.
* tests/openpgp/verify.scm: Likewise.
* tests/openpgp/version.scm: Likewise.
* tests/openpgp/issue2346.scm: Likewise and simplify.
--
The previous Bourne Shell-based test suite created the environment
before running all tests, and tore it down after executing them. When
we created the Scheme-based test suite, we kept this design at first,
but introduced a way to run each test in its own environment to
prevent tests from interfering with each other. Nevertheless, every
test started out with the same environment.
Move the creation of the test environment into each test. This gives
us finer control over the environment each test is run in. It also
makes it possible to run each test by simply executing it using gpgscm
without the use of the runner. Furthermore, it has the neat
side-effect of speeding up the test suite if run in parallel.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/ffi.c (ffi_init): New Scheme variable '*win32*'.
* tests/gpgscm/tests.scm (canonical-path): Correctly handle paths with
drive letter on Windows. Use 'path-join'.
(path-expand): Use 'path-join'.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/init.scm (catch): Bind all arguments to '*error*' in
the error handler, update and fix comment.
(*error-hook*): Revert to original definition.
* tests/gpgscm/tests.scm (tr:do): Adapt accordingly.
* tests/openpgp/issue2419.scm: Likewise.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/ffi.c (do_get_isotime): New function.
(ffi_init): Add parameter 'scriptname', bind new function and
scriptname.
* tests/gpgscm/ffi.h (ffi_init): Update prototype.
* tests/gpgscm/main.c (main): Hand in the script name.
* tests/gpgscm/tests.scm (mkdtemp): Use current time and script name
for the names of temporary directories.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/ffi.c (ffi_init): Rename 'mkdtemp'.
* tests/gpgscm/tests.scm (mkdtemp): New function that uses a sensible
location and template if no arguments are given.
(with-temporary-working-directory): Simplify accordingly.
(make-temporary-file): Likewise.
* tests/openpgp/run-tests.scm (run-tests-parallel-isolated): Likewise.
(run-tests-sequential-isolated): Likewise.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/tests.scm (call-check): Capture stdout and stderr, and
return stdout if the child exited successfully, or include stderr in
the error.
* tests/openpgp/version.scm: Demonstrate this by checking the stdout.
Signed-off-by: Justus Winter <justus@g10code.com>
This fixes problems with long socket names, e.g. when doing distcheck.
* tests/gpgscm/tests.scm (path-join): New function.
(with-temporary-working-directory): Honor 'TMP'.
(make-temporary-file): Likewise.
* tests/migrations/Makefile.am (TMP): Default to '/tmp'.
(TESTS_ENVIRONMENT): Set 'TMP'.
* tests/openpgp/Makefile.am (TMP): Default to '/tmp'.
(TESTS_ENVIRONMENT): Set 'TMP'.
Signed-off-by: Justus Winter <justus@g10code.com>
* tests/gpgscm/lib.scm (echo): Move...
* tests/gpgscm/tests.scm (echo): ... here.
(info, error, skip): And use echo here.
(file-exists?): New function.
(tr:spawn): Check that source exists and if the sink has been created.
(tr:call-with-content): Hand in optional arguments.
Signed-off-by: Justus Winter <justus@g10code.com>
