* kbx/keybox-init.c (ll_buffer_size): New var intialized to 128k
(stream_buffers): New var.
(keybox_set_buffersize): New.
(_keybox_ll_open, _keybox_ll_close): Implement buffering.
* sm/gpgsm.c (oKbxBufferSize): New.
(opts): Add option --kbx-buffer-size.
(main): Call keybox_set_buffersize.
* g10/gpg.c: Include keybox.h.
* (oKbxBufferSize): New.
(opts): Add option --kbx-buffer-size.
(main): Call keybox_set_buffersize.
--
Running a test on Windows using a pubring.kbx with
Total number of blobs: 2098
openpgp: 1294
x509: 803
and a size of 42MiB with
gpgsm -k --with-validation --disable-dirmngr --kbx-buffer-size N >nul
gives these performance figures using procmon
| N(k) | file events | time(s) |
|------+-------------+---------|
| 0 | 4900000 | 86 |
| 16 | 2456000 | 58 |
| 32 | 1233000 | 43 |
| 64 | 622000 | 37 |
| 128 | 317000 | 32 |
| 256 | 164000 | 31 |
| 512 | 88000 | 30 |
Using _open instead of CreateFile give the same number of file events
but increased the time slight by one second for the measured buffer
size of 64k and 128k. Benchmarks for gpg have not been conducted.
--
We need to use es_fopen on Windows to cope with non-ascii file names.
This is quite a large but fortunately straightforward change. At a
very few places we keep using stdio (for example due to the use of
popen).
GnuPG-bug-id: 5098
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/util.h (UBID_LEN): New. Use it at all places.
* kbx/keybox-blob.c (create_blob_finish): Do not write the UBID item.
* kbx/keybox-dump.c (print_ubib): Remove.
(_keybox_dump_blob): Do not print the now removed ubid flag.
* kbx/keybox-search-desc.h (struct keydb_search_desc): Use constants
for the size of the ubid and grip.
* kbx/keybox-search.c (blob_cmp_ubid): New.
(has_ubid): Make it a simple wrapper around blob_cmp_ubid.
(keybox_get_data): Add arg 'r_ubid'.
* kbx/frontend.h (enum kbxd_store_modes): New.
* kbx/kbxserver.c (cmd_store): Add new option --insert.
* kbx/backend-cache.c (be_cache_initialize): New.
(be_cache_add_resource): Call it here.
* kbx/backend-kbx.c (be_kbx_seek): Remove args 'fpr' and 'fprlen'.
(be_kbx_search): Get the UBID from keybox_get_data.
* kbx/backend-support.c (be_fingerprint_from_blob): Replace by ...
(be_ubid_from_blob): new. Change all callers.
* kbx/frontend.c (kbxd_add_resource): Temporary disable the cache but
use the new cache init function.
(kbxd_store): Replace arg 'only_update' by 'mode'. Seek using the
ubid. Take care of the mode.
--
It turned out that using the hash of the entire blob was not helpful.
Thus we redefine the Unique-Blob-ID (UBID) as the primary fingerprint
of the blob. In case this is a v5 OpenPGP key a left truncated
version of the SHA-256 hash is used; in all other cases the full SHA-1
hash. Using a SHA-256 hash does not make sense because v4 keys are
and will for some time be the majority of keys and thus padding them
with zeroes won't make any difference. Even if fingerprint collisions
can eventually be created we will assume that the keys are bogus and
that it does not make sense to store its twin also in our key storage.
We can also easily extend the update code to detect a collision and
reject the update.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox.h: Fix naming of arguments.
* scd/ccid-driver.c (print_error) [TEST]: Add missing break. Note
that this is anyway an impossible case.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-init.c (keybox_lock): New arg TIMEOUT. Change all
callers to pass -1 when locking.
* sm/keydb.c (struct resource_item): Remove LOCKANDLE.
(struct keydb_handle): Add KEEP_LOCK.
(keydb_add_resource): Use keybox locking instead of a separate dotlock
for testing whether we can run a compress.
(keydb_release): Reset KEEP_LOCK.
(keydb_lock): Set KEEP_LOCK.
(unlock_all): Take care of KEEP_LOCK.
(lock_all): Use keybox_lock instead of dotlock fucntions.
(keydb_delete): Remove arg UNLOCK.
* sm/delete.c (delete_one): Adjust keydb_delete. Due to the KEEP_LOCK
the keydb_release takes care of unlocking.
--
This aligns the code more with g10/keydb.c and avoids the separate
calls to dotlock_take.
GnuPG-bug-id: 4505
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-util.c (keybox_set_malloc_hooks): Remove.
(_keybox_malloc, _keybox_calloc, keybox_realloc)
(_keybox_free): Remove.
(keybox_file_rename): Remove. Was not used.
* sm/gpgsm.c (main): Remove call to keybox_set_malloc_hooks.
* kbx/kbxutil.c (main): Ditto.
* kbx/keybox-defs.h: Remove all separate includes. Include util.h.
remove convenience macros.
* common/logging.h (return_if_fail): New. Originally from
keybox-defs.h but now using log_debug.
(return_null_if_fail): Ditto.
(return_val_if_fail): Ditto.
(never_reached): Ditto.
--
Originally the KBX code was written to allow standalone use. However
this required lot of ugliness like separate memory allocators and
such. It also precludes the use of some standard functions from
common due to their use of the common gnupg malloc functions.
Dropping all that makes things easier. Minor disadvantages: the kbx
call done for gpg will now use gcry malloc fucntions and not the
standard malloc functions. This might be a bit slower but removing
them even fixes a possible bug in keybox_tmp_names which is used in
gpg and uses gpg's xfree which is actually gcry_free.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keyblock_cache): Remove field SIGSTATUS.
(keyblock_cache_clear): Adjust for that removal.
(parse_keyblock_image): Remove arg SIGSTATUS. Remove the signature
cache setting; this is now done in the parser.
(keydb_get_keyblock): Do not set SIGSTATUS.
(build_keyblock_image): Remove arg SIGSTATUS and simplify. Change
caller.
* kbx/keybox-blob.c: Explain that the signature information is not
anymore used.
(_keybox_create_openpgp_blob): Remove arg SIGSTATUS and change
callers.
* kbx/keybox-search.c (keybox_get_keyblock): Remove arg R_SIGSTATUS
and change callers.
* kbx/keybox-update.c (keybox_insert_keyblock): Likewise.
--
This thing was too complicated and has been replaced by the new ring
trust packet code.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-util.c (keybox_file_rename): Add arg BLOCK_SIGNALS.
* kbx/keybox-update.c (rename_tmp_file): Block all signals when doing
a double rename.
* g10/keyring.c (rename_tmp_file): Block all signals during the double
rename.
--
This might fix
Debian-bug-id: 831510
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-update.c (create_tmp_file): Move some code to...
* kbx/keybox-util.c (keybox_tmp_names): new.
* g10/keyring.c: Include keybox.h.
(create_tmp_file): Replace parts by keybox_tmp_names.
--
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-defs.h: Include dotlock.h and logging.h.
(CONST_KB_NAME): Remove. Replace usage by KB_NAME.
(struct keybox_name): Add field "lockhd".
* kbx/keybox-init.c (keybox_register_file): Init LOCKHD.
(keybox_lock): Chnage to return gpg_error_t. Implement locking.
--
The keybox locking for gpg was not implemented - This needs to be
fixed of course.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-init.c (keybox_register_file): Change interface to return
the token even if the file has already been registered.
* g10/keydb.c (primary_keyring): Rename to primary_keydb.
(maybe_create_keyring_or_box): Change return type to gpg_error_t.
(keydb_add_resource): Ditto. s/rc/err/.
(keydb_add_resource): Mark an already registered as primary.
* sm/keydb.c (maybe_create_keybox): Change return type to gpg_error_t.
(keydb_add_resource): Ditto. s/rc/err/.
(keydb_add_resource): Adjust for changed keybox_register_file.
--
This change aligns the registering of keyboxes with those of
keyrings. This fixes a potential bug:
gpg --keyring foo.kbx --keyring bar.gpg --keyring foo.kbx
would have marked bar.gpg as primary resource and thus inserting new
keys there. The correct and now fixed behavior is to insert to
foo.kbx.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_seek): New function.
* g10/keydb.c (keydb_search): When reading from the cache, seek to
just after the cached record.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* kbx/keybox-search.c (keybox_offset): New function.
* g10/keydb.c (struct keyblock_cache): Add fields resource and offset.
(keyblock_cache_clear): Reset HD->KEYBLOCK_CACHE.RESOURCE and
HD->KEYBLOCK_CACHE.OFFSET.
(keydb_search): Don't use the cached result if it comes before the
current file position. When caching an entry, also record the
position at which it was found.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
GnuPG-bug-id: 2187
* kbx/keybox.h (KEYBOX_WITH_X509): Do not define.
* sm/Makefile.am (AM_CPPFLAGS): Define it here.
(common_libs): Change to libkeybox509.a
* g10/Makefile.am (AM_CFLAGS): remove KSBA_CFLAGS.
(gpg2_LDADD, gpgv2_LDADD): Remove KSBA_LIBS
* kbx/Makefile.am (noinst_LIBRARIES): Add libkeybox509.a.
(libkeybox509_a_SOURCES): New.
(libkeybox_a_CFLAGS): New.
(libkeybox509_a_CFLAGS): New.
(kbxutil_CFLAGS): New.
* kbx/keybox-search.c (has_keygrip) [!KEYBOX_WITH_X509]: Declare args
as unused.
--
There is no real need to link to Libksba in gpg.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox.h (keybox_blobtype_t): New.
* kbx/keybox-defs.h (BLOBTYPE_*): Replace by KEYBOX_BLOBTYPE_*.
* kbx/keybox-search.c (keybox_search): Add arg want_blobtype and skip
non-matching blobs.
* sm/keydb.c (keydb_search): Pass KEYBOX_BLOBTYPE_X509 to keybox_search.
* g10/keydb.c (keydb_search): Pass KEYBOX_BLOBTYPE_PGP to keybox_search.
--
Without this fix a listing of all keys would fail because the wrong
blob type would be returned for the gpg or gpgsm.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_search): Add arg r_skipped and skip too
long blobs.
* sm/keydb.c (keydb_search): Call keybox_search with a dummy param.
* g10/keydb.c (struct keydb_handle): Add field skipped_long_blobs.
(keydb_search_reset): Reset that field.
(keydb_search): Update that field.
(keydb_get_skipped_counter): New.
* g10/keylist.c (list_all): Print count of skipped keys.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-defs.h (struct keybox_handle): Add field for_openpgp.
* kbx/keybox-file.c (_keybox_write_header_blob): Set openpgp header
flag.
* kbx/keybox-blob.c (_keybox_update_header_blob): Add arg for_openpgp
and set header flag.
* kbx/keybox-init.c (keybox_new): Rename to do_keybox_new, make static
and add arg for_openpgp.
(keybox_new_openpgp, keybox_new_x509): New. Use them instead of the
former keybox_new.
* kbx/keybox-update.c (blob_filecopy): Add arg for_openpgp and set the
openpgp header flags.
* g10/keydb.c (rt_from_file): New. Factored out and extended from
keydb_add_resource.
(keydb_add_resource): Switch to the kbx file if it has the openpgp
flag set.
* kbx/keybox-dump.c (dump_header_blob): Print header flags.
--
The problem was reported by dkg on gnupg-devel (2014-10-07):
I just discovered a new problem, though, which will affect people on
systems that have gpg and gpg2 coinstalled:
0) create a new keyring with gpg2, and use it exclusively with gpg2
for a while.
1) somehow (accidentally?) use gpg (1.4.x) again -- this creates
~/.gnupg/pubring.gpg
2) future runs of gpg2 now only look at pubring.gpg and ignore
pubring.kbx -- the keys you had accumulated in the keybox are no
longer listed in the output of gpg2 --list-keys
Note that gpgsm has always used pubring.kbx and thus this file might
already be there but without gpg ever inserted a key. The new flag in
the KBX header gives us an indication whether a KBX file has ever been
written by gpg >= 2.1. If that is the case we will use it instead of
the default pubring.gpg.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_search): Add arg R_DESCINDEX. Chnage
both callers.
* g10/keydb.c (keydb_search): Always set DESCINDEX.
--
This only affects the new keybox for OpenPGP keys in 2.1. The bug
exhibited itself by running GPA's backup command on Windows.
* g10/keydb.c (parse_keyblock_image): Add args PK_NO and UID_NO and
set the note flags accordingly.
(keydb_get_keyblock): Transfer PK_NO and UID_NO to parse_keyblock_image.
* kbx/keybox-search.c (blob_cmp_fpr, blob_cmp_fpr_part)
(blob_cmp_name, blob_cmp_mail): Return the key/user number.
(keybox_search): Set the key and user number into the found struct.
(keybox_get_keyblock): Add args R_PK_NO and R_UID_NO and set them from
the found struct.
--
getkey.c needs to know whether the correct subkey was found. Thus we
need to set the node flags the same way we did it with the keyring
storage.
* g10/keydb.c (parse_keyblock_image): Add arg SIGSTATUS.
(keydb_get_keyblock): Handle it.
(build_keyblock_image): Add arg SIGSTATUS.
(keydb_insert_keyblock): Handle it.
* kbx/keybox-blob.c (pgp_create_sig_part): Add arg SIGSTATUS.
(_keybox_create_openpgp_blob): Ditto.
* kbx/kbxutil.c (import_openpgp): Adjust for above change.
* kbx/keybox.h (KEYBOX_FLAG_SIG_INFO): New.
* kbx/keybox-search.c (_keybox_get_flag_location): Handle new flag.
(keybox_get_keyblock): Add arg R_SIGSTATUS.
* kbx/keybox-update.c (keybox_insert_keyblock): Add arg SIGSTATUS.
--
With this change a key listing using the keybox format is now double
as fast as using a keyring. The memory use dropped as well. Measured
with about 1500 keys.
* g10/getkey.c (get_pubkey_fast): Improve the assertion.
* kbx/keybox.h: Include iobuf.h.
* kbx/keybox-blob.c (keyboxblob_uid): Add field OFF.
(KEYBOX_WITH_OPENPGP): Remove use of this macro.
(pgp_create_key_part_single): New.
(pgp_temp_store_kid): Change to use the keybox-openpgp parser.
(pgp_create_key_part): Ditto.
(pgp_create_uid_part): Ditto.
(pgp_create_sig_part): Ditto.
(pgp_create_blob_keyblock): Ditto.
(_keybox_create_openpgp_blob): Ditto.
* kbx/keybox-search.c (keybox_get_keyblock): New.
* kbx/keybox-update.c (keybox_insert_keyblock): New.
* g10/keydb.c (parse_keyblock_image):
(keydb_get_keyblock): Support keybox.
(build_keyblock_image): New.
(keydb_insert_keyblock): Support keybox.
* kbx/kbxutil.c (import_openpgp, main): Add option --dry-run and print
a kbx file to stdout.
* kbx/keybox-file.c (_keybox_read_blob2): Allow keyblocks up to 10^6
bytes.
--
Import and key listing does now work with the keybox format. It is
still quite slow and signature caching is completely missing.
Increasing the maximum allowed length for a keyblock was required due
to a 700k keyblock which inhibited kbxutil to list the file.
kbxutil's option name --import-openpgp is not quite appropriate
because it only creates KBX blobs from OpenPGP data.
* kbx/keybox-defs.h (_keybox_write_header_blob): Move prototype to ..
* kbx/keybox.h: here.
* kbx/keybox-init.c (keybox_lock): Add dummy function
* g10/keydb.c: Include keybox.h.
(KeydbResourceType): Add KEYDB_RESOURCE_TYPE_KEYBOX.
(struct resource_item): Add field kb.
(maybe_create_keyring_or_box): Add error descriptions to diagnostics.
Add arg IS_BOX. Write a header for a new keybox file.
(keydb_add_resource): No more need for the force flag. Rename the
local variable "force" to "create". Add URL scheme "gnupg-kbx". Add
magic test to detect a keybox file. Add basic support for keybox.
(keydb_new, keydb_get_resource_name, keydb_delete_keyblock)
(keydb_locate_writable, keydb_search_reset, keydb_search2): Add
support for keybox.
(lock_all, unlock_all): Ditto.
* g10/Makefile.am (needed_libs): Add libkeybox.a.
(gpg2_LDADD, gpgv2_LDADD): Add KSBA_LIBS as a workaround.
* g10/keydb.h (KEYDB_RESOURCE_FLAG_PRIMARY)
KEYDB_RESOURCE_FLAG_DEFAULT, KEYDB_RESOURCE_FLAG_READONLY): New.
* g10/gpg.c, g10/gpgv.c (main): Use new constants.
--
I did most of these changes back in 2011 and only cleaned them up
now. More to follow soon.
We better do this once and for all instead of cluttering all future
commits with diffs of trailing white spaces. In the majority of cases
blank or single lines are affected and thus this change won't disturb
a git blame too much. For future commits the pre-commit scripts
checks that this won't happen again.
* keybox.h (KEYBOX_FLAG_BLOB_SECRET, KEYBOX_FLAG_BLOB_EPHEMERAL):
New.
* keybox-update.c (keybox_compress): Use it here instead of a
magic constant.
sm/
* fingerprint.c (gpgsm_get_fingerprint): Add caching.
(gpgsm_get_fingerprint_string): Use bin2hexcolon().
(gpgsm_get_fingerprint_hexstring): Use bin2hex and allocate only
as much memory as required.
(gpgsm_get_keygrip_hexstring): Use bin2hex.
* certchain.c (gpgsm_validate_chain): Keep track of the
certificate chain and reset the ephemeral flags.
* keydb.c (keydb_set_cert_flags): New args EPHEMERAL and MASK.
Changed caller to use a mask of ~0. Return a proper error code if
the certificate is not available.
* keybox-search.c (get_flag_from_image, keybox_get_flags): New.
(_keybox_get_flag_location): New.
* certchain.c (gpgsm_validate_chain): Mark revoked certs in the
keybox.
* keylist.c (list_cert_colon): New arg VALIDITY; use it to print a
revoked flag.
(list_internal_keys): Retrieve validity flag.
(list_external_cb): Pass 0 as validity flag.
* keydb.c (keydb_get_flags, keydb_set_flags): New.
(keydb_set_cert_flags): New.
(lock_all): Return a proper error code.
(keydb_lock): New.
(keydb_delete): Don't lock but check that it has been locked.
(keydb_update_keyblock): Ditto.
* delete.c (delete_one): Take a lock.
more than one name.
(has_subject_or_alt): New.
(blob_cmp_mail): New.
(has_mail): New.
(keybox_search): Implemented exact search and exact mail search.
* kbx/keybox-blob.c (_keybox_create_x509_blob): Insert alternate
names.
Werner Koch