* kbx/keybox-defs.h (KEYBOX_LL_OPEN_READ)
(KEYBOX_LL_OPEN_UPDATE, KEYBOX_LL_OPEN_CREATE): New.
* kbx/keybox-init.c (_keybox_ll_open): New. Replace all keybox use of
es_fopen by this function.
(_keybox_ll_close): New. Replace all keybox use of es_fclose by this
function.
--
Note that this has not been done for the utilities and the backend-kbx
of keyboxd.
* kbx/keybox-search.c (open_file): Use sysopen and sequential.
* kbx/keybox-update.c (create_tmp_file): Ditto.
(blob_filecopy): Ditto.
(keybox_set_flags): Ditto.
(keybox_delete): Ditto.
(keybox_compress): Ditto.
--
Under Windows "sysopen" requests that direct API calls (CreateFile et
al.) are used instead of the libc wrappers. This may or may not
improve the performance.
Using "sequential" is a hint to Windows to assume that a file is in
general access in a sequential manner. This will have an affect only
with a future libgpg-error.
* kbx/keybox-search.c (blob_cmp_fpr_part): Don't change FPROFF, since
it's caller which tweaks the offset.
(has_short_kid, has_long_kid): Examine the key flags to determine if
fingerprint 32 or 20.
--
GnuPG-bug-id: 5888
Signed-off-by: NIIBE Yutaka <gniibe@fsij.org>
--
We need to use es_fopen on Windows to cope with non-ascii file names.
This is quite a large but fortunately straightforward change. At a
very few places we keep using stdio (for example due to the use of
popen).
GnuPG-bug-id: 5098
Signed-off-by: Werner Koch <wk@gnupg.org>
No functional changes, just fixing minor spelling issues.
---
Most of these were identified from the command line by running:
codespell \
--ignore-words-list fpr,stati,keyserver,keyservers,asign,cas,iff,ifset \
--skip '*.po,ChangeLog*,help.*.txt,*.jpg,*.eps,*.pdf,*.png,*.gpg,*.asc' \
doc g13 g10 kbx agent artwork scd tests tools am common dirmngr sm \
NEWS README README.maint TODO
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
* common/util.h (UBID_LEN): New. Use it at all places.
* kbx/keybox-blob.c (create_blob_finish): Do not write the UBID item.
* kbx/keybox-dump.c (print_ubib): Remove.
(_keybox_dump_blob): Do not print the now removed ubid flag.
* kbx/keybox-search-desc.h (struct keydb_search_desc): Use constants
for the size of the ubid and grip.
* kbx/keybox-search.c (blob_cmp_ubid): New.
(has_ubid): Make it a simple wrapper around blob_cmp_ubid.
(keybox_get_data): Add arg 'r_ubid'.
* kbx/frontend.h (enum kbxd_store_modes): New.
* kbx/kbxserver.c (cmd_store): Add new option --insert.
* kbx/backend-cache.c (be_cache_initialize): New.
(be_cache_add_resource): Call it here.
* kbx/backend-kbx.c (be_kbx_seek): Remove args 'fpr' and 'fprlen'.
(be_kbx_search): Get the UBID from keybox_get_data.
* kbx/backend-support.c (be_fingerprint_from_blob): Replace by ...
(be_ubid_from_blob): new. Change all callers.
* kbx/frontend.c (kbxd_add_resource): Temporary disable the cache but
use the new cache init function.
(kbxd_store): Replace arg 'only_update' by 'mode'. Seek using the
ubid. Take care of the mode.
--
It turned out that using the hash of the entire blob was not helpful.
Thus we redefine the Unique-Blob-ID (UBID) as the primary fingerprint
of the blob. In case this is a v5 OpenPGP key a left truncated
version of the SHA-256 hash is used; in all other cases the full SHA-1
hash. Using a SHA-256 hash does not make sense because v4 keys are
and will for some time be the majority of keys and thus padding them
with zeroes won't make any difference. Even if fingerprint collisions
can eventually be created we will assume that the keys are bogus and
that it does not make sense to store its twin also in our key storage.
We can also easily extend the update code to detect a collision and
reject the update.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-blob.c (create_blob_header): New blob flag UBIB.
(create_blob_finish): Write the UBIB.
* kbx/keybox-dump.c (print_ubib): New.
(_keybox_dump_blob): Print UBIB flag.
* kbx/keybox-search.c (has_ubid): Compare the stored UBIB if
available.
--
This make scanning the keybox for a given UBIB much faster once it has
been stored.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/backend.h (enum database_types): Add DB_TYPE_CACHE.
(struct db_request_part_s): Add seqno fields.
(struct db_request_s): Add infos for the cache backend.
* kbx/backend-support.c (struct backend_handle_s): Add 'backend_id'.
(strdbtype): Support DB_TYPE_CACHE.
(be_generic_release_backend): Ditto.
(be_find_request_part): New.
(be_return_pubkey): New arg UBID and chnage status name.
* kbx/backend-cache.c: New.
* kbx/backend-kbx.c (be_kbx_init_request_part): New.
(be_kbx_search): Factor some code out to a support function.
(be_kbx_seek): New.
* kbx/frontend.c (kbxd_add_resource): Support DB_TYPE_CACHE.
(kbxd_search): Support the NEXR operation with the cache.
* kbx/keybox-search-desc.h (KEYDB_SEARCH_MODE_UBID): New.
(struct keydb_search_desc): Add field u.ubid.
* kbx/keybox-search.c (has_ubid): New.
(keybox_search): Support the UBID search.
--
This adds a caching backend to the keyboxd. This tries to accommodate
for duplicate use of fingerprints and thus be correct in case a
fingerprint is used in several keys. It also turned out that we need
to have a unique identifier (UBID) to identify a keyblock or X.509
certificate. In particular with an OpenPGP keyblob we can't easily
use the primary fingerprint as an identifier because that fingerprint
may also be used as subkey in another key. Thus using a hash of the
entire keyblock is a better identifier to be used to address a
keyblock for restarting a search or for identifying the keyblock to be
updated. Note that this new UBID is not a permanent identifier
because it changes with all keyblock update; it should be viewed as a
handle to the keyblock or X509 cert.
* kbx/keybox-search.c (keybox_search): We need to seek to the last
position in all cases not just when doing a NEXT.
--
This is because search from the beginning needs a keybox_search_reset.
We can only make an exception for KEYDB_SEARCH_MODE_FIRST..
Fixes-commit: 49b236af0ecbb6df67513feb4b63851f2e159ea2
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-init.c (keybox_lock) [W32]: Use _keybox_close_file
instead of fclose so that a close is done if the file is opened by
another handle.
* kbx/keybox-search.c (keybox_search): Remember the last offset and
use that in NEXT search mode if we had to re-open the file.
--
GnuPG-bug-id: 4505
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search-desc.h (KEYDB_SEARCH_MODE_FPR16)
(KEYDB_SEARCH_MODE_FPR20, KEYDB_SEARCH_MODE_FPR32): Remove. Switch
all users to KEYDB_SEARCH_MODE_FPR along with the fprlen value.
--
These search modes were added over time and there has until recently
be no incentive to remove the cruft. With the change for v5 keys I
finally went over all places and allowed the generic fingerprint mode
along with a given length of the fingerprint at all places.
Consequently the other modes can now be removed.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/userids.c (classify_user_id): Support 32 byte fingerprints.
* kbx/keybox-search-desc.h (KEYDB_SEARCH_MODE_FPR32): New.
(struct keydb_search_desc): Add field fprlen.
* kbx/keybox-defs.h (struct _keybox_openpgp_key_info): Add field
version and increase size of fpr to 32.
* kbx/keybox-blob.c: Define new version 2 for PGP and X509 blobs.
(struct keyboxblob_key): Add field fprlen and increase size of fpr.
(pgp_create_key_part_single): Allow larger fingerprints.
(create_blob_header): Implement blob version 2 and add arg want_fpr32.
(_keybox_create_openpgp_blob): Detect the need for blob version 2.
* kbx/keybox-search.c (blob_get_first_keyid): Support 32 byte
fingerprints.
(blob_cmp_fpr): Ditto.
(blob_cmp_fpr_part): Ditto.
(has_fingerprint): Add arg fprlen and pass on.
(keybox_search): Support KEYDB_SEARCH_MODE_FPR32 and adjust for
changed has_fingerprint.
* kbx/keybox-openpgp.c (parse_key): Support version 5 keys.
* kbx/keybox-dump.c (_keybox_dump_blob): Support blob version 2.
* g10/delkey.c (do_delete_key): Support KEYDB_SEARCH_MODE_FPR32.
* g10/export.c (exact_subkey_match_p): Ditto.
* g10/gpg.c (main): Ditto.
* g10/getkey.c (get_pubkey_byfprint): Adjust for changed
KEYDB_SEARCH_MODE_FPR.
* g10/keydb.c (keydb_search_desc_dump): Support
KEYDB_SEARCH_MODE_FPR32 and adjust for changed KEYDB_SEARCH_MODE_FPR.
(keydb_search): Add new arg fprlen and change all callers.
* g10/keyedit.c (find_by_primary_fpr): Ditto.
* g10/keyid.c (keystr_from_desc): Ditto.
* g10/keyring.c (keyring_search): Ditto.
* g10/keyserver.c (print_keyrec): Ditto.
(parse_keyrec): Ditto.
(keyserver_export): Ditto.
(keyserver_retrieval_screener): Ditto.
(keyserver_import): Ditto.
(keyserver_import_fprint): Ditto.
(keyidlist): Ditto.
(keyserver_get_chunk): Ditto.
* g10/keydb.c (keydb_search): Add new arg fprlen and change all
callers.
* sm/keydb.c (keydb_search_fpr): Adjust for changed
KEYDB_SEARCH_MODE_FPR.
--
This prepares the support for OpenPGP v5 keys. The new version 2 blob
format is needed for the longer fingerprints and we also use this
opportunity to prepare for storing the keygrip in the blob for faster
lookup by keygrip. Right now this is not yet functional.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-defs.h (struct _keybox_openpgp_key_info): Add field grip.
* kbx/keybox-openpgp.c (struct keyparm_s): New.
(keygrip_from_keyparm): New.
(parse_key): Compute keygrip.
* kbx/keybox-search.c (blob_openpgp_has_grip): New.
(has_keygrip): Call it.
--
This has been marked for too long as not yet working. However, it is
a pretty useful feature and will come pretty handy when looking for
all keys matching one keygrip.
Can be optimized a lot by storing the keygrip in the meta data. This
will be done along with the upgrade of KBX for v5 fingerprints.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (blob_cmp_fpr): Avoid overflow in OFF+LEN
checking.
(blob_cmp_fpr_part): Ditto.
(blob_cmp_name): Ditto.
(blob_cmp_mail): Ditto.
(blob_x509_has_grip): Ditto.
(keybox_get_keyblock): Check OFF and LEN using a 64 bit var.
(keybox_get_cert): Ditto.
--
On most 32 bit systems size_t is 32 bit and thus the check
size_t cert_off = get32 (buffer+8);
size_t cert_len = get32 (buffer+12);
if (cert_off+cert_len > length)
return gpg_error (GPG_ERR_TOO_SHORT);
does not work as intended for all supplied values. The simplest
solution here is to cast them to 64 bit.
In general it will be better to avoid size_t at all and work with
uint64_t. We did not do this in the past because uint64_t was not
universally available.
GnuPG-bug-id: 3770
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-file.c (_keybox_read_blob): Remove.
(_keybox_read_blob2): Rename to ....
(_keybox_read_blob): this. Make arg options. Change all callers.
* kbx/keybox-search.c (keybox_search): Factor fopen call out to ...
(open_file): new.
(keybox_seek): Als use open_file.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.h (KEYDB_HANDLE): Move typedef to ...
* g10/gpg.h: here.
(struct server_control_s): Add field 'cached_getkey_kdb'.
* g10/gpg.c (gpg_deinit_default_ctrl): Release that keydb handle.
* g10/getkey.c (getkey_end): Cache keydb handle.
(get_pubkey): Use cached keydb handle.
* kbx/keybox-search.c (keybox_search_reset): Use lseek instead of
closing the file.
--
Before this patch a "gpg --check-sigs" opened and closed the keybox
file for almost every signature check. By caching the keydb handle
and using lseek(2) this can be limited to just 2 times. This might
speed up things on Windows.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keyblock_cache): Remove field SIGSTATUS.
(keyblock_cache_clear): Adjust for that removal.
(parse_keyblock_image): Remove arg SIGSTATUS. Remove the signature
cache setting; this is now done in the parser.
(keydb_get_keyblock): Do not set SIGSTATUS.
(build_keyblock_image): Remove arg SIGSTATUS and simplify. Change
caller.
* kbx/keybox-blob.c: Explain that the signature information is not
anymore used.
(_keybox_create_openpgp_blob): Remove arg SIGSTATUS and change
callers.
* kbx/keybox-search.c (keybox_get_keyblock): Remove arg R_SIGSTATUS
and change callers.
* kbx/keybox-update.c (keybox_insert_keyblock): Likewise.
--
This thing was too complicated and has been replaced by the new ring
trust packet code.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_seek): New function.
* g10/keydb.c (keydb_search): When reading from the cache, seek to
just after the cached record.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* kbx/keybox-search.c (keybox_offset): New function.
* g10/keydb.c (struct keyblock_cache): Add fields resource and offset.
(keyblock_cache_clear): Reset HD->KEYBLOCK_CACHE.RESOURCE and
HD->KEYBLOCK_CACHE.OFFSET.
(keydb_search): Don't use the cached result if it comes before the
current file position. When caching an entry, also record the
position at which it was found.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
GnuPG-bug-id: 2187
* common/util.h: Remove replacement macros for libgpg-error<1.21.
* common/types.h: Ditto.
* common/mischelp.h: Ditto.
* common/t-mapstrings.c: Include t-support.h before stringhelp.h
* common/t-stringhelp.c: Ditto.
* common/t-support.h: Always include gpg-error.h.
* kbx/keybox-search.c: Do not include stringhelp.h so that keybox-defs
comes first.
--
This patch enhances commit d6e0149 from Dec 10.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search-desc.h (struct keydb_search_desc.skipfnc): Change
third parameter to be the index of the user id packet in the keyblock
rather than the packet itself. Update users.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
The keybox code doesn't work directly with keyblocks. As such, the
matched user packet is not readily available to pass to
DESC[n].SKIPFNC. But, we do know the index of the user id packet that
matched. Thus, pass that instead. If the skip function needs the
user id packet, it can use the key id to look up the key block and
find the appropriate packet.
* kbx/keybox.h (KEYBOX_WITH_X509): Do not define.
* sm/Makefile.am (AM_CPPFLAGS): Define it here.
(common_libs): Change to libkeybox509.a
* g10/Makefile.am (AM_CFLAGS): remove KSBA_CFLAGS.
(gpg2_LDADD, gpgv2_LDADD): Remove KSBA_LIBS
* kbx/Makefile.am (noinst_LIBRARIES): Add libkeybox509.a.
(libkeybox509_a_SOURCES): New.
(libkeybox_a_CFLAGS): New.
(libkeybox509_a_CFLAGS): New.
(kbxutil_CFLAGS): New.
* kbx/keybox-search.c (has_keygrip) [!KEYBOX_WITH_X509]: Declare args
as unused.
--
There is no real need to link to Libksba in gpg.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (blob_cmp_mail): Stop comparing at the '>'.
--
This change allows to find mail addresses like
Joe Doe <joe@example.org> bar
Joe Doe <joe@example.org> (comment)
using the command
gpg -k '<joe@example.org'
or (with syntactic sugar)
gpg -k '<joe@example.org>'
These UIDs are ill-formed according to gpg checks but nevertheless are
seen in the wild.
Note, that it does only work with the new keybox format.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/host2net.h (buf16_to_ulong, buf16_to_uint): New.
(buf16_to_ushort, buf16_to_u16): New.
(buf32_to_size_t, buf32_to_ulong, buf32_to_uint, buf32_to_u32): New.
--
Commit 91b826a38880fd8a989318585eb502582636ddd8 was not enough to
avoid all sign extension on shift problems. Hanno Böck found a case
with an invalid read due to this problem. To fix that once and for
all almost all uses of "<< 24" and "<< 8" are changed by this patch to
use an inline function from host2net.h.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (blob_get_keyid): Rename to
blob_get_first_keyid. Check number of keys and remove blob type check.
--
There is no need to check the blob type. We already know that it is a
key blob type and keyids are used for X.509 and OpenPGP. Also added
check for number of keys because the other parser functions do it as
well.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (blob_get_keyid): New.
(keybox-search): Call skipfnc callback function.
--
This patch (tentatively) fixes
GnuPG-bug-id: 1794
The keybox_search function in kbx/keybox-search.c currently ignores
the skipfnc callback, but the validate_key_list function in
g10/trustdb.c uses such a callback to exclude ultimately trusted keys.
* kbx/keybox.h (keybox_blobtype_t): New.
* kbx/keybox-defs.h (BLOBTYPE_*): Replace by KEYBOX_BLOBTYPE_*.
* kbx/keybox-search.c (keybox_search): Add arg want_blobtype and skip
non-matching blobs.
* sm/keydb.c (keydb_search): Pass KEYBOX_BLOBTYPE_X509 to keybox_search.
* g10/keydb.c (keydb_search): Pass KEYBOX_BLOBTYPE_PGP to keybox_search.
--
Without this fix a listing of all keys would fail because the wrong
blob type would be returned for the gpg or gpgsm.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_search): Add arg r_skipped and skip too
long blobs.
* sm/keydb.c (keydb_search): Call keybox_search with a dummy param.
* g10/keydb.c (struct keydb_handle): Add field skipped_long_blobs.
(keydb_search_reset): Reset that field.
(keydb_search): Update that field.
(keydb_get_skipped_counter): New.
* g10/keylist.c (list_all): Print count of skipped keys.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_search): Add arg R_DESCINDEX. Chnage
both callers.
* g10/keydb.c (keydb_search): Always set DESCINDEX.
--
This only affects the new keybox for OpenPGP keys in 2.1. The bug
exhibited itself by running GPA's backup command on Windows.
* kbx/keybox-update.c (keybox_update_keyblock): Implement.
* kbx/keybox-search.c (get_blob_flags): Move to ...
* kbx/keybox-defs.h (blob_get_type): here.
* kbx/keybox-file.c (_keybox_read_blob2): Fix calling without R_BLOB.
* g10/keydb.c (build_keyblock_image): Allow calling without
R_SIGSTATUS.
(keydb_update_keyblock): Implement for keybox.
* kbx/keybox-dump.c (_keybox_dump_blob): Fix printing of the unhashed
size. Print "does not expire" also on 64 bit platforms.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (parse_keyblock_image): Add args PK_NO and UID_NO and
set the note flags accordingly.
(keydb_get_keyblock): Transfer PK_NO and UID_NO to parse_keyblock_image.
* kbx/keybox-search.c (blob_cmp_fpr, blob_cmp_fpr_part)
(blob_cmp_name, blob_cmp_mail): Return the key/user number.
(keybox_search): Set the key and user number into the found struct.
(keybox_get_keyblock): Add args R_PK_NO and R_UID_NO and set them from
the found struct.
--
getkey.c needs to know whether the correct subkey was found. Thus we
need to set the node flags the same way we did it with the keyring
storage.
* g10/keydb.c (parse_keyblock_image): Add arg SIGSTATUS.
(keydb_get_keyblock): Handle it.
(build_keyblock_image): Add arg SIGSTATUS.
(keydb_insert_keyblock): Handle it.
* kbx/keybox-blob.c (pgp_create_sig_part): Add arg SIGSTATUS.
(_keybox_create_openpgp_blob): Ditto.
* kbx/kbxutil.c (import_openpgp): Adjust for above change.
* kbx/keybox.h (KEYBOX_FLAG_SIG_INFO): New.
* kbx/keybox-search.c (_keybox_get_flag_location): Handle new flag.
(keybox_get_keyblock): Add arg R_SIGSTATUS.
* kbx/keybox-update.c (keybox_insert_keyblock): Add arg SIGSTATUS.
--
With this change a key listing using the keybox format is now double
as fast as using a keyring. The memory use dropped as well. Measured
with about 1500 keys.
* g10/getkey.c (get_pubkey_fast): Improve the assertion.
* kbx/keybox.h: Include iobuf.h.
* kbx/keybox-blob.c (keyboxblob_uid): Add field OFF.
(KEYBOX_WITH_OPENPGP): Remove use of this macro.
(pgp_create_key_part_single): New.
(pgp_temp_store_kid): Change to use the keybox-openpgp parser.
(pgp_create_key_part): Ditto.
(pgp_create_uid_part): Ditto.
(pgp_create_sig_part): Ditto.
(pgp_create_blob_keyblock): Ditto.
(_keybox_create_openpgp_blob): Ditto.
* kbx/keybox-search.c (keybox_get_keyblock): New.
* kbx/keybox-update.c (keybox_insert_keyblock): New.
* g10/keydb.c (parse_keyblock_image):
(keydb_get_keyblock): Support keybox.
(build_keyblock_image): New.
(keydb_insert_keyblock): Support keybox.
* kbx/kbxutil.c (import_openpgp, main): Add option --dry-run and print
a kbx file to stdout.
* kbx/keybox-file.c (_keybox_read_blob2): Allow keyblocks up to 10^6
bytes.
--
Import and key listing does now work with the keybox format. It is
still quite slow and signature caching is completely missing.
Increasing the maximum allowed length for a keyblock was required due
to a 700k keyblock which inhibited kbxutil to list the file.
kbxutil's option name --import-openpgp is not quite appropriate
because it only creates KBX blobs from OpenPGP data.
We better do this once and for all instead of cluttering all future
commits with diffs of trailing white spaces. In the majority of cases
blank or single lines are affected and thus this change won't disturb
a git blame too much. For future commits the pre-commit scripts
checks that this won't happen again.
