* g10/keydb.c (keydb_rebuild_caches): Only mark the cached as prepared
if it is actually prepared, which it only is if the resource is a
keybox.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* g10/keydb.c (keyblock_cache): Don't declare this variable. Instead...
(struct keyblock_cache): ... turn its type into this first class
object...
(struct keydb_handle): ... and instantiate it once per database
handle. Update all users.
(keydb_rebuild_caches): Don't invalidate the keyblock cache.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* g10/keydb.c (keydb_get_keyblock): If the iobuf_seek fails when
reading from the cache, then simply clear the cache and try reading
from the database.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* g10/keydb.c (keydb_new): If we fail to open a keyring or keybox
correctly release all resources.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* g10/keydb.c: Improve code comments and documentation of internal
interfaces. Improve documentation of public APIs and move that to...
* g10/keydb.h: ... this file.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* g10/keydb.c (struct keydb_handle): Add new field is_reset.
(keydb_new): Initialize hd->is_reset to 1.
(keydb_locate_writable): Set hd->is_reset to 1.
(keydb_search): Set hd->is_reset to 0. Don't cache a key not found if
the search started from the beginning of the database.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* g10/keydb.c (keydb_search_first): Reset the handle before starting
the search.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
This bug hasn't shown up yet in practice, because keydb_search_first
is always called immediately after a keydb_new. This changes cleans
up the semantics and will hopefully prevent future bugs.
* g10/keydb.c (struct kid_list_s): Rename from this...
(struct kid_not_found_cache_bucket): ... to this. Update users.
Remove field state.
(kid_list_t): Remove type.
(KID_NOT_FOUND_CACHE_BUCKETS): Define. Use this instead of a literal.
(kid_found_table): Rename from this...
(kid_not_found_cache_bucket): ... to this. Update users.
(kid_found_table_count): Rename from this...
(kid_not_found_cache_count): ... to this. Update users.
(kid_not_found_p): Only return whether a key with the specified key id
is definitely not in the database.
(kid_not_found_insert): Remove parameter found. Update callers.
(keydb_search): Only insert a key id in the not found cache if it is
not found. Rename local variable once_found to already_in_cache.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
Commit e0873a33 started tracking whether key ids where definitely in
the database. This information is, however, never used and thus just
unnecessarily inflates the cache. This patch effectively reverts that
change (however, e0873a33 contains two separate changes and this only
reverts that change).
* g10/keydb.h (KEYDB_RESOURCE_FLAG_GPGVDEF): New.
* g10/keydb.c (keydb_add_resource): Take care of new flag.
* g10/gpgv.c (main): Use new flag.
--
GnuPG-bug-id: 2025
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keydb_search_fpr): Skip legacy keys.
--
A test case for this problem can be found at
GnuPG-bug-id: 2031
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (kid_list_s): Keep a state in the table.
(kid_not_found_table): Rename to kid_found_table.
(n_kid_not_found_table): Rename to kid_found_table_count.
(kid_not_found_p): Return found state.
(kid_not_found_insert): Add arg found.
(keydb_search): Store found state in the table.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (kid_list_t): New.
(kid_not_found_table, n_kid_not_found_table): New.
(kid_not_found_p, kid_not_found_insert, kid_not_found_flush): New.
(keydb_insert_keyblock): Flush the new cache.
(keydb_delete_keyblock): Ditto.
(keydb_update_keyblock): Ditto.
(keydb_search): Use the new cache.
(keydb_dump_stats): New.
* g10/gpg.c (g10_exit): Dump keydb stats.
--
What we do here is to keep track of key searches by long keyids (as
stored in all signatures) so that we do not need to scan the keybox
again after we already found that this keyid will result in
not-found. As soon as we change gpg to run as a co-process we should
store this table per session because other instances of gpg may have
updated the keybox without us knowing.
On a test ring with
gpg: 94721 good signatures
gpg: 6831 bad signatures
gpg: 150703 signatures not checked due to missing keys
gpg: 5 signatures not checked due to errors
gpg: keydb: kid_not_found_table: total: 14132
this new cache speeds a --check-sigs listing up from 28 minutes to
less than 3 minutes.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keydb_handle): Add field saved_found.
(keydb_new): Init new field.
(keydb_push_found_state, keydb_pop_found_state): New.
* g10/keyring.c (kyring_handle): Add field saved_found.
(keyring_push_found_state, keyring_pop_found_state): New.
--
We have the same feature in gpgsm. It is very useful to check for an
unambiguous user id with a follow up update of the keyblock.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keyring.c (keyring_get_keyblock): Whitelist allowed packet
types.
* g10/keydb.c (parse_keyblock_image): Ditto.
--
The keyring DB code did not reject packets which don't belong into a
keyring. If for example the keyblock contains a literal data packet
it is expected that the processing code stops at the data packet and
reads from the input stream which is referenced from the data packets.
Obviously the keyring processing code does not and cannot do that.
However, when exporting this messes up the IOBUF and leads to an
invalid read of sizeof (int).
We now skip all packets which are not allowed in a keyring.
Reported-by: Hanno Böck <hanno@hboeck.de>
Test data:
gpg2 --no-default-keyring --keyring FILE --export >/dev/null
With this unpacked data for FILE:
-----BEGIN PGP ARMORED FILE-----
mI0EVNP2zQEEALvETPVDCJDBXkegF4esiV1fqlne40yJnCmJeDEJYocwFPXfFA86
sSGjInzgDbpbC9gQPwq91Qe9x3Vy81CkyVonPOejhINlzfpzqAAa3A6viJccZTwt
DJ8E/I9jg53sbYW8q+VgfLn1hlggH/XQRT0HkXMP5y9ClURYnTsNwJhXABEBAAGs
CXRlc3QgdGVzdIi5BBMBCgAjBQJU0/bNAhsDBwsJCAcDAgEGFQgCCQoLBBYCAwEC
HgECF4AACgkQlsmuCapsqYLvtQP/byY0tM0Lc3moftbHQZ2eHj9ykLjsCjeMDfPx
kZUUtUS3HQaqgZLZOeqPjM7XgGh5hJsd9pfhmRWJ0x+iGB47XQNpRTtdLBV/WMCS
l5z3uW7e9Md7QVUVuSlJnBgQHTS6EgP8JQadPkAiF+jgpJZXP+gFs2j3gobS0qUF
eyTtxs+wAgAD
=uIt9
-----END PGP ARMORED FILE-----
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keydb_search_first, keydb_search_next): Skip legacy
keys.
* g10/keyring.c (keyring_get_keyblock): Handle GPG_ERR_LEGACY_KEY.
(prepare_search): Ditto.
(keyring_rebuild_cache): Skip legacy keys.
* g10/keyserver.c (keyidlist): Ditto.
* g10/trustdb.c (validate_key_list): Ditto.
--
This is not the most elegant way to handle it but it reduces the
chance for unwanted side effects.
GnuPG-bug-id: 1816
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/gpg.h (g10_errstr): Remove macro and change all occurrences by
gpg_strerror.
(G10ERR_): Remove all macros and change all occurrences by their
GPG_ERR_ counterparts.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/getkey.c (get_pubkey_next): Divert to getkey_next.
(get_pubkey_end): Move code to getkey_end.
* g10/keydb.c (keydb_search_reset): Add a debug statement.
(dump_search_desc): Add arg HD and print the handle.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox.h (keybox_blobtype_t): New.
* kbx/keybox-defs.h (BLOBTYPE_*): Replace by KEYBOX_BLOBTYPE_*.
* kbx/keybox-search.c (keybox_search): Add arg want_blobtype and skip
non-matching blobs.
* sm/keydb.c (keydb_search): Pass KEYBOX_BLOBTYPE_X509 to keybox_search.
* g10/keydb.c (keydb_search): Pass KEYBOX_BLOBTYPE_PGP to keybox_search.
--
Without this fix a listing of all keys would fail because the wrong
blob type would be returned for the gpg or gpgsm.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keydb_handle): Add field no_caching.
(keyblock_cache): Repalce field kid by fpr.
(keydb_disable_caching): New.
(keydb_search): Use the fingerprint as cache index.
* g10/import.c (import_one): Use the fingerprint and not the kid to
lookup the key. Call keydb_disable_caching beofre re-searching for
update.
* tests/openpgp/import.test: Add a test case.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_search): Add arg r_skipped and skip too
long blobs.
* sm/keydb.c (keydb_search): Call keybox_search with a dummy param.
* g10/keydb.c (struct keydb_handle): Add field skipped_long_blobs.
(keydb_search_reset): Reset that field.
(keydb_search): Update that field.
(keydb_get_skipped_counter): New.
* g10/keylist.c (list_all): Print count of skipped keys.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-defs.h (struct keybox_handle): Add field for_openpgp.
* kbx/keybox-file.c (_keybox_write_header_blob): Set openpgp header
flag.
* kbx/keybox-blob.c (_keybox_update_header_blob): Add arg for_openpgp
and set header flag.
* kbx/keybox-init.c (keybox_new): Rename to do_keybox_new, make static
and add arg for_openpgp.
(keybox_new_openpgp, keybox_new_x509): New. Use them instead of the
former keybox_new.
* kbx/keybox-update.c (blob_filecopy): Add arg for_openpgp and set the
openpgp header flags.
* g10/keydb.c (rt_from_file): New. Factored out and extended from
keydb_add_resource.
(keydb_add_resource): Switch to the kbx file if it has the openpgp
flag set.
* kbx/keybox-dump.c (dump_header_blob): Print header flags.
--
The problem was reported by dkg on gnupg-devel (2014-10-07):
I just discovered a new problem, though, which will affect people on
systems that have gpg and gpg2 coinstalled:
0) create a new keyring with gpg2, and use it exclusively with gpg2
for a while.
1) somehow (accidentally?) use gpg (1.4.x) again -- this creates
~/.gnupg/pubring.gpg
2) future runs of gpg2 now only look at pubring.gpg and ignore
pubring.kbx -- the keys you had accumulated in the keybox are no
longer listed in the output of gpg2 --list-keys
Note that gpgsm has always used pubring.kbx and thus this file might
already be there but without gpg ever inserted a key. The new flag in
the KBX header gives us an indication whether a KBX file has ever been
written by gpg >= 2.1. If that is the case we will use it instead of
the default pubring.gpg.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (maybe_create_keyring_or_box): Rename arg for clarity.
(keydb_add_resource): Fix order of args to maybe_create_keyring_or_box
and check and create .kbx.
* common/iobuf.c (direct_open): Add arg MODE700.
(iobuf_create): Ditto.
* g10/openfile.c (open_outfile): Add arg RESTRICTEDPERM. Change call
callers to pass 0 for it.
* g10/revoke.c (gen_desig_revoke, gen_revoke): Here pass true for new
arg.
* g10/export.c (do_export): Pass true for new arg if SECRET is true.
--
GnuPG-bug-id: 1653.
Note that this works only if --output has been used.
* kbx/keybox-search.c (keybox_search): Add arg R_DESCINDEX. Chnage
both callers.
* g10/keydb.c (keydb_search): Always set DESCINDEX.
--
This only affects the new keybox for OpenPGP keys in 2.1. The bug
exhibited itself by running GPA's backup command on Windows.
* kbx/keybox-update.c (keybox_update_keyblock): Implement.
* kbx/keybox-search.c (get_blob_flags): Move to ...
* kbx/keybox-defs.h (blob_get_type): here.
* kbx/keybox-file.c (_keybox_read_blob2): Fix calling without R_BLOB.
* g10/keydb.c (build_keyblock_image): Allow calling without
R_SIGSTATUS.
(keydb_update_keyblock): Implement for keybox.
* kbx/keybox-dump.c (_keybox_dump_blob): Fix printing of the unhashed
size. Print "does not expire" also on 64 bit platforms.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (iobuf_seek): Fix for temp streams.
* g10/pubkey-enc.c (get_session_key, get_it): Add some log_clock calls.
* g10/keydb.c (dump_search_desc): New.
(enum_keyblock_states, struct keyblock_cache): New.
(keyblock_cache_clear): New.
(keydb_get_keyblock, keydb_search): Implement a keyblock cache.
(keydb_update_keyblock, keydb_insert_keyblock, keydb_delete_keyblock)
(keydb_rebuild_caches, keydb_search_reset): Clear the cache.
--
Gpg uses the key database at several places without a central
coordination. This leads to several scans of the keybox for the same
key. To improve that we now use a simple cache to store a retrieved
keyblock in certain cases. In theory this caching could also be done
for old keyrings, but it is a bit more work and questionable whether
it is needed; the keybox scheme is anyway much faster than keyrings.
Using a keybox with 20000 384 bit ECDSA/ECHD keypairs and a 252 byte
sample text we get these values for encrypt and decrypt operations on
an Core i5 4*3.33Ghz system. The option --trust-model=always is used.
Times are given in milliseconds wall time.
| | enc | dec | dec,q |
|-----------+-----+-----+-------|
| key 1 | 48 | 96 | 70 |
| key 10000 | 60 | 98 | 80 |
| key 20000 | 69 | 106 | 88 |
| 10 keys | 540 | 290 | 70 |
The 10 keys test uses a mix of keys, the first one is used for
decryption but all keys are looked up so that information about are
printed. The last column gives decryption results w/o information
printing (--quiet).
The keybox is always scanned sequentially without using any index. By
adding an index to the keybox it will be possible to further reduce
the time required for keys stored to the end of the file.
* g10/keydb.c (parse_keyblock_image): Add args PK_NO and UID_NO and
set the note flags accordingly.
(keydb_get_keyblock): Transfer PK_NO and UID_NO to parse_keyblock_image.
* kbx/keybox-search.c (blob_cmp_fpr, blob_cmp_fpr_part)
(blob_cmp_name, blob_cmp_mail): Return the key/user number.
(keybox_search): Set the key and user number into the found struct.
(keybox_get_keyblock): Add args R_PK_NO and R_UID_NO and set them from
the found struct.
--
getkey.c needs to know whether the correct subkey was found. Thus we
need to set the node flags the same way we did it with the keyring
storage.
* common/logging.c (log_clock): New.
* g10/gpg.c (set_debug): Print clock debug flag.
* g10/options.h (DBG_CLOCK_VALUE, DBG_CLOCK): New.
--
To actually use log_clock you need to enable the code in
logginc.c:log_check() and link against librt. --debug 4096 may then
be used to enable it at runtime.
* g10/keydb.c (parse_keyblock_image): Add arg SIGSTATUS.
(keydb_get_keyblock): Handle it.
(build_keyblock_image): Add arg SIGSTATUS.
(keydb_insert_keyblock): Handle it.
* kbx/keybox-blob.c (pgp_create_sig_part): Add arg SIGSTATUS.
(_keybox_create_openpgp_blob): Ditto.
* kbx/kbxutil.c (import_openpgp): Adjust for above change.
* kbx/keybox.h (KEYBOX_FLAG_SIG_INFO): New.
* kbx/keybox-search.c (_keybox_get_flag_location): Handle new flag.
(keybox_get_keyblock): Add arg R_SIGSTATUS.
* kbx/keybox-update.c (keybox_insert_keyblock): Add arg SIGSTATUS.
--
With this change a key listing using the keybox format is now double
as fast as using a keyring. The memory use dropped as well. Measured
with about 1500 keys.
* g10/getkey.c (get_pubkey_fast): Improve the assertion.
* kbx/keybox.h: Include iobuf.h.
* kbx/keybox-blob.c (keyboxblob_uid): Add field OFF.
(KEYBOX_WITH_OPENPGP): Remove use of this macro.
(pgp_create_key_part_single): New.
(pgp_temp_store_kid): Change to use the keybox-openpgp parser.
(pgp_create_key_part): Ditto.
(pgp_create_uid_part): Ditto.
(pgp_create_sig_part): Ditto.
(pgp_create_blob_keyblock): Ditto.
(_keybox_create_openpgp_blob): Ditto.
* kbx/keybox-search.c (keybox_get_keyblock): New.
* kbx/keybox-update.c (keybox_insert_keyblock): New.
* g10/keydb.c (parse_keyblock_image):
(keydb_get_keyblock): Support keybox.
(build_keyblock_image): New.
(keydb_insert_keyblock): Support keybox.
* kbx/kbxutil.c (import_openpgp, main): Add option --dry-run and print
a kbx file to stdout.
* kbx/keybox-file.c (_keybox_read_blob2): Allow keyblocks up to 10^6
bytes.
--
Import and key listing does now work with the keybox format. It is
still quite slow and signature caching is completely missing.
Increasing the maximum allowed length for a keyblock was required due
to a 700k keyblock which inhibited kbxutil to list the file.
kbxutil's option name --import-openpgp is not quite appropriate
because it only creates KBX blobs from OpenPGP data.
* kbx/keybox-defs.h (_keybox_write_header_blob): Move prototype to ..
* kbx/keybox.h: here.
* kbx/keybox-init.c (keybox_lock): Add dummy function
* g10/keydb.c: Include keybox.h.
(KeydbResourceType): Add KEYDB_RESOURCE_TYPE_KEYBOX.
(struct resource_item): Add field kb.
(maybe_create_keyring_or_box): Add error descriptions to diagnostics.
Add arg IS_BOX. Write a header for a new keybox file.
(keydb_add_resource): No more need for the force flag. Rename the
local variable "force" to "create". Add URL scheme "gnupg-kbx". Add
magic test to detect a keybox file. Add basic support for keybox.
(keydb_new, keydb_get_resource_name, keydb_delete_keyblock)
(keydb_locate_writable, keydb_search_reset, keydb_search2): Add
support for keybox.
(lock_all, unlock_all): Ditto.
* g10/Makefile.am (needed_libs): Add libkeybox.a.
(gpg2_LDADD, gpgv2_LDADD): Add KSBA_LIBS as a workaround.
* g10/keydb.h (KEYDB_RESOURCE_FLAG_PRIMARY)
KEYDB_RESOURCE_FLAG_DEFAULT, KEYDB_RESOURCE_FLAG_READONLY): New.
* g10/gpg.c, g10/gpgv.c (main): Use new constants.
--
I did most of these changes back in 2011 and only cleaned them up
now. More to follow soon.
The asymmetric quotes used by GNU in the past (`...') don't render
nicely on modern systems. We now use two \x27 characters ('...').
The proper solution would be to use the correct Unicode symmetric
quotes here. However this has the disadvantage that the system
requires Unicode support. We don't want that today. If Unicode is
available a generated po file can be used to output proper quotes. A
simple sed script like the one used for en@quote is sufficient to
change them.
The changes have been done by applying
sed -i "s/\`\([^'\`]*\)'/'\1'/g"
to most files and fixing obvious problems by hand. The msgid strings in
the po files were fixed with a similar command.
Returning -1 as an error code is not very clean given that gpg error
has more descriptive error codes. Thus we now return
GPG_ERR_NOT_FOUND for all search operations and adjusted all callers.
We better do this once and for all instead of cluttering all future
commits with diffs of trailing white spaces. In the majority of cases
blank or single lines are affected and thus this change won't disturb
a git blame too much. For future commits the pre-commit scripts
checks that this won't happen again.
to libgcrypt functions, using shared error codes from libgpg-error,
replacing the old functions we used to have in ../util by those in
../jnlib and ../common, renaming the malloc functions and a couple of
types. Note, that not all changes are listed below becuause they are
too similar and done at far too many places. As of today the code
builds using the current libgcrypt from CVS but it is very unlikely
that it actually works.
section for random gatherers.
* keyring.c (create_tmp_file, rename_tmp_file): Create tmp files with
user-only permissions, but restore the original permissions if the user
has something special set.
* openfile.c (copy_options_file): Create new options file (gpg.conf) with
user-only permissions.
* keydb.c (keydb_add_resource): Create new keyrings with user-only
permissions.
restarting gpg. This is Debian bug 124219, though their supplied patch
will not do the right thing.
* main.h, tdbio.c (tdbio_set_dbname), misc.c (removed check_permissions),
keydb.c (keydb_add_resource), g10.c (main, check_permissions): Significant
reworking of the permission check mechanism. The new behavior is to check
everything in the homedir by checking the homedir itself. If the user
wants to put (possibly shared) keyrings outside the homedir, they are not
checked. The options file and any extension files are checked wherever
they are, as well as their enclosing directories. This is Debian bug
147760.
