* kbx/keybox-dump.c (_keybox_dump_find_dups): Close FP on the error
paths.
--
GnuPG-bug-id: 6495
Signed-off-by: zhangguangzhi <zhangguangzhi3@huawei.com>
* kbx/keybox-init.c (DEFAULT_LL_BUFFER_SIZE): New.
--
A simple gpg --check-sigs benchmark showed on Linux a small
performance peak at around 64k (5m52 vs. 6m8 for 128k and 6m33 for
system size).
* kbx/keybox-init.c (ll_buffer_size): New var intialized to 128k
(stream_buffers): New var.
(keybox_set_buffersize): New.
(_keybox_ll_open, _keybox_ll_close): Implement buffering.
* sm/gpgsm.c (oKbxBufferSize): New.
(opts): Add option --kbx-buffer-size.
(main): Call keybox_set_buffersize.
* g10/gpg.c: Include keybox.h.
* (oKbxBufferSize): New.
(opts): Add option --kbx-buffer-size.
(main): Call keybox_set_buffersize.
--
Commit message from 2.4:
Running a test on Windows using a pubring.kbx with
Total number of blobs: 2098
openpgp: 1294
x509: 803
and a size of 42MiB with
gpgsm -k --with-validation --disable-dirmngr --kbx-buffer-size N >nul
gives these performance figures using procmon
| N(k) | file events | time(s) |
|------+-------------+---------|
| 0 | 4900000 | 86 |
| 16 | 2456000 | 58 |
| 32 | 1233000 | 43 |
| 64 | 622000 | 37 |
| 128 | 317000 | 32 |
| 256 | 164000 | 31 |
| 512 | 88000 | 30 |
Using _open instead of CreateFile give the same number of file events
but increased the time slight by one second for the measured buffer
size of 64k and 128k. Benchmarks for gpg have not been conducted.
* kbx/keybox-defs.h (KEYBOX_LL_OPEN_READ)
(KEYBOX_LL_OPEN_UPDATE, KEYBOX_LL_OPEN_CREATE): New.
* kbx/keybox-init.c (_keybox_ll_open): New. Replace all keybox use of
es_fopen by this function.
(_keybox_ll_close): New. Replace all keybox use of es_fclose by this
function.
--
Note that this has not been done for the utilities and the backend-kbx
of keyboxd.
* kbx/keybox-search.c (open_file): Use sysopen and sequential.
* kbx/keybox-update.c (create_tmp_file): Ditto.
(blob_filecopy): Ditto.
(keybox_set_flags): Ditto.
(keybox_delete): Ditto.
(keybox_compress): Ditto.
--
Under Windows "sysopen" requests that direct API calls (CreateFile et
al.) are used instead of the libc wrappers. This may or may not
improve the performance.
Using "sequential" is a hint to Windows to assume that a file is in
general access in a sequential manner. This will have an affect only
with a future libgpg-error.
* kbx/keybox-search.c (keybox_search_reset) [W32]: Always close.
* kbx/keybox-init.c (keybox_close_all_files): New.
* sm/keydb.c (keydb_close_all_files): New.
* sm/call-dirmngr.c (gpgsm_dirmngr_isvalid): Call new function.
(gpgsm_dirmngr_lookup): Ditto.
(gpgsm_dirmngr_run_command): Ditto.
--
We need to make sure that there are no open files on Windows. Thus we
close them at several strategic locations.
GnuPG-bug-id: 4505
* configure.ac (GPGRT_ENABLE_ARGPARSE_MACROS): Define.
* common/argparse.c, common/argparse.h: Rewrite.
* tests/gpgscm/main.c: Switch to the new option parser.
* g10/gpg.c: Switch to the new option parser and enable a global conf
file.
* g10/gpgv.c: Ditto.
* agent/gpg-agent.c: Ditto.
* agent/preset-passphrase.c: Ditto.
* agent/protect-tool.c: Ditto.
* scd/scdaemon.c: Ditto.
* dirmngr/dirmngr.c: Ditto.
* dirmngr/dirmngr_ldap.c: Ditto
* dirmngr/dirmngr-client.c: Ditto.
* kbx/kbxutil.c: Ditto.
* tools/gpg-card.c: Ditto.
* tools/gpg-check-pattern.c: Ditto.
* tools/gpg-connect-agent.c: Ditto.
* tools/gpg-pair-tool.c: Ditto.
* tools/gpg-wks-client.c: Ditto.
* tools/gpg-wks-server.c: Ditto.
* tools/gpgconf.c: Ditto.
* tools/gpgsplit.c: Ditto.
* tools/gpgtar.c: Ditto.
* g13/g13.c: Ditto.
* g13/g13-syshelp.c: Ditto. Do not force verbose mode.
* sm/gpgsm.c: Ditto. Add option --no-options.
--
This is backport from master
commit cdbe10b762
commit ba463128ce
commit 3bc004decd
commit 2c823bd878
commit 0e8f6e2aa9
but without changing all functions names to gpgrt. Instead we use
wrapper functions which, when building against old Libgpg-error
versions, are implemented in argparse.c using code from the current
libgpg-error. This allows to keep the dependency requirement at
libgpg-error 1.27 to support older distributions. Tested builds
against 1.27 and 1.40-beta.
Note that g13-syshelp does not anymore default to --verbose because
that can now be enabled in /etc/gnupg/g13-syshelp.conf.
GnuPG-bug-id: 4788
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/sysutils.c (gnupg_fopen) [W32]: Use _wfopen if needed. Use
new function in most places where fopen is used.
--
The config files in 2.2 are still read using fopen - we need to change
this to allow Unicode directory names. There is also one case where
files are written using the old fopen. The new option parser in 2.3
does not have this problem but at some places fopen is also still used.
GnuPG-bug-id: 5098
Signed-off-by: Werner Koch <wk@gnupg.org>
--
We need to use es_fopen on Windows to cope with non-ascii file names.
This is quite a large but fortunately straightforward change. At a
very few places we keep using stdio (for example due to the use of
popen).
GnuPG-bug-id: 5098
Signed-off-by: Werner Koch <wk@gnupg.org>
Backported-from-master: 390497ea11
* common/sysutils.c (gnupg_access): New. Replace all calls to access
by this wrapper.
* common/homedir.c (w32_shgetfolderpath): Change to return UTF-8
directory name.
(standard_homedir): Adjust for change.
(w32_commondir, gnupg_cachedir): Ditto.
--
Also use SHGetFolderPathW instead of SHGetFolderPathA on Windows.
This is required to correctly handle non-ascii filenames on Windows.
GnuPG-bug-id: 5098
(cherry picked from commit c94ee1386e)
* kbx/keybox-dump.c (_keybox_dump_blob): Fix the fixmes. Add support
for 32 byte fingerprints.
--
kbxutil is a debug tool but nevertheless it should behave well and not
read beyond allocated buffers and in turn crash.
Vincent Ulitzsch and Dominik Maier were kind enough to report these
crashes along with a couple of test keyboxes and crash analysis.
Signed-off-by: Werner Koch <wk@gnupg.org>
Backported from master and while doing that adding also the support
for OpenPGP v5 keys.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-init.c (keybox_lock): Add arg TIMEOUT and change all
callers to pass -1.
* g10/keydb.c (keydb_add_resource): Call keybox_compress.
--
Note that here in the 2.2 branch the way we call the locking in gpgsm
is different from the one in gpg. So we could not cherry-pick from
master.
GnuPG-bug-id: 4644
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-dump.c (_keybox_dump_file): Take deleted records in
account.
--
This also changes the numbering of the records to reflect the real
record number.
Signed-off-by: Werner Koch <wk@gnupg.org>
(cherry picked from commit 5ef0d7a795)
* kbx/keybox-update.c (keybox_compress): Use make_timestamp.
--
We have implemented the same in master, albeit we needed two commits
for that.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-search.c (keybox_search): We need to seek to the last
position in all cases not just when doing a NEXT.
--
This is because search from the beginning needs a keybox_search_reset.
We can only make an exception for KEYDB_SEARCH_MODE_FIRST..
Fixes-commit: 6f72aa8214
GnuPG-bug-id: 4505
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-init.c (keybox_lock) [W32]: Use _keybox_close_file
instead of fclose so that a close is done if the file is opened by
another handle.
* kbx/keybox-search.c (keybox_search): Remember the last offset and
use that in NEXT search mode if we had to re-open the file.
--
GnuPG-bug-id: 4505
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-defs.h (struct _keybox_openpgp_key_info): Add field grip.
* kbx/keybox-openpgp.c (struct keyparm_s): New.
(keygrip_from_keyparm): New.
(parse_key): Compute keygrip.
* kbx/keybox-search.c (blob_openpgp_has_grip): New.
(has_keygrip): Call it.
--
This has been marked for too long as not yet working. However, it is
a pretty useful feature and will come pretty handy when looking for
all keys matching one keygrip.
Can be optimized a lot by storing the keygrip in the meta data. This
will be done along with the upgrade of KBX for v5 fingerprints.
Signed-off-by: Werner Koch <wk@gnupg.org>
(cherry picked from commit c128667b3c)
* kbx/keybox-search.c (blob_cmp_fpr): Avoid overflow in OFF+LEN
checking.
(blob_cmp_fpr_part): Ditto.
(blob_cmp_name): Ditto.
(blob_cmp_mail): Ditto.
(blob_x509_has_grip): Ditto.
(keybox_get_keyblock): Check OFF and LEN using a 64 bit var.
(keybox_get_cert): Ditto.
--
On most 32 bit systems size_t is 32 bit and thus the check
size_t cert_off = get32 (buffer+8);
size_t cert_len = get32 (buffer+12);
if (cert_off+cert_len > length)
return gpg_error (GPG_ERR_TOO_SHORT);
does not work as intended for all supplied values. The simplest
solution here is to cast them to 64 bit.
In general it will be better to avoid size_t at all and work with
uint64_t. We did not do this in the past because uint64_t was not
universally available.
GnuPG-bug-id: 3770
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-util.c (keybox_set_malloc_hooks): Remove.
(_keybox_malloc, _keybox_calloc, keybox_realloc)
(_keybox_free): Remove.
(keybox_file_rename): Remove. Was not used.
* sm/gpgsm.c (main): Remove call to keybox_set_malloc_hooks.
* kbx/kbxutil.c (main): Ditto.
* kbx/keybox-defs.h: Remove all separate includes. Include util.h.
remove convenience macros.
* common/logging.h (return_if_fail): New. Originally from
keybox-defs.h but now using log_debug.
(return_null_if_fail): Ditto.
(return_val_if_fail): Ditto.
(never_reached): Ditto.
--
Originally the KBX code was written to allow standalone use. However
this required lot of ugliness like separate memory allocators and
such. It also precludes the use of some standard functions from
common due to their use of the common gnupg malloc functions.
Dropping all that makes things easier. Minor disadvantages: the kbx
call done for gpg will now use gcry malloc fucntions and not the
standard malloc functions. This might be a bit slower but removing
them even fixes a possible bug in keybox_tmp_names which is used in
gpg and uses gpg's xfree which is actually gcry_free.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-file.c (_keybox_read_blob): Remove.
(_keybox_read_blob2): Rename to ....
(_keybox_read_blob): this. Make arg options. Change all callers.
* kbx/keybox-search.c (keybox_search): Factor fopen call out to ...
(open_file): new.
(keybox_seek): Als use open_file.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.h (KEYDB_HANDLE): Move typedef to ...
* g10/gpg.h: here.
(struct server_control_s): Add field 'cached_getkey_kdb'.
* g10/gpg.c (gpg_deinit_default_ctrl): Release that keydb handle.
* g10/getkey.c (getkey_end): Cache keydb handle.
(get_pubkey): Use cached keydb handle.
* kbx/keybox-search.c (keybox_search_reset): Use lseek instead of
closing the file.
--
Before this patch a "gpg --check-sigs" opened and closed the keybox
file for almost every signature check. By caching the keydb handle
and using lseek(2) this can be limited to just 2 times. This might
speed up things on Windows.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keyblock_cache): Remove field SIGSTATUS.
(keyblock_cache_clear): Adjust for that removal.
(parse_keyblock_image): Remove arg SIGSTATUS. Remove the signature
cache setting; this is now done in the parser.
(keydb_get_keyblock): Do not set SIGSTATUS.
(build_keyblock_image): Remove arg SIGSTATUS and simplify. Change
caller.
* kbx/keybox-blob.c: Explain that the signature information is not
anymore used.
(_keybox_create_openpgp_blob): Remove arg SIGSTATUS and change
callers.
* kbx/keybox-search.c (keybox_get_keyblock): Remove arg R_SIGSTATUS
and change callers.
* kbx/keybox-update.c (keybox_insert_keyblock): Likewise.
--
This thing was too complicated and has been replaced by the new ring
trust packet code.
Signed-off-by: Werner Koch <wk@gnupg.org>
* agent/preset-passphrase.c, agent/protect-tool.c, dirmngr/dirmngr.c
* dirmngr/t-http.c, g10/gpg.c, g10/gpgv.c, g13/g13-syshelp.c
* g13/g13.c, kbx/kbxutil.c, scd/scdaemon.c, sm/gpgsm.c
* tests/gpgscm/main.c, tools/gpg-check-pattern.c
* tools/gpg-connect-agent.c, tools/gpgconf.c, tools/gpgtar.c
* tools/symcryptrun.c: Invoke log_set_prefix() with
human-readable labels.
--
Some invocations of log_set_prefix() were done with raw numeric values
instead of values that humans can understand. Use symbolic
representations instead of numeric for better readability.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
* common/init.c (init_common_subsystems): Initialize libgcrypt.
* dirmngr/Makefile.am (dirmngr_ldap): Link with libgcrypt.
--
Most other modules already call gcry_check_version() after
init_common_subsystems() so may as well move initialization of libgcrypt
to here. Also fixes a warning in the system log from gpgconf --homedir.
Signed-off-by: Ben Kibbey <bjk@luxsci.net>
* kbx/keybox-util.c (keybox_file_rename): Add arg BLOCK_SIGNALS.
* kbx/keybox-update.c (rename_tmp_file): Block all signals when doing
a double rename.
* g10/keyring.c (rename_tmp_file): Block all signals during the double
rename.
--
This might fix
Debian-bug-id: 831510
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keydb.c (keydb_release): Clear keyblock cache.
(keydb_get_keyblock): Revert previous change.
* kbx/keybox-blob.c (create_blob_finish): Free previous buffer, free
fixups after applying them.
(_keybox_release_blob): Free buffer. Currently, the buffer has been
extracted before the keybox is released, but this is the right thing
to do here.
Fixes-commit: c57501cc
Signed-off-by: Justus Winter <justus@g10code.com>
* kbx/keybox-util.c (keybox_file_rename): Restart retry intervals
after 800ms.
--
The common use case is that the process waiting for a rename does an
import while another process does a key listing with only short lock
periods. Thus it does not make sense to set the final backoff time to
8s. It would actually be okay to retry every 100ms but that would
spill the console with "waiting..." messages. This change prints the
waiting message only every 1.5s.
Signed-off-by: Werner Koch <wk@gnupg.org>
* g10/keyring.c (keyring_lock) [W32]: Flush the close cache before
locking.
* kbx/keybox-init.c (keybox_lock) [W32]: Close the file before
locking.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-update.c (create_tmp_file): Move some code to...
* kbx/keybox-util.c (keybox_tmp_names): new.
* g10/keyring.c: Include keybox.h.
(create_tmp_file): Replace parts by keybox_tmp_names.
--
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-defs.h: Include dotlock.h and logging.h.
(CONST_KB_NAME): Remove. Replace usage by KB_NAME.
(struct keybox_name): Add field "lockhd".
* kbx/keybox-init.c (keybox_register_file): Init LOCKHD.
(keybox_lock): Chnage to return gpg_error_t. Implement locking.
--
The keybox locking for gpg was not implemented - This needs to be
fixed of course.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-init.c (keybox_register_file): Change interface to return
the token even if the file has already been registered.
* g10/keydb.c (primary_keyring): Rename to primary_keydb.
(maybe_create_keyring_or_box): Change return type to gpg_error_t.
(keydb_add_resource): Ditto. s/rc/err/.
(keydb_add_resource): Mark an already registered as primary.
* sm/keydb.c (maybe_create_keybox): Change return type to gpg_error_t.
(keydb_add_resource): Ditto. s/rc/err/.
(keydb_add_resource): Adjust for changed keybox_register_file.
--
This change aligns the registering of keyboxes with those of
keyrings. This fixes a potential bug:
gpg --keyring foo.kbx --keyring bar.gpg --keyring foo.kbx
would have marked bar.gpg as primary resource and thus inserting new
keys there. The correct and now fixed behavior is to insert to
foo.kbx.
Signed-off-by: Werner Koch <wk@gnupg.org>
