* kbx/keybox-init.c (DEFAULT_LL_BUFFER_SIZE): New.
--
A simple gpg --check-sigs benchmark showed on Linux a small
performance peak at around 64k (5m52 vs. 6m8 for 128k and 6m33 for
system size).
* kbx/keybox-init.c (ll_buffer_size): New var intialized to 128k
(stream_buffers): New var.
(keybox_set_buffersize): New.
(_keybox_ll_open, _keybox_ll_close): Implement buffering.
* sm/gpgsm.c (oKbxBufferSize): New.
(opts): Add option --kbx-buffer-size.
(main): Call keybox_set_buffersize.
* g10/gpg.c: Include keybox.h.
* (oKbxBufferSize): New.
(opts): Add option --kbx-buffer-size.
(main): Call keybox_set_buffersize.
--
Running a test on Windows using a pubring.kbx with
Total number of blobs: 2098
openpgp: 1294
x509: 803
and a size of 42MiB with
gpgsm -k --with-validation --disable-dirmngr --kbx-buffer-size N >nul
gives these performance figures using procmon
| N(k) | file events | time(s) |
|------+-------------+---------|
| 0 | 4900000 | 86 |
| 16 | 2456000 | 58 |
| 32 | 1233000 | 43 |
| 64 | 622000 | 37 |
| 128 | 317000 | 32 |
| 256 | 164000 | 31 |
| 512 | 88000 | 30 |
Using _open instead of CreateFile give the same number of file events
but increased the time slight by one second for the measured buffer
size of 64k and 128k. Benchmarks for gpg have not been conducted.
* kbx/keybox-defs.h (KEYBOX_LL_OPEN_READ)
(KEYBOX_LL_OPEN_UPDATE, KEYBOX_LL_OPEN_CREATE): New.
* kbx/keybox-init.c (_keybox_ll_open): New. Replace all keybox use of
es_fopen by this function.
(_keybox_ll_close): New. Replace all keybox use of es_fclose by this
function.
--
Note that this has not been done for the utilities and the backend-kbx
of keyboxd.
* kbx/keybox-search.c (open_file): Use sysopen and sequential.
* kbx/keybox-update.c (create_tmp_file): Ditto.
(blob_filecopy): Ditto.
(keybox_set_flags): Ditto.
(keybox_delete): Ditto.
(keybox_compress): Ditto.
--
Under Windows "sysopen" requests that direct API calls (CreateFile et
al.) are used instead of the libc wrappers. This may or may not
improve the performance.
Using "sequential" is a hint to Windows to assume that a file is in
general access in a sequential manner. This will have an affect only
with a future libgpg-error.
* kbx/keyboxd.h (kbxd_status_printf): New.
* kbx/backend-support.c (be_return_pubkey): Use kbxd_status_printf.
* kbx/kbxserver.c (kbxd_status_printf): New.
* kbxd_start_command_handler (kbxd_start_command_handler): Don't
use set_assuan_context_func, because the function pointer is
shared by multiple threads.
--
GnuPG-bug-id: 5948
Signed-off-by: NIIBE Yutaka <gniibe@fsij.org>
* kbx/keybox-search.c (blob_cmp_fpr_part): Don't change FPROFF, since
it's caller which tweaks the offset.
(has_short_kid, has_long_kid): Examine the key flags to determine if
fingerprint 32 or 20.
--
GnuPG-bug-id: 5888
Signed-off-by: NIIBE Yutaka <gniibe@fsij.org>
* kbx/keyboxd.c (oStealSocket): New const.
(opts): Add option.
(steal_socket): New file global flag.
(main): Set option.
(create_server_socket): Implement option.
--
* kbx/backend-kbx.c (be_kbx_search): Initialize skipped_long_blobs
value which is passed to keybox_search and incremented there.
(be_kbx_seek): Likewise.
--
GnuPG-bug-id: 5393
Co-authored-by: NIIBE Yutaka <gniibe@fsij.org>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
* kbx/keybox-blob.c (create_blob_header): Fix the flag to match no
UBID entry,
(create_blob_finish): Fix the length of data to be hashed.
--
GnuPG-bug-id: 5573
Fixes-commit: 915297705a
Signed-off-by: NIIBE Yutaka <gniibe@fsij.org>
* kbx/keybox-search-desc.h (struct keydb_search_desc): New flag
name_used.
* common/userids.c (classify_user_id): Set flag.
* kbx/kbxserver.c (struct search_backing_store_s): New.
(cmd_search): use a backing store for the const pointers.
(kbxd_start_command_handler): Release the backing store.
--
Well, the search object partly uses buffers but also const
pointers (for strings and the serial number). This when assigning
such objects to an another one we should really take a deep copy and
not just copy the pointer. The more clean solution would have been to
provide a storage option the search object but that needs checking the
code at too many places so that I decided to use a separate backing
store array here.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/kbxserver.c (cmd_search): Use the openpgp hack for calling
classify_user_id.
* kbx/backend-sqlite.c (run_select_statement): Remove angle brackets
in exact addrspec mode.
* g10/call-keyboxd.c (keydb_search): Do not duplicate the left angle
bracket.
* sm/keydb.c (keydb_search): Ditto.
--
Note that the openpgp hack flag of classify_user_id is actually a
misnomer because we actually hack a round a problem in gpgsm. And it
is only over there that we don't set it there. In keyboxd the flag
should be set. And we need to remove the angle brackets of course
because that is how we create the addrspec column values.
Signed-off-by: Werner Koch <wk@gnupg.org>
* configure.ac (NEED_GPG_ERROR_VERSION): Rename to NEED_GPGRT_VERSION
and set to 1.41.
* common/sysutils.c (gnupg_access): Remove code for older gpgrt
versions.
* kbx/backend-sqlite.c: Ditto.
* sm/gpgsm.c (main): Ditto.
--
We already have a requirement for a newer Libgcrypt and thus we can
also require a more recent libgpgrt (aka libgpg-error) which was
released before Libgcrypt.
* common/sysutils.c (gnupg_stat): New.
* common/sysutils.h: Include sys/stat.h.
--
Yet another wrapper for Unicode support on Windows.
GnuPG-bug-id: 5098
Signed-off-by: Werner Koch <wk@gnupg.org>
--
We need to use es_fopen on Windows to cope with non-ascii file names.
This is quite a large but fortunately straightforward change. At a
very few places we keep using stdio (for example due to the use of
popen).
GnuPG-bug-id: 5098
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/sysutils.c (gnupg_access): New. Replace all calls to access
by this wrapper.
* common/homedir.c (w32_shgetfolderpath): Change to return UTF-8
directory name.
(standard_homedir): Adjust for change.
(w32_commondir, gnupg_cachedir): Ditto.
--
Also use SHGetFolderPathW instead of SHGetFolderPathA on Windows.
This is required to correctly handle non-ascii filenames on Windows.
GnuPG-bug-id: 5098
* kbx/backend-sqlite.c (struct be_sqlite_local_s): Add fields
lastubid_valid and lastubid.
(run_sql_prepare): Add optional extra2 arg and chage callers.
(run_select_statement): Add an ORDER BY clause to most SELECTs.
(be_sqlite_search): Skip duplicated keyblocks in a search.
--
See the comment in the code for the background.
Beware: This change lets tests/openpgp/key-selection.scm fail. Needs
to be fixed.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/backend-sqlite.c (DATABASE_VERSION): New.
(table_definitions): Make column name of table config unique.
(create_or_open_database): Read and set the database version.
(get_config_value, set_config_value): New.
--
Well, kind of. For now we just store the database version. Note that
for existing databases the config table should first be dropped so
that the UNIQUE constraint can be used.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/backend-sqlite.c (be_sqlite_rollback): New.
(be_sqlite_commit): New.
(be_sqlite_search): Take care of global transactions.
(be_sqlite_store): Ditto.
(be_sqlite_delete): Ditto.
* kbx/frontend.c (kbxd_rollback, kbxd_commit): New.
* kbx/keyboxd.h (opt): Add vars for transactions.
* kbx/kbxserver.c (struct server_local_s): Add fields next_session and
client_pid.
(session_list): New var.
(cmd_transaction): New.
(register_commands): Register command.
(kbxd_start_command_handler): Store pids and track sessions. Do a
final rollback.
--
This command is currently an experiment to allow a client to run
everything in one session.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/kbx-client-util.c (kbx_client_data_simple): New.
* kbx/backend-sqlite.c (struct be_sqlite_local_s): Add field descidx.
(be_sqlite_search): Use that.
* g10/call-keyboxd.c (keydb_search): Implement multi mode.
--
With that change the keyboxd is at par with the keybox code.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/backend-sqlite.c: Always use ?NNN for SQL parameters.
--
Using :NNN is obviously wrong to get the parameters in the right
order.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/backend-sqlite.c (table_definitions): Add column UINO to
userids.
(be_sqlite_local_s): Add fields select_col_uidno and
select_col_subkey.
(run_select_statement): Also select subkey or uidno column.
(be_sqlite_search): Return their values.
(store_into_userid): Store the UIDNO.
* kbx/backend-support.c (be_return_pubkey): Extend PUBKEY_INFO.
--
For an existing database adding the new column to the table userid is
straightforward. However if the original version of the schema used an
integer for the keyid column, that column has likely be renamed. Make
sure that the NOT NULL constraint has also be removed; check the
SQLite documentation on how this can be done.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keybox-dump.c (_keybox_dump_blob): Fix the fixmes.
--
kbxutil is a debug tool but nevertheless it should behave well and not
read beyond allocated buffers and in turn crash.
Vincent Ulitzsch and Dominik Maier were kind enough to report these
crashes along with a couple of test keyboxes and crash analysis.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/backend-sqlite.c: Change definition of column KID.
(kid_from_mem): Remove.
(kid_from_u32): Rewrite.
(run_sql_bind_int64): Remove.
(run_select_statement): Implement lookup by short keyid. Fix lookup
by long keyid.
(store_into_fingerprint): Adjust kid arg.
--
The original idea of using an INTEGER, which has the required 64 bits,
didn't worked out due to problems with signed/unsigned mismatch and
the required truncation in a short keyid lookup. Thus we change the
definition to a blob.
For a database currently in use the change can be done by hand:
alter table fingerprint rename column kid to deletedkid;
alter table fingerprint add column kid not null
default X'0000000000000000';
update fingerprint set kid = substr (fpr, 13);
The update does only work with 20 octet fingerprints; that is not with
the new v5 OpenPGP keys. Note that sqlite does not allow to drip a
column, thus we rename it.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/backend-support.c (be_return_pubkey): Add args is_ephemeral and
is_revoked. Adjust callers.
* kbx/backend-sqlite.c: Alter table pubkey to add new columns.
(run_select_statement): Add new column to all selects.
(be_sqlite_search): Return the new flags.
--
For existing test databases the new column can be added with:
alter table pubkey add ephemeral integer not null default 0;
alter table pubkey add revoked integer not null default 0;
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/kbx-client-util.c (kbx_client_data_new): Add arg 'dlines'.
* g10/call-keyboxd.c (open_context): Set DLINES to true.
* sm/keydb.c (open_context): Ditto.
--
This allows to compile time switch between the D-line and the
fd-passing data communication between gpg/gpgsm and keyboxd. A quick
test with about 3000 OpenPGP keys showed that D-lines are only 10%
slower than the fd-passing based implementation. Given that the
thread adds extra complexity we go for now with the D-line approach.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/keyboxd.h (struct server_control_s): Replace the two request
objects by just one. Add filter flags.
* kbx/kbxserver.c (cmd_search): Add options --openpgp and --x509.
(cmd_killkeyboxd): Do not return GPG_ERR_EOF.
* kbx/frontend.c (kbxd_release_session_info): Adjust for the new
request object.
(kbxd_search, kbxd_store, kbxd_delete): Ditto.
* kbx/backend-sqlite.c (struct be_sqlite_local_s): Add filter flags.
(run_sql_prepare): Add optional arg 'extra'. Change callers.
(run_sql_bind_ntext): New.
(run_sql_bind_text): Just call run_sql_bind_ntext.
(run_select_statement): Add ctrl arg. Implement the filter flags.
* g10/call-keyboxd.c (keydb_search): Use the --openpgp option.
--
As soon as we implement X.509 we need to have a way to return only
openpgp or x.509 certificates. Gpg/gpgsm will then use the respective
flag.
Signed-off-by: Werner Koch <wk@gnupg.org>
* kbx/kbx-client-util.c: New.
* kbx/kbx-client-util.h: New.
* kbx/Makefile.am (client_sources): New.
* g10/keydb.c (parse_keyblock_image): Rename to keydb_parse_keyblock
and make global.
* g10/call-keyboxd.c: Include kbx-client-util.h.
(struct keyboxd_local_s): Remove struct datastream. Add field kcd.
Remove per_session_init_done.
(lock_datastream, unlock_datastream): Remove.
(prepare_data_pipe, datastream_thread): Remove.
(keydb_get_keyblock_do_parse): Remove.
(gpg_keyboxd_deinit_session_data): Release the KCD object.
(open_context): Use of kbx_client_data_new.
(keydb_get_keyblock): Simplify.
(keydb_search): Use kbx_client_data_cmd and _wait.
--
The data specific part of the code has been moved from gpg to a new
module in kbx/ so that it can also be used by gpgsm. The OpenPGP
parsing while reading the data has been replaced by storing the data
in memory and parse it later. That makes a nice interface and
abstracts the fd-passing/D-lines handling away.
Signed-off-by: Werner Koch <wk@gnupg.org>
