* common/iobuf.c (iobuf_read_line): Add fast path for finding '\n'
character in buffer.
--
This patch reduce per byte overhead in iobuf_read_line by avoiding
using iobuf_get when possible and use memchr to find '\n'. This
speeds armored decryption.
Benchmark results below, tested on Intel Core i7-4790K (turbo off).
Encrypted 2 GiB through pipe to ramfs file using AES128. Decrypt
ramfs file out through pipe to /dev/null.
before patch-set
----------------
gpg process
armor: user time pipe transfer rate
encrypt-aead: 13.8 140 MB/s
decrypt-aead: 30.6 68 MB/s
encrypt-cfb: 17.4 114 MB/s
decrypt-cfb: 32.6 64 MB/s
after (decrypt+iobuf opt)
-------------------------
gpg process
armor: user time pipe transfer rate
decrypt-aead: 22.5 92 MB/s
decrypt-cfb: 24.4 85 MB/s
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/iobuf.c (block_filter): Use iobuf_get_noeof.
--
When singed integer has negative value, left shift computation is
undefined in C.
GnuPG-bug-id: 4093
Reported-by: Philippe Antoine
Signed-off-by: NIIBE Yutaka <gniibe@fsij.org>
* g10/gpg.c (opts): Add new option.
(opt_set_iobuf_size): New var.
(set_debug): Set the option.
* tests/openpgp/armor.scm: Use this option to revert the buffer size
to the one which used to exhibit the tested bugs.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (IOBUF_BUFFER_SIZE): Rename to
DEFAULT_IOBUF_BUFFER_SIZE and increase to 64k.
(iobuf_buffer_size): New var. Always use this instead of the macro.
(iobuf_set_buffer_size): New.
(struct file_filter_ctx_t): Add field delayed_rc.
(file_filter) [!W32]: Try to fill the supplied buffer.
--
I did some test to see whether this has an effect. A test program
piped 100 million random bytes to gpg to symmetric encryption only w/0
compression. Single read means the old behaviour, multi read the new
behaviour which fills up the buffer when the read(2) returned only 4k
in once call.
8k buffer single read
User time (seconds): 0.09
System time (seconds): 0.04
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.04
8k buffer multi read
User time (seconds): 0.08
System time (seconds): 0.05
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.04
64k buffer single read
User time (seconds): 0.09
System time (seconds): 0.06
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.20
64k buffer multi read
User time (seconds): 0.11
System time (seconds): 0.06
Percent of CPU this job got: 8%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.01
128k buffer single read
User time (seconds): 0.09
System time (seconds): 0.05
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.05
128k buffer multi read
User time (seconds): 0.11
System time (seconds): 0.05
Percent of CPU this job got: 8%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.01
512k buffer single read:
User time (seconds): 0.08
System time (seconds): 0.08
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.21
512k buffer multi read:
User time (seconds): 0.10
System time (seconds): 0.06
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.05
Does not make much of a difference :-(. Maybe it changes depending on
the type of used filters.
Signed-off-by: Werner Koch <wk@gnupg.org>
* sm/gpgsm.c (check_special_filename): Move to ..
* common/sysutils.c (check_special_filename): here. Add arg
NOTRANSLATE.
(allow_special_filenames): New local var.
(enable_special_filenames): New public functions.
* sm/gpgsm.c (allow_special_filenames): Remove var.
(main): Call enable_special_filenames instead of setting the var.
(open_read, open_es_fread, open_es_fwrite): Call
check_special_filename with 0 for NOTRANSLATE.
* common/iobuf.c (special_names_enabled): Remove var.
(iobuf_enable_special_filenames): Remove func.
(check_special_filename): Remove func.
(iobuf_is_pipe_filename): Call new version of the function with
NOTRANSLATE set.
(do_open): Ditto.
* g10/gpg.c (main): Call enable_special_filenames instead of
iobuf_enable_special_filenames.
* g10/gpgv.c (main): Ditto.
--
Note that we keep the iobuf.c:translate_file_handle because it is a
bit different (for whatever reasons) than the translate function from
sysutils.
Signed-off-by: Werner Koch <wk@gnupg.org>
* README, agent/command.c, agent/keyformat.txt, common/i18n.c,
common/iobuf.c, common/keyserver.h, dirmngr/cdblib.c,
dirmngr/ldap-wrapper.c, doc/DETAILS, doc/TRANSLATE,
doc/announce-2.1.txt, doc/gpg.texi, doc/gpgsm.texi,
doc/scdaemon.texi, doc/tools.texi, doc/whats-new-in-2.1.txt,
g10/export.c, g10/getkey.c, g10/import.c, g10/keyedit.c, m4/ksba.m4,
m4/libgcrypt.m4, m4/ntbtls.m4, po/ca.po, po/cs.po, po/da.po,
po/de.po, po/el.po, po/eo.po, po/es.po, po/et.po, po/fi.po,
po/fr.po, po/gl.po, po/hu.po, po/id.po, po/it.po, po/ja.po,
po/nb.po, po/pl.po, po/pt.po, po/ro.po, po/ru.po, po/sk.po,
po/sv.po, po/tr.po, po/uk.po, po/zh_CN.po, po/zh_TW.po,
scd/app-p15.c, scd/ccid-driver.c, scd/command.c, sm/gpgsm.c,
sm/sign.c, tools/gpgconf-comp.c, tools/gpgtar.h: replace "Allow to"
with clearer text.
In standard English, the normal construction is "${XXX} allows ${YYY}
to" -- that is, the subject (${XXX}) of the sentence is allowing the
object (${YYY}) to do something. When the object is missing, the
phrasing sounds awkward, even if the object is implied by context.
There's almost always a better construction that isn't as awkward.
These changes should make the language a bit clearer.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Previously, iobuf_peek on a file smaller than 'buflen' would hang.
* common/iobuf.c (underflow): Generalize by adding a target parameter.
(iobuf_peek): Use this to prevent looping here.
* tests/openpgp/Makefile.am (TESTS): Add new test.
* tests/openpgp/setup.scm (dearmor): Move function...
* tests/openpgp/defs.scm (dearmor): ... here.
* tests/openpgp/issue2419.scm: New file.
* tests/openpgp/samplemsgs/issue2419.asc: Likewise.
GnuPG-bug-id: 2419
Signed-off-by: Justus Winter <justus@g10code.com>
* common/iobuf.c (pop_filter): Rename from this...
(iobuf_pop_filter): ... to this. Don't mark it as static.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* common/iobuf.c (iobuf_copy): Change buffer size from 1 MB to 32 KB.
--
Change suggested by Werner based on the observation that other buffers
are of a similar size.
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* g10/build-packet.c (do_symkey_enc): Use iobuf_write instead of
iobuf_put in a loop. Use iobuf_copy instead of iobuf_read and
iobuf_write in a loop. Move the memory wiping from here...
* common/iobuf.c (iobuf_copy): ... to here.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* common/b64enc.c (b64enc_finish): Remove var assignment which is not
used later.
* common/iobuf.c (file_filter): Ditto.
* common/tlv.c (do_find_tlv): Ditto.
* common/userids.c (classify_user_id): Ditto.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (iobuf_esopen): Initialize 'len' as 'file_es_filter'
will make use of it.
--
Found using the Clang Static Analyzer.
Signed-off-by: Justus Winter <justus@g10code.com>
* common/iobuf.h (enum iobuf_use): Add new member, IOBUF_INPUT_TEMP.
* common/iobuf.c (iobuf_temp_with_content): Create the iobuf as an
IOBUF_INPUT_TEMP, not an IOBUF_INPUT buffer. Assert that LENGTH ==
A->D.SIZE.
(iobuf_push_filter2): If A is an IOBUF_INPUT_TEMP, then make the new
filter an IOBUF_INPUT filter and set its buffer size to
IOBUF_BUFFER_SIZE.
(underflow): If A is an IOBUF_INPUT_TEMP, then just return EOF; don't
remove already read data.
(iobuf_seek): If A is an IOBUF_INPUT_TEMP, don't discard the buffered
data.
(iobuf_alloc): Allow USE == IOBUF_INPUT_TEMP.
(pop_filter): Allow USE == IOBUF_INPUT_TEMP.
(iobuf_peek): Allow USE == IOBUF_INPUT_TEMP.
(iobuf_writebyte): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_write): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_writestr): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_flush_temp): Fail if USE == IOBUF_INPUT_TEMP.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
Introduce a new iobuf type, IOBUF_INPUT_TEMP. Use this for the iobuf
created by iobuf_temp_with_content instead of IOBUF_INPUT. This was
necessary so that seeking and peeking correctly work on this type of
iobuf. In particular, seeking didn't work because we discarded the
buffered data and peeking didn't work because we discarded data which
was already read, which made seeking later impossible.
* common/iobuf.c (iobuf_read): Make control flow more obvious.
(iobuf_get_filelength): Likewise.
(iobuf_get_fd): Likewise.
(iobuf_seek): Likewise.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_alloc): Check that BUFSIZE is not 0.
(iobuf_readbyte): Check that A is an input filter. Check that the
amount of read data is at most the amount of buffered data.
(iobuf_read): Check that A is an input filter.
(iobuf_writebyte): Check that A is not an input filter.
(iobuf_writestr): Check that A is not an input filter.
(iobuf_flush_temp): Check that A is not an input filter.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_push_filter2): If the head filter is a temp
filter, use IOBUF_BUFFER_SIZE for the new filter.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_push_filter2): If the pipeline is an output or
temp pipeline, the new filter shouldn't assume ownership of the old
head's internal buffer: the data was written before the filter was
added.
* common/t-iobuf.c (double_filter): New function.
(main): Add test cases for the above bug.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_temp_to_buffer): Flush each filter in the
pipeline and copy the data from the last (not the first) filter's
internal buffer.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (do_open): New function, which is a generalization of
iobuf_open, iobuf_Create, iobuf_openrw.
(iobuf_open): Call do_open.
(iobuf_create): Likewise.
(iobuf_openrw): Likewise.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.h (iobuf_open_fd_or_name): Remove prototype. Replace
use with either iobuf_open or iobuf_fdopen_nc, as appropriate.
* common/iobuf.c (iobuf_open): Remove function.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_cancel): Don't abort freeing a pipeline if
freeing a filter fails. This needs to a memory leak. Instead, keep
freeing and return the error code of the first filter that fails.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (underflow): Take additional parameter
clear_pending_eof. If not set, don't clear a pending eof when
returning EOF. Update callers.
(iobuf_peek): Fill the internal buffer, if needed, to be able to
better satisfy any request.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (underflow): Don't require that the buffer be empty.
When called, fill any available space.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_read_line): Be more careful with boundary
conditions.
* common/iobuf.h: Include <gpg-error.h>.
* common/t-iobuf.c: New file.
* common/Makefile.am (module_tests): Add t-iobuf.
(t_mbox_util_LDADD): New variable.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_temp_with_content): Set the filter type to
IOBUF_INPUT, not IOBUF_TEMP, which is only for output filters that
write into a dynamic buffer.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.h (struct iobuf_struct): Remove field desc.
* common/iobuf.c (iobuf_desc): New function. When a filter's
description is needed, use this instead of the filter's desc field.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.h (struct iobuf_struct): Clarify semantics of nofast.
Simplify use of nofast to implement just these semantics.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c: Move BLOCK_FILTER_INPUT,
BLOCK_FILTER_OUTPUT_BLOCK_FILTER_TEMP from here...
* common/iobuf.h: ... to here and rename to IOBUF_INPUT, IOBUF_OUTPUT
and IOBUF_TEMP, respectively. Where appropriate, use these macros
instead of a literal.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (BLOCK_FILTER_INPUT): Define. Where appropriate, use
this instead of a literal.
(BLOCK_FILTER_OUTPUT): Likewise.
(BLOCK_FILTER_TEMP): Likewise.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
