* common/iobuf.c (file_filter_ctx_t): Add fields for the peek feature.
(file_filter): Implement peeking.
(iobuf_ioctl): Add new IOBUF_IOCTL_PEEK.
* common/iobuf.h (IOBUF_IOCTL_PEEK, IOBUFCTRL_PEEK): New.
* common/miscellaneous.c (is_file_compressed): Rewrite. Detect PDF.
* g10/encrypt.c (encrypt_simple): Peek before detecting compression.
(encrypt_crypt): Ditto.
* g10/sign.c (sign_file): Also detect already compressed data.
* g10/options.h (opt): Add explicit_compress_option.
* g10/gpg.c (main): Set opt.explicit_compress_option for -z.
--
Note that this patch also introduces a compression check for signing
which was never done in the past.
GnuPG-bug-id: 6332
* common/iobuf.c (iobuf_cancel): Use gnupg_remove
* common/mischelp.c (same_file_p): Allow for Unicode names.
--
Note that the second patch is used to handle Unicode filenames which
are symbolic links.
* common/iobuf.c (iobuf_read): Handle a case with IOBUF_INPUT_TEMP.
(iobuf_write): Handle a case with IOBUF_OUTPUT_TEMP.
--
GnuPG-bug-id: 5941
Signed-off-by: NIIBE Yutaka <gniibe@fsij.org>
* common/iobuf.c (filter_flush): Remove "src_len == 0" check.
* g10/compress-bz2.c (do_compress): Exit early if flush not
forced and input length is zero.
* g10/compress.c (do_compress): Likewise.
--
Remove "(src_len == 0)" check in filter_flush which was
introduced to fix compress failure caused by zero length
flush from iobuf_close. However this check broke enarmoring
file with length of zero. Patch instead fixes zero length
flush problem in compress filters.
GnuPG-bug-id: T5828
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/iobuf.c (filter_flush): Use 'iobuf->e_d' if configured.
(iobuf_write): Configure 'iobuf->e_d' for 'filter_flush' if
'iobuf->d.buf' is empty and external buffer is larger than threshold.
--
Zero-copy operation in iobuf_write() and filter_flush() allow bypassing
'iobuf->d.buf' for greater performance. This mainly helps OCB
performance where additional memory copies through iobuf stack
can take significant portion of program time.
GnuPG-bug-id: T5828
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/iobuf.h (iobuf_struct): Add 'e_d' substructure and members.
* common/iobuf.c (IOBUF_ZEROCOPY_THRESHOLD): New.
(iobuf_alloc): Clear 'iobuf->e_d'.
(underflow_target): Use 'iobuf->e_d' when configured to bypass copying
through 'iobuf->d.buf'.
(iobuf_read): Configure 'iobuf->e_d' for 'underflow' if 'iobuf->d.buf'
is empty and external buffer is larger than threshold.
--
Zero-copy operation in iobuf_read() and underflow() allow bypassing
'iobuf->d.buf' for greater performance. This mainly helps OCB
performance where additional memory copies through iobuf stack
can take significant portion of program time.
GnuPG-bug-id: T5828
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/iobuf.c (iobuf_copy): Use iobuf buffer size for temporary
buffers.
* g10/plaintext.c (handle_plaintext, do_hash): Likewise.
* g10/sign.c (sign_file): Likewise.
--
As iobuf will have zerocopy operation for read/write, it is better to
use same size buffers as iobuf for temporary copy buffers.
GnuPG-bug-id: T5828
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/sysutils.c (any8bitchar) [W32]: New.
(gnupg_open): New. Replace most calls to open by this.
* common/iobuf.c (any8bitchar) [W32]: New.
(direct_open) [W32]: Use CreateFileW if needed.
--
This is yet another step for full Unicode support on Windows.
GnuPG-bug-id: 5098
* common/stringhelp.c (w32_strerror): Strip trailing CR,LF.
* common/iobuf.c (iobuf_get_filelength): Use -1 and not 0 for the
arg to w32_strerror.
--
This is in particular annoying since we started to use a string
argument sanitizer in the logging code. Before that we just add an
extra blank line.
The second patch corrects a never yet seen error message.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/util.h (EXTERN_UNLESS_MAIN_MODULE): Add the definion only
here but now without the Norcroft-C. Change all other places where it
gets defined.
* common/iobuf.h (iobuf_debug_mode): Declare unconditionally as
extern.
* common/iobuf.c (iobuf_debug_mode): Define it here.
* agent/gpg-agent.c (INCLUDED_BY_MAIN_MODULE): Define here and also in
all main modules of all other programs.
* g10/main.h: Put util.h before the local header files.
--
This change is required for use with gcc/ld's LTO feature which does
not allow common blocks. Further gcc 10 will make -fno-common the
default and thus this chnage is always needed. What a pitty.
Co-authored-by: Tomáš Mráz
GnuPG-bug-id: 4831
Signed-off-by: Werner Koch <wk@gnupg.org>
(cherry picked from commit 21d9bd8b87a9f793a106095e3838eb71825189d7)
- Applied respective chnages also to gpg-card and keyboxd.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (iobuf_copy): Wipe used area of buffer instead of
first sizeof(char*) bytes.
--
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/iobuf.c (iobuf_read_line): Add fast path for finding '\n'
character in buffer.
--
This patch reduce per byte overhead in iobuf_read_line by avoiding
using iobuf_get when possible and use memchr to find '\n'. This
speeds armored decryption.
Benchmark results below, tested on Intel Core i7-4790K (turbo off).
Encrypted 2 GiB through pipe to ramfs file using AES128. Decrypt
ramfs file out through pipe to /dev/null.
before patch-set
----------------
gpg process
armor: user time pipe transfer rate
encrypt-aead: 13.8 140 MB/s
decrypt-aead: 30.6 68 MB/s
encrypt-cfb: 17.4 114 MB/s
decrypt-cfb: 32.6 64 MB/s
after (decrypt+iobuf opt)
-------------------------
gpg process
armor: user time pipe transfer rate
decrypt-aead: 22.5 92 MB/s
decrypt-cfb: 24.4 85 MB/s
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/iobuf.c (block_filter): Use iobuf_get_noeof.
--
When singed integer has negative value, left shift computation is
undefined in C.
GnuPG-bug-id: 4093
Reported-by: Philippe Antoine
Signed-off-by: NIIBE Yutaka <gniibe@fsij.org>
* g10/gpg.c (opts): Add new option.
(opt_set_iobuf_size): New var.
(set_debug): Set the option.
* tests/openpgp/armor.scm: Use this option to revert the buffer size
to the one which used to exhibit the tested bugs.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (IOBUF_BUFFER_SIZE): Rename to
DEFAULT_IOBUF_BUFFER_SIZE and increase to 64k.
(iobuf_buffer_size): New var. Always use this instead of the macro.
(iobuf_set_buffer_size): New.
(struct file_filter_ctx_t): Add field delayed_rc.
(file_filter) [!W32]: Try to fill the supplied buffer.
--
I did some test to see whether this has an effect. A test program
piped 100 million random bytes to gpg to symmetric encryption only w/0
compression. Single read means the old behaviour, multi read the new
behaviour which fills up the buffer when the read(2) returned only 4k
in once call.
8k buffer single read
User time (seconds): 0.09
System time (seconds): 0.04
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.04
8k buffer multi read
User time (seconds): 0.08
System time (seconds): 0.05
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.04
64k buffer single read
User time (seconds): 0.09
System time (seconds): 0.06
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.20
64k buffer multi read
User time (seconds): 0.11
System time (seconds): 0.06
Percent of CPU this job got: 8%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.01
128k buffer single read
User time (seconds): 0.09
System time (seconds): 0.05
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.05
128k buffer multi read
User time (seconds): 0.11
System time (seconds): 0.05
Percent of CPU this job got: 8%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.01
512k buffer single read:
User time (seconds): 0.08
System time (seconds): 0.08
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.21
512k buffer multi read:
User time (seconds): 0.10
System time (seconds): 0.06
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.05
Does not make much of a difference :-(. Maybe it changes depending on
the type of used filters.
Signed-off-by: Werner Koch <wk@gnupg.org>
* sm/gpgsm.c (check_special_filename): Move to ..
* common/sysutils.c (check_special_filename): here. Add arg
NOTRANSLATE.
(allow_special_filenames): New local var.
(enable_special_filenames): New public functions.
* sm/gpgsm.c (allow_special_filenames): Remove var.
(main): Call enable_special_filenames instead of setting the var.
(open_read, open_es_fread, open_es_fwrite): Call
check_special_filename with 0 for NOTRANSLATE.
* common/iobuf.c (special_names_enabled): Remove var.
(iobuf_enable_special_filenames): Remove func.
(check_special_filename): Remove func.
(iobuf_is_pipe_filename): Call new version of the function with
NOTRANSLATE set.
(do_open): Ditto.
* g10/gpg.c (main): Call enable_special_filenames instead of
iobuf_enable_special_filenames.
* g10/gpgv.c (main): Ditto.
--
Note that we keep the iobuf.c:translate_file_handle because it is a
bit different (for whatever reasons) than the translate function from
sysutils.
Signed-off-by: Werner Koch <wk@gnupg.org>
* README, agent/command.c, agent/keyformat.txt, common/i18n.c,
common/iobuf.c, common/keyserver.h, dirmngr/cdblib.c,
dirmngr/ldap-wrapper.c, doc/DETAILS, doc/TRANSLATE,
doc/announce-2.1.txt, doc/gpg.texi, doc/gpgsm.texi,
doc/scdaemon.texi, doc/tools.texi, doc/whats-new-in-2.1.txt,
g10/export.c, g10/getkey.c, g10/import.c, g10/keyedit.c, m4/ksba.m4,
m4/libgcrypt.m4, m4/ntbtls.m4, po/ca.po, po/cs.po, po/da.po,
po/de.po, po/el.po, po/eo.po, po/es.po, po/et.po, po/fi.po,
po/fr.po, po/gl.po, po/hu.po, po/id.po, po/it.po, po/ja.po,
po/nb.po, po/pl.po, po/pt.po, po/ro.po, po/ru.po, po/sk.po,
po/sv.po, po/tr.po, po/uk.po, po/zh_CN.po, po/zh_TW.po,
scd/app-p15.c, scd/ccid-driver.c, scd/command.c, sm/gpgsm.c,
sm/sign.c, tools/gpgconf-comp.c, tools/gpgtar.h: replace "Allow to"
with clearer text.
In standard English, the normal construction is "${XXX} allows ${YYY}
to" -- that is, the subject (${XXX}) of the sentence is allowing the
object (${YYY}) to do something. When the object is missing, the
phrasing sounds awkward, even if the object is implied by context.
There's almost always a better construction that isn't as awkward.
These changes should make the language a bit clearer.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Previously, iobuf_peek on a file smaller than 'buflen' would hang.
* common/iobuf.c (underflow): Generalize by adding a target parameter.
(iobuf_peek): Use this to prevent looping here.
* tests/openpgp/Makefile.am (TESTS): Add new test.
* tests/openpgp/setup.scm (dearmor): Move function...
* tests/openpgp/defs.scm (dearmor): ... here.
* tests/openpgp/issue2419.scm: New file.
* tests/openpgp/samplemsgs/issue2419.asc: Likewise.
GnuPG-bug-id: 2419
Signed-off-by: Justus Winter <justus@g10code.com>
* common/iobuf.c (pop_filter): Rename from this...
(iobuf_pop_filter): ... to this. Don't mark it as static.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* common/iobuf.c (iobuf_copy): Change buffer size from 1 MB to 32 KB.
--
Change suggested by Werner based on the observation that other buffers
are of a similar size.
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* g10/build-packet.c (do_symkey_enc): Use iobuf_write instead of
iobuf_put in a loop. Use iobuf_copy instead of iobuf_read and
iobuf_write in a loop. Move the memory wiping from here...
* common/iobuf.c (iobuf_copy): ... to here.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* common/b64enc.c (b64enc_finish): Remove var assignment which is not
used later.
* common/iobuf.c (file_filter): Ditto.
* common/tlv.c (do_find_tlv): Ditto.
* common/userids.c (classify_user_id): Ditto.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (iobuf_esopen): Initialize 'len' as 'file_es_filter'
will make use of it.
--
Found using the Clang Static Analyzer.
Signed-off-by: Justus Winter <justus@g10code.com>
* common/iobuf.h (enum iobuf_use): Add new member, IOBUF_INPUT_TEMP.
* common/iobuf.c (iobuf_temp_with_content): Create the iobuf as an
IOBUF_INPUT_TEMP, not an IOBUF_INPUT buffer. Assert that LENGTH ==
A->D.SIZE.
(iobuf_push_filter2): If A is an IOBUF_INPUT_TEMP, then make the new
filter an IOBUF_INPUT filter and set its buffer size to
IOBUF_BUFFER_SIZE.
(underflow): If A is an IOBUF_INPUT_TEMP, then just return EOF; don't
remove already read data.
(iobuf_seek): If A is an IOBUF_INPUT_TEMP, don't discard the buffered
data.
(iobuf_alloc): Allow USE == IOBUF_INPUT_TEMP.
(pop_filter): Allow USE == IOBUF_INPUT_TEMP.
(iobuf_peek): Allow USE == IOBUF_INPUT_TEMP.
(iobuf_writebyte): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_write): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_writestr): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_flush_temp): Fail if USE == IOBUF_INPUT_TEMP.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
Introduce a new iobuf type, IOBUF_INPUT_TEMP. Use this for the iobuf
created by iobuf_temp_with_content instead of IOBUF_INPUT. This was
necessary so that seeking and peeking correctly work on this type of
iobuf. In particular, seeking didn't work because we discarded the
buffered data and peeking didn't work because we discarded data which
was already read, which made seeking later impossible.
* common/iobuf.c (iobuf_read): Make control flow more obvious.
(iobuf_get_filelength): Likewise.
(iobuf_get_fd): Likewise.
(iobuf_seek): Likewise.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_alloc): Check that BUFSIZE is not 0.
(iobuf_readbyte): Check that A is an input filter. Check that the
amount of read data is at most the amount of buffered data.
(iobuf_read): Check that A is an input filter.
(iobuf_writebyte): Check that A is not an input filter.
(iobuf_writestr): Check that A is not an input filter.
(iobuf_flush_temp): Check that A is not an input filter.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
