* common/iobuf.c (file_filter_ctx_t): Add fields for the peek feature.
(file_filter): Implement peeking.
(iobuf_ioctl): Add new IOBUF_IOCTL_PEEK.
* common/iobuf.h (IOBUF_IOCTL_PEEK, IOBUFCTRL_PEEK): New.
* common/miscellaneous.c (is_file_compressed): Rewrite. Detect PDF.
* g10/encrypt.c (encrypt_simple): Peek before detecting compression.
(encrypt_crypt): Ditto.
* g10/sign.c (sign_file): Also detect already compressed data.
* g10/options.h (opt): Add explicit_compress_option.
* g10/gpg.c (main): Set opt.explicit_compress_option for -z.
--
Note that this patch also introduces a compression check for signing
which was never done in the past.
GnuPG-bug-id: 6332
* common/iobuf.h (iobuf_struct): Add 'e_d' substructure and members.
* common/iobuf.c (IOBUF_ZEROCOPY_THRESHOLD): New.
(iobuf_alloc): Clear 'iobuf->e_d'.
(underflow_target): Use 'iobuf->e_d' when configured to bypass copying
through 'iobuf->d.buf'.
(iobuf_read): Configure 'iobuf->e_d' for 'underflow' if 'iobuf->d.buf'
is empty and external buffer is larger than threshold.
--
Zero-copy operation in iobuf_read() and underflow() allow bypassing
'iobuf->d.buf' for greater performance. This mainly helps OCB
performance where additional memory copies through iobuf stack
can take significant portion of program time.
GnuPG-bug-id: T5828
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
* common/util.h (EXTERN_UNLESS_MAIN_MODULE): Add the definion only
here but now without the Norcroft-C. Change all other places where it
gets defined.
* common/iobuf.h (iobuf_debug_mode): Declare unconditionally as
extern.
* common/iobuf.c (iobuf_debug_mode): Define it here.
* agent/gpg-agent.c (INCLUDED_BY_MAIN_MODULE): Define here and also in
all main modules of all other programs.
* g10/main.h: Put util.h before the local header files.
--
This change is required for use with gcc/ld's LTO feature which does
not allow common blocks. Further gcc 10 will make -fno-common the
default and thus this chnage is always needed. What a pitty.
Co-authored-by: Tomáš Mráz
GnuPG-bug-id: 4831
Signed-off-by: Werner Koch <wk@gnupg.org>
(cherry picked from commit 21d9bd8b87)
- Applied respective chnages also to gpg-card and keyboxd.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (IOBUF_BUFFER_SIZE): Rename to
DEFAULT_IOBUF_BUFFER_SIZE and increase to 64k.
(iobuf_buffer_size): New var. Always use this instead of the macro.
(iobuf_set_buffer_size): New.
(struct file_filter_ctx_t): Add field delayed_rc.
(file_filter) [!W32]: Try to fill the supplied buffer.
--
I did some test to see whether this has an effect. A test program
piped 100 million random bytes to gpg to symmetric encryption only w/0
compression. Single read means the old behaviour, multi read the new
behaviour which fills up the buffer when the read(2) returned only 4k
in once call.
8k buffer single read
User time (seconds): 0.09
System time (seconds): 0.04
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.04
8k buffer multi read
User time (seconds): 0.08
System time (seconds): 0.05
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.04
64k buffer single read
User time (seconds): 0.09
System time (seconds): 0.06
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.20
64k buffer multi read
User time (seconds): 0.11
System time (seconds): 0.06
Percent of CPU this job got: 8%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.01
128k buffer single read
User time (seconds): 0.09
System time (seconds): 0.05
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.05
128k buffer multi read
User time (seconds): 0.11
System time (seconds): 0.05
Percent of CPU this job got: 8%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.01
512k buffer single read:
User time (seconds): 0.08
System time (seconds): 0.08
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.21
512k buffer multi read:
User time (seconds): 0.10
System time (seconds): 0.06
Percent of CPU this job got: 7%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.05
Does not make much of a difference :-(. Maybe it changes depending on
the type of used filters.
Signed-off-by: Werner Koch <wk@gnupg.org>
* sm/gpgsm.c (check_special_filename): Move to ..
* common/sysutils.c (check_special_filename): here. Add arg
NOTRANSLATE.
(allow_special_filenames): New local var.
(enable_special_filenames): New public functions.
* sm/gpgsm.c (allow_special_filenames): Remove var.
(main): Call enable_special_filenames instead of setting the var.
(open_read, open_es_fread, open_es_fwrite): Call
check_special_filename with 0 for NOTRANSLATE.
* common/iobuf.c (special_names_enabled): Remove var.
(iobuf_enable_special_filenames): Remove func.
(check_special_filename): Remove func.
(iobuf_is_pipe_filename): Call new version of the function with
NOTRANSLATE set.
(do_open): Ditto.
* g10/gpg.c (main): Call enable_special_filenames instead of
iobuf_enable_special_filenames.
* g10/gpgv.c (main): Ditto.
--
Note that we keep the iobuf.c:translate_file_handle because it is a
bit different (for whatever reasons) than the translate function from
sysutils.
Signed-off-by: Werner Koch <wk@gnupg.org>
* common/iobuf.c (pop_filter): Rename from this...
(iobuf_pop_filter): ... to this. Don't mark it as static.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>
* common/iobuf.h (enum iobuf_use): Add new member, IOBUF_INPUT_TEMP.
* common/iobuf.c (iobuf_temp_with_content): Create the iobuf as an
IOBUF_INPUT_TEMP, not an IOBUF_INPUT buffer. Assert that LENGTH ==
A->D.SIZE.
(iobuf_push_filter2): If A is an IOBUF_INPUT_TEMP, then make the new
filter an IOBUF_INPUT filter and set its buffer size to
IOBUF_BUFFER_SIZE.
(underflow): If A is an IOBUF_INPUT_TEMP, then just return EOF; don't
remove already read data.
(iobuf_seek): If A is an IOBUF_INPUT_TEMP, don't discard the buffered
data.
(iobuf_alloc): Allow USE == IOBUF_INPUT_TEMP.
(pop_filter): Allow USE == IOBUF_INPUT_TEMP.
(iobuf_peek): Allow USE == IOBUF_INPUT_TEMP.
(iobuf_writebyte): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_write): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_writestr): Fail if USE == IOBUF_INPUT_TEMP.
(iobuf_flush_temp): Fail if USE == IOBUF_INPUT_TEMP.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
Introduce a new iobuf type, IOBUF_INPUT_TEMP. Use this for the iobuf
created by iobuf_temp_with_content instead of IOBUF_INPUT. This was
necessary so that seeking and peeking correctly work on this type of
iobuf. In particular, seeking didn't work because we discarded the
buffered data and peeking didn't work because we discarded data which
was already read, which made seeking later impossible.
* common/iobuf.h (enum iobuf_use): Name the IOBUF_OUTPUT, etc. enum.
(struct iobuf_struct): Change the field use's type to it.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.h (iobuf_open_fd_or_name): Remove prototype. Replace
use with either iobuf_open or iobuf_fdopen_nc, as appropriate.
* common/iobuf.c (iobuf_open): Remove function.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (iobuf_read_line): Be more careful with boundary
conditions.
* common/iobuf.h: Include <gpg-error.h>.
* common/t-iobuf.c: New file.
* common/Makefile.am (module_tests): Add t-iobuf.
(t_mbox_util_LDADD): New variable.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.h (struct iobuf_struct): Remove field desc.
* common/iobuf.c (iobuf_desc): New function. When a filter's
description is needed, use this instead of the filter's desc field.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.h (struct iobuf_struct): Clarify semantics of nofast.
Simplify use of nofast to implement just these semantics.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c: Move BLOCK_FILTER_INPUT,
BLOCK_FILTER_OUTPUT_BLOCK_FILTER_TEMP from here...
* common/iobuf.h: ... to here and rename to IOBUF_INPUT, IOBUF_OUTPUT
and IOBUF_TEMP, respectively. Where appropriate, use these macros
instead of a literal.
--
Signed-off-by: Neal H. Walfield <neal@g10code.com>.
* common/iobuf.c (direct_open): Add arg MODE700.
(iobuf_create): Ditto.
* g10/openfile.c (open_outfile): Add arg RESTRICTEDPERM. Change call
callers to pass 0 for it.
* g10/revoke.c (gen_desig_revoke, gen_revoke): Here pass true for new
arg.
* g10/export.c (do_export): Pass true for new arg if SECRET is true.
--
GnuPG-bug-id: 1653.
Note that this works only if --output has been used.
We better do this once and for all instead of cluttering all future
commits with diffs of trailing white spaces. In the majority of cases
blank or single lines are affected and thus this change won't disturb
a git blame too much. For future commits the pre-commit scripts
checks that this won't happen again.
char * vs. unsigned char * warnings. The GNU coding standards used to
say that these mismatches are okay and better than a bunch of casts.
Obviously this has changed now.
