0170f0e37e
Improve a bit the comments in the code
...
This is related to the previous commit
2019-09-01 13:52:02 +02:00
0cf0541ad9
Remove nsid fields from MSOffice documents
...
nsids are random identifiers, usually used to ease merging
between documents, and can trivially be used for fingerprinting.
2019-09-01 13:52:02 +02:00
0c75cd15dc
Remove a mypy workaround to bump coverage back to 100%
2019-07-22 23:28:51 +02:00
5280b6c2b3
Add a test for svg namespace
2019-07-22 23:21:06 +02:00
8bb2826f7a
CI: Add job to run codespell, a spell checking software
2019-07-22 13:31:40 -07:00
5c33b290ae
Fix mypy
2019-07-20 16:05:55 +02:00
dc5603eb1d
Please mypy
2019-07-13 23:25:44 +02:00
4999209f9c
Add support for svg
2019-07-13 21:26:05 +02:00
bdd5581033
Compress cleaned zip archives by default
2019-07-13 15:04:43 +02:00
47f9cb33bf
Please mypy
2019-07-13 15:03:40 +02:00
35d550d229
Use memoization get _*_path() functions
...
This shouldn't make a big difference in the CLI/extension
usage, but might improve the performances of long-running
instances, or people misusing the API.
2019-05-16 00:31:40 +02:00
aa52a5c91c
Please mypy wrt. the last two commits
2019-05-14 00:50:17 +02:00
f19f6ed8b6
Rework the dependency checks to distinguish required/optional ones
...
Rework the dependencies definition to include a 'required' flags, which
is passed by the check_dependencies helper to the callers, so that they
can distinguish between required and optional dependencies.
This help in two ways:
- The unit test for the dependencies was now failing when an optional
one was missing, due to a previous rework.
- Mat2's --check-dependencies was referring to "required dependencies"
and was misleading for the user as some of them could be optional.
Signed-off-by: Antoine Tenart <antoine.tenart@ack.tf>
2019-05-13 23:35:26 +02:00
97abafdc58
Minor code cleanup
2019-05-09 09:41:05 +02:00
7e031c9757
typo
2019-05-03 02:39:15 -07:00
9516990693
Add some verification for "dangerous" tarfiles
2019-05-01 17:55:35 +02:00
a7ebb587e1
Handle weird permissions in tar archives
2019-04-27 22:48:40 +02:00
14a4cddb8b
Improve the display of tarfile's members mtime
2019-04-27 21:15:06 +02:00
8e41b098d6
Add support for compressed tar files
2019-04-27 06:03:09 -07:00
82cc822a1d
Add tar archive support
2019-04-27 04:05:36 -07:00
05f429b197
Add support for xhtml files
2019-04-14 20:36:33 +02:00
1e325c5b5b
Please mypy
...
Apparently, mypy isn't able (yet?) to deal
with variables that are changing their types
at runtime.
Python is wonderful.
2019-03-30 10:33:16 +01:00
d454ef5b8e
libmat2: fix dependency checks for cmd line utilities
...
The command line checks for command line utilities are done by trying to
access the executables and by throwing an exception when not found. This
lead to:
- The mat2 cmd line --check-dependencies option failing.
- The ffmpeg unit tests failing when ffmpeg isn't installed (even though
it's an optional dependency).
This patch fixes it.
Signed-off-by: Antoine Tenart <antoine.tenart@ack.tf>
2019-03-29 19:29:28 +01:00
c824a68dd8
libmat2: reshape the dependencies list
...
Invert the keys and values in DEPENDENCIES. It seems more natural to use
the key as a key in check_dependencies(), and the value as the value.
This also help in preparing for reworking the check_dependencies()
helper.
Signed-off-by: Antoine Tenart <antoine.tenart@ack.tf>
2019-03-29 19:29:28 +01:00
b8c92fec09
Fix the testsuite
2019-03-23 00:41:23 +01:00
0e3c2c9b1b
libmat2: audio: not all id3 types have a text attribute
...
Not all id3 types have a text attribute (such as mutagen.id3.APIC or
mutagen.id3.UFID). This leads to the get_meta helper to crash when
trying to access the text attribute of an object which does not have it.
Fixes it by checking the text attribute is available before accessing
it.
Signed-off-by: Antoine Tenart <antoine.tenart@ack.tf>
2019-03-23 00:32:44 +01:00
5ac91cd4f9
Refactor {black,white}list into {block,allow}list
...
Closes #96
2019-03-05 23:13:42 +00:00
c3f097a82b
fix typo
2019-03-01 22:00:23 +00:00
55214206b5
Improve the previous commit
...
- More tests
- More documentation
- Minor code cleanup
2019-02-27 23:53:07 +01:00
73d2966e8c
Improve epub support
2019-02-27 23:04:38 +01:00
eb2e702f37
Document the previous commit
2019-02-25 15:37:44 +01:00
545dccc352
In archive-based formats, the `mimetype` file comes first
...
This should improve epub compatibility,
along with other formats as a side-effect
2019-02-24 23:32:32 +01:00
524bae5972
<title> is also an html metadata
2019-02-23 20:47:26 +01:00
c757a9b7ef
Fix a bug in css cleaning
...
It's not mandatory to actually have a comment inside
comment delimiter, like `/**/`.
2019-02-23 20:21:11 +01:00
02ff21b158
Implement epub support
2019-02-20 16:28:11 -08:00
a81b7658a8
Make the mandatory metadata warning generic
...
This should close #95 .
2019-02-10 21:46:13 +01:00
6e63e03b86
Streamline a bit the previous commit
2019-02-09 15:23:16 +01:00
a71488d459
bind mount /etc/ld.so.cache to the sandbox
...
without /etc/ld.so.cache available in the sandbox, tests fail on gentoo with:
/usr/bin/ffmpeg: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory
2019-02-09 09:49:51 +01:00
6ef6aaa222
Improve a bit get_meta for libreoffice files
2019-02-08 23:23:56 +01:00
6cc034e81b
Add support for html files
2019-02-08 23:05:18 +01:00
e1dd439fc8
Use of the archive refactoring for the office documents too
2019-02-07 22:19:37 +01:00
b9a62d798a
Refactor a bit office get_meta handling
...
This should make easier to get more metadata from
archive-based file formats.
2019-02-04 00:31:26 +01:00
433609f8ea
Implement .gif support
2019-02-03 21:01:58 +01:00
e8c1bb0e3c
Whenever possible, use bwrap for subprocesses
...
This should closes #90
2019-02-03 19:18:41 +01:00
8e84ba547a
Add support for wmv
2019-02-02 19:19:36 +01:00
04bb8c8ccf
Add mp4 support
2018-10-28 07:41:04 -07:00
3a070b0ab7
Add support for zip files
2018-10-25 11:56:46 +02:00
283e5e5787
Improve archive-based parser's robustness against corrupted embedded files
2018-10-25 11:56:12 +02:00
513d897ea0
Implement get_meta() for archives
2018-10-25 11:29:50 +02:00
5a9dc388ad
Minor refactorisation of how we're checking for exiftool's presence
2018-10-25 11:05:06 +02:00
fe885babee
Implement lightweight cleaning for jpg
2018-10-24 19:35:07 +02:00
9a81b3adfd
Improve type annotation coverage
2018-10-23 16:32:28 +02:00
f1a071d460
Implement lightweight cleaning for png and tiff
2018-10-23 16:22:11 +02:00
38df679a88
Optimize the handling of problematic files
2018-10-23 13:49:58 +02:00
44f267a596
Improve problematic filenames support
2018-10-22 16:56:05 +02:00
83389a63e9
Test mat2's reliability wrt. corrupted video files
2018-10-22 13:42:04 +02:00
e70ea811c9
Implement support for .avi files, via ffmpeg
...
- This commit introduces optional dependencies (namely ffmpeg):
mat2 will spit a warning when trying to process an .avi file
if ffmpeg isn't installed.
- Since metadata are obtained via exiftool, this commit
also refactors a bit our exfitool wrapper.
2018-10-22 12:58:01 +02:00
2ba38dd2a1
Bump mypy typing coverage
2018-10-12 14:32:09 +02:00
b832a59414
Refactor lightweight mode implementation
2018-10-12 11:49:24 +02:00
b9dbd12ef9
Implement recursive metadata for FLAC files
...
Since FLAC files can contain covers, it makes sense
to parse their metadata
2018-10-11 19:52:47 +02:00
b2e153b69c
Delete pictures of FLAC files
2018-10-11 18:15:11 +02:00
0d25b18d26
Improve both the typing and the comments
2018-10-05 17:07:58 +02:00
d0f3534eff
Hide unsupported extensions in `mat2 -l`
2018-10-05 12:43:21 +02:00
8e98593b02
Trash word/people.xml in office files
2018-10-04 16:28:20 +02:00
34fbd633fd
libmat2: fix shebang
...
Relates 0a2a398c9c
2018-10-03 18:38:28 +00:00
5a5c642a46
Don't break office files for MS Office
...
We didn't take the whitelist into account while
removing dangling files from [Content_types].xml
2018-10-03 16:38:05 +02:00
1b356b8c6f
Improve mat2's cli reliability
...
- Replace some class members by instance members
- Don't thread the cleaning process anymore for now
2018-10-03 15:22:36 +02:00
c67bbafb2c
Use [Content_Types].xml to improve MS Office coverage
2018-10-02 11:55:42 -07:00
5b606f939d
fix typo
2018-10-02 16:01:24 +00:00
652b8e519f
Files processed via MAT2 are now accepted without warnings by MS Office
2018-10-01 12:25:37 -07:00
81a3881aa4
Please mypy
2018-09-30 19:55:17 +02:00
e342671ead
Remove dangling references in MS Office's [Content_types].xml
2018-09-30 19:53:18 +02:00
719cdf20fa
Second pass of minor formatting
2018-09-24 20:15:07 +02:00
2e243355f5
Fix some minor formatting issues
2018-09-24 19:50:24 +02:00
174d4a0ac0
Implement rsid stripping for office files
...
MS Office XML rsid is a "unique identifier used to track the editing session
when the physical character representing this section mark was last formatted."
See the following links for details:
- https://msdn.microsoft.com/en-us/library/office/documentformat.openxml.wordprocessing.previoussectionproperties.rsidrpr.aspx
- https://blogs.msdn.microsoft.com/brian_jones/2006/12/11/whats-up-with-all-those-rsids/ .
2018-09-24 18:03:59 +02:00
fbcf68c280
Lexicographical sort on xml attributes for office files
...
In XML, the order of the attributes shouldn't be meaningful,
however, MS Office sorts attributes for a given XML tag
differently than LibreOffice.
2018-09-24 17:45:09 +02:00
a1a06d023e
Insert archive members in lexicographic order
2018-09-18 22:44:21 +02:00
5cf94bd256
Bump coverage back to 100%
2018-09-12 14:54:54 +02:00
de65f4f4d4
Improve the resilience of MAT2 wrt. corrupted PNG
2018-09-09 19:09:05 +02:00
9fe6f1023b
Make pylint happy
2018-09-06 11:36:04 +02:00
e3d817f57e
Split office and archives
2018-09-06 11:34:14 +02:00
120b204988
Change a bit the previous commit
2018-09-06 11:13:11 +02:00
f3cef319b9
Unknown Members: make policy use an Enum
...
Closes #60
Note: this changeset also ensures that clean.cleaned.docx is removed
up after the pytest is over.
2018-09-05 18:59:33 -04:00
072ee1814d
Remove defusedxml support and document why
2018-09-05 18:41:08 +02:00
46bb1b83ea
Improve the previous commit
2018-09-05 17:26:09 +02:00
1d7e374e5b
office: try all members, even when one fails
...
the end result will be the same -- an abort -- but the user will get
to see all the warnings for a particular file, instead of getting them
one at a time.
2018-09-04 18:28:04 -04:00
915dc634c4
document all unknown/unhandlable files even on abort
...
This makes it easy to get a list of all files that mat2 doesn't know
how to handle, without having to choose -u keep or -u omit.
2018-09-04 18:28:04 -04:00
4192a2daa3
office: create policy for what to do about unknown members
...
previously, encountering an unknown member meant that any parser of
this type would abort.
now, the user can set parser.unknown_member_policy to either 'omit' or
'keep' if they don't want the current action of 'abort'
note that this causes pylint to complain about branching depth for
remove_all() because of the nuanced error-handling. I've disabled
this check.
2018-09-04 16:13:33 -04:00
907fc591cc
Bump the coverage back to 100%
2018-09-01 16:58:34 +02:00
3e2890eb9e
three minor spelling fixes
2018-09-01 06:47:22 -07:00
91e80527fc
Add archlinux to the CI
2018-09-01 15:41:22 +02:00
7877ba0da5
Fix a minor formatting issue
2018-09-01 14:16:55 +02:00
e2634f7a50
Logging cleanup
2018-09-01 05:14:32 -07:00
1c72448e58
Improve the detection of unsupported extensions in uppercase
2018-08-23 21:28:37 +02:00
f068621628
libmat2: images: fix handling of .JPG files
...
Pixbuf only supports .jpeg files, not .jpg, so libmat2 looks for such an
extension and converts it if necessary. As this check is case sensitive,
processing .JPG files does not work.
Fixes #47 .
Signed-off-by: Antoine Tenart <antoine.tenart@ack.tf>
2018-08-23 20:43:27 +02:00
71b1ced842
AbstractParser: Fix typos
2018-07-21 00:46:48 +00:00
942859601d
Improve the code's documentation
2018-07-19 23:10:27 +02:00
565cb66d14
Minor simplification in how we're handling xml for office files
2018-07-19 22:55:08 +02:00
84d50f97c0
Add a check for a missed dependency in `./mat2 -c`
2018-07-15 17:00:01 +02:00
5a7c7f35f7
Remove `print` from libmat, and use the `logging` module instead
...
This should close #28
2018-07-10 21:30:38 +02:00
d5861e4653
Implement a check for dependencies in mat2
...
Example use:
```
$ mat2 -c
Dependencies required for MAT2 0.1.3:
- Cairo: yes
- Exiftool: yes
- GdkPixbuf from PyGobject: yes
- Mutagen: yes
- Poppler from PyGobject: yes
- PyGobject: yes
```
This should close #35
2018-07-10 21:24:26 +02:00
080d6769ca
Make pylint even happier
2018-07-09 01:11:44 +02:00
8c21006e6c
Fix some pep8 issues spotted by pyflakes
2018-07-08 22:40:36 +02:00
f49aa5cab7
Achieve 100% coverage!
2018-07-08 22:27:37 +02:00
ad3e7ccee8
Bump coverage for office files and fix some related crashes
2018-07-08 21:35:45 +02:00
ca01484126
Silence a mypy's stupid warning
2018-07-08 17:12:17 +02:00
f9bc022c96
Add defusedxml as an (optional) way to prevent XML-based attacks
...
Those attacks are DoS-only.
2018-07-08 17:07:26 +02:00
72e1fda18d
Remove a leftover print
2018-07-08 15:19:18 +02:00
3cd4f9111f
Bump coverage for torrent handling
2018-07-08 15:13:03 +02:00
b5fcddd6a6
Simplify how torrent files are handled
...
- Rework the testsuite wrt. torrent
- fail at parser's instantiation on corrupted torrent,
instead of during `get_meta` or `remove_all` call
2018-07-08 13:49:11 +02:00
7ea362d908
Bump the coverage for pdf
2018-07-07 18:12:33 +02:00
85455a4419
Fix a mistake in office file revisions handling
2018-07-07 18:05:54 +02:00
3d80f97524
Simplify BMP handling
2018-07-06 00:49:17 +02:00
53271495f7
Add support for .txt files
2018-07-06 00:42:09 +02:00
893f58554a
Improve a bit the formatting of the code thanks to pyflakes3
2018-07-02 00:22:05 +02:00
bee56a57ce
Remove docx revisions
2018-07-01 23:16:14 +02:00
02f7605ac1
MAT2 is now cleaning revisions from odt files!
2018-07-01 21:09:20 +02:00
80fc4ffb40
Remove the thumbnails from libreoffice files
2018-07-01 17:29:05 +02:00
177184ac67
Massively simplify how we're cleaning office files
2018-06-27 21:48:46 +02:00
f44769df41
Ensure Poppler's minimal version
...
We're using methods that aren't available in Poppler
below 0.46, so we're checking for this upon import.
This commit is based on ideas from @LogicalDash ♥
2018-06-24 22:40:57 +02:00
74f2d50433
Split the testsuite a bit and add more tests
2018-06-22 21:16:55 +02:00
b4ef0c9622
Improve reliability against corrupted image files
2018-06-22 20:38:29 +02:00
5b38bd7ccd
Improve the reliability of the office parser
2018-06-21 23:18:59 +02:00
846a261465
Fix some linter warnings
2018-06-21 23:07:21 +02:00
09e748fa4c
Refactor how offices files are handled
...
- xml files are no longer considered harmless
- Factorization of the `remove_all` method for office files
- Explicit whitelist are used
- Blacklist are used to skip files completely
- Non-blacklisted files are _still cleaned_
- Unsupported files are still triggering an error
2018-06-21 23:02:41 +02:00
a89dae054a
Minor simplification of the office-related code
2018-06-21 21:24:53 +02:00
cce5de82e5
libmat2: harmless: add the text/xml mime type
...
Fedora defines the 'text/xml' mime type for xml files. Adds this mime
type to the harmless parser.
Fixes #36 .
Signed-off-by: Antoine Tenart <antoine.tenart@ack.tf>
2018-06-12 21:34:47 +02:00
484e26dd9c
libmat2: audio: add the audio/x-flac mime type
...
The FLAC parser looks for the 'audio/flac' mime type, but Fedora
defines 'audio/x-flac' in /etc/mime.types for FLAC files. Add this mime
type to the audio parser.
Fixes #36 .
Signed-off-by: Antoine Tenart <antoine.tenart@ack.tf>
2018-06-12 21:34:47 +02:00
545887af98
Minor code simplification
2018-06-10 20:20:32 +02:00
7dad77a785
Make the parsing of office format's metadata more robust
2018-06-10 20:20:00 +02:00
8c7979aae3
Add some tests for non-supported embedded fileformats
2018-06-10 20:19:35 +02:00
87bdcd1a95
Improve a bit our coverage wrt. torrent files handling
2018-06-10 00:56:55 +02:00
3c56fa3237
Improve a bit the performances wrt. image's metadata display
2018-06-10 00:43:38 +02:00
9c7aa34f50
Bump a bit the coverage
2018-06-10 00:43:25 +02:00
e81ce6cd1a
Fix and add a test for explicitly non-supported formats
2018-06-10 00:28:43 +02:00
633654376a
Improve a bit parsers autoloading
2018-06-10 00:28:26 +02:00
aa42b905d5
Speed up a bit the processing of get_meta for images with a "regular" name
2018-06-08 23:30:12 +02:00
e86e8e3c23
Improve the code to handle problematic filenames
2018-06-08 17:34:53 +02:00
6a832a4104
Prevent exiftool-based parameter-injection
2018-06-06 23:50:25 +02:00
8368de7fa7
Sort the output of `mat2 -l`
2018-06-04 23:32:13 +02:00
6a1b0b31f0
Add more typing and use mypy in the CI
2018-06-04 23:20:30 +02:00
4ebf9754f8
Import the dynamic import system
...
The dynamic import should now work when MAT2 is
installed system-wide, either via the distribution's
packaging system, or via pip.
2018-06-04 20:53:21 +02:00
d1392de6f5
Make pyflakes happier
2018-06-04 20:43:28 +02:00
183667a7f9
Improve a bit the typing, again
2018-06-04 20:39:27 +02:00
8143b63ee3
Improve a return type annotation
2018-06-04 20:29:41 +02:00
38fae60b8b
Rename some files to simplify packaging
...
- the `src` folder is now `libmat2`
- the `main.py` script is now `mat2.py`
2018-05-18 23:52:40 +02:00
12e2330ca6
Remove some useless files
2018-03-19 00:04:00 +01:00
df3c27d79d
Improve the testsuite
2018-03-18 21:42:12 +01:00
jvoisin