MeiliSearch/crates
meili-bors[bot] fb1caa4724
Merge #5148
5148: Do not duplicate NDJson data when unecessary r=dureuill a=Kerollmops

This PR improves the NDJSON support. Usually, we save all of the user's document content into a temporary file, validate its content, and then convert everything into NDJSON in the file store (update files in the tasks).

It is a waste of time when users are already sending NDJSON. So, this PR removes the last copy and directly stores the user content in the file store, validating it from the file store. If an issue arises, the file will not persist and will be dropped/deleted instead.

Related to #5078.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-11 13:00:50 +00:00
..
benchmarks remove mimalloc on Windows 2024-12-02 18:13:56 +01:00
build-info Move crates under a sub folder to clean up the code 2024-10-21 08:18:43 +02:00
dump Clean up dependencies 2024-11-27 14:30:34 +01:00
file-store Do not duplicate NDJson when unecessary 2024-12-11 12:02:48 +01:00
filter-parser Move crates under a sub folder to clean up the code 2024-10-21 08:18:43 +02:00
flatten-serde-json Move crates under a sub folder to clean up the code 2024-10-21 08:18:43 +02:00
fuzzers Plug the NoPanicThreadPool in the tests and benchmarks 2024-11-27 17:04:49 +01:00
index-scheduler Reintroduce the document addition logs 2024-12-11 10:39:04 +01:00
json-depth-checker Move crates under a sub folder to clean up the code 2024-10-21 08:18:43 +02:00
meili-snap Add comment explaining why we fixed the version of insta 2024-11-21 16:56:56 +01:00
meilisearch Wrap the read NDJSON pass into a tokio blocking 2024-12-11 12:18:36 +01:00
meilisearch-auth Clean up dependencies 2024-11-27 14:30:34 +01:00
meilisearch-types Use a nonrandom hasher when decoding JSON 2024-12-11 12:04:29 +01:00
meilitool Merge branch 'main' into indexer-edition-2024 2024-11-20 16:59:58 +01:00
milli Make sure we use an FxHashBuilder on the Value 2024-12-10 15:52:22 +01:00
permissive-json-pointer Add indices field to _matchesPosition to specify where in an array a match comes from (#5005) 2024-11-20 01:00:43 +01:00
tracing-trace Remove orphan span 2024-11-21 12:12:07 +01:00
xtask Allow xtask bench to proceed without a commit message 2024-12-09 09:36:59 +01:00