mirror of
https://github.com/meilisearch/MeiliSearch
synced 2024-12-12 22:36:29 +01:00
cac355bfa7
5124: Optimize Prefixes and Merges r=ManyTheFish a=Kerollmops In this PR, we plan to optimize the read of LMDB to use read the entries in lexicographic order and better use the memory-mapping OS cache: - Optimize the prefix generation for word position docids (`@manythefish)` - Optimize the parallel merging of the caches to sort entries before merging the caches (`@kerollmops)` ## Benchmarks on 1cpu 2gb gpo3 (5k IOps) Before on the tag meilisearch-v1.12.0-rc.3. ``` word_position_docids:merge_and_send_docids: 988s compute_word_fst: 23.3s word_pair_proximity_docids:merge_and_send_docids: 428s compute_word_prefix_fid_docids:recompute_modified_prefixes: 76.3s compute_word_prefix_position_docids:recompute_modified_prefixes:from_prefixes: 429s ``` After sorting the whole `HashMap`s in a `Vec` on this branch. ``` word_position_docids:merge_and_send_docids: 202s compute_word_fst: 20.4s word_pair_proximity_docids:merge_and_send_docids: 427s compute_word_prefix_fid_docids:recompute_modified_prefixes: 65.5s compute_word_prefix_position_docids:recompute_modified_prefixes:from_prefixes: 62.5s ``` Co-authored-by: ManyTheFish <many@meilisearch.com> Co-authored-by: Kerollmops <clement@meilisearch.com> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| build-info | ||
| dump | ||
| file-store | ||
| filter-parser | ||
| flatten-serde-json | ||
| fuzzers | ||
| index-scheduler | ||
| json-depth-checker | ||
| meili-snap | ||
| meilisearch | ||
| meilisearch-auth | ||
| meilisearch-types | ||
| meilitool | ||
| milli | ||
| permissive-json-pointer | ||
| tracing-trace | ||
| xtask | ||