25123af3b8
436: Speed up the word prefix databases computation time r=Kerollmops a=Kerollmops This PR depends on the fixes done in #431 and must be merged after it. In this PR we will bring the `WordPrefixPairProximityDocids`, `WordPrefixDocids` and, `WordPrefixPositionDocids` update structures to a new era, a better era, where computing the word prefix pair proximities costs much fewer CPU cycles, an era where this update structure can use the, previously computed, set of new word docids from the newly indexed batch of documents. --- The `WordPrefixPairProximityDocids` is an update structure, which means that it is an object that we feed with some parameters and which modifies the LMDB database of an index when asked for. This structure specifically computes the list of word prefix pair proximities, which correspond to a list of pairs of words associated with a proximity (the distance between both words) where the second word is not a word but a prefix e.g. `s`, `se`, `a`. This word prefix pair proximity is associated with the list of documents ids which contains the pair of words and prefix at the given proximity. The origin of the performances issue that this struct brings is related to the fact that it starts its job from the beginning, it clears the LMDB database before rewriting everything from scratch, using the other LMDB databases to achieve that. I hope you understand that this is absolutely not an optimized way of doing things. Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Kerollmops <clement@meilisearch.com> |
||
|---|---|---|
| .github | ||
| benchmarks | ||
| cli | ||
| filter-parser | ||
| helpers | ||
| http-ui | ||
| infos | ||
| milli | ||
| script | ||
| .gitignore | ||
| .rustfmt.toml | ||
| bors.toml | ||
| Cargo.toml | ||
| LICENSE | ||
| README.md | ||
a concurrent indexer combined with fast and relevant search algorithms
Introduction
This repository contains the core engine used in Meilisearch.
It contains a library that can manage one and only one index. Meilisearch manages the multi-index itself. Milli is unable to store updates in a store: it is the job of something else above and this is why it is only able to process one update at a time.
This repository contains crates to quickly debug the engine:
- There are benchmarks located in the
benchmarkscrate. - The
http-uicrate is a simple HTTP dashboard to tests the features like for real! - The
infoscrate is used to dump the internal data-structure and ensure correctness. - The
searchcrate is a simple command-line that helps run flamegraph on top of it. - The
helperscrate is only used to modify the database inplace, sometimes.
Compile and run the HTTP debug server
You can specify the number of threads to use to index documents and many other settings too.
cd http-ui
cargo run --release -- --db my-database.mdb -vvv --indexing-jobs 8
Index your documents
It can index a massive amount of documents in not much time, I already achieved to index:
- 115m songs (song and artist name) in ~48min and take 81GiB on disk.
- 12m cities (name, timezone and country ID) in ~4min and take 6GiB on disk.
These metrics are done on a MacBook Pro with the M1 processor.
You can feed the engine with your CSV (comma-separated, yes) data like this:
printf "id,name,age\n1,hello,32\n2,kiki,24\n" | http POST 127.0.0.1:9700/documents content-type:text/csv
Don't forget to specify the id of the documents. Also, note that it supports JSON and JSON
streaming: you can send them to the engine by using the content-type:application/json and
content-type:application/x-ndjson headers respectively.
Querying the engine via the website
You can query the engine by going to the HTML page itself.
Contributing
You can setup a git-hook to stop you from making a commit too fast. It'll stop you if:
- Any of the workspaces does not build
- Your code is not well-formatted
These two things are also checked in the CI, so ignoring the hook won't help you merge your code.
But if you need to, you can still add --no-verify when creating your commit to ignore the hook.
To enable the hook, run the following command from the root of the project:
cp script/pre-commit .git/hooks/pre-commit