f25f454bd4
275: Fix the benchmarks dependencies r=Kerollmops a=irevoire Import exactly the same dependency as milli instead of a wildcard that can do anything Co-authored-by: Tamo <tamo@meilisearch.com> Co-authored-by: Irevoire <irevoire@protonmail.ch>
Benchmarks
TOC
Run the benchmarks
On our private server
The Meili team has self-hosted his own GitHub runner to run benchmarks on our dedicated bare metal server.
To trigger the benchmark workflow:
- Go to the
Actionstab of this repository. - Select the
Benchmarksworkflow on the left. - Click on
Run workflowin the blue banner. - Select the branch on which you want to run the benchmarks and select the dataset you want (default:
songs). - Finally, click on
Run workflow.
This GitHub workflow will run the benchmarks and push the critcmp report to a DigitalOcean Space (= S3).
The name of the uploaded file is displayed in the workflow.
💡 To compare the just-uploaded benchmark with another one, check out the next section.
On your machine
To run all the benchmarks (~4h):
cargo bench
To run only the songs (~1h) or wiki (~3h) benchmark:
cargo bench --bench <dataset name>
By default, the benchmarks will be downloaded and uncompressed automatically in the target directory.
If you don't want to download the datasets every time you update something on the code, you can specify a custom directory with the environment variable MILLI_BENCH_DATASETS_PATH:
mkdir ~/datasets
MILLI_BENCH_DATASETS_PATH=~/datasets cargo bench --bench songs # the two datasets are downloaded
touch build.rs
MILLI_BENCH_DATASETS_PATH=~/datasets cargo bench --bench songs # the code is compiled again but the datasets are not downloaded
Comparison between benchmarks
The benchmark reports we push are generated with critcmp. Thus, we use critcmp to generate comparison results between 2 benchmarks.
We provide a script to download and display the comparison report.
Requirements:
grepcurlcritcmp
List the available file in the DO Space:
./benchmarks/script/list.sh
songs_main_09a4321.json
songs_geosearch_24ec456.json
Run the comparison script:
./benchmarks/scripts/compare.sh songs_main_09a4321.json songs_geosearch_24ec456.json
Datasets
The benchmarks are available for the following datasets:
songswiki
Songs
songs is a subset of the songs.csv dataset.
It was generated with this command:
xsv sample --seed 42 1000000 songs.csv -o smol-songs.csv
Download the generated songs dataset.
Wiki
wiki is a subset of the wikipedia-articles.csv dataset.
It was generated with the following command:
xsv sample --seed 42 500000 wiki-articles.csv -o smol-wiki-articles.csv