Commit Graph

137 Commits

Author SHA1 Message Date
Clémentine Urquizar
d5e9b7305b
Update version for next release (v0.32.0) 2022-07-21 13:20:02 +04:00
bors[bot]
941af58239
Merge #561
561: Enriched documents batch reader r=curquiza a=Kerollmops

~This PR is based on #555 and must be rebased on main after it has been merged to ease the review.~
This PR contains the work in #555 and can be merged on main as soon as reviewed and approved.

- [x] Create an `EnrichedDocumentsBatchReader` that contains the external documents id.
- [x] Extract the primary key name and make it accessible in the `EnrichedDocumentsBatchReader`.
- [x] Use the external id from the `EnrichedDocumentsBatchReader` in the `Transform::read_documents`.
- [x] Remove the `update_primary_key` from the _transform.rs_ file.
- [x] Really generate the auto-generated documents ids.
- [x] Insert the (auto-generated) document ids in the document while processing it in `Transform::read_documents`.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-07-21 07:08:50 +00:00
Loïc Lecrenier
8270e2b768 Fix name of "release_date" facet in movies benchmarks 2022-07-18 10:34:12 +02:00
Kerollmops
448114cc1c
Fix the benchmarks with the new indexation API 2022-07-12 15:22:09 +02:00
Kerollmops
a892a4a79c
Introduce a function to extend from a JSON array of objects 2022-07-12 15:14:06 +02:00
Kerollmops
ea852200bb
Fix the format used for a geo deleting benchmark 2022-07-12 14:55:52 +02:00
Kerollmops
399eec5c01
Fix the indexation tests 2022-07-12 14:55:51 +02:00
Kerollmops
fcfc4caf8c
Move the Object type in the lib.rs file and use it everywhere 2022-07-12 14:55:51 +02:00
Kerollmops
a97d4d63b9
Fix the benchmarks 2022-07-12 14:55:50 +02:00
Loïc Lecrenier
aae03356cb Use BufReader to read datasets in benchmarks 2022-07-06 18:20:15 +02:00
bors[bot]
eeba196053
Merge #572
572: Add reindexing benchmarks r=Kerollmops a=irevoire

With #557 coming, we should add benchmarks that measure our impact on the reindexing process.

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-07-05 14:43:01 +00:00
Loïc Lecrenier
9bc7627e27 Fix deserialisation of NDJson documents in benchmarks 2022-07-05 15:51:06 +02:00
Tamo
2700d8dc67
Add reindexing benchmarks 2022-07-05 14:46:46 +02:00
Kerollmops
cc48992e79
Bump the milli version to 0.31.1 2022-06-22 17:05:51 +02:00
Kerollmops
f5c3b951bc
Bump the milli version to 0.31.0 2022-06-22 12:08:16 +02:00
Clémentine Urquizar
31f749b5d8
Update version for next release (v0.30.0) 2022-06-20 12:09:57 +02:00
ManyTheFish
2652310f2a Change delete benchmark names 2022-06-16 10:32:58 +02:00
ManyTheFish
adbb0ff318 Add deletion benchmarks 2022-06-16 10:17:58 +02:00
Tamo
676187ba43
bump milli version 2022-06-09 16:53:32 +02:00
Kerollmops
56ee9cc21f
Bump the version to 0.29.2 2022-06-08 16:00:06 +02:00
Clémentine Urquizar
478dbfa45a
Update version for next release (v0.29.1) 2022-06-07 18:59:33 +02:00
bors[bot]
05ae6dbfa4
Merge #541
541: Update version for next release (v0.29.0) r=ManyTheFish a=curquiza

Need to update the version since #540 was merged and breaking

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2022-06-02 16:53:28 +00:00
Clémentine Urquizar
6ce1c6487a
Update version for next release (v0.29.0) 2022-06-02 18:07:55 +02:00
ManyTheFish
727d663f28 Update benchmarks 2022-06-02 18:07:10 +02:00
ManyTheFish
4dd7b20c32 Update benchmarks 2022-06-02 17:33:25 +02:00
Clémentine Urquizar
c19c17eddb
Update version to v0.28.1 2022-06-01 18:31:02 +02:00
ManyTheFish
895f5d8a26 Bump milli version 2022-05-18 10:37:12 +02:00
Tamo
f820c9804d
add one nested benchmark 2022-05-02 19:35:57 +02:00
Clémentine Urquizar
d138b3c704
Update version 2022-04-25 18:43:46 +02:00
Tamo
152a10344c
Get rid of the threshold when comparing benchmarks
It just hide things
2022-04-19 15:39:58 +02:00
bors[bot]
ea4bb9402f
Merge #483
483: Enhance matching words r=Kerollmops a=ManyTheFish

# Summary

Enhance milli word-matcher making it handle match computing and cropping.

# Implementation

## Computing best matches for cropping

Before we were considering that the first match of the attribute was the best one, this was accurate when only one word was searched but was missing the target when more than one word was searched.

Now we are searching for the best matches interval to crop around, the chosen interval is the one:
1) that have the highest count of unique matches
> for example, if we have a query `split the world`, then the interval `the split the split the` has 5 matches but only 2 unique matches (1 for `split` and 1 for `the`) where the interval `split of the world` has 3 matches and 3 unique matches. So the interval `split of the world` is considered better.
2) that have the minimum distance between matches
> for example, if we have a query `split the world`, then the interval `split of the world` has a distance of 3 (2 between `split` and `the`, and 1 between `the` and `world`) where the interval `split the world` has a distance of 2. So the interval `split the world` is considered better.
3) that have the highest count of ordered matches
> for example, if we have a query `split the world`, then the interval `the world split` has 2 ordered words where the interval `split the world` has 3. So the interval `split the world` is considered better.

## Cropping around the best matches interval

Before we were cropping around the interval without checking the context.

Now we are cropping around words in the same context as matching words.
This means that we will keep words that are farther from the matching words but are in the same phrase, than words that are nearer but separated by a dot.

> For instance, for the matching word `Split` the text:
`Natalie risk her future. Split The World is a book written by Emily Henry. I never read it.`
will be cropped like:
`…. Split The World is a book written by Emily Henry. …`
and  not like:
`Natalie risk her future. Split The World is a book …`


Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-04-19 11:42:32 +00:00
ManyTheFish
5809d3ae0d Add first benchmarks on formatting 2022-04-12 16:31:58 +02:00
Irevoire
4f3ce6d9cd
nested fields 2022-04-07 16:58:46 +02:00
Kerollmops
63682c2c9a
Upgrade the dependencies 2022-03-15 11:17:44 +01:00
Clémentine Urquizar
d9ed9de2b0
Update heed link in cargo toml 2022-03-01 19:45:29 +01:00
Clément Renault
acfc96525c
Apply GitHub suggestions 2022-02-23 16:20:29 +01:00
Clément Renault
a820aa11e6
Add a new movies benchmark to test multi batch indexing 2022-02-23 16:20:29 +01:00
Kerollmops
8d2e3e4aba
Add a new wiki benchmark to test multi batch indexing 2022-02-23 16:20:29 +01:00
Kerollmops
ab5247dc64
Add a new songs benchmark to test multi batch indexing 2022-02-23 16:20:28 +01:00
Marin Postma
0c84a40298 document batch support
reusable transform

rework update api

add indexer config

fix tests

review changes

Co-authored-by: Clément Renault <clement@meilisearch.com>

fmt
2022-01-19 12:40:20 +01:00
Clément Renault
94011bb9a8
Fix the benchmarks to work with optional filters 2021-12-09 12:14:16 +01:00
Marin Postma
6eb47ab792 remove update_id in UpdateBuilder 2021-11-16 13:07:04 +01:00
Tamo
6831c23449
merge with main 2021-11-06 16:34:30 +01:00
Tamo
5c01e9bf7c
fix the benchmarks 2021-11-06 16:03:49 +01:00
marin postma
baddd80069
implement review suggestions 2021-10-25 18:29:12 +02:00
Tamo
ed7fd855af
fix the wiki indexing benchmark 2021-10-11 14:26:36 +02:00
many
d2427f18e5
Enhance CSV document parsing 2021-09-29 10:25:33 +02:00
Tamo
176160d32f
fix all benchmarks and add the compile time checking of the benhcmarks in the ci 2021-09-22 12:10:21 +02:00
bors[bot]
700318dc62
Merge #357
357: Add benchmarks for the geosearch r=Kerollmops a=irevoire

closes #336

Should I merge this PR in #322 and then we merge everything in `main` or should we wait for #322 to be merged and then merge this one in `main` later?

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-09-21 16:08:06 +00:00
Tamo
eaba772f21
update the README to better match the new critcmp usage
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-09-20 10:59:55 +02:00
Irevoire
9a920d1f93
Fix datasets links in the readme
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-09-20 10:44:37 +02:00
Tamo
5e683ba472
add benchmarks for the geosearch 2021-09-20 10:44:37 +02:00
Irevoire
f6c6b026bb
improve the comparison script 2021-09-16 11:25:51 +02:00
Irevoire
4b99d8cb91
rewrite the indexing benchmarks 2021-08-19 15:02:43 +02:00
Kerollmops
5b88df508e
Use the new Asc/Desc syntax everywhere 2021-08-17 14:15:22 +02:00
Tamo
7eb2d71009
fix the benchmarks 2021-07-29 16:27:05 +02:00
Tamo
976dc1f4bc
prefix the search benchmarks with 'search' 2021-07-29 16:27:05 +02:00
Tamo
931021fe57
add benchmarks for indexing 2021-07-07 13:09:05 +02:00
Kerollmops
91c5d0c042
Use the AlwaysFreePages flag when opening an index 2021-07-05 16:36:13 +02:00
bors[bot]
f25f454bd4
Merge #275
275: Fix the benchmarks dependencies r=Kerollmops a=irevoire

Import exactly the same dependency as milli instead of a wildcard that can do anything

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <irevoire@protonmail.ch>
2021-07-01 11:07:01 +00:00
Irevoire
ec87bf3dd5
Update benchmarks/Cargo.toml
Co-authored-by: Clément Renault <renault.cle@gmail.com>
2021-07-01 11:45:05 +02:00
Tamo
ef965aa3f3
fix the fmt of the auto-generated file 2021-07-01 11:43:09 +02:00
Tamo
fc09d77e89
fix the benchmarks dependcies 2021-07-01 11:38:30 +02:00
Kerollmops
98285b4b18
Bump milli to 0.6.0 2021-06-23 17:30:26 +02:00
Tamo
77eb37934f
add jemalloc to http-ui and the benchmarks 2021-06-22 14:17:56 +02:00
Tamo
9716fb3b36
format the whole project 2021-06-16 18:33:33 +02:00
Kerollmops
713acc408b
Introduce the primary key to the Settings builder structure 2021-06-16 11:03:36 +02:00
Clémentine Urquizar
7e93811fbc
Update dataset links 2021-06-08 18:18:54 +02:00
Irevoire
d912c94034
improve the benchmark’s readme 2021-06-08 10:38:23 +02:00
Irevoire
563492f1e5
update the TOC order 2021-06-07 17:29:22 +02:00
Kerollmops
29824d05ab
Reduce the length of the benchmarks names 2021-06-03 15:59:43 +02:00
Kerollmops
76a2343639
Fix the compare script of the benchmarks 2021-06-03 15:39:52 +02:00
Kerollmops
82fb5f0bef
Fix the benchmarks compilation 2021-06-03 10:33:42 +02:00
Tamo
6dc08bf45e
remove the nop function 2021-06-02 17:09:21 +02:00
Tamo
087ae64899
add a gitignore to avoid pushing the autogenerated file 2021-06-02 17:03:30 +02:00
Tamo
3db25153e5
fix the faceted_fields one last time 2021-06-02 17:00:58 +02:00
Irevoire
f346805c0c
Update benchmarks/Cargo.toml
Co-authored-by: Clément Renault <clement@meilisearch.com>
2021-06-02 15:47:03 +02:00
Clémentine Urquizar
ef1ac8a0cb
Update README 2021-06-02 11:13:22 +02:00
Clémentine Urquizar
edfcdb171c
Update benchmarks/scripts/list.sh
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-06-02 11:13:22 +02:00
Clémentine Urquizar
3c91a9a551
Update following reviews 2021-06-02 11:13:22 +02:00
Tamo
bc4f4ee829
remove s3cmd as a dependency and provide a script to list all the available benchmarks 2021-06-02 11:13:22 +02:00
Clémentine Urquizar
61fe422a88
Update benchmarks/scripts/compare.sh
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-06-02 11:13:22 +02:00
Clémentine Urquizar
57ed96622b
Update benchmarks/scripts/compare.sh
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-06-02 11:13:22 +02:00
Clémentine Urquizar
b3c0d43890
Update benchmarks/scripts/compare.sh
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-06-02 11:13:22 +02:00
Clémentine Urquizar
0d0e900158
Add CI for benchmarks 2021-06-02 11:13:22 +02:00
tamo
4536dfccd0
add a way to provide primary_key or autogenerate documents ids 2021-06-02 11:13:20 +02:00
tamo
06c414a753
move the benchmarks to another crate so we can download the datasets automatically without adding overhead to the build of milli 2021-06-02 11:11:50 +02:00