32 Commits

Author SHA1 Message Date
Tamo
f820c9804d
add one nested benchmark 2022-05-02 19:35:57 +02:00
bors[bot]
ea4bb9402f
Merge #483
483: Enhance matching words r=Kerollmops a=ManyTheFish

# Summary

Enhance milli word-matcher making it handle match computing and cropping.

# Implementation

## Computing best matches for cropping

Before we were considering that the first match of the attribute was the best one, this was accurate when only one word was searched but was missing the target when more than one word was searched.

Now we are searching for the best matches interval to crop around, the chosen interval is the one:
1) that have the highest count of unique matches
> for example, if we have a query `split the world`, then the interval `the split the split the` has 5 matches but only 2 unique matches (1 for `split` and 1 for `the`) where the interval `split of the world` has 3 matches and 3 unique matches. So the interval `split of the world` is considered better.
2) that have the minimum distance between matches
> for example, if we have a query `split the world`, then the interval `split of the world` has a distance of 3 (2 between `split` and `the`, and 1 between `the` and `world`) where the interval `split the world` has a distance of 2. So the interval `split the world` is considered better.
3) that have the highest count of ordered matches
> for example, if we have a query `split the world`, then the interval `the world split` has 2 ordered words where the interval `split the world` has 3. So the interval `split the world` is considered better.

## Cropping around the best matches interval

Before we were cropping around the interval without checking the context.

Now we are cropping around words in the same context as matching words.
This means that we will keep words that are farther from the matching words but are in the same phrase, than words that are nearer but separated by a dot.

> For instance, for the matching word `Split` the text:
`Natalie risk her future. Split The World is a book written by Emily Henry. I never read it.`
will be cropped like:
`…. Split The World is a book written by Emily Henry. …`
and  not like:
`Natalie risk her future. Split The World is a book …`


Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-04-19 11:42:32 +00:00
ManyTheFish
5809d3ae0d Add first benchmarks on formatting 2022-04-12 16:31:58 +02:00
Irevoire
4f3ce6d9cd
nested fields 2022-04-07 16:58:46 +02:00
Clément Renault
acfc96525c
Apply GitHub suggestions 2022-02-23 16:20:29 +01:00
Clément Renault
a820aa11e6
Add a new movies benchmark to test multi batch indexing 2022-02-23 16:20:29 +01:00
Kerollmops
8d2e3e4aba
Add a new wiki benchmark to test multi batch indexing 2022-02-23 16:20:29 +01:00
Kerollmops
ab5247dc64
Add a new songs benchmark to test multi batch indexing 2022-02-23 16:20:28 +01:00
Marin Postma
0c84a40298 document batch support
reusable transform

rework update api

add indexer config

fix tests

review changes

Co-authored-by: Clément Renault <clement@meilisearch.com>

fmt
2022-01-19 12:40:20 +01:00
Clément Renault
94011bb9a8
Fix the benchmarks to work with optional filters 2021-12-09 12:14:16 +01:00
Marin Postma
6eb47ab792 remove update_id in UpdateBuilder 2021-11-16 13:07:04 +01:00
Tamo
6831c23449
merge with main 2021-11-06 16:34:30 +01:00
Tamo
5c01e9bf7c
fix the benchmarks 2021-11-06 16:03:49 +01:00
marin postma
baddd80069
implement review suggestions 2021-10-25 18:29:12 +02:00
Tamo
ed7fd855af
fix the wiki indexing benchmark 2021-10-11 14:26:36 +02:00
many
d2427f18e5
Enhance CSV document parsing 2021-09-29 10:25:33 +02:00
Tamo
176160d32f
fix all benchmarks and add the compile time checking of the benhcmarks in the ci 2021-09-22 12:10:21 +02:00
Tamo
5e683ba472
add benchmarks for the geosearch 2021-09-20 10:44:37 +02:00
Irevoire
4b99d8cb91
rewrite the indexing benchmarks 2021-08-19 15:02:43 +02:00
Kerollmops
5b88df508e
Use the new Asc/Desc syntax everywhere 2021-08-17 14:15:22 +02:00
Tamo
7eb2d71009
fix the benchmarks 2021-07-29 16:27:05 +02:00
Tamo
976dc1f4bc
prefix the search benchmarks with 'search' 2021-07-29 16:27:05 +02:00
Tamo
931021fe57
add benchmarks for indexing 2021-07-07 13:09:05 +02:00
Tamo
77eb37934f
add jemalloc to http-ui and the benchmarks 2021-06-22 14:17:56 +02:00
Tamo
9716fb3b36
format the whole project 2021-06-16 18:33:33 +02:00
Kerollmops
713acc408b
Introduce the primary key to the Settings builder structure 2021-06-16 11:03:36 +02:00
Kerollmops
29824d05ab
Reduce the length of the benchmarks names 2021-06-03 15:59:43 +02:00
Kerollmops
82fb5f0bef
Fix the benchmarks compilation 2021-06-03 10:33:42 +02:00
Tamo
6dc08bf45e
remove the nop function 2021-06-02 17:09:21 +02:00
Tamo
3db25153e5
fix the faceted_fields one last time 2021-06-02 17:00:58 +02:00
tamo
4536dfccd0
add a way to provide primary_key or autogenerate documents ids 2021-06-02 11:13:20 +02:00
tamo
06c414a753
move the benchmarks to another crate so we can download the datasets automatically without adding overhead to the build of milli 2021-06-02 11:11:50 +02:00