Tamo
d1db495119
add a settings for the search cutoff
2024-03-19 10:28:23 +01:00
meili-bors[bot]
abd954755d
Merge #4476
...
4476: Make the `/facet-search` route use the `sortFacetValuesBy` setting r=irevoire a=Kerollmops
This PR fixes #4423 by ensuring that the `/facet-search` route uses the `sortFacetValuesBy` setting.
Note for the documentation team (to be moved in the tracking issue): Using the new `sortFacetValuesBy` setting can slow down the facet-search requests as Meilisearch iterates over the whole list of facet values and computes the count of documents on every entry. That is hardly or even impossible to optimize correctly.
### TODO
- [x] Create a custom HashMap wrapper for the facet `OrderBy` settings.
This wrapper will return the `OrderBy` setting of the facet, if not defined will use the default `*` one, and if not there either (strange) will fall back on the lexicographic one.
- [x] Create a `ValuesCollection` wrapper that implements the logic for the lexicographic and count order by.
- [x] Use it when there is no search query.
- [x] Use it when there is a search query with and without allowed typos.
- [x] Do not change the original logic, only use a wrapper.
- [x] Add tests
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-03-13 14:36:14 +00:00
Clément Renault
6c9823d7bb
Add tests to sortFacetValuesBy count
2024-03-13 11:59:39 +01:00
Tamo
8ec3e30d2b
Merge branch 'main' into tmp-release-v1.7.0
2024-03-11 15:39:51 +01:00
Tamo
f053c280e1
add tests when the field limit is reached
2024-03-06 18:42:41 +01:00
Tamo
b130917933
add the content type in the webhook + improve the test
2024-03-05 11:22:29 +01:00
meili-bors[bot]
b87485e80d
Merge #4433
...
4433: Enhance facet incremental r=Kerollmops a=ManyTheFish
# Pull Request
## Related issue
Fixes #4367
Fixes #4409
## What does this PR do?
- Add a test reproducing #4409
- Fix #4409 by removing a document from a level only if it is no more present in all the linked sub-level nodes
- Optimize facet Incremental indexing by creating or deleting a complete level once per field id instead of for each facet value
- Optimize facet Incremental indexing by doing the additions and the deletions in the same process instead of doing them separately
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-02-28 15:28:46 +00:00
Tamo
a478392b7a
create a test with the dry-run parameter enabled
2024-02-26 13:59:41 +01:00
Tamo
bbf3fb88ca
rename the cli parameter
2024-02-26 13:59:40 +01:00
Tamo
507739bd98
add an experimental cli parameter to allow specifying your task id
2024-02-26 13:58:03 +01:00
Tamo
eb25b07390
let you specify your task id
2024-02-26 13:56:31 +01:00
Tamo
c2e2003a80
create a test with the dry-run parameter enabled
2024-02-22 15:51:47 +01:00
Tamo
693ba8dd15
rename the cli parameter
2024-02-21 14:33:40 +01:00
Tamo
01ae46dd80
add an experimental cli parameter to allow specifying your task id
2024-02-20 11:24:44 +01:00
Tamo
9ee4f55e6c
let you specify your task id
2024-02-19 14:29:33 +01:00
ManyTheFish
865b415b3f
Add test rerpoducing bug
2024-02-15 16:00:48 +01:00
Tamo
a081da0d90
add support for the json format in the stream route
2024-02-14 15:34:39 +01:00
Tamo
3b6544db6d
Implement the experimental log mode cli flag
2024-02-13 18:09:15 +01:00
Tamo
2c88131bb1
rename the fmt mode to human
2024-02-08 15:04:06 +01:00
Tamo
35aa9d5904
fix an error message
2024-02-08 15:04:06 +01:00
Louis Dureuil
ef994d84d0
Change error messages and fix tests
2024-02-08 15:04:06 +01:00
Louis Dureuil
1b74010e9e
Remove "with_line_numbers"
2024-02-08 15:04:06 +01:00
Tamo
bcf7909bba
add a profile_memory parameter disabled by default
2024-02-08 15:04:05 +01:00
Tamo
ceb211c515
move the /logs route to the /logs/stream route
2024-02-08 15:04:05 +01:00
Tamo
f158e96fe7
fix the auth
2024-02-08 15:04:05 +01:00
Tamo
e23ec4886d
fix the tests and add tests on the experimental features
2024-02-08 15:04:03 +01:00
Tamo
80774148fd
handle and tests errors
2024-02-08 15:03:33 +01:00
Tamo
bf5cea8b10
add a test
2024-02-08 15:03:33 +01:00
Tamo
77254765e8
get rids of env loggegr and fix the tests
2024-02-08 15:03:33 +01:00
Louis Dureuil
05edd85d75
Stabilize scoreDetails
2024-02-06 11:15:19 +01:00
Tamo
318843aacd
add a bunch of tests and fix the error message when adding the geosearch as filterable/sortable while there is malformed documents in the DB
2024-02-01 10:33:27 +01:00
Louis Dureuil
6d111139b5
Add test
2024-02-01 10:33:27 +01:00
Tamo
c1bf33a112
Revert "Remove panic on the geosearch"
2024-01-25 18:51:19 +01:00
Tamo
7d190d8078
add a bunch of tests and fix the error message when adding the geosearch as filterable/sortable while there is malformed documents in the DB
2024-01-17 15:51:52 +01:00
Louis Dureuil
38abfec611
Fix tests
2024-01-11 21:35:30 +01:00
Clément Renault
3f3462ab62
Limit the number of values returned by the facet search
2024-01-10 16:54:08 +01:00
ManyTheFish
f75f22e026
Display default value when proximityPrecision is not set
2024-01-08 11:09:37 +01:00
Louis Dureuil
94b9f3b310
Add test
2024-01-03 15:56:20 +01:00
meili-bors[bot]
43e822e802
Merge #4238
...
4238: Task queue webhook r=dureuill a=irevoire
# Prototype `prototype-task-queue-webhook-1`
The prototype is available through Docker by using the following command:
```bash
docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-task-queue-webhook-1
```
# Pull Request
Implements the task queue webhook.
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4236
## What does this PR do?
- Provide a new cli and env var for the webhook, respectively called `--task-webhook-url` and `MEILI_TASK_WEBHOOK_URL`
- Also supports sending the requests with a custom `Authorization` header by specifying the optional `--task-webhook-authorization-header` CLI parameter or `MEILI_TASK_WEBHOOK_AUTHORIZATION_HEADER` env variable.
- Throw an error if the specified URL is invalid
- Every time a batch is processed, send all the finished tasks into the webhook with our public `TaskView` type as a JSON Line GZIPed body.
- Add one test.
## PR checklist
### Before becoming ready to review
- [x] Add a test
- [x] Compress the data we send
- [x] Chunk and stream the data we send
- [x] Remove the unwrap in the index-scheduler when sending the data fails
- [x] The analytics are missing
### Before merging
- [x] Release a prototype
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-12-21 14:43:46 +00:00
Louis Dureuil
6ff81de401
Fix tests
2023-12-20 17:16:46 +01:00
Tamo
c83a33017e
stream and chunk the data
2023-12-19 10:35:51 +01:00
Tamo
be72326c0a
gzip the tasks
2023-12-19 10:35:51 +01:00
Tamo
0b2fff27f2
update and fix the test
2023-12-19 10:35:51 +01:00
Tamo
3adbc2b942
return a task view instead of a task
2023-12-19 10:35:51 +01:00
Tamo
fbea721378
add a first working test with actixweb
2023-12-19 10:35:51 +01:00
Tamo
391eb72137
start writing a test with actix but it doesn't works
2023-12-19 10:35:50 +01:00
ManyTheFish
3a99a555a2
Fix experimental features snapshots in tests
2023-12-18 10:05:51 +01:00
Many the fish
9e1b458010
Merge branch 'main' into change-proximity-precision-settings
2023-12-18 09:08:47 +01:00
ManyTheFish
6425996e36
Change the naming of attributeScale and wordScale into byAttribute and byWord
2023-12-14 16:31:00 +01:00
Louis Dureuil
87bba98bd8
Various changes
...
- fixed seed for arroy
- check vector dimensions as soon as it is provided to search
- don't embed whitespace
2023-12-14 16:08:42 +01:00
Louis Dureuil
217105b7da
hybrid search uses semantic ratio, error handling
2023-12-14 16:08:42 +01:00
ManyTheFish
f3f3944469
Fix error checking
2023-12-14 16:08:42 +01:00
ManyTheFish
ac68f33194
Add simple test
2023-12-14 16:08:42 +01:00
Louis Dureuil
806e5b6899
Tests pass
2023-12-14 16:08:41 +01:00
Louis Dureuil
12940d79a9
WIP
...
- manual embedder
- multi embedders OK
- clippy + tests OK
2023-12-14 16:08:41 +01:00
ManyTheFish
35e1981488
Remove proximityPrecision form the experimental feature
2023-12-14 15:52:42 +01:00
Clément Renault
56571f762a
Merge remote-tracking branch 'origin/main' into tmp-release-v1.5.1
2023-12-13 11:57:01 +01:00
ManyTheFish
c9860c7913
Small test fixes
2023-12-06 15:49:05 +01:00
ManyTheFish
03ffabe889
Add a new dump test
2023-12-06 15:49:05 +01:00
ManyTheFish
1f4fc9c229
Make the feature experimental
2023-12-06 15:49:05 +01:00
ManyTheFish
8cc3c54117
Add proximityPrecision setting in settings route
2023-12-06 15:49:05 +01:00
ManyTheFish
0c3fa8cbc4
Add tests on proximityPrecision setting
2023-12-06 14:59:23 +01:00
ManyTheFish
dc07790133
Add test reproducing #4232
2023-11-27 11:39:11 +01:00
Clément Renault
0dbf1a16ff
Make clippy happy
2023-11-23 14:11:38 +01:00
Clément Renault
7cb7e37ba8
Merge branch 'main' into tmp-release-v1.5.0
2023-11-21 16:30:46 +01:00
Louis Dureuil
9fedd8101a
Fix tests
2023-10-30 15:11:07 +01:00
Tamo
e7244aa485
fix warnings
2023-10-30 11:00:46 +01:00
Vivek Kumar
32c78ac8b1
add/update tests when search with distinct attribute & pagination with no ranking
2023-10-23 12:06:27 +02:00
meili-bors[bot]
eae9eab181
Merge #4126
...
4126: Make the experimental route /metrics activable via HTTP r=dureuill a=braddotcoffee
# Pull Request
## Related issue
Closes #4086
## What does this PR do?
- [x] Make `/metrics` available via HTTP as described in #4086
- [x] The users can still launch Meilisearch using the `--experimental-enable-metrics` flag.
- [x] If the flag `--experimental-enable-metrics` is activated, a call to the `GET /experimental-features` route right after the launch will show `"metrics": true` even if the user has not called the `PATCH /experimental-features` route yet.
- [x] Even if the --experimental-enable-metrics flag is present at launch, calling the `PATCH /experimental-features` route with `"metrics": false` disables the experimental feature.
- [x] Update the spec
- I was unable to find docs in this repository to update about the `/experimental-features` endpoint. I'll happily update if you point me in the right direction!
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Co-authored-by: bwbonanno <bradfordbonanno@gmail.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-10-23 08:51:37 +00:00
bwbonanno
dd619913da
Use RwLock to never persist cli state to db
2023-10-19 12:45:57 -07:00
meili-bors[bot]
5e0485d8dd
Merge #4131
...
4131: Reduce proximity range from 7 to 3 r=Kerollmops a=ManyTheFish
## Summary
This PR aims to reduce the impact of the proximity databases on the indexing time and on the database size by reducing the maximum distance between two words to be indexed in the proximity database.
## Stats
### Impact on database size and indexing time
### Impact on search relevancy
<details>
| dataset_name | host_name | Relevancy rate (Precision) | completion_rate 25.00% | completion_rate 50.00% | completion_rate 75.00% | completion_rate 100.00% |
|--------------|------------------|------------------------------------|-----------------|-----------------|-----------------|-----------------|
| FBIS | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 5.56% |
| FBIS | 1_4_0 | percentile-75 | 0.00% | 12.50% | 35.00% | 45.00% |
| FBIS | 1_4_0 | percentile-90 | 20.00% | 40.00% | | 100.00% |
| FBIS | 1_4_0 | average | 5.78% | 11.16% | 21.90% | 26.29% |
| FBIS | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 5.56% |
| FBIS | reduce_proximity | percentile-75 | 0.00% | 15.00% | 35.00% | 40.00% |
| FBIS | reduce_proximity | percentile-90 | 20.00% | 40.00% | 85.00% | 100.00% |
| FBIS | reduce_proximity | average | 5.55% | 11.34% | 21.75% | 26.14% |
| FR94 | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | 1_4_0 | percentile-50 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | 1_4_0 | percentile-75 | 0.00% | 5.00% | 15.00% | 42.11% |
| FR94 | 1_4_0 | percentile-90 | 15.00% | 54.55% | 100.00% | 100.00% |
| FR94 | 1_4_0 | average | 5.95% | 12.07% | 18.70% | 25.57% |
| FR94 | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | reduce_proximity | percentile-50 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | reduce_proximity | percentile-75 | 0.00% | 5.00% | 15.00% | 42.11% |
| FR94 | reduce_proximity | percentile-90 | 15.00% | 54.55% | 100.00% | 100.00% |
| FR94 | reduce_proximity | average | 5.79% | 12.00% | 18.70% | 25.53% |
| FT | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 10.00% |
| FT | 1_4_0 | percentile-75 | 0.00% | 15.00% | 30.00% | 40.00% |
| FT | 1_4_0 | percentile-90 | 20.00% | 50.00% | 65.00% | 100.00% |
| FT | 1_4_0 | average | 5.08% | 12.58% | 20.00% | 25.49% |
| FT | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 10.00% |
| FT | reduce_proximity | percentile-75 | 0.00% | 15.00% | 30.00% | 40.00% |
| FT | reduce_proximity | percentile-90 | 10.00% | 45.00% | 60.00% | 100.00% |
| FT | reduce_proximity | average | 5.01% | 12.64% | 20.10% | 25.53% |
| LAT | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 5.00% |
| LAT | 1_4_0 | percentile-75 | 5.00% | 15.00% | 30.00% | 30.00% |
| LAT | 1_4_0 | percentile-90 | 15.00% | 45.00% | 60.00% | 80.00% |
| LAT | 1_4_0 | average | 4.80% | 11.80% | 17.88% | 21.62% |
| LAT | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 5.00% |
| LAT | reduce_proximity | percentile-75 | 0.00% | 11.11% | 25.00% | 35.00% |
| LAT | reduce_proximity | percentile-90 | 15.00% | 45.00% | 55.00% | 80.00% |
| LAT | reduce_proximity | average | 4.43% | 11.23% | 17.32% | 21.45% |
</details>
### Impact on Search time
| dataset_name | host_name | 25.00% | 50.00% | 75.00% | 100.00% | Average |
|--------------|------------------|------------:|------------:|------------:|------------:|-------------|
| FBIS | 1_4_0 | 3.45 | 7.446666667 | 9.773489933 | 9.620300752 | 7.572614338 |
| FBIS | reduce_proximity | 2.983333333 | 5.316666667 | 6.911073826 | 7.637218045 | 5.712072968 |
| FR94 | 1_4_0 | 2.236666667 | 4.45 | 5.523489933 | 4.560150376 | 4.192576744 |
| FR94 | reduce_proximity | 2.09 | 3.991666667 | 4.981543624 | 4.266917293 | 3.832531896 |
| FT | 1_4_0 | 5.956666667 | 9.656666667 | 13.86912752 | 10.83270677 | 10.0787919 |
| FT | reduce_proximity | 4.51 | 5.981666667 | 7.701342282 | 6.766917293 | 6.23998156 |
| LAT | 1_4_0 | 5.856666667 | 9.233333333 | 12.98322148 | 10.78759398 | 9.715203865 |
| LAT | reduce_proximity | 6.91 | 6.706666667 | 8.463087248 | 8.265037594 | 7.586197877 |
## Technical approach
- Ensure the MAX_DISTANCE constant is used everywhere needed
- Reduce the MAX_DISTANCE from 8 to 4
## Related
TBD
Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-10-18 14:56:08 +00:00
ManyTheFish
27eec21415
Fix tests
2023-10-18 16:03:22 +02:00
bwbonanno
12fc878640
Merge remote-tracking branch 'origin/main' into enable-metrics-http
2023-10-16 13:48:01 -07:00
bwbonanno
689ec7c7ad
Make the experimental route /metrics activable via HTTP
2023-10-13 22:12:54 +00:00
Kerollmops
1b8871a585
Make cargo insta happy
2023-10-13 13:11:30 +02:00
meili-bors[bot]
0913373a5e
Merge #4122
...
4122: Bring back changes from `release-v1.4.1` into `main` r=Kerollmops a=curquiza
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Vivek Kumar <vivek.26@outlook.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-10-12 15:57:47 +00:00
Clément Renault
1a7f1282af
Fix test to use new common Value type
2023-10-12 17:37:04 +02:00
Vivek Kumar
d1331d8abf
add integration test for distinct search with no ranking
2023-10-11 19:12:56 +05:30
meili-bors[bot]
86b314626d
Merge #4080
...
4080: Bring back changes from v1.4.0 into main r=Kerollmops a=curquiza
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com>
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: curquiza <clementine@meilisearch.com>
Co-authored-by: Vivek Kumar <vivek.26@outlook.com>
Co-authored-by: dogukanakkaya <doguakkaya27@hotmail.com>
2023-09-26 08:13:49 +00:00
Tamo
584b772248
enable metrics in debug builds
2023-09-21 17:01:05 +02:00
Tamo
791c5cd874
makes clippy happy
2023-09-11 17:02:01 +02:00
Tamo
5bea1092fb
fix the flaky test
2023-09-11 16:56:26 +02:00
Tamo
056b2c387d
refactor the tests suite slightly
2023-09-11 16:56:26 +02:00
Tamo
2c1d60f79b
get rid of a warning
2023-09-11 14:40:22 +02:00
Tamo
08af69a33b
improve a test to understand what's going on with the ci
2023-09-11 14:23:57 +02:00
Tamo
9258e5b5bf
Fix the stats of the documents deletion by filter
...
The issue was that the operation « DocumentDeletionByFilter » was not
declared as an index operation. That means the indexes stats were not
reprocessed after the application of the operation.
2023-09-11 14:04:10 +02:00
Tamo
ddd34a488a
update the api-key tests
2023-09-11 13:52:07 +02:00
Tamo
e8c9367686
implement the snapshots on demand
2023-09-11 12:35:57 +02:00
meili-bors[bot]
dc3d9c90d9
Merge #3994
...
3994: Fix synonyms with separators r=Kerollmops a=ManyTheFish
# Pull Request
## Related issue
Fixes #3977
## Available prototype
```
$ docker pull getmeili/meilisearch:prototype-fix-synonyms-with-separators-0
```
## What does this PR do?
- add a new test
- filter the empty synonyms after normalization
Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-09-05 14:42:46 +00:00
meili-bors[bot]
e4e49e63d0
Merge #3993
...
3993: Bringing back changes from v1.3.1 to `main` r=irevoire a=curquiza
Co-authored-by: irevoire <irevoire@users.noreply.github.com>
Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-08-10 14:30:02 +00:00
ManyTheFish
fc2590fc9d
Add a test
2023-08-08 16:43:08 +02:00
Tamo
4988199bb9
ensure the geoboundingbox works with strings and int geofields in milli and meilisearch
2023-08-08 16:29:25 +02:00
ManyTheFish
4a21fecf67
Merge branch 'main' into settings-customizing-tokenization
2023-08-08 16:08:16 +02:00
ManyTheFish
ae8e69c030
Add API route for the new settings
2023-08-08 16:03:16 +02:00
meili-bors[bot]
939b2fc6fd
Merge #3949
...
3949: Fix score details casing r=Kerollmops a=ManyTheFish
# Pull Request
Fixes #3941
Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-07-26 14:14:59 +00:00
ManyTheFish
b0c1a9504a
ensure the synonyms are updated when the tokenizer settings are changed
2023-07-26 09:33:42 +02:00
ManyTheFish
88559a2d54
Fix score details casing
2023-07-25 15:49:33 +02:00
Clément Renault
59201a7852
Use snapshot instead of asserts
...
Co-authored-by: Many the fish <many@meilisearch.com>
2023-07-25 15:34:05 +02:00
ManyTheFish
d57026cd96
Support synonyms sinergies
2023-07-25 15:01:42 +02:00
ManyTheFish
41c9e8856a
Fix test
2023-07-25 10:55:37 +02:00
