meili-bors[bot]
56bf8503db
Merge #4537
...
4537: Expose distribution shift in settings r=ManyTheFish a=dureuill
See [usage page](https://meilisearch.notion.site/v1-8-AI-search-API-usage-135552d6e85a4a52bc7109be82aeca42#d652adc0890445658aaf36352dbc8802 )
# Changes
- Distribution shift added to all embedders.
- Exposed in settings
- Changed the reindexing logic to not trigger a reindex operation when only the distribution shift or API key change
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-04-03 09:08:58 +00:00
meili-bors[bot]
78668584cd
Merge #4533
...
4533: Hide api key in settings and task queue r=dureuill a=dureuill
# Pull Request
See [Usage page](https://meilisearch.notion.site/v1-8-AI-search-API-usage-135552d6e85a4a52bc7109be82aeca42#117f5ff7b19f4d95bb3ae0005f6c6633 )
## Motivation
See [slack discussion (internal link)](https://meilisearch.slack.com/archives/C06GQP7FQ6P/p1709804022298749 )
## Changes
- The value of the `apiKey` parameter is now hidden in the settings and the details of the task queue.
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-03-28 16:02:53 +00:00
Louis Dureuil
cde7ce4f44
Add test
2024-03-27 14:02:09 +01:00
Tamo
3a1f458139
fix a flaky test
2024-03-26 21:06:55 +01:00
Tamo
2e36f069c2
fmt imports
2024-03-26 19:23:55 +01:00
Tamo
8127c9a115
handle the case of a queue of zero elements
2024-03-26 19:04:39 +01:00
Tamo
e7704f1fc1
add a test to ensure we effectively returns a retry-after when the search queue is full
2024-03-26 18:08:59 +01:00
Tamo
e433fd53e6
rename the method to get a permit and use it in all search requests
2024-03-26 17:28:03 +01:00
Tamo
c41e1274dc
push and test the search queue datastructure
2024-03-26 15:56:43 +01:00
Louis Dureuil
9a95ed619d
Add tests
2024-03-26 10:36:56 +01:00
Tamo
d8fe4fe49d
return the order in the score details
2024-03-19 15:45:04 +01:00
Tamo
4369e9e97c
add an error code test on the setting
2024-03-19 11:14:28 +01:00
Tamo
6a0c399c2f
rename the search_cutoff parameter to search_cutoff_ms
2024-03-19 10:35:47 +01:00
Tamo
038c26c118
stop returning the degraded boolean when a search was cutoff
2024-03-19 10:35:47 +01:00
Tamo
ad9192fbbf
reduce the size of an integration test
2024-03-19 10:35:47 +01:00
Tamo
b8cda6c300
fix the search cutoff and add a test
2024-03-19 10:35:47 +01:00
Tamo
b72495eb58
fix the settings tests
2024-03-19 10:28:23 +01:00
Tamo
d1db495119
add a settings for the search cutoff
2024-03-19 10:28:23 +01:00
meili-bors[bot]
abd954755d
Merge #4476
...
4476: Make the `/facet-search` route use the `sortFacetValuesBy` setting r=irevoire a=Kerollmops
This PR fixes #4423 by ensuring that the `/facet-search` route uses the `sortFacetValuesBy` setting.
Note for the documentation team (to be moved in the tracking issue): Using the new `sortFacetValuesBy` setting can slow down the facet-search requests as Meilisearch iterates over the whole list of facet values and computes the count of documents on every entry. That is hardly or even impossible to optimize correctly.
### TODO
- [x] Create a custom HashMap wrapper for the facet `OrderBy` settings.
This wrapper will return the `OrderBy` setting of the facet, if not defined will use the default `*` one, and if not there either (strange) will fall back on the lexicographic one.
- [x] Create a `ValuesCollection` wrapper that implements the logic for the lexicographic and count order by.
- [x] Use it when there is no search query.
- [x] Use it when there is a search query with and without allowed typos.
- [x] Do not change the original logic, only use a wrapper.
- [x] Add tests
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-03-13 14:36:14 +00:00
Clément Renault
6c9823d7bb
Add tests to sortFacetValuesBy count
2024-03-13 11:59:39 +01:00
Tamo
8ec3e30d2b
Merge branch 'main' into tmp-release-v1.7.0
2024-03-11 15:39:51 +01:00
Tamo
f053c280e1
add tests when the field limit is reached
2024-03-06 18:42:41 +01:00
Tamo
b130917933
add the content type in the webhook + improve the test
2024-03-05 11:22:29 +01:00
meili-bors[bot]
b87485e80d
Merge #4433
...
4433: Enhance facet incremental r=Kerollmops a=ManyTheFish
# Pull Request
## Related issue
Fixes #4367
Fixes #4409
## What does this PR do?
- Add a test reproducing #4409
- Fix #4409 by removing a document from a level only if it is no more present in all the linked sub-level nodes
- Optimize facet Incremental indexing by creating or deleting a complete level once per field id instead of for each facet value
- Optimize facet Incremental indexing by doing the additions and the deletions in the same process instead of doing them separately
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-02-28 15:28:46 +00:00
Tamo
a478392b7a
create a test with the dry-run parameter enabled
2024-02-26 13:59:41 +01:00
Tamo
bbf3fb88ca
rename the cli parameter
2024-02-26 13:59:40 +01:00
Tamo
507739bd98
add an experimental cli parameter to allow specifying your task id
2024-02-26 13:58:03 +01:00
Tamo
eb25b07390
let you specify your task id
2024-02-26 13:56:31 +01:00
Tamo
c2e2003a80
create a test with the dry-run parameter enabled
2024-02-22 15:51:47 +01:00
Tamo
693ba8dd15
rename the cli parameter
2024-02-21 14:33:40 +01:00
Tamo
01ae46dd80
add an experimental cli parameter to allow specifying your task id
2024-02-20 11:24:44 +01:00
Tamo
9ee4f55e6c
let you specify your task id
2024-02-19 14:29:33 +01:00
ManyTheFish
865b415b3f
Add test rerpoducing bug
2024-02-15 16:00:48 +01:00
Tamo
a081da0d90
add support for the json format in the stream route
2024-02-14 15:34:39 +01:00
Tamo
3b6544db6d
Implement the experimental log mode cli flag
2024-02-13 18:09:15 +01:00
Tamo
2c88131bb1
rename the fmt mode to human
2024-02-08 15:04:06 +01:00
Tamo
35aa9d5904
fix an error message
2024-02-08 15:04:06 +01:00
Louis Dureuil
ef994d84d0
Change error messages and fix tests
2024-02-08 15:04:06 +01:00
Louis Dureuil
1b74010e9e
Remove "with_line_numbers"
2024-02-08 15:04:06 +01:00
Tamo
bcf7909bba
add a profile_memory parameter disabled by default
2024-02-08 15:04:05 +01:00
Tamo
ceb211c515
move the /logs route to the /logs/stream route
2024-02-08 15:04:05 +01:00
Tamo
f158e96fe7
fix the auth
2024-02-08 15:04:05 +01:00
Tamo
e23ec4886d
fix the tests and add tests on the experimental features
2024-02-08 15:04:03 +01:00
Tamo
80774148fd
handle and tests errors
2024-02-08 15:03:33 +01:00
Tamo
bf5cea8b10
add a test
2024-02-08 15:03:33 +01:00
Tamo
77254765e8
get rids of env loggegr and fix the tests
2024-02-08 15:03:33 +01:00
Louis Dureuil
05edd85d75
Stabilize scoreDetails
2024-02-06 11:15:19 +01:00
Tamo
318843aacd
add a bunch of tests and fix the error message when adding the geosearch as filterable/sortable while there is malformed documents in the DB
2024-02-01 10:33:27 +01:00
Louis Dureuil
6d111139b5
Add test
2024-02-01 10:33:27 +01:00
Tamo
c1bf33a112
Revert "Remove panic on the geosearch"
2024-01-25 18:51:19 +01:00
Tamo
7d190d8078
add a bunch of tests and fix the error message when adding the geosearch as filterable/sortable while there is malformed documents in the DB
2024-01-17 15:51:52 +01:00
Louis Dureuil
38abfec611
Fix tests
2024-01-11 21:35:30 +01:00
Clément Renault
3f3462ab62
Limit the number of values returned by the facet search
2024-01-10 16:54:08 +01:00
ManyTheFish
f75f22e026
Display default value when proximityPrecision is not set
2024-01-08 11:09:37 +01:00
Louis Dureuil
94b9f3b310
Add test
2024-01-03 15:56:20 +01:00
meili-bors[bot]
43e822e802
Merge #4238
...
4238: Task queue webhook r=dureuill a=irevoire
# Prototype `prototype-task-queue-webhook-1`
The prototype is available through Docker by using the following command:
```bash
docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-task-queue-webhook-1
```
# Pull Request
Implements the task queue webhook.
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4236
## What does this PR do?
- Provide a new cli and env var for the webhook, respectively called `--task-webhook-url` and `MEILI_TASK_WEBHOOK_URL`
- Also supports sending the requests with a custom `Authorization` header by specifying the optional `--task-webhook-authorization-header` CLI parameter or `MEILI_TASK_WEBHOOK_AUTHORIZATION_HEADER` env variable.
- Throw an error if the specified URL is invalid
- Every time a batch is processed, send all the finished tasks into the webhook with our public `TaskView` type as a JSON Line GZIPed body.
- Add one test.
## PR checklist
### Before becoming ready to review
- [x] Add a test
- [x] Compress the data we send
- [x] Chunk and stream the data we send
- [x] Remove the unwrap in the index-scheduler when sending the data fails
- [x] The analytics are missing
### Before merging
- [x] Release a prototype
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-12-21 14:43:46 +00:00
Louis Dureuil
6ff81de401
Fix tests
2023-12-20 17:16:46 +01:00
Tamo
c83a33017e
stream and chunk the data
2023-12-19 10:35:51 +01:00
Tamo
be72326c0a
gzip the tasks
2023-12-19 10:35:51 +01:00
Tamo
0b2fff27f2
update and fix the test
2023-12-19 10:35:51 +01:00
Tamo
3adbc2b942
return a task view instead of a task
2023-12-19 10:35:51 +01:00
Tamo
fbea721378
add a first working test with actixweb
2023-12-19 10:35:51 +01:00
Tamo
391eb72137
start writing a test with actix but it doesn't works
2023-12-19 10:35:50 +01:00
ManyTheFish
3a99a555a2
Fix experimental features snapshots in tests
2023-12-18 10:05:51 +01:00
Many the fish
9e1b458010
Merge branch 'main' into change-proximity-precision-settings
2023-12-18 09:08:47 +01:00
ManyTheFish
6425996e36
Change the naming of attributeScale and wordScale into byAttribute and byWord
2023-12-14 16:31:00 +01:00
Louis Dureuil
87bba98bd8
Various changes
...
- fixed seed for arroy
- check vector dimensions as soon as it is provided to search
- don't embed whitespace
2023-12-14 16:08:42 +01:00
Louis Dureuil
217105b7da
hybrid search uses semantic ratio, error handling
2023-12-14 16:08:42 +01:00
ManyTheFish
f3f3944469
Fix error checking
2023-12-14 16:08:42 +01:00
ManyTheFish
ac68f33194
Add simple test
2023-12-14 16:08:42 +01:00
Louis Dureuil
806e5b6899
Tests pass
2023-12-14 16:08:41 +01:00
Louis Dureuil
12940d79a9
WIP
...
- manual embedder
- multi embedders OK
- clippy + tests OK
2023-12-14 16:08:41 +01:00
ManyTheFish
35e1981488
Remove proximityPrecision form the experimental feature
2023-12-14 15:52:42 +01:00
Clément Renault
56571f762a
Merge remote-tracking branch 'origin/main' into tmp-release-v1.5.1
2023-12-13 11:57:01 +01:00
ManyTheFish
c9860c7913
Small test fixes
2023-12-06 15:49:05 +01:00
ManyTheFish
03ffabe889
Add a new dump test
2023-12-06 15:49:05 +01:00
ManyTheFish
1f4fc9c229
Make the feature experimental
2023-12-06 15:49:05 +01:00
ManyTheFish
8cc3c54117
Add proximityPrecision setting in settings route
2023-12-06 15:49:05 +01:00
ManyTheFish
0c3fa8cbc4
Add tests on proximityPrecision setting
2023-12-06 14:59:23 +01:00
ManyTheFish
dc07790133
Add test reproducing #4232
2023-11-27 11:39:11 +01:00
Clément Renault
0dbf1a16ff
Make clippy happy
2023-11-23 14:11:38 +01:00
Clément Renault
7cb7e37ba8
Merge branch 'main' into tmp-release-v1.5.0
2023-11-21 16:30:46 +01:00
Louis Dureuil
9fedd8101a
Fix tests
2023-10-30 15:11:07 +01:00
Tamo
e7244aa485
fix warnings
2023-10-30 11:00:46 +01:00
Vivek Kumar
32c78ac8b1
add/update tests when search with distinct attribute & pagination with no ranking
2023-10-23 12:06:27 +02:00
meili-bors[bot]
eae9eab181
Merge #4126
...
4126: Make the experimental route /metrics activable via HTTP r=dureuill a=braddotcoffee
# Pull Request
## Related issue
Closes #4086
## What does this PR do?
- [x] Make `/metrics` available via HTTP as described in #4086
- [x] The users can still launch Meilisearch using the `--experimental-enable-metrics` flag.
- [x] If the flag `--experimental-enable-metrics` is activated, a call to the `GET /experimental-features` route right after the launch will show `"metrics": true` even if the user has not called the `PATCH /experimental-features` route yet.
- [x] Even if the --experimental-enable-metrics flag is present at launch, calling the `PATCH /experimental-features` route with `"metrics": false` disables the experimental feature.
- [x] Update the spec
- I was unable to find docs in this repository to update about the `/experimental-features` endpoint. I'll happily update if you point me in the right direction!
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Co-authored-by: bwbonanno <bradfordbonanno@gmail.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-10-23 08:51:37 +00:00
bwbonanno
dd619913da
Use RwLock to never persist cli state to db
2023-10-19 12:45:57 -07:00
meili-bors[bot]
5e0485d8dd
Merge #4131
...
4131: Reduce proximity range from 7 to 3 r=Kerollmops a=ManyTheFish
## Summary
This PR aims to reduce the impact of the proximity databases on the indexing time and on the database size by reducing the maximum distance between two words to be indexed in the proximity database.
## Stats
### Impact on database size and indexing time
![Impact on datasets](https://github.com/meilisearch/meilisearch/assets/6482087/28ed3d96-bdde-41c1-bdac-e90c1b1dbb23 )
### Impact on search relevancy
<details>
| dataset_name | host_name | Relevancy rate (Precision) | completion_rate 25.00% | completion_rate 50.00% | completion_rate 75.00% | completion_rate 100.00% |
|--------------|------------------|------------------------------------|-----------------|-----------------|-----------------|-----------------|
| FBIS | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 5.56% |
| FBIS | 1_4_0 | percentile-75 | 0.00% | 12.50% | 35.00% | 45.00% |
| FBIS | 1_4_0 | percentile-90 | 20.00% | 40.00% | | 100.00% |
| FBIS | 1_4_0 | average | 5.78% | 11.16% | 21.90% | 26.29% |
| FBIS | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FBIS | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 5.56% |
| FBIS | reduce_proximity | percentile-75 | 0.00% | 15.00% | 35.00% | 40.00% |
| FBIS | reduce_proximity | percentile-90 | 20.00% | 40.00% | 85.00% | 100.00% |
| FBIS | reduce_proximity | average | 5.55% | 11.34% | 21.75% | 26.14% |
| FR94 | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | 1_4_0 | percentile-50 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | 1_4_0 | percentile-75 | 0.00% | 5.00% | 15.00% | 42.11% |
| FR94 | 1_4_0 | percentile-90 | 15.00% | 54.55% | 100.00% | 100.00% |
| FR94 | 1_4_0 | average | 5.95% | 12.07% | 18.70% | 25.57% |
| FR94 | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | reduce_proximity | percentile-50 | 0.00% | 0.00% | 0.00% | 0.00% |
| FR94 | reduce_proximity | percentile-75 | 0.00% | 5.00% | 15.00% | 42.11% |
| FR94 | reduce_proximity | percentile-90 | 15.00% | 54.55% | 100.00% | 100.00% |
| FR94 | reduce_proximity | average | 5.79% | 12.00% | 18.70% | 25.53% |
| FT | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 10.00% |
| FT | 1_4_0 | percentile-75 | 0.00% | 15.00% | 30.00% | 40.00% |
| FT | 1_4_0 | percentile-90 | 20.00% | 50.00% | 65.00% | 100.00% |
| FT | 1_4_0 | average | 5.08% | 12.58% | 20.00% | 25.49% |
| FT | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| FT | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 10.00% |
| FT | reduce_proximity | percentile-75 | 0.00% | 15.00% | 30.00% | 40.00% |
| FT | reduce_proximity | percentile-90 | 10.00% | 45.00% | 60.00% | 100.00% |
| FT | reduce_proximity | average | 5.01% | 12.64% | 20.10% | 25.53% |
| LAT | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 5.00% |
| LAT | 1_4_0 | percentile-75 | 5.00% | 15.00% | 30.00% | 30.00% |
| LAT | 1_4_0 | percentile-90 | 15.00% | 45.00% | 60.00% | 80.00% |
| LAT | 1_4_0 | average | 4.80% | 11.80% | 17.88% | 21.62% |
| LAT | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% |
| LAT | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 5.00% |
| LAT | reduce_proximity | percentile-75 | 0.00% | 11.11% | 25.00% | 35.00% |
| LAT | reduce_proximity | percentile-90 | 15.00% | 45.00% | 55.00% | 80.00% |
| LAT | reduce_proximity | average | 4.43% | 11.23% | 17.32% | 21.45% |
</details>
### Impact on Search time
| dataset_name | host_name | 25.00% | 50.00% | 75.00% | 100.00% | Average |
|--------------|------------------|------------:|------------:|------------:|------------:|-------------|
| FBIS | 1_4_0 | 3.45 | 7.446666667 | 9.773489933 | 9.620300752 | 7.572614338 |
| FBIS | reduce_proximity | 2.983333333 | 5.316666667 | 6.911073826 | 7.637218045 | 5.712072968 |
| FR94 | 1_4_0 | 2.236666667 | 4.45 | 5.523489933 | 4.560150376 | 4.192576744 |
| FR94 | reduce_proximity | 2.09 | 3.991666667 | 4.981543624 | 4.266917293 | 3.832531896 |
| FT | 1_4_0 | 5.956666667 | 9.656666667 | 13.86912752 | 10.83270677 | 10.0787919 |
| FT | reduce_proximity | 4.51 | 5.981666667 | 7.701342282 | 6.766917293 | 6.23998156 |
| LAT | 1_4_0 | 5.856666667 | 9.233333333 | 12.98322148 | 10.78759398 | 9.715203865 |
| LAT | reduce_proximity | 6.91 | 6.706666667 | 8.463087248 | 8.265037594 | 7.586197877 |
## Technical approach
- Ensure the MAX_DISTANCE constant is used everywhere needed
- Reduce the MAX_DISTANCE from 8 to 4
## Related
TBD
Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-10-18 14:56:08 +00:00
ManyTheFish
27eec21415
Fix tests
2023-10-18 16:03:22 +02:00
bwbonanno
12fc878640
Merge remote-tracking branch 'origin/main' into enable-metrics-http
2023-10-16 13:48:01 -07:00
bwbonanno
689ec7c7ad
Make the experimental route /metrics activable via HTTP
2023-10-13 22:12:54 +00:00
Kerollmops
1b8871a585
Make cargo insta happy
2023-10-13 13:11:30 +02:00
meili-bors[bot]
0913373a5e
Merge #4122
...
4122: Bring back changes from `release-v1.4.1` into `main` r=Kerollmops a=curquiza
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Vivek Kumar <vivek.26@outlook.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-10-12 15:57:47 +00:00
Clément Renault
1a7f1282af
Fix test to use new common Value type
2023-10-12 17:37:04 +02:00
Vivek Kumar
d1331d8abf
add integration test for distinct search with no ranking
2023-10-11 19:12:56 +05:30
meili-bors[bot]
86b314626d
Merge #4080
...
4080: Bring back changes from v1.4.0 into main r=Kerollmops a=curquiza
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com>
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: curquiza <clementine@meilisearch.com>
Co-authored-by: Vivek Kumar <vivek.26@outlook.com>
Co-authored-by: dogukanakkaya <doguakkaya27@hotmail.com>
2023-09-26 08:13:49 +00:00
Tamo
584b772248
enable metrics in debug builds
2023-09-21 17:01:05 +02:00
Tamo
791c5cd874
makes clippy happy
2023-09-11 17:02:01 +02:00
Tamo
5bea1092fb
fix the flaky test
2023-09-11 16:56:26 +02:00
Tamo
056b2c387d
refactor the tests suite slightly
2023-09-11 16:56:26 +02:00