Commit Graph

942 Commits

Author SHA1 Message Date
Louis Dureuil
8d6ac261ae
Add tests on various failure modes for embedders 2024-07-16 13:39:02 +02:00
Louis Dureuil
b4c8b01c88
Update existing snapshots 2024-07-16 13:39:01 +02:00
meili-bors[bot]
1582c7e788
Merge #4769
4769: Federated search r=ManyTheFish a=dureuill

# Pull Request

## Related issue
Fixes #4747 

[Usage](https://meilisearch.notion.site/v1-10-federated-search-698dfe36ab6b4668b044f735fb40f0b2)

## What does this PR do?
- multi-search now allows a top-level federation object. When not `null`, the results of multi-search are modified to be a single list of results rather than a list of a list of results
- changed lifetimes around tokenizer et al. to be able to make hits one by one rather than using a vector
- adds `roaring` to Meilisearch itself. As the federated search happens at the Meilisearch level (reuses the search functions declared at the Meilisearch level + merge happens after the hits were created), `RoaringBitmap`s are needed to track the candidates: hits that were seen,  all candidates.
- Refactor `make_hits` to allow for an individual, optimized `make_hit` 
- Score details comparison no longer fail when sorting on different field names or target point (for geo)

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-16 08:14:46 +00:00
Louis Dureuil
20094eba06
Apply review comments 2024-07-15 12:43:29 +02:00
Louis Dureuil
c35904d6e8
search::federated::ranking_rules -> search::ranking_rules 2024-07-15 08:43:22 +02:00
Louis Dureuil
2cacc448b6
Rename src/search.rs -> src/search/mod.rs 2024-07-15 08:43:21 +02:00
Louis Dureuil
a61b852695
Add tests 2024-07-15 08:43:21 +02:00
Louis Dureuil
3167411e98
Analytics 2024-07-15 08:43:21 +02:00
Louis Dureuil
83d71662aa
Changes to multi_search route 2024-07-15 08:43:21 +02:00
Louis Dureuil
5c323cecc7
search: introduce federated search 2024-07-15 08:43:21 +02:00
Louis Dureuil
d3a6d2a6fa
search: introduce hitmaker 2024-07-11 16:35:59 +02:00
Louis Dureuil
2123d76089
search: introduce "search_from_kind" 2024-07-11 16:35:11 +02:00
Louis Dureuil
edab4e75b0
Make SearchKind cloneable 2024-07-11 16:33:24 +02:00
Louis Dureuil
b9982587d4
Add new errors to meilisearch 2024-07-11 16:31:44 +02:00
Louis Dureuil
12a7a45930
Add roaring to meilisearch 2024-07-11 16:27:50 +02:00
meili-bors[bot]
677ed6bbf6
Merge #4787
4787: Add index exists function in index_scheduler which stops opening indexes to only check if they exist. r=Kerollmops a=Karribalu

# Pull Request

## Related issue
Fixes #4784

## What does this PR do?
- Added index_exists function in the index_scheduler.
- Resolved opening indexes to only check if they exist.
- Made changes to existing tests to test this function.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: karribalu <karri.balu123456@gmail.com>
2024-07-11 13:05:20 +00:00
Clément Renault
6e80364c50
Apply review comments 2024-07-11 11:00:27 +02:00
karribalu
603676cb3b Address PR review changes 2024-07-10 19:42:16 +01:00
karribalu
23e102ca71 Address PR review changes 2024-07-10 19:33:16 +01:00
Clément Renault
487997f6ad
Support the new editDocumentsByFunction experimental feature 2024-07-10 16:29:18 +02:00
Clément Renault
94809090a3
Support not specifying a context 2024-07-10 16:29:18 +02:00
Clément Renault
01144b2c74
Make the edit documents by function route experimental 2024-07-10 16:29:18 +02:00
Clément Renault
e97600eead
Improve the analytics for the document edition by function 2024-07-10 16:29:18 +02:00
Clément Renault
767553519d
Create errors for the HTTP route issues 2024-07-10 16:29:18 +02:00
Clément Renault
e706023969
Fix some analytics issues 2024-07-10 16:29:17 +02:00
Clément Renault
862d49e4af
Editing documents requires the documents.all action (add, get, and del) 2024-07-10 16:29:05 +02:00
Clément Renault
400e6b93ce
Support user-provided context for documents edition 2024-07-10 16:28:15 +02:00
Clément Renault
f32e6c32fc
Rename editionCode to function 2024-07-10 16:28:15 +02:00
Clément Renault
f07256971a
Fix tests 2024-07-10 16:28:14 +02:00
Clément Renault
246f0e7130
Make the filter field really optional 2024-07-10 16:28:14 +02:00
Clément Renault
45af18ae9c
Check the Rhai syntax before accepting the script 2024-07-10 16:28:13 +02:00
Clément Renault
efc156a4a4
Executing Lua works correctly 2024-07-10 16:27:36 +02:00
Clément Renault
ba85959642
Support filtering the documents to edit with lua 2024-07-10 16:23:21 +02:00
Clément Renault
9d6885793e
Upgrade dependencies 2024-07-10 13:46:24 +02:00
Clément Renault
5f4530ce57
Remove more unused dependencies 2024-07-10 13:36:34 +02:00
Tamo
4d5005b01a make clippy happy 2024-07-10 10:06:59 +02:00
Tamo
9feba5028d update byte-unit 2024-07-09 23:41:29 +02:00
karribalu
ea21b948b1 Address PR review changes 2024-07-09 09:18:57 +01:00
karribalu
47e526f5ea Add index exists function in index_scheduler 2024-07-08 22:27:10 +01:00
Tamo
4aa7d386d8 remove http and uses actix_web::http instead 2024-07-08 21:17:10 +02:00
Tamo
ef8d9a20f8 update actix-web 2024-07-08 18:36:32 +02:00
Tamo
6afa578688 update most incompatible dependencies 2024-07-08 18:31:15 +02:00
Tamo
300bdfc2a7 update most dependencies 2024-07-08 18:09:12 +02:00
Louis Dureuil
128e6c7502
Search: spans with a finer granularity 2024-07-02 16:13:53 +02:00
Tamo
3d9befd64f fix warning 2024-07-02 15:30:16 +02:00
Tamo
ee14d5196c fix the tests 2024-07-02 15:18:30 +02:00
ManyTheFish
015d90a962 merge main 2024-07-01 11:50:36 +02:00
Louis Dureuil
e53de15b8e
Fix behavior of limit and offset for hybrid search when keyword results are returned early
The test is fixed
2024-06-27 14:25:33 +02:00
Louis Dureuil
8c4921b9dd
Add failing test on limit+offset for hybrid search 2024-06-27 14:21:34 +02:00
Tamo
ce08dc509b add more tests and improve the location of the error 2024-06-27 11:51:45 +02:00
Tamo
1daaed163a Make _vectors.:embedding.regenerate mandatory + tests + error messages 2024-06-27 11:04:58 +02:00
meili-bors[bot]
809e742253
Merge #4731
4731: Fix the missing geo distance when one or both of the lat / lng are string r=irevoire a=irevoire

# Pull Request

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4193

## What does this PR do?
- Properly extract the lat / lng when one or both of them are string
- Add a test 


Co-authored-by: Tamo <tamo@meilisearch.com>
2024-06-27 07:33:22 +00:00
meili-bors[bot]
decdfe03bc
Merge #4724
4724: Improve tenant token error messages r=ManyTheFish a=irevoire

# Pull Request

## Related issue
Fixes  #4727

## What does this PR do?
- Introduce a bunch of new error messages around tenant tokens
- Ignore the error messages in most tests that were doing for loop over multiple kinds of errors
- Introduce new tests that specifically test these error messages


Co-authored-by: Tamo <tamo@meilisearch.com>
2024-06-27 06:47:40 +00:00
Tamo
a108d8f6f3 update yaup 2024-06-26 16:03:51 +02:00
Tamo
eb292a7a62 Fix the missing geo distance when one or both of the lat / lng are string 2024-06-26 14:50:15 +02:00
Tamo
a1dcde6b9a
Update meilisearch/src/extractors/authentication/mod.rs
Co-authored-by: Many the fish <many@meilisearch.com>
2024-06-26 14:00:21 +02:00
meili-bors[bot]
1e4699b82c
Merge #4716
4716: Fix bad http status and error message on wrong payload  r=irevoire a=Karribalu

# Pull Request

## Related issue
Fixes #4698

## What does this PR do?
- Fixes bad http status when bad payload with gzip Content-Encoding

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: karribalu <karri.balu123456@gmail.com>
2024-06-26 08:00:51 +00:00
meili-bors[bot]
2c09c324f7
Merge #4730
4730: fix a possibly flaky test r=irevoire a=irevoire

On slow CI, it was possible for a document addition to _not_ to be processed and then get autobatched with an index deletion, which changed their task summary details in the end.
Now, I wait for the task to finish, and the result will always be the same

Co-authored-by: Tamo <tamo@meilisearch.com>
2024-06-26 07:32:51 +00:00
Tamo
3d6b61d8d2 fix flakyness for real 2024-06-26 09:24:09 +02:00
Tamo
1374b661d1 fix a possibly flaky test 2024-06-26 09:14:59 +02:00
meili-bors[bot]
7e3c306c54
Merge #4725
4725: Store primary key as String when Number exceeds i64 range r=irevoire a=JWSong

# Pull Request

## Related issue
Fixes #4696 

## What does this PR do?
- When a Number value exceeding the range of i64 is received as a primary key, it will be stored as a String.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: JWSong <thdwjddn123@gmail.com>
2024-06-26 07:06:04 +00:00
karribalu
2608a596a0 Update error message and add tests for incomplete compressed document 2024-06-25 18:36:29 +01:00
JWSong
6fb36ed30e get rid of the redundant info in document_addition_with_huge_int_primary_key 2024-06-25 23:54:27 +09:00
JWSong
dcdc83946f accept large number as string 2024-06-25 21:41:47 +09:00
Tamo
7da21bb601 introduce as many custom error message as possible 2024-06-25 12:40:51 +02:00
Junho Choi
2e0ff56f3f Add missing Korean support
Some configuration is missing `korean` features and
add a test case in `milli/src/search/mod.rs`.
2024-06-25 12:45:21 +09:00
Tamo
a74fb87d1e start introducing new error messages 2024-06-24 19:00:53 +02:00
Tamo
558b66e535 makes most tests works with variable error messages 2024-06-24 19:00:44 +02:00
Tamo
606e108420 fix all the flaky snapshots 2024-06-24 11:13:45 +02:00
Tamo
7be17b7e4c add the missing snapshots 2024-06-24 10:52:57 +02:00
Tamo
1693332cab Update arroy and always build the tree that need to be built 2024-06-24 10:14:03 +02:00
karribalu
2a38f5c757 Run Rustfmt 2024-06-21 00:14:26 +01:00
karribalu
fb683fe88b Fix bad http status and error message on wrong payload 2024-06-20 23:55:09 +01:00
curquiza
bad28cc9e2 Update mini-dashboard 2.14 2024-06-20 10:01:36 +02:00
meili-bors[bot]
e580d6b98f
Merge #4693
4693: Introduce distinct attributes at search time r=irevoire a=Kerollmops

This PR fixes #4611.

### To Do
- [x] Remove the `distinguishableAttributes` settings (not even a commit about that).
- [x] Use the `filterableAttributes` to be able to use the `distinct` parameter at search.
- [x] Work on the errors and make tests.

Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-06-18 07:45:03 +00:00
Tamo
8ba65e333b add snapshot files 2024-06-17 16:50:26 +02:00
Tamo
43875e6758 fix bug around nested fields 2024-06-17 15:59:30 +02:00
Tamo
d7844a6e45 add a bunch of tests on the errors of the distinct at search time 2024-06-17 15:37:32 +02:00
meili-bors[bot]
e9bf4c43a4
Merge #4649
4649: Don't store the vectors in the documents database r=dureuill a=irevoire

# Pull Request

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4607

## What does this PR do?
- Ensure that anything falling under `_vectors` is NOT searchable, filterable or sortable
- [x] per embedder, add a roaring bitmap of documents that provide "userProvided" embeddings
- [x] in the indexing process in extract_vector_points, set the bit corresponding to the document depending on the "userProvided" subfield in the _vectors field.
- [x] in the document DB in typed chunks, when writing the _vectors field, remove all keys corresponding to an embedder

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-06-17 12:32:03 +00:00
Tamo
a8a0854421
Update meilisearch/src/analytics/segment_analytics.rs 2024-06-17 14:30:50 +02:00
Louis Dureuil
09d9b63e1c
- test case where all vectors were generated
- update tests following changes in behavior from previous commit
2024-06-13 17:16:41 +02:00
Louis Dureuil
b9b938c902
Change retrieveVectors behavior:
- when the feature is disabled, documents are never modified
- when the feature is enabled and `retrieveVectors` is disabled, `_vectors` is removed from documents
- when the feature is enabled and `retrieveVectors` is enabled, vectors from the vectors DB are merged with `_vectors` in documents

Additionally `_vectors` is never displayed when the `displayedAttributes` list does not contain either `*` or `_vectors`

- fixed an issue where `_vectors` was not injected when all vectors in the dataset where always generated
2024-06-13 17:13:36 +02:00
Tamo
6bf07d969e add failing test 2024-06-13 15:49:42 +02:00
Louis Dureuil
3f212a8202
Update tests 2024-06-12 18:13:34 +02:00
Louis Dureuil
3bc8f81abc
user_provided => regenerate 2024-06-12 18:12:20 +02:00
Louis Dureuil
fca9fe39b3
Update test snapshots 2024-06-12 14:50:55 +02:00
meili-bors[bot]
e0eff08095
Merge #4685
4685: Fix ci tests r=dureuill a=ManyTheFish

# Pull Request
Make the all following CI succeed:
https://github.com/meilisearch/meilisearch/actions/runs/9477183091

## Related issue
Fixes #4629

## What does this PR do?
- Change the test behavior for `swedish-recomposition` feature flag
- Remove the `-v` parameter from grep

Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2024-06-12 07:58:33 +00:00
Clément Renault
ee39309aae
Improve errors and introduce a new InvalidSearchDistinct error code 2024-06-11 16:03:39 -04:00
Clément Renault
0d31be1494
Make the distinct work at search 2024-06-11 11:39:35 -04:00
Tamo
3493093c4f add a batch of tests 2024-06-11 16:03:54 +02:00
Tamo
600e97d9dc gate the retrieveVectors parameter behind the vectors feature flag 2024-06-10 18:26:12 +02:00
Louis Dureuil
7559dfc814
Merge tag 'v1.8.2' into release-v1.9.0 2024-06-10 15:07:34 +02:00
Louis Dureuil
50f8218a5d
Asynchronously drop permits 2024-06-10 10:19:57 +02:00
ManyTheFish
57d066595b fix Tests almost all features 2024-06-06 17:24:50 +02:00
Tamo
734d1c53ad fix a panic in yaup 2024-06-06 16:31:07 +02:00
Tamo
63dded3961 implements the new analytics for the get documents routes 2024-06-06 11:39:29 +02:00
Tamo
2cdcb703d9 fix the deletion of vectors and add a test 2024-06-06 11:39:29 +02:00
Tamo
6607875f49 add the retrieveVectors parameter to the get and fetch documents route 2024-06-06 11:39:29 +02:00
Tamo
31a793d226 fix the regeneration of the embeddings in the search 2024-06-06 11:39:29 +02:00
Tamo
d85ab23b82 rename all occurences of user_defined to user_provided for consistency 2024-06-06 11:39:29 +02:00