ManyTheFish
35bbe1c2a2
Add failing test on settings changes
2024-11-20 14:48:12 +01:00
Louis Dureuil
7d64e8dbd3
Fix Windows compilation
2024-11-20 14:40:38 +01:00
Tamo
ec06879d28
apply review changes
2024-11-20 14:40:36 +01:00
Tamo
83d1f858c1
Update crates/index-scheduler/src/lib.rs
...
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-11-20 14:36:05 +01:00
Louis Dureuil
cae8c89467
"fix" last warnings
2024-11-20 14:03:52 +01:00
Tamo
a7ac590e9e
implements the reverse query parameter for the batches
2024-11-20 13:29:52 +01:00
Clément Renault
7cb8732b45
Introduce a new bincode internal error
2024-11-20 13:23:11 +01:00
Tamo
8ad68dd708
stop leaking the update files of the canceled tasks
2024-11-20 13:17:54 +01:00
ManyTheFish
fe5d50969a
Fix filed selector in extrators
2024-11-20 13:16:44 +01:00
Clément Renault
56c7c5d5f0
Fix comments
2024-11-20 13:16:44 +01:00
Louis Dureuil
4cdfdddd6d
Fix one more
2024-11-20 13:16:43 +01:00
Louis Dureuil
2afa33011a
Fix tokenize_document
2024-11-20 13:16:43 +01:00
Louis Dureuil
61feca1f41
More tests pass
2024-11-20 13:16:43 +01:00
Louis Dureuil
f893b5153e
Don't mark [""] as empty facet
2024-11-20 13:16:42 +01:00
Louis Dureuil
ca779c21f9
facets: Handle boolean and skip empty strings
2024-11-20 13:16:42 +01:00
Louis Dureuil
477077bdc2
Remove _vectors from fid map when there are no vectors in sight
2024-11-20 13:16:42 +01:00
ManyTheFish
b1f8aec348
Fix index_documents_check_exists_database
2024-11-20 13:16:41 +01:00
ManyTheFish
ba7f091db3
Use tokenizer on numbers and booleans
2024-11-20 13:16:41 +01:00
Louis Dureuil
8049df125b
Add depth to facet extraction so that null inside an array doesn't mark the entire field as null
2024-11-20 13:16:40 +01:00
Clément Renault
50d1bd01df
We no longer index geo lat and lng
2024-11-20 13:16:40 +01:00
Louis Dureuil
a28d4f5d0c
Fix setup_search_index_with_criteria
2024-11-20 13:16:40 +01:00
Louis Dureuil
fc14f4bc66
Attempt to fix setup_search_index_with_criteria
2024-11-20 13:16:39 +01:00
Clément Renault
5f8a82d6f5
Improve test
2024-11-20 13:16:39 +01:00
Clément Renault
fe04e51a49
One more
2024-11-20 13:16:38 +01:00
Clément Renault
01b27e40ad
Fix a bit of the placeholder search tests
2024-11-20 13:16:38 +01:00
Louis Dureuil
8076d98544
Fix stats_should_not_return_deleted_documents
2024-11-20 13:16:37 +01:00
Louis Dureuil
9e951baad5
One more test passing
2024-11-20 13:16:37 +01:00
Louis Dureuil
52f2fc4c46
Fail in case of user error in tests
2024-11-20 13:16:37 +01:00
Clément Renault
3957917e0b
Correctly count indexed documents
2024-11-20 13:16:36 +01:00
Louis Dureuil
651c30899e
Allow fetching embedders from inside tests
2024-11-20 13:16:36 +01:00
Clément Renault
2c7a7fe4e8
Count the number of documents correctly
2024-11-20 13:16:35 +01:00
Clément Renault
23f0c2c29b
Generate internal ids only when needed
2024-11-20 13:16:35 +01:00
Louis Dureuil
6641c3f59b
Remove all autogenerated tests
2024-11-20 13:16:34 +01:00
Louis Dureuil
07a72824b7
Subfields of _vectors are no longer part of the fid map
2024-11-20 13:16:34 +01:00
Louis Dureuil
000eb55c4e
fix one
2024-11-20 13:16:34 +01:00
Clément Renault
b4bf7ce9b0
Increase the number of readers as the indexer uses readers too
2024-11-20 13:16:33 +01:00
Louis Dureuil
1aef0e4037
documents! macro accepts a single object again
2024-11-20 13:16:33 +01:00
Clément Renault
32d0e50a75
Fix all the benchmark compilation errors
2024-11-20 13:16:32 +01:00
Louis Dureuil
df5884b0c1
Fix settings test
2024-11-20 13:16:32 +01:00
Louis Dureuil
9e0eb5ebb0
Removed some warnings
2024-11-20 13:16:32 +01:00
Clément Renault
3cf1352ae1
Fix the benchmark tests
2024-11-20 13:16:31 +01:00
Clément Renault
aba8a0e9e0
Fix some tests but not all of them
2024-11-20 13:16:31 +01:00
Clément Renault
670aff5553
Remove useless Transform methods
2024-11-20 13:16:08 +01:00
Tamo
7e379b3d14
remove useless prints
2024-11-20 12:27:12 +01:00
Tamo
56eacd221f
update the tests after the rebase
2024-11-20 10:54:38 +01:00
Tamo
bdb51a85fe
now that the task cancelation shares their started at with all the tasks of their batch we don't need the trick of retrieving the previous batch anymore
2024-11-20 10:51:07 +01:00
Tamo
b24a34830d
fix the dump test -> the only change is that we now have a null batch_uid in all the tasks
2024-11-20 10:51:06 +01:00
Tamo
e145d71a62
implements the two last TODOs
2024-11-20 10:51:06 +01:00
Tamo
d9a4e69990
push a missing snapshot
2024-11-20 10:51:06 +01:00
Tamo
b906e3ed70
improve the way we access the mutex
2024-11-20 10:51:06 +01:00
Tamo
4abcd9c04e
add some stats on the batches
2024-11-20 10:51:06 +01:00
Tamo
229fa0f902
implements the batch details
2024-11-20 10:51:06 +01:00
Tamo
5d10c2312b
remove unused file
2024-11-20 10:51:06 +01:00
Tamo
f1d38581e5
add the front end tests on the batches routes
2024-11-20 10:51:06 +01:00
Tamo
62646af7b9
implements the automatic batch deletion
2024-11-20 10:51:06 +01:00
Tamo
1fcb9526f5
fix the task cancelation
2024-11-20 10:51:06 +01:00
Tamo
15eefa4fcc
fixes a lot of small issue, the test about the cancellation is still failing
2024-11-20 10:51:05 +01:00
Tamo
ad9763ffcd
copy multiple task query tests to batches. Currently, they fails
2024-11-20 10:49:25 +01:00
Tamo
d489f5635f
add the mapping between the task and batches
2024-11-20 10:49:23 +01:00
Tamo
a1251c3c83
Implements the get all batches route with filters working
2024-11-20 10:42:55 +01:00
Tamo
6062914654
add the batch_id to the tasks
2024-11-20 10:42:54 +01:00
Lukas Kalbertodt
057fcb3993
Add indices field to _matchesPosition to specify where in an array a match comes from ( #5005 )
...
* Remove unreachable code
* Add `indices` field to `MatchBounds`
For matches inside arrays, this field holds the indices of the array
elements that matched. For example, searching for `cat` inside
`{ "a": ["dog", "cat", "fox"] }` would return `indices: [1]`. For nested
arrays, this contains multiple indices, starting with the one for the
top-most array. For matches in fields without arrays, `indices` is not
serialized (does not exist) to save space.
2024-11-20 01:00:43 +01:00
ManyTheFish
41dbdd2d18
Fix filtered_placeholder_search_should_not_return_deleted_documents and word_scale_set_and_reset
2024-11-19 16:08:25 +01:00
Louis Dureuil
bfefaf71c2
Progress displayed in logs
2024-11-19 09:32:52 +01:00
Louis Dureuil
c782c09208
Move step to a dedicated mod and replace it with an enum
2024-11-18 18:22:13 +01:00
Louis Dureuil
75943a5a9b
Add TODO to remember replacing steps with an enum
2024-11-18 17:40:51 +01:00
Louis Dureuil
04c38220ca
Move MostlySend, ThreadLocal, FullySend to their own commit
2024-11-18 16:43:05 +01:00
Louis Dureuil
5f93651cef
fixes
2024-11-18 16:23:11 +01:00
ManyTheFish
510ca99996
Fixes #4974
2024-11-18 16:08:55 +01:00
ManyTheFish
8924d486db
Add a test reproducing the bug
2024-11-18 16:08:55 +01:00
ManyTheFish
e0c3f3d560
Fix #4984
2024-11-18 16:08:53 +01:00
Louis Dureuil
0a21d9bfb3
Fix double borrow of new fields id map
2024-11-18 15:56:01 +01:00
Louis Dureuil
1f8b01a598
Fix snap since _vectors is no longer part of the field distributions
2024-11-18 12:50:59 +01:00
Louis Dureuil
e736a74729
Remove infinite loop in import_vectors
2024-11-18 12:50:56 +01:00
Louis Dureuil
e9d17136b2
Add deadline of 3 seconds to embedding requests made in the context of hybrid search
2024-11-18 12:15:11 +01:00
Louis Dureuil
a05e448cf8
Add test
2024-11-18 12:15:11 +01:00
ManyTheFish
cd796b0f4b
Fix SDK test
2024-11-18 11:46:00 +01:00
Louis Dureuil
6570da3bcb
Retry in case where the JSON deserialization fails
2024-11-18 11:33:09 +01:00
Clément Renault
5b4c06c24c
Plug the grenad max memory parameter
2024-11-18 11:28:04 +01:00
Louis Dureuil
3a8051866a
Use return_keyword_results function instead of returning raw keyword results when the embedder is broken
2024-11-18 11:17:15 +01:00
Louis Dureuil
9150c8f052
Accept changes to vector format
2024-11-18 11:04:57 +01:00
Louis Dureuil
c202f3dbe2
fix tests and revert change in behavior when primary_key_from_op != primary_key_from_db && index.is_empty()
2024-11-18 10:59:05 +01:00
Clément Renault
677d7293f5
Fix a lot of primary key related tests
2024-11-18 10:59:05 +01:00
Clément Renault
bd31ea2174
Check for at least one valid task after setting their statuses
2024-11-18 10:59:05 +01:00
Clément Renault
83865d2ebd
Expose intermediate errors when processing batches
2024-11-18 10:59:05 +01:00
ManyTheFish
72ba353498
reproduce sdk fail
2024-11-18 10:03:23 +01:00
ManyTheFish
4ff2b3c2ee
Fix test on locales
2024-11-14 15:45:04 +01:00
ManyTheFish
91c58cfa38
Fix positional databases
2024-11-14 11:40:12 +01:00
Clément Renault
9e8367f1e6
Move the rayon thread pool outside the extract method
2024-11-14 10:40:32 +01:00
ManyTheFish
0dd321afc7
reproduce #4984
2024-11-14 10:02:51 +01:00
Louis Dureuil
0e3c5d91ab
Document deletion test passes
2024-11-14 08:42:56 +01:00
Louis Dureuil
695c2c6b99
Cosmetic fix
2024-11-14 08:42:39 +01:00
Louis Dureuil
40dd25d6b2
Fix issue with Replace document method when adding and deleting a document in the same batch
2024-11-13 22:10:00 +01:00
Clément Renault
8e5b1a3ec1
Compute the field distribution and convert _geo into an f64s
2024-11-13 17:44:05 +01:00
ManyTheFish
e627e182ce
Fix facet strings
2024-11-13 17:43:02 +01:00
ManyTheFish
51b6293738
Add linear facet databases
2024-11-13 17:43:02 +01:00
Clément Renault
b17896d899
Finialize the GeoExtractor
2024-11-13 17:43:02 +01:00
Louis Dureuil
a01bc7b454
Fix error_document_field_limit_reached_in_one_document test
2024-11-13 10:34:54 +01:00
Louis Dureuil
7accfea624
Don't short circuit when we encounter a semantic error while extracting fields and external docid
2024-11-13 10:33:59 +01:00
Louis Dureuil
82dcaba6ca
Fix test: somehow on main vectors where displayed even though retrieveVectors: false
2024-11-12 23:58:25 +01:00
Louis Dureuil
cb1d6613dd
Adjust snapshots
2024-11-12 23:26:30 +01:00
Louis Dureuil
3b0cb5b487
Fix vector error messages
2024-11-12 23:26:16 +01:00
Louis Dureuil
bfdcd1cf33
Space changes
2024-11-12 22:52:45 +01:00
Louis Dureuil
1d13e804f7
Adjust test snapshots
2024-11-12 22:52:41 +01:00
Louis Dureuil
c4e9f761e9
Emit better error messages when parsing vectors
2024-11-12 22:49:22 +01:00
Louis Dureuil
8a6e61c77f
InvalidVectorsEmbedderConf error takes a String rather than a deserr error
2024-11-12 22:47:57 +01:00
Louis Dureuil
68bbf674c9
Make REST mock thread independent
2024-11-12 16:31:31 +01:00
Louis Dureuil
980921e078
Vector fixes
2024-11-12 16:31:22 +01:00
Louis Dureuil
1fcd5f091e
Remove progress from task
2024-11-12 12:23:13 +01:00
Louis Dureuil
6094bb299a
Fix user_provided vectors
2024-11-12 10:15:55 +01:00
Louis Dureuil
bef8fc6cf1
Fix hf embedder
2024-11-08 13:10:17 +01:00
Louis Dureuil
e32677999f
Adapt some snapshots
2024-11-08 00:06:33 +01:00
Louis Dureuil
5185aa21b8
Know if your vectors are implicit when writing them back in documents + don't write empty _vectors
2024-11-08 00:05:36 +01:00
Louis Dureuil
8a314ab81d
Fix primary key fid order
2024-11-08 00:05:12 +01:00
Louis Dureuil
4706a0eb49
Fix vector parsing
2024-11-07 23:26:20 +01:00
Louis Dureuil
d97af4d8e6
fix field order of JSON documents
2024-11-07 22:36:52 +01:00
Tamo
2eb1801e85
reverse the order of the task queue
2024-11-07 19:19:44 +01:00
meili-bors[bot]
a5d7ae23bd
Merge #5044
...
5044: Adds new metrics to prometheus r=irevoire a=PedroTurik
not 100% confident in this solution, especially because i couldn't make the "Search Queue searches waiting" metric give me any value other than 0 with my local testing 😆 . But i believe it solves the Issue.
# Pull Request
## Related issue
Fixes #4998
## What does this PR do?
### Adds new metrics to prometheus;
- SearchQueue size,
- SearchQueue searches running,
- and Search Queue searches waiting.
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Co-authored-by: Pedro Turik Firmino <pedroturik@gmail.com>
2024-11-07 17:05:43 +00:00
ManyTheFish
1f5d801271
Fix crashes in facet search indexing
2024-11-07 17:22:30 +01:00
Clément Renault
7864530589
Make the word prefix integer multi-threaded
2024-11-07 16:39:14 +01:00
PedroTurik
03886d0012
Applies optimizations to formatted integration tests ( #5043 )
2024-11-07 15:58:55 +01:00
Clément Renault
700757c01f
Adding a new step
2024-11-07 15:32:04 +01:00
Clément Renault
01f8f30a7a
Fix indentation
2024-11-07 15:08:56 +01:00
Clément Renault
0e4e9e866a
Move the RefCellExt trait in a dedicated module
2024-11-07 11:36:09 +01:00
Louis Dureuil
1477b81d38
Support cancelation in merge and send
2024-11-07 11:23:49 +01:00
Clément Renault
c9f478bc45
Fix bbbul merger
2024-11-07 10:53:46 +01:00
meili-bors[bot]
b427b9e88f
Merge #5025
...
5025: test: improve performance of get_documents.rs r=irevoire a=PedroTurik
# Pull Request
## Related issue
Fixes one item from #4840
## What does this PR do?
- Applies the changes recommended on the issue for `meilisearch/tests/documents/get_documents.rs`
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Pedro Turik Firmino <pedroturik@gmail.com>
2024-11-07 09:46:34 +00:00
Louis Dureuil
39366a67c4
Top level fields don't return vector fields
2024-11-07 10:39:58 +01:00
Louis Dureuil
e2138170ad
some warning fix
2024-11-07 10:06:07 +01:00
Louis Dureuil
03650e3217
Reverse order of computation
2024-11-07 09:39:46 +01:00
Pedro Turik Firmino
8b95f5ccc6
Adds new metrics to prometheus: SearchQueue size, SearchQueue searches running, and Search Queue searches waiting.
2024-11-06 15:37:16 -03:00
Louis Dureuil
10f49f0d75
Post processing of the merge
2024-11-06 17:50:12 +01:00
Louis Dureuil
ee03743355
Merge branch 'indexer-edition-2024' into indexer-edition-2024-doc-chunks
2024-11-06 15:50:53 +01:00
ManyTheFish
10feeb88f2
Merge branch 'main' into indexer-edition-2024
2024-11-06 15:19:18 +01:00
Pedro Turik Firmino
da59a043ba
Fixes formatting issues
2024-11-06 09:55:48 -03:00
Pedro Turik Firmino
da4d47b5d0
Fixes formatting issues
2024-11-06 09:54:20 -03:00
meili-bors[bot]
0507f5d99b
Merge #4928
...
4928: Make matches consider phrases as a single `Match` r=ManyTheFish a=flevi29
# Pull Request
## Related issue
Fixes #4732
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: F. Levi <55688616+flevi29@users.noreply.github.com>
2024-11-06 08:22:01 +00:00
meili-bors[bot]
ede086bc30
Merge #5034
...
5034: Upgrade from v1 10 to v1 11 r=irevoire a=irevoire
# Pull Request
## Related issue
Parts of https://github.com/meilisearch/meilisearch/issues/4978
## What does this PR do?
- Move the code around the offline upgrade to its own module with a file per version
- Fix the upgrade from v1.9 to v1.10 because I couldn’t make it work anymore. It now uses a specified format instead of relying on cargo to get the right set of feature
- ☝️ must be checked against docker
- Provide an update path from v1.10 to v1.11. Most of the code is boilerplate in meilitool, the real code is located here: 053807bf38/src/lib.rs (L161-L269)
2024-11-05 14:49:56 +00:00
Tamo
7415ef7ff5
Update crates/meilitool/src/upgrade/v1_11.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-05 15:37:59 +01:00
Tamo
a5d138ac34
use a tag while importing arroy instead of a loose branch or rev
2024-11-05 15:24:02 +01:00
Tamo
0f74a93346
Update crates/meilitool/src/upgrade/v1_11.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-05 15:14:02 +01:00
Tamo
e4993aa705
Update crates/meilitool/src/upgrade/mod.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-05 15:13:50 +01:00
Tamo
66b7e0824e
Update crates/meilitool/src/upgrade/mod.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-05 15:13:40 +01:00
Tamo
f193c3a67c
Update crates/meilitool/src/main.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-05 15:13:32 +01:00
Tamo
48ab898ca2
fix the datetime of v1.9
2024-11-05 10:30:53 +01:00
F. Levi
a5dc783ffa
Merge with main branch
2024-11-05 10:56:17 +02:00
meili-bors[bot]
1b49b60486
Merge #5026
...
5026: test: improve performance of update_documents.rs r=dureuill a=PedroTurik
# Pull Request
## Related issue
Fixes one item from #4840
## What does this PR do?
- Applies the changes recommended on the issue for `meilisearch/tests/documents/update_documents.rs`
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Pedro Turik Firmino <pedroturik@gmail.com>
2024-11-05 08:37:44 +00:00
Pedro Turik Firmino
d0b1ba20cb
Improves usage of shared indexes
2024-11-04 17:26:50 -03:00
Tamo
99a9fde37f
push back the removed files
2024-11-04 17:55:55 +01:00
Tamo
106cc7fe3a
fmt
2024-11-04 17:51:40 +01:00
