72ebac1fbb
Merge #4388
...
4388: Cap the maximum memory of the grenad sorters r=curquiza a=Kerollmops
This PR clamps the memory usage of the grenad sorters to a reasonable maximum. Grenad sorters are opened on multiple threads at a time. This can result in higher memory usage than expected, even though it shouldn't consume more than the memory available.
Fixes #4152 .
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-02-08 13:19:28 +00:00
a1caac9bfb
Correct distribution shifts for new models
2024-02-07 15:09:16 +01:00
88d03c56ab
Don't accept dimensions of 0 (ever) or dimensions greater than the default dimensions of the model
2024-02-07 11:52:09 +01:00
32ee05ccef
Fix default dimensions for models
2024-02-07 11:52:09 +01:00
74c180267e
pass dimensions only when defined
2024-02-07 11:52:08 +01:00
517f5332d6
Allow actually passing `dimensions` for OpenAI source
...
-> make sure the settings change is rejected or the settings task fails when the specified model doesn't support
overriding `dimensions` and the passed `dimensions` differs from the model's default dimensions.
2024-02-07 11:51:44 +01:00
9ac5750096
Retrieve the overriden dimensions from the configuration when fetching settings
2024-02-07 11:51:44 +01:00
7ae4013478
Make sure the overriden dimensions are always used when embedding
2024-02-07 11:51:44 +01:00
fb705116a6
feat: add new models and ability to override dimensions
2024-02-07 11:51:42 +01:00
053306c0e7
Try with 500MiB
2024-02-07 11:24:43 +01:00
9eeb75d501
Clamp the max memory of the grenad sorters to a reasonable maximum
2024-02-06 10:47:04 +01:00
fbf5f2a392
Don't use a runtime in extract_embedder, use it only for OpenAI
2024-02-01 10:33:27 +01:00
1555870088
Truncate HuggingFace vectors that are too long
2024-02-01 10:33:27 +01:00
9f8f3105d5
make clippy happy
2024-02-01 10:33:27 +01:00
318843aacd
add a bunch of tests and fix the error message when adding the geosearch as filterable/sortable while there is malformed documents in the DB
2024-02-01 10:33:27 +01:00
dff2707471
Use MatchingWords from keyword search instead of the one from vector search
2024-02-01 10:33:27 +01:00
c1bf33a112
Revert "Remove panic on the geosearch"
2024-01-25 18:51:19 +01:00
f692021bfc
Implement PR comments
2024-01-22 10:25:56 +01:00
84f49d76cd
Add cuda feature
2024-01-22 10:25:16 +01:00
0887186ecf
make clippy happy
2024-01-17 16:07:10 +01:00
7d190d8078
add a bunch of tests and fix the error message when adding the geosearch as filterable/sortable while there is malformed documents in the DB
2024-01-17 15:51:52 +01:00
01e2c3d6bb
Bump arroy to v0.2.0
2024-01-16 16:45:55 +01:00
9f9ad4cc05
Fix Clippy warnings
2024-01-16 15:27:24 +01:00
3ee7682fa7
Fix some integer comparisons
2024-01-16 15:22:23 +01:00
7f125bfb12
Update incompatible dependencies
2024-01-16 15:15:54 +01:00
5869ca7716
Upgrade all compatible dependencies
2024-01-16 15:05:03 +01:00
e93d36d5b9
Merge #4313
...
4313: Fix document formatting performances r=Kerollmops a=ManyTheFish
reduce the formatted option list to the attributes that should be formatted,
instead of all the attributes to display.
The time to compute the `format` list scales with the number of fields to format;
cumulated with `map_leaf_values` that iterates over all the nested fields, it gives a quadratic complexity:
`d*f` where `d` is the total number of fields to display and `f` is the total number of fields to format.
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-01-11 14:19:44 +00:00
5f5a486895
Reduce formatting time
2024-01-11 11:36:41 +01:00
5f4fc6c955
Add timer logs
2024-01-11 09:44:16 +01:00
3f3462ab62
Limit the number of values returned by the facet search
2024-01-10 16:54:08 +01:00
54ae6951eb
fix warning
2024-01-02 15:19:30 +01:00
0bf879fb88
Fix warning on rust stable
2023-12-20 17:48:09 +01:00
6ff81de401
Fix tests
2023-12-20 17:16:46 +01:00
9123370e90
Validate fused settings in settings task after fusing with existing setting
2023-12-20 17:16:46 +01:00
14b396d302
Add new errors
2023-12-20 17:16:45 +01:00
393216bf30
Flatten embedders settings
2023-12-20 17:16:43 +01:00
e249e4db7b
Change Setting::apply function signature
2023-12-20 17:15:24 +01:00
333ce12eb2
Fixed issue where the default revision is always the one we picked for the default model
2023-12-20 10:17:49 +01:00
942d49314c
Remove dependency that requires libstdc++
2023-12-18 22:17:18 +01:00
9e1b458010
Merge branch 'main' into change-proximity-precision-settings
2023-12-18 09:08:47 +01:00
6425996e36
Change the naming of attributeScale and wordScale into byAttribute and byWord
2023-12-14 16:31:00 +01:00
eb5cb91da2
Switch default from hf to openai
2023-12-14 16:19:46 +01:00
87bba98bd8
Various changes
...
- fixed seed for arroy
- check vector dimensions as soon as it is provided to search
- don't embed whitespace
2023-12-14 16:08:42 +01:00
217105b7da
hybrid search uses semantic ratio, error handling
2023-12-14 16:08:42 +01:00
9991152bbe
Add TODOs
2023-12-14 16:08:42 +01:00
a4536b1381
Small adjustments to respect the spec
2023-12-14 16:08:42 +01:00
5b51cb04af
Remove some settings
2023-12-14 16:08:42 +01:00
b8e4709dfa
Remove prompt strategy and fallback
2023-12-14 16:08:41 +01:00
806e5b6899
Tests pass
2023-12-14 16:08:41 +01:00
e0cc775dc4
Various changes
...
- DistributionShift in Search object (to be set from model in embed?)
- Fix issue where embedder index wasn't computed at search time
- Accept as default embedder either the "default" one, or the only embedder when there is only one
2023-12-14 16:08:41 +01:00
12940d79a9
WIP
...
- manual embedder
- multi embedders OK
- clippy + tests OK
2023-12-14 16:08:41 +01:00
922a640188
WIP multi embedders
...
fixed template bugs
2023-12-14 16:08:41 +01:00
d4715e0c4d
Fix same vector sort bug
2023-12-14 16:08:41 +01:00
11e2a2c1aa
Fix geosort bug
2023-12-14 16:08:41 +01:00
65e49b7092
Remove stuff, add distribution shift (WIP)
2023-12-14 16:08:38 +01:00
e56f160032
Actually pass embedders on reindex
2023-12-14 16:07:49 +01:00
687d92f217
prompt bifluor+
2023-12-14 16:07:49 +01:00
fb539f61fe
WIP
2023-12-14 16:07:49 +01:00
cb4ebe163e
WIP
2023-12-14 16:07:49 +01:00
dde3a04679
WIP arroy integration
2023-12-14 16:07:49 +01:00
13c2c6c16b
Small commit to add hybrid search and autoembedding
2023-12-14 16:07:48 +01:00
21bcf32109
Add candle and hg_hub, updating a lot of deps in the process
2023-12-14 16:07:48 +01:00
56571f762a
Merge remote-tracking branch 'origin/main' into tmp-release-v1.5.1
2023-12-13 11:57:01 +01:00
467b49153d
Implement proximityPrecision setting on milli side
2023-12-06 15:49:02 +01:00
bddc168d83
List TODOs
2023-12-06 14:59:23 +01:00
3b3fa38f27
Put the restrict list in a sub-struct
2023-11-28 18:37:57 +01:00
170e063b80
Remove the actix-web dependency from milli
2023-11-28 17:19:57 +01:00
d6c2ee15a9
Filter on attributes before computing the docids when attribute restriction is on
2023-11-28 14:55:29 +01:00
ec9b52d608
Rename copy_to_path to copy_to_file
2023-11-28 14:32:30 +01:00
34c67ac389
Remove the possibility to fail fetching the env info
2023-11-28 14:31:23 +01:00
d050c9b4ae
Only remap the main database once
2023-11-28 14:27:30 +01:00
7dd1226faf
Clarify an unreachable unwrap
2023-11-28 14:26:31 +01:00
548c8247c2
Create and use real error types in the codecs
2023-11-28 10:11:17 +01:00
d32eb11329
Move to the v0.20.0-alpha.9 of heed
2023-11-27 11:52:22 +01:00
58dac8af42
Remove the panics and unwraps
2023-11-23 15:00:48 +01:00
0dbf1a16ff
Make clippy happy
2023-11-23 14:11:38 +01:00
462b4c0080
Fix the tests
2023-11-23 12:07:35 +01:00
0d4482625a
Make the changes to use heed v0.20-alpha.6
2023-11-23 11:43:58 +01:00
56a0d91ecd
Update the heed dependency and lock file
2023-11-22 15:11:09 +01:00
7cb7e37ba8
Merge branch 'main' into tmp-release-v1.5.0
2023-11-21 16:30:46 +01:00
d3575fb028
Make into_del_add_obkv parameters more human readable
2023-11-20 16:10:39 +01:00
39cbb499c2
Small fixes
2023-11-20 10:20:39 +01:00
ebef6bc24d
Simplify documents database writing
2023-11-20 10:14:57 +01:00
d59b7db8d0
remove unused code
2023-11-20 10:10:45 +01:00
263e825619
Fix typos in comments
2023-11-20 10:06:29 +01:00
b0adc73ce6
Merge pull request #4207 from meilisearch/diff-indexing-prefix-databases
...
Diff indexing prefix databases
2023-11-14 16:04:05 +01:00
772964125d
Factor removal of document from DB
2023-11-13 13:51:22 +01:00
378deb0bef
Rename trait
2023-11-13 13:38:36 +01:00
1f36410541
Update tests
2023-11-13 13:36:39 +01:00
8c649d8061
Throw error when the vector search is sent with the wrong size
2023-11-13 09:57:42 +01:00
264b10ec20
Fixup documentation
2023-11-09 16:23:20 +01:00
3053e01c05
Batch::remove_documents_from_db_no_batch
2023-11-09 14:23:02 +01:00
b11c2afac0
Index::external_id_of
2023-11-09 14:22:43 +01:00
9cef800b2a
Enrich uses the new type
2023-11-09 14:22:05 +01:00
db2fb86b8b
Extract PrimaryKey logic to a type
2023-11-09 14:19:16 +01:00
882ab9cc85
remove warnings
2023-11-09 11:35:33 +01:00
5a9c96e1db
Compute word integer prefix cache
2023-11-09 11:34:26 +01:00
70ce40828c
Compute word docids prefix cache
2023-11-08 17:01:00 +01:00
688266c83e
Remove word pair proximity prefix cache and compute it at search time
2023-11-08 14:16:01 +01:00
6dab826908
Reactivate prefix databases
2023-11-08 13:58:01 +01:00
1e2fbc6a42
revert "REVERT ME: ignore prefix pair databases tests"
...
This reverts commit 1b2ea6cf19
2023-11-08 11:50:52 +01:00
cbaa54cafd
Fix clippy issues
2023-11-06 11:19:31 +01:00
1bccf2079e
Correctly mark non-tests as non-tests
2023-11-06 11:03:56 +01:00
1b2ea6cf19
REVERT ME: ignore prefix pair databases tests
2023-11-06 10:46:22 +01:00
1ad1fcc8c8
Remove all warnings
2023-11-06 10:31:14 +01:00
87610a5f98
Don't try to delete a document that is not in the database
2023-11-02 16:49:03 +01:00
ff522c919d
Fix the vector extractions for the diff indexing
2023-11-02 15:58:08 +01:00
bf0651f23c
Implement iter method on ExternalDocumentsIds
2023-11-02 15:38:00 +01:00
5b20e625f3
fix merge
2023-11-02 15:31:37 +01:00
bc51d6157a
Fix transform reindexing path
2023-11-02 15:26:20 +01:00
1b4ff991c0
update typed chunks
2023-11-02 15:26:20 +01:00
4b64c33aa2
update vector extractor
2023-11-02 15:26:20 +01:00
12323d610e
Change the original document sorter key from the internal docid to a concatenation of the internal and the external docid
2023-11-02 15:26:20 +01:00
4d864f0702
Always sort internal Sorter entries in parallel
2023-11-02 14:47:43 +01:00
b10c060bf7
Cleanup TOML
2023-11-01 14:03:04 +01:00
c71b1d33ae
Sort entries using rayon in the transform sorters
2023-11-01 11:07:16 +01:00
0fc446c62f
Add more timing logs to the Transform
2023-11-01 11:07:16 +01:00
0fb6acefc3
Add snapshots for facets
2023-10-31 17:11:08 +01:00
b1d1355b69
remove tests on soft-deleted
2023-10-31 16:36:27 +01:00
f19332466e
Extract field value as values instead of Option<Value>
2023-10-31 16:36:27 +01:00
03ddb4f310
use deladd in facet update tests
2023-10-31 16:36:27 +01:00
c855cc2721
Remove unused test
2023-10-31 16:36:27 +01:00
da0503ef80
Fix document count
2023-10-31 16:36:27 +01:00
94206b0055
Update tests
2023-10-31 13:48:47 +01:00
b40253bf18
update snapshots
2023-10-31 10:30:48 +01:00
d8bf3f3fc2
Remove unused snapshots
2023-10-31 10:12:49 +01:00
9d59e8011a
fix some tests
2023-10-31 10:08:36 +01:00
dad78cbf8d
Bulk facet remove deletes keys from DB when value empty
2023-10-31 09:53:55 +01:00
4e91707a06
Rename test
2023-10-31 09:41:17 +01:00
de10f20732
Fix field distribution again
2023-10-30 17:47:22 +01:00
be395c7944
Change order of arguments to tokenizer_builder
2023-10-30 16:26:29 +01:00
9fedd8101a
Fix tests
2023-10-30 15:11:07 +01:00
54d07a8da3
Update field distribution taking into account both deletions and additions
2023-10-30 14:47:51 +01:00
58690dfb19
Fix tests compilation after changes to ExternalDocumentsIds API
2023-10-30 13:34:07 +01:00
abf424ebfc
Remove unused FromIterator
2023-10-30 11:41:56 +01:00
dfab6293c9
Use an LMDB database to store the external documents ids
2023-10-30 11:41:23 +01:00
fdf3f7f627
Fix facet distribution test
2023-10-30 11:41:23 +01:00
6260cff65f
Actually delete documents from DB when the merge function says so
2023-10-30 11:41:22 +01:00
8e0d9c9a5e
Recover delete_documents tests that were too eagerly deleted
2023-10-30 11:41:22 +01:00
ae4ec8ea55
Add delete_document_using_wtxn to TempIndex
2023-10-30 11:41:22 +01:00
9a2dccc3bc
Add iterator to find external ids of a bitmap of internal ids
2023-10-30 11:41:22 +01:00
a35988550c
Fix some snapshots
2023-10-30 11:41:22 +01:00
e78281785c
Actually execute the transform even if there are only documents to delete
2023-10-30 11:41:22 +01:00
3c15881818
Add simple delete test
2023-10-30 11:41:22 +01:00
73c06d31d9
snapshot always display stuff in consistent order
2023-10-30 11:41:22 +01:00
290e773d23
remove more warnings and fix some tests
2023-10-30 11:41:22 +01:00
fa6c7f65ca
Add TmpIndex::delete_documents
2023-10-30 11:41:22 +01:00
113527f466
Remove soft-deleted related methods from Index
2023-10-30 11:41:22 +01:00
c534a1b687
Stop using delete documents pipeline in batch runner
2023-10-30 11:41:22 +01:00
2263dff02b
Stop using removed delete pipelines almost everywhere
2023-10-30 11:41:22 +01:00
d651b3ef01
Remove delete documents files
2023-10-30 11:41:20 +01:00
762b0b47e6
Use deladd merging function in chunks mergers
2023-10-30 11:40:20 +01:00
01d5eedf2f
Remove some warnings
2023-10-30 11:40:20 +01:00
073f89db79
Fix facet tests
2023-10-30 11:40:20 +01:00
8370fbc92b
Fix snaps
2023-10-30 11:40:20 +01:00
85f42fbc03
Handle external to internal id mapping from TypedChunk::Documents
2023-10-30 11:40:20 +01:00
c6b3c18c85
WIP: Comment out document deletion in other pipelines than update
...
TODO: fix calls to DELETE route
2023-10-30 11:40:20 +01:00
bafeb892a7
Modify Index after changes to ExternalDocumentsIds
2023-10-30 11:40:20 +01:00
8fb221dae3
Refactor ExternalDocumentsIds
...
- Remove soft deleted
- Add apply method that takes a list of operations to encapsulate modifications to the external -> internal mapping
2023-10-30 11:40:20 +01:00
946c762d28
WIP: reset documents in TypedChunk::Documents
2023-10-30 11:40:20 +01:00
cda6ca1ee6
Remove TypedChunk::NewDocumentIds
2023-10-30 11:40:18 +01:00
696fcf4d18
Fix document insertion into LMDB
2023-10-30 11:39:31 +01:00
476e4d3dbe
Use value buffer instead of the initial value when writting the final result in the sorter
2023-10-30 11:39:31 +01:00
576fa9c6da
Remove useless comment
2023-10-30 11:39:31 +01:00
77dcbff6b2
Remove and Insert the DelAdd geo points
2023-10-30 11:39:31 +01:00
544440c363
Ignore geo fields when the Del and Add content is the same
2023-10-30 11:39:31 +01:00
a3dae4db9b
Extract the geo fields DelAdd and generate a new DelAdd obkv with it
2023-10-30 11:39:31 +01:00
ba90a5ec0e
update extract fid word count docids
2023-10-30 11:39:31 +01:00
b26dc9aabe
Explanatory code comment
2023-10-30 11:39:31 +01:00
66abac9364
Use specialized `KvReaderDelAdd` type
...
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-10-30 11:39:31 +01:00
59f88c14b3
Simplify facet update after removing `Index::faceted_documents_ids`
2023-10-30 11:39:29 +01:00
14832cb324
Remove Index::faceted_documents_ids
2023-10-30 11:37:32 +01:00
04ec293024
Facet Incremental update
2023-10-30 11:37:30 +01:00
f67ff3a738
Facets Bulk update
2023-10-30 11:36:40 +01:00
560e8f5613
Introduce the CboRoaringBitmapCodec merge_deladd_into and use it
2023-10-30 11:34:55 +01:00
2d3f15f82c
Introduce a function to only serialize the Add side of a DelAdd obkv
2023-10-30 11:34:55 +01:00
40186bf403
Rename FieldIdWordCountDocids correctly
2023-10-30 11:34:50 +01:00
87e3d27878
update extract word pair proximity to support deladd obkvs
2023-10-30 11:34:02 +01:00
6bcf8b4f8c
update extract word position docids
2023-10-30 11:34:02 +01:00
46aa75abdb
update extract word docids
2023-10-30 11:34:02 +01:00
2597bbd107
Make script language docids map taking a tuple of roaring bitmaps expressing the deletions and the additions
2023-10-30 11:34:00 +01:00
e2bc054604
Update extract_facet_string_docids to support deladd obkvs
2023-10-30 11:32:36 +01:00
fcd3a1434d
Update extract_facet_number_docids to support deladd obkvs
2023-10-30 11:31:04 +01:00
a82dee21e0
Rename docid_fid into fid_docid
2023-10-30 11:31:02 +01:00
bc45c1206d
Implement all the facet extraction paths and simplify them
2023-10-30 11:29:08 +01:00
6ae4100f07
Generate the DelAdd for is_null, is_empty, and exists
2023-10-30 11:29:08 +01:00
0c47defeee
Work on fid docid facet values rewrite
2023-10-30 11:29:06 +01:00
313b16bec2
Support diff indexing on extract_docid_word_positions
2023-10-30 11:24:19 +01:00
1dd97578a8
Make the transform struct return diff-based documents obkvs
2023-10-30 11:22:07 +01:00
f5ef69293b
deactivate prefix dbs
2023-10-30 11:22:07 +01:00
1c5705c164
clean PR warnings
2023-10-30 11:22:05 +01:00
66c2c82a18
Split wpp in several sorters
2023-10-30 11:15:02 +01:00
28a8d0ccda
Fix word pair proximity
2023-10-30 11:15:02 +01:00
96be85396d
Use a vecDeque in wpp database
2023-10-30 11:15:02 +01:00
df9e5c8651
Generalize usage of CboRoaringBitmap codec to ease the use
2023-10-30 11:15:02 +01:00
b541d48847
Add buffer to the obkv writter
2023-10-30 11:15:02 +01:00
8ccf32d1a0
Compute word_fid_docids before word_docids and exact_word_docids
2023-10-30 11:15:02 +01:00
db1ca21231
add puffin in sorter into reeder function
2023-10-30 11:15:00 +01:00
11ea5acff9
Fix
2023-10-30 11:13:10 +01:00
8d77736a67
Fix fid_word_docids
2023-10-30 11:13:10 +01:00
meili-bors[bot]