Commit Graph

8668 Commits

Author SHA1 Message Date
ManyTheFish
bc51d6157a Fix transform reindexing path 2023-11-02 15:26:20 +01:00
ManyTheFish
1b4ff991c0 update typed chunks 2023-11-02 15:26:20 +01:00
ManyTheFish
4b64c33aa2 update vector extractor 2023-11-02 15:26:20 +01:00
ManyTheFish
12323d610e Change the original document sorter key from the internal docid to a concatenation of the internal and the external docid 2023-11-02 15:26:20 +01:00
Clément Renault
44e9033b3a
Merge pull request #4181 from meilisearch/diff-indexing-parallel-transform
Use rayon to sort entries in parallel
2023-11-02 15:16:10 +01:00
Clément Renault
4d864f0702
Always sort internal Sorter entries in parallel 2023-11-02 14:47:43 +01:00
Clément Renault
b10c060bf7
Cleanup TOML 2023-11-01 14:03:04 +01:00
Clément Renault
e507ef5932
Slow the logging down 2023-11-01 13:49:32 +01:00
Clément Renault
c71b1d33ae
Sort entries using rayon in the transform sorters 2023-11-01 11:07:16 +01:00
Clément Renault
0fc446c62f
Add more timing logs to the Transform 2023-11-01 11:07:16 +01:00
Louis Dureuil
0fb6acefc3
Add snapshots for facets 2023-10-31 17:11:08 +01:00
Louis Dureuil
b1d1355b69
remove tests on soft-deleted 2023-10-31 16:36:27 +01:00
Louis Dureuil
f19332466e
Extract field value as values instead of Option<Value> 2023-10-31 16:36:27 +01:00
Louis Dureuil
03ddb4f310
use deladd in facet update tests 2023-10-31 16:36:27 +01:00
Louis Dureuil
c855cc2721
Remove unused test 2023-10-31 16:36:27 +01:00
Louis Dureuil
da0503ef80
Fix document count 2023-10-31 16:36:27 +01:00
ManyTheFish
94206b0055 Update tests 2023-10-31 13:48:47 +01:00
Louis Dureuil
b40253bf18
update snapshots 2023-10-31 10:30:48 +01:00
Louis Dureuil
d8bf3f3fc2
Remove unused snapshots 2023-10-31 10:12:49 +01:00
Louis Dureuil
9d59e8011a
fix some tests 2023-10-31 10:08:36 +01:00
Louis Dureuil
dad78cbf8d
Bulk facet remove deletes keys from DB when value empty 2023-10-31 09:53:55 +01:00
Louis Dureuil
4e91707a06
Rename test 2023-10-31 09:41:17 +01:00
Louis Dureuil
de10f20732
Fix field distribution again 2023-10-30 17:47:22 +01:00
Louis Dureuil
be395c7944
Change order of arguments to tokenizer_builder 2023-10-30 16:26:29 +01:00
Louis Dureuil
9fedd8101a
Fix tests 2023-10-30 15:11:07 +01:00
Louis Dureuil
54d07a8da3
Update field distribution taking into account both deletions and additions 2023-10-30 14:47:51 +01:00
Louis Dureuil
58690dfb19
Fix tests compilation after changes to ExternalDocumentsIds API 2023-10-30 13:34:07 +01:00
Louis Dureuil
abf424ebfc
Remove unused FromIterator 2023-10-30 11:41:56 +01:00
Clément Renault
dfab6293c9
Use an LMDB database to store the external documents ids 2023-10-30 11:41:23 +01:00
Louis Dureuil
fdf3f7f627
Fix facet distribution test 2023-10-30 11:41:23 +01:00
Louis Dureuil
6260cff65f
Actually delete documents from DB when the merge function says so 2023-10-30 11:41:22 +01:00
Louis Dureuil
8e0d9c9a5e
Recover delete_documents tests that were too eagerly deleted 2023-10-30 11:41:22 +01:00
Louis Dureuil
ae4ec8ea55
Add delete_document_using_wtxn to TempIndex 2023-10-30 11:41:22 +01:00
Louis Dureuil
652ac3052d
use new iterator in batch 2023-10-30 11:41:22 +01:00
Louis Dureuil
9a2dccc3bc
Add iterator to find external ids of a bitmap of internal ids 2023-10-30 11:41:22 +01:00
Louis Dureuil
a35988550c
Fix some snapshots 2023-10-30 11:41:22 +01:00
Louis Dureuil
e78281785c
Actually execute the transform even if there are only documents to delete 2023-10-30 11:41:22 +01:00
Louis Dureuil
3c15881818
Add simple delete test 2023-10-30 11:41:22 +01:00
Louis Dureuil
73c06d31d9
snapshot always display stuff in consistent order 2023-10-30 11:41:22 +01:00
Louis Dureuil
290e773d23
remove more warnings and fix some tests 2023-10-30 11:41:22 +01:00
Louis Dureuil
fa6c7f65ca
Add TmpIndex::delete_documents 2023-10-30 11:41:22 +01:00
Louis Dureuil
113527f466
Remove soft-deleted related methods from Index 2023-10-30 11:41:22 +01:00
Louis Dureuil
c534a1b687
Stop using delete documents pipeline in batch runner 2023-10-30 11:41:22 +01:00
Louis Dureuil
2263dff02b
Stop using removed delete pipelines almost everywhere 2023-10-30 11:41:22 +01:00
Louis Dureuil
d651b3ef01
Remove delete documents files 2023-10-30 11:41:20 +01:00
ManyTheFish
762b0b47e6
Use deladd merging function in chunks mergers 2023-10-30 11:40:20 +01:00
Louis Dureuil
01d5eedf2f
Remove some warnings 2023-10-30 11:40:20 +01:00
Louis Dureuil
073f89db79
Fix facet tests 2023-10-30 11:40:20 +01:00
Louis Dureuil
8370fbc92b
Fix snaps 2023-10-30 11:40:20 +01:00
Louis Dureuil
85f42fbc03
Handle external to internal id mapping from TypedChunk::Documents 2023-10-30 11:40:20 +01:00