Commit Graph

9199 Commits

Author SHA1 Message Date
Louis Dureuil
6260cff65f
Actually delete documents from DB when the merge function says so 2023-10-30 11:41:22 +01:00
Louis Dureuil
8e0d9c9a5e
Recover delete_documents tests that were too eagerly deleted 2023-10-30 11:41:22 +01:00
Louis Dureuil
ae4ec8ea55
Add delete_document_using_wtxn to TempIndex 2023-10-30 11:41:22 +01:00
Louis Dureuil
652ac3052d
use new iterator in batch 2023-10-30 11:41:22 +01:00
Louis Dureuil
9a2dccc3bc
Add iterator to find external ids of a bitmap of internal ids 2023-10-30 11:41:22 +01:00
Louis Dureuil
a35988550c
Fix some snapshots 2023-10-30 11:41:22 +01:00
Louis Dureuil
e78281785c
Actually execute the transform even if there are only documents to delete 2023-10-30 11:41:22 +01:00
Louis Dureuil
3c15881818
Add simple delete test 2023-10-30 11:41:22 +01:00
Louis Dureuil
73c06d31d9
snapshot always display stuff in consistent order 2023-10-30 11:41:22 +01:00
Louis Dureuil
290e773d23
remove more warnings and fix some tests 2023-10-30 11:41:22 +01:00
Louis Dureuil
fa6c7f65ca
Add TmpIndex::delete_documents 2023-10-30 11:41:22 +01:00
Louis Dureuil
113527f466
Remove soft-deleted related methods from Index 2023-10-30 11:41:22 +01:00
Louis Dureuil
c534a1b687
Stop using delete documents pipeline in batch runner 2023-10-30 11:41:22 +01:00
Louis Dureuil
2263dff02b
Stop using removed delete pipelines almost everywhere 2023-10-30 11:41:22 +01:00
Louis Dureuil
d651b3ef01
Remove delete documents files 2023-10-30 11:41:20 +01:00
ManyTheFish
762b0b47e6
Use deladd merging function in chunks mergers 2023-10-30 11:40:20 +01:00
Louis Dureuil
01d5eedf2f
Remove some warnings 2023-10-30 11:40:20 +01:00
Louis Dureuil
073f89db79
Fix facet tests 2023-10-30 11:40:20 +01:00
Louis Dureuil
8370fbc92b
Fix snaps 2023-10-30 11:40:20 +01:00
Louis Dureuil
85f42fbc03
Handle external to internal id mapping from TypedChunk::Documents 2023-10-30 11:40:20 +01:00
Louis Dureuil
c6b3c18c85
WIP: Comment out document deletion in other pipelines than update
TODO: fix calls to DELETE route
2023-10-30 11:40:20 +01:00
Louis Dureuil
bafeb892a7
Modify Index after changes to ExternalDocumentsIds 2023-10-30 11:40:20 +01:00
Louis Dureuil
8fb221dae3
Refactor ExternalDocumentsIds
- Remove soft deleted
- Add apply method that takes a list of operations to encapsulate modifications to the external -> internal mapping
2023-10-30 11:40:20 +01:00
Louis Dureuil
5be569e3e2
Update obkv 2023-10-30 11:40:20 +01:00
Louis Dureuil
946c762d28
WIP: reset documents in TypedChunk::Documents 2023-10-30 11:40:20 +01:00
Louis Dureuil
cda6ca1ee6
Remove TypedChunk::NewDocumentIds 2023-10-30 11:40:18 +01:00
Louis Dureuil
696fcf4d18
Fix document insertion into LMDB 2023-10-30 11:39:31 +01:00
ManyTheFish
476e4d3dbe
Use value buffer instead of the initial value when writting the final result in the sorter 2023-10-30 11:39:31 +01:00
Clément Renault
576fa9c6da
Remove useless comment 2023-10-30 11:39:31 +01:00
Kerollmops
77dcbff6b2
Remove and Insert the DelAdd geo points 2023-10-30 11:39:31 +01:00
Kerollmops
544440c363
Ignore geo fields when the Del and Add content is the same 2023-10-30 11:39:31 +01:00
Clément Renault
a3dae4db9b
Extract the geo fields DelAdd and generate a new DelAdd obkv with it 2023-10-30 11:39:31 +01:00
ManyTheFish
ba90a5ec0e
update extract fid word count docids 2023-10-30 11:39:31 +01:00
Louis Dureuil
b26dc9aabe
Explanatory code comment 2023-10-30 11:39:31 +01:00
Louis Dureuil
66abac9364
Use specialized KvReaderDelAdd type
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-10-30 11:39:31 +01:00
Louis Dureuil
59f88c14b3
Simplify facet update after removing Index::faceted_documents_ids 2023-10-30 11:39:29 +01:00
Louis Dureuil
14832cb324
Remove Index::faceted_documents_ids 2023-10-30 11:37:32 +01:00
Louis Dureuil
04ec293024
Facet Incremental update 2023-10-30 11:37:30 +01:00
Louis Dureuil
f67ff3a738
Facets Bulk update 2023-10-30 11:36:40 +01:00
Clément Renault
560e8f5613
Introduce the CboRoaringBitmapCodec merge_deladd_into and use it 2023-10-30 11:34:55 +01:00
Clément Renault
2d3f15f82c
Introduce a function to only serialize the Add side of a DelAdd obkv 2023-10-30 11:34:55 +01:00
Clément Renault
40186bf403
Rename FieldIdWordCountDocids correctly 2023-10-30 11:34:50 +01:00
ManyTheFish
87e3d27878
update extract word pair proximity to support deladd obkvs 2023-10-30 11:34:02 +01:00
ManyTheFish
6bcf8b4f8c
update extract word position docids 2023-10-30 11:34:02 +01:00
ManyTheFish
46aa75abdb
update extract word docids 2023-10-30 11:34:02 +01:00
ManyTheFish
2597bbd107
Make script language docids map taking a tuple of roaring bitmaps expressing the deletions and the additions 2023-10-30 11:34:00 +01:00
Clément Renault
e2bc054604
Update extract_facet_string_docids to support deladd obkvs 2023-10-30 11:32:36 +01:00
Clément Renault
fcd3a1434d
Update extract_facet_number_docids to support deladd obkvs 2023-10-30 11:31:04 +01:00
Clément Renault
a82dee21e0
Rename docid_fid into fid_docid 2023-10-30 11:31:02 +01:00
Clément Renault
bc45c1206d
Implement all the facet extraction paths and simplify them 2023-10-30 11:29:08 +01:00