703 Commits

Author	SHA1	Message	Date
ManyTheFish	bc51d6157a	Fix transform reindexing path	2023-11-02 15:26:20 +01:00
ManyTheFish	1b4ff991c0	update typed chunks	2023-11-02 15:26:20 +01:00
ManyTheFish	4b64c33aa2	update vector extractor	2023-11-02 15:26:20 +01:00
ManyTheFish	12323d610e	Change the original document sorter key from the internal docid to a concatenation of the internal and the external docid	2023-11-02 15:26:20 +01:00
Clément Renault	4d864f0702	Always sort internal Sorter entries in parallel	2023-11-02 14:47:43 +01:00
Clément Renault	c71b1d33ae	Sort entries using rayon in the transform sorters	2023-11-01 11:07:16 +01:00
Clément Renault	0fc446c62f	Add more timing logs to the Transform	2023-11-01 11:07:16 +01:00
Louis Dureuil	0fb6acefc3	Add snapshots for facets	2023-10-31 17:11:08 +01:00
Louis Dureuil	b1d1355b69	remove tests on soft-deleted	2023-10-31 16:36:27 +01:00
Louis Dureuil	f19332466e	Extract field value as values instead of Option<Value>	2023-10-31 16:36:27 +01:00
Louis Dureuil	03ddb4f310	use deladd in facet update tests	2023-10-31 16:36:27 +01:00
Louis Dureuil	da0503ef80	Fix document count	2023-10-31 16:36:27 +01:00
Louis Dureuil	b40253bf18	update snapshots	2023-10-31 10:30:48 +01:00
Louis Dureuil	d8bf3f3fc2	Remove unused snapshots	2023-10-31 10:12:49 +01:00
Louis Dureuil	9d59e8011a	fix some tests	2023-10-31 10:08:36 +01:00
Louis Dureuil	dad78cbf8d	Bulk facet remove deletes keys from DB when value empty	2023-10-31 09:53:55 +01:00
Louis Dureuil	4e91707a06	Rename test	2023-10-31 09:41:17 +01:00
Louis Dureuil	de10f20732	Fix field distribution again	2023-10-30 17:47:22 +01:00
Louis Dureuil	be395c7944	Change order of arguments to tokenizer_builder	2023-10-30 16:26:29 +01:00
Louis Dureuil	9fedd8101a	Fix tests	2023-10-30 15:11:07 +01:00
Louis Dureuil	54d07a8da3	Update field distribution taking into account both deletions and additions	2023-10-30 14:47:51 +01:00
Louis Dureuil	58690dfb19	Fix tests compilation after changes to ExternalDocumentsIds API	2023-10-30 13:34:07 +01:00
Louis Dureuil	abf424ebfc	Remove unused FromIterator	2023-10-30 11:41:56 +01:00
Clément Renault	dfab6293c9	Use an LMDB database to store the external documents ids	2023-10-30 11:41:23 +01:00
Louis Dureuil	fdf3f7f627	Fix facet distribution test	2023-10-30 11:41:23 +01:00
Louis Dureuil	6260cff65f	Actually delete documents from DB when the merge function says so	2023-10-30 11:41:22 +01:00
Louis Dureuil	8e0d9c9a5e	Recover delete_documents tests that were too eagerly deleted	2023-10-30 11:41:22 +01:00
Louis Dureuil	a35988550c	Fix some snapshots	2023-10-30 11:41:22 +01:00
Louis Dureuil	e78281785c	Actually execute the transform even if there are only documents to delete	2023-10-30 11:41:22 +01:00
Louis Dureuil	290e773d23	remove more warnings and fix some tests	2023-10-30 11:41:22 +01:00
Louis Dureuil	113527f466	Remove soft-deleted related methods from Index	2023-10-30 11:41:22 +01:00
Louis Dureuil	c534a1b687	Stop using delete documents pipeline in batch runner	2023-10-30 11:41:22 +01:00
Louis Dureuil	2263dff02b	Stop using removed delete pipelines almost everywhere	2023-10-30 11:41:22 +01:00
Louis Dureuil	d651b3ef01	Remove delete documents files	2023-10-30 11:41:20 +01:00
ManyTheFish	762b0b47e6	Use deladd merging function in chunks mergers	2023-10-30 11:40:20 +01:00
Louis Dureuil	01d5eedf2f	Remove some warnings	2023-10-30 11:40:20 +01:00
Louis Dureuil	073f89db79	Fix facet tests	2023-10-30 11:40:20 +01:00
Louis Dureuil	85f42fbc03	Handle external to internal id mapping from TypedChunk::Documents	2023-10-30 11:40:20 +01:00
Louis Dureuil	c6b3c18c85	WIP: Comment out document deletion in other pipelines than update ... TODO: fix calls to DELETE route	2023-10-30 11:40:20 +01:00
Louis Dureuil	946c762d28	WIP: reset documents in TypedChunk::Documents	2023-10-30 11:40:20 +01:00
Louis Dureuil	cda6ca1ee6	Remove TypedChunk::NewDocumentIds	2023-10-30 11:40:18 +01:00
Louis Dureuil	696fcf4d18	Fix document insertion into LMDB	2023-10-30 11:39:31 +01:00
ManyTheFish	476e4d3dbe	Use value buffer instead of the initial value when writting the final result in the sorter	2023-10-30 11:39:31 +01:00
Clément Renault	576fa9c6da	Remove useless comment	2023-10-30 11:39:31 +01:00
Kerollmops	77dcbff6b2	Remove and Insert the DelAdd geo points	2023-10-30 11:39:31 +01:00
Kerollmops	544440c363	Ignore geo fields when the Del and Add content is the same	2023-10-30 11:39:31 +01:00
Clément Renault	a3dae4db9b	Extract the geo fields DelAdd and generate a new DelAdd obkv with it	2023-10-30 11:39:31 +01:00
ManyTheFish	ba90a5ec0e	update extract fid word count docids	2023-10-30 11:39:31 +01:00
Louis Dureuil	b26dc9aabe	Explanatory code comment	2023-10-30 11:39:31 +01:00
Louis Dureuil	66abac9364	Use specialized KvReaderDelAdd type ... Co-authored-by: Clément Renault <clement@meilisearch.com>	2023-10-30 11:39:31 +01:00