48 Commits

Author	SHA1	Message	Date
ManyTheFish	35f78b5423	TO REMOVE: usefull debug prints	2024-10-03 11:13:01 +02:00
ManyTheFish	14261f8f04	Integrate facet level bulk update ... Only the facet bulk update has been added so far, the incremental must be completely rewritten Factorize facet merging Fix facet level extraction	2024-10-03 11:13:00 +02:00
ManyTheFish	d79f75f630	Compute and Write external-documents-ids database	2024-10-03 11:11:56 +02:00
Clément Renault	b7a5ba100e	Move the ParallelIteratorExt into the parallel_iterator_ext module	2024-10-01 11:11:52 +02:00
Clément Renault	00e045b249	Rename and use the try_arc_for_each_try_init method	2024-10-01 11:11:25 +02:00
Clément Renault	d83c9a4074	Introduce the try_for_each_try_init method to be used with Arced Errors	2024-10-01 11:11:25 +02:00
Clément Renault	f3356ddaa4	Fix the errors when using the try_map_try_init method	2024-10-01 11:11:10 +02:00
Clément Renault	31de5c747e	WIP using try_map_try_init	2024-10-01 11:10:53 +02:00
Louis Dureuil	8cb5e7437d	try using try_map_try_init	2024-10-01 11:10:53 +02:00
ManyTheFish	bb7a503e5d	Compute prefix databases ... We are now computing the prefix FST and a prefix delta in the Merger thread, after all the databases are written, the main thread will recompute the prefix databases based on the prefix delta without needing any grenad temporary file anymore	2024-10-01 09:57:06 +02:00
Louis Dureuil	64589278ac	Appease *some* of clippy warnings	2024-09-30 16:08:29 +02:00
ManyTheFish	2b51a63418	Remove dead code	2024-09-30 11:42:36 +02:00
Louis Dureuil	3d8024fb2b	write the weighted fields ids map	2024-09-30 11:35:03 +02:00
Clément Renault	e0c7067355	Expose an IndexedParallelIterator to the index function	2024-09-24 17:24:59 +02:00
Clément Renault	013acb3d93	Measure merger writer channel contention	2024-09-23 11:07:59 +02:00
ManyTheFish	7ba49b849e	Extract and write facet databases	2024-09-16 09:35:16 +02:00
Clément Renault	f7652186e1	WIP geo fields	2024-09-12 18:01:02 +02:00
Clément Renault	b2f4e67c9a	Do not store useless updates	2024-09-12 15:38:31 +02:00
Clément Renault	ff5d3b59f5	Move the document id extraction to the primary key code	2024-09-12 12:01:42 +02:00
Clément Renault	3e9198ebaa	Support guessing primary key again	2024-09-11 17:25:40 +02:00
Clément Renault	2a0ad0982f	Fix the document counter	2024-09-11 15:59:36 +02:00
ManyTheFish	39b5990f64	Mutualize tokenization	2024-09-11 10:22:38 +02:00
Clément Renault	8287c2644f	Support CSV again	2024-09-10 21:10:28 +01:00
Clément Renault	c1c44a0b81	Impl serialize on TopLevelMap	2024-09-10 19:32:03 +01:00
Clément Renault	04596f3616	Move the TopLevelMap into a dedicated module	2024-09-10 18:01:17 +01:00
Clément Renault	24cb5839ad	Move the document changes sorting logic to a new trait	2024-09-10 17:37:52 +01:00
Clément Renault	8fd0afaaaa	Make sure we iterate over the payload documents in order	2024-09-06 08:09:08 +02:00
Clément Renault	72c6a21a30	Use raw JSON to read the payloads	2024-09-05 20:08:23 +02:00
Clément Renault	8412be4a7d	Cleanup CowStr and TopLevelMap struct	2024-09-05 18:32:55 +02:00
Louis Dureuil	10f09c531f	add some commented code to read from json with raw values	2024-09-05 18:22:16 +02:00
ManyTheFish	8fd99b111b	Add tracing timers logs	2024-09-05 18:00:22 +02:00
Clément Renault	f6b3d1f9a5	Increase some channel sizes	2024-09-05 15:12:07 +02:00
Clément Renault	73ce67862d	Use the word pair proximity and fid word count docids extractors ... Co-authored-by: ManyTheFish <many@meilisearch.com>	2024-09-05 10:56:22 +02:00
Clément Renault	0b061f1e70	Introduce the FieldIdFacetIsEmptyDocidsExtractor	2024-09-04 17:40:24 +02:00
Clément Renault	19d937ab21	Introduce the facet extractors	2024-09-04 17:03:54 +02:00
Clément Renault	98e48371c3	Factorize some stuff	2024-09-04 12:17:13 +02:00
Clément Renault	6d74fb0229	Introduce the WordFidWordDocids database	2024-09-04 11:40:55 +02:00
Clément Renault	27b4cab857	Extract and write the documents and words fst in the database	2024-09-04 09:59:19 +02:00
Clément Renault	52d32b4ee9	Move the channel sender in the closure to stop the merger thread	2024-09-03 16:08:33 +02:00
Clément Renault	c1557734dc	Use the GlobalFieldsIdsMap everywhere and write it to disk ... Co-authored-by: Dureuill <louis@meilisearch.com> Co-authored-by: ManyTheFish <many@meilisearch.com>	2024-09-03 12:01:01 +02:00
ManyTheFish	c50d3edc4a	Integrate first searchable exctrator	2024-09-03 11:02:39 +02:00
Clément Renault	5369bf4a62	Change some lifetimes	2024-09-02 19:51:22 +02:00
Clément Renault	bcb1aa3d22	Find a temporary solution to par into iter on an HashMap ... Spoiler: Do not use an HashMap but drain it into a Vec	2024-09-02 19:39:48 +02:00
Clément Renault	ab01679a8f	Remove the useless option from the document changes	2024-09-02 15:21:00 +02:00
Clément Renault	521775f788	I push for Many	2024-09-02 15:10:21 +02:00
Clément Renault	72e7b7846e	Renaming the indexers	2024-09-02 14:42:27 +02:00
Clément Renault	6526ce1208	Fix the merging of documents	2024-09-02 14:41:20 +02:00
Clément Renault	e639ec79d1	Move the indexers into their own modules	2024-09-02 10:42:19 +02:00