2612 Commits

Author	SHA1	Message	Date
ManyTheFish	d675e73af1	Finish prefix databases	2024-10-14 11:12:10 +02:00
ManyTheFish	a2fbf2ea21	set updated at at the end of the indexing	2024-10-14 11:05:25 +02:00
ManyTheFish	132916f62c	Only run word pair proximity docids extraction if proximity_precision enables it	2024-10-14 11:05:25 +02:00
Louis Dureuil	8371819114	Some clippy related fixes	2024-10-14 10:58:37 +02:00
Louis Dureuil	6028d6ba43	Remove somme warnings	2024-10-10 22:42:37 +02:00
Louis Dureuil	68a2502388	Introduce indexer level bumpalo	2024-10-10 22:23:05 +02:00
Clément Renault	39b27e42be	Plug the deletion pipeline	2024-10-08 16:04:19 +02:00
Clément Renault	470c2272dd	Show much more stats about the LRU caches	2024-10-08 15:29:24 +02:00
Clément Renault	eb09dfed04	Avoid reallocation with the ThreadLocal pool	2024-10-07 16:41:17 +02:00
Clément Renault	83c09d0db0	Remove the now, useless AppendOnlyVec library	2024-10-07 16:38:45 +02:00
ManyTheFish	c11b7e5c0f	Reduce number of cache created by using thread_local	2024-10-07 15:58:16 +02:00
Clément Renault	58d96fbea3	Rename Node parent to next	2024-10-03 16:15:05 +02:00
Clément Renault	4665bfcb19	Move the parent assignation before the exchange operation	2024-10-03 16:14:23 +02:00
Clément Renault	a7a01646cf	Remove the useless Manually drop	2024-10-03 15:57:31 +02:00
Clément Renault	0409a26cd8	Replace the concurrent vec by a linked list	2024-10-03 15:15:29 +02:00
ManyTheFish	35f78b5423	TO REMOVE: usefull debug prints	2024-10-03 11:13:01 +02:00
ManyTheFish	14261f8f04	Integrate facet level bulk update ... Only the facet bulk update has been added so far, the incremental must be completely rewritten Factorize facet merging Fix facet level extraction	2024-10-03 11:13:00 +02:00
ManyTheFish	774ed28539	Fix Prefix FST when a document is modified	2024-10-03 11:12:26 +02:00
ManyTheFish	d79f75f630	Compute and Write external-documents-ids database	2024-10-03 11:11:56 +02:00
Clément Renault	b7a5ba100e	Move the ParallelIteratorExt into the parallel_iterator_ext module	2024-10-01 11:11:52 +02:00
Clément Renault	dead7a56a3	Keep the caches in the AppendOnlyVec	2024-10-01 11:11:39 +02:00
Clément Renault	0a8cb471df	Introduce the AppendOnlyVec struct for the parallel computing	2024-10-01 11:11:25 +02:00
Clément Renault	00e045b249	Rename and use the try_arc_for_each_try_init method	2024-10-01 11:11:25 +02:00
Clément Renault	d83c9a4074	Introduce the try_for_each_try_init method to be used with Arced Errors	2024-10-01 11:11:25 +02:00
Clément Renault	f3356ddaa4	Fix the errors when using the try_map_try_init method	2024-10-01 11:11:10 +02:00
Clément Renault	31de5c747e	WIP using try_map_try_init	2024-10-01 11:10:53 +02:00
Clément Renault	3843240940	Prefer using Ars instead of Options	2024-10-01 11:10:53 +02:00
Louis Dureuil	8cb5e7437d	try using try_map_try_init	2024-10-01 11:10:53 +02:00
Louis Dureuil	5b776556fe	Add ParallelIteratorExt	2024-10-01 11:10:53 +02:00
ManyTheFish	bb7a503e5d	Compute prefix databases ... We are now computing the prefix FST and a prefix delta in the Merger thread, after all the databases are written, the main thread will recompute the prefix databases based on the prefix delta without needing any grenad temporary file anymore	2024-10-01 09:57:06 +02:00
Louis Dureuil	64589278ac	Appease *some* of clippy warnings	2024-09-30 16:08:29 +02:00
ManyTheFish	8df6daf308	Remove fid_wordcount_docids.rs	2024-09-30 11:52:31 +02:00
ManyTheFish	5b552caf42	Fix position in insertions	2024-09-30 11:46:32 +02:00
ManyTheFish	2b51a63418	Remove dead code	2024-09-30 11:42:36 +02:00
Louis Dureuil	3d8024fb2b	write the weighted fields ids map	2024-09-30 11:35:03 +02:00
Louis Dureuil	4b0da0ff24	Fix inversion of field_id and position	2024-09-30 11:34:50 +02:00
Louis Dureuil	079f2b5de0	Format error messages consistently	2024-09-30 11:34:31 +02:00
ManyTheFish	960060ebdf	Fix fst builder when their is no previous FST	2024-09-25 16:53:00 +02:00
Clément Renault	3d244451df	Reduce the lru key size from 8 to 12 bytes	2024-09-25 16:14:13 +02:00
Clément Renault	5f53935c8a	Fix a bug in the Lru	2024-09-25 16:09:34 +02:00
Clément Renault	29a7623c3f	Fxi some logs	2024-09-25 15:57:50 +02:00
Clément Renault	e97041f7d0	Replace the Lru free list by a simple increment	2024-09-25 15:55:52 +02:00
Clément Renault	52d7f3ed1c	Reduce the lru key size from 20 to 8 bytes	2024-09-25 15:37:13 +02:00
Clément Renault	86d5e6d9ff	Use the new Lru	2024-09-25 14:54:56 +02:00
Clément Renault	759b9b1546	Introduce a new custom Lru	2024-09-25 14:49:12 +02:00
ManyTheFish	3f7a500f3b	Build prefix fst	2024-09-25 14:36:06 +02:00
ManyTheFish	974272f2e9	Merge branch 'main' into indexer-edition-2024	2024-09-25 07:41:16 +02:00
Clément Renault	7ad037841f	Move the tracing info to eprintln	2024-09-24 18:21:58 +02:00
Clément Renault	e0c7067355	Expose an IndexedParallelIterator to the index function	2024-09-24 17:24:59 +02:00
ManyTheFish	6e87332410	Change the way the FST is built	2024-09-24 16:28:31 +02:00