1052 Commits

Author	SHA1	Message	Date
ManyTheFish	774ed28539	Fix Prefix FST when a document is modified	2024-10-03 11:12:26 +02:00
ManyTheFish	d79f75f630	Compute and Write external-documents-ids database	2024-10-03 11:11:56 +02:00
Clément Renault	b7a5ba100e	Move the ParallelIteratorExt into the parallel_iterator_ext module	2024-10-01 11:11:52 +02:00
Clément Renault	dead7a56a3	Keep the caches in the AppendOnlyVec	2024-10-01 11:11:39 +02:00
Clément Renault	0a8cb471df	Introduce the AppendOnlyVec struct for the parallel computing	2024-10-01 11:11:25 +02:00
Clément Renault	00e045b249	Rename and use the try_arc_for_each_try_init method	2024-10-01 11:11:25 +02:00
Clément Renault	d83c9a4074	Introduce the try_for_each_try_init method to be used with Arced Errors	2024-10-01 11:11:25 +02:00
Clément Renault	f3356ddaa4	Fix the errors when using the try_map_try_init method	2024-10-01 11:11:10 +02:00
Clément Renault	31de5c747e	WIP using try_map_try_init	2024-10-01 11:10:53 +02:00
Clément Renault	3843240940	Prefer using Ars instead of Options	2024-10-01 11:10:53 +02:00
Louis Dureuil	8cb5e7437d	try using try_map_try_init	2024-10-01 11:10:53 +02:00
Louis Dureuil	5b776556fe	Add ParallelIteratorExt	2024-10-01 11:10:53 +02:00
ManyTheFish	bb7a503e5d	Compute prefix databases ... We are now computing the prefix FST and a prefix delta in the Merger thread, after all the databases are written, the main thread will recompute the prefix databases based on the prefix delta without needing any grenad temporary file anymore	2024-10-01 09:57:06 +02:00
Louis Dureuil	64589278ac	Appease *some* of clippy warnings	2024-09-30 16:08:29 +02:00
ManyTheFish	8df6daf308	Remove fid_wordcount_docids.rs	2024-09-30 11:52:31 +02:00
ManyTheFish	5b552caf42	Fix position in insertions	2024-09-30 11:46:32 +02:00
ManyTheFish	2b51a63418	Remove dead code	2024-09-30 11:42:36 +02:00
Louis Dureuil	3d8024fb2b	write the weighted fields ids map	2024-09-30 11:35:03 +02:00
Louis Dureuil	4b0da0ff24	Fix inversion of field_id and position	2024-09-30 11:34:50 +02:00
ManyTheFish	960060ebdf	Fix fst builder when their is no previous FST	2024-09-25 16:53:00 +02:00
Clément Renault	3d244451df	Reduce the lru key size from 8 to 12 bytes	2024-09-25 16:14:13 +02:00
Clément Renault	5f53935c8a	Fix a bug in the Lru	2024-09-25 16:09:34 +02:00
Clément Renault	29a7623c3f	Fxi some logs	2024-09-25 15:57:50 +02:00
Clément Renault	e97041f7d0	Replace the Lru free list by a simple increment	2024-09-25 15:55:52 +02:00
Clément Renault	52d7f3ed1c	Reduce the lru key size from 20 to 8 bytes	2024-09-25 15:37:13 +02:00
Clément Renault	86d5e6d9ff	Use the new Lru	2024-09-25 14:54:56 +02:00
Clément Renault	759b9b1546	Introduce a new custom Lru	2024-09-25 14:49:12 +02:00
ManyTheFish	3f7a500f3b	Build prefix fst	2024-09-25 14:36:06 +02:00
ManyTheFish	974272f2e9	Merge branch 'main' into indexer-edition-2024	2024-09-25 07:41:16 +02:00
Clément Renault	7ad037841f	Move the tracing info to eprintln	2024-09-24 18:21:58 +02:00
Clément Renault	e0c7067355	Expose an IndexedParallelIterator to the index function	2024-09-24 17:24:59 +02:00
ManyTheFish	6e87332410	Change the way the FST is built	2024-09-24 16:28:31 +02:00
Clément Renault	2d1caf27df	Use eprintln to log	2024-09-24 15:59:50 +02:00
Clément Renault	7f148c127c	Measure the SmallVec efficacity	2024-09-24 15:32:15 +02:00
Clément Renault	4ce5d3d66d	Do not check before pushing in bitmaps	2024-09-24 09:43:16 +02:00
Clément Renault	42b093687d	Introduce the new PushOptimizedBitmap	2024-09-23 16:38:21 +02:00
Clément Renault	f00664247d	Add more stats about the channel message sent	2024-09-23 15:13:52 +02:00
Clément Renault	013acb3d93	Measure merger writer channel contention	2024-09-23 11:07:59 +02:00
Tamo	1113c42de0	fix broken comments	2024-09-19 16:18:36 +02:00
Tamo	b6b73fe41c	Update milli/src/update/settings.rs ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2024-09-19 15:41:14 +02:00
Tamo	163f8023a1	remove debug println	2024-09-19 12:13:25 +02:00
Tamo	633537ccd7	fix updating documents without updating the settings	2024-09-19 12:00:58 +02:00
Tamo	3f6301dbc9	fix the missing embedder name in the error message when trying to disable the binary quantization	2024-09-19 12:00:58 +02:00
Tamo	2b6952eda1	rename the ArroyReader to an ArroyWrapper since it can read and write	2024-09-19 12:00:58 +02:00
Tamo	79f29eed3c	fix the tests and the arroy_readers method	2024-09-19 12:00:58 +02:00
Tamo	cc45e264ca	implement the binary quantization in meilisearch	2024-09-19 12:00:56 +02:00
Clément Renault	f4ab1f168e	Prefer using Rc<str> than String when cloning a lot	2024-09-16 15:41:29 +02:00
ManyTheFish	1a0e962299	Replace hashmap by vectors in wpp	2024-09-16 15:01:20 +02:00
ManyTheFish	f13e076b8a	Use hashmap instead of Btree in wpp extractor	2024-09-16 14:40:40 +02:00
ManyTheFish	7ba49b849e	Extract and write facet databases	2024-09-16 09:35:16 +02:00