51 Commits

Author	SHA1	Message	Date
Clément Renault	b7a5ba100e	Move the ParallelIteratorExt into the parallel_iterator_ext module	2024-10-01 11:11:52 +02:00
Clément Renault	dead7a56a3	Keep the caches in the AppendOnlyVec	2024-10-01 11:11:39 +02:00
Clément Renault	00e045b249	Rename and use the try_arc_for_each_try_init method	2024-10-01 11:11:25 +02:00
Clément Renault	d83c9a4074	Introduce the try_for_each_try_init method to be used with Arced Errors	2024-10-01 11:11:25 +02:00
Clément Renault	f3356ddaa4	Fix the errors when using the try_map_try_init method	2024-10-01 11:11:10 +02:00
Clément Renault	31de5c747e	WIP using try_map_try_init	2024-10-01 11:10:53 +02:00
Louis Dureuil	64589278ac	Appease *some* of clippy warnings	2024-09-30 16:08:29 +02:00
ManyTheFish	8df6daf308	Remove fid_wordcount_docids.rs	2024-09-30 11:52:31 +02:00
ManyTheFish	5b552caf42	Fix position in insertions	2024-09-30 11:46:32 +02:00
ManyTheFish	2b51a63418	Remove dead code	2024-09-30 11:42:36 +02:00
Louis Dureuil	4b0da0ff24	Fix inversion of field_id and position	2024-09-30 11:34:50 +02:00
Clément Renault	3d244451df	Reduce the lru key size from 8 to 12 bytes	2024-09-25 16:14:13 +02:00
Clément Renault	5f53935c8a	Fix a bug in the Lru	2024-09-25 16:09:34 +02:00
Clément Renault	29a7623c3f	Fxi some logs	2024-09-25 15:57:50 +02:00
Clément Renault	e97041f7d0	Replace the Lru free list by a simple increment	2024-09-25 15:55:52 +02:00
Clément Renault	52d7f3ed1c	Reduce the lru key size from 20 to 8 bytes	2024-09-25 15:37:13 +02:00
Clément Renault	86d5e6d9ff	Use the new Lru	2024-09-25 14:54:56 +02:00
Clément Renault	e0c7067355	Expose an IndexedParallelIterator to the index function	2024-09-24 17:24:59 +02:00
Clément Renault	2d1caf27df	Use eprintln to log	2024-09-24 15:59:50 +02:00
Clément Renault	7f148c127c	Measure the SmallVec efficacity	2024-09-24 15:32:15 +02:00
Clément Renault	4ce5d3d66d	Do not check before pushing in bitmaps	2024-09-24 09:43:16 +02:00
Clément Renault	42b093687d	Introduce the new PushOptimizedBitmap	2024-09-23 16:38:21 +02:00
Clément Renault	f4ab1f168e	Prefer using Rc<str> than String when cloning a lot	2024-09-16 15:41:29 +02:00
ManyTheFish	1a0e962299	Replace hashmap by vectors in wpp	2024-09-16 15:01:20 +02:00
ManyTheFish	f13e076b8a	Use hashmap instead of Btree in wpp extractor	2024-09-16 14:40:40 +02:00
ManyTheFish	7ba49b849e	Extract and write facet databases	2024-09-16 09:35:16 +02:00
ManyTheFish	eb9a20ff0b	Fix fid_word_docids extraction	2024-09-12 11:08:18 +02:00
ManyTheFish	2b317c681b	Build mergers in parallel	2024-09-11 11:49:26 +02:00
ManyTheFish	39b5990f64	Mutualize tokenization	2024-09-11 10:22:38 +02:00
ManyTheFish	f69688e8f7	Fix several warnings in extractors and remove unreachable macros	2024-09-09 14:52:50 +02:00
ManyTheFish	8fd99b111b	Add tracing timers logs	2024-09-05 18:00:22 +02:00
Clément Renault	f6b3d1f9a5	Increase some channel sizes	2024-09-05 15:12:07 +02:00
Clément Renault	73ce67862d	Use the word pair proximity and fid word count docids extractors ... Co-authored-by: ManyTheFish <many@meilisearch.com>	2024-09-05 10:56:22 +02:00
Clément Renault	0fc02f7351	Move the facet extraction to dedicated modules	2024-09-05 10:32:27 +02:00
ManyTheFish	34f11e3380	Implement word count and word pair proximity extractors	2024-09-05 10:30:39 +02:00
Clément Renault	27308eaab1	Import the facet extractors	2024-09-04 17:58:15 +02:00
Clément Renault	b33ec9ba3f	Introduce the FieldIdFacetIsNullDocidsExtractor	2024-09-04 17:50:08 +02:00
Clément Renault	9c0a1cd9fd	Introduce the FieldIdFacetExistsDocidsExtractor	2024-09-04 17:48:49 +02:00
Clément Renault	0b061f1e70	Introduce the FieldIdFacetIsEmptyDocidsExtractor	2024-09-04 17:40:24 +02:00
Clément Renault	19d937ab21	Introduce the facet extractors	2024-09-04 17:03:54 +02:00
Clément Renault	6d74fb0229	Introduce the WordFidWordDocids database	2024-09-04 11:40:55 +02:00
ManyTheFish	1eb75a1040	remove milli/src/update/new/extract/tokenize_document.rs	2024-09-04 11:40:26 +02:00
Clément Renault	3b82d8b5b9	Fix the cache to serialize entries correctly	2024-09-04 10:55:36 +02:00
ManyTheFish	781a186f75	remove milli/src/update/new/extract/extract_word_docids.rs	2024-09-04 10:28:31 +02:00
ManyTheFish	6a399556b5	Implement more searchable extractor	2024-09-04 10:20:18 +02:00
Clément Renault	27b4cab857	Extract and write the documents and words fst in the database	2024-09-04 09:59:19 +02:00
ManyTheFish	da61408e52	Remove unimplemented from document changes	2024-09-03 15:14:16 +02:00
ManyTheFish	fe69385bd7	Fix tokenizer test	2024-09-03 14:24:37 +02:00
Clément Renault	c1557734dc	Use the GlobalFieldsIdsMap everywhere and write it to disk ... Co-authored-by: Dureuill <louis@meilisearch.com> Co-authored-by: ManyTheFish <many@meilisearch.com>	2024-09-03 12:01:01 +02:00
ManyTheFish	c50d3edc4a	Integrate first searchable exctrator	2024-09-03 11:02:39 +02:00