104 Commits

Author	SHA1	Message	Date
ManyTheFish	974272f2e9	Merge branch 'main' into indexer-edition-2024	2024-09-25 07:41:16 +02:00
Tamo	3f6301dbc9	fix the missing embedder name in the error message when trying to disable the binary quantization	2024-09-19 12:00:58 +02:00
Tamo	2b6952eda1	rename the ArroyReader to an ArroyWrapper since it can read and write	2024-09-19 12:00:58 +02:00
Tamo	cc45e264ca	implement the binary quantization in meilisearch	2024-09-19 12:00:56 +02:00
Louis Dureuil	3c5e363554	Remove default embedders	2024-09-17 16:30:43 +02:00
Louis Dureuil	f18e9cb7b3	Change openai default model	2024-09-09 13:09:35 +02:00
Louis Dureuil	ed19b7c3c3	Only reindex if the size increased	2024-09-03 12:07:59 +02:00
Louis Dureuil	1ac008926b	Add maxBytes parameter	2024-09-03 12:07:15 +02:00
Louis Dureuil	21296190a3	Reindex embedders	2024-09-02 13:00:53 +02:00
Clément Renault	0c57cf7565	Replace obkv with the temporary new version of it	2024-08-30 11:53:58 +02:00
Louis Dureuil	5aa6cb3600	Specialize authorized error message depending on config source	2024-07-31 15:03:44 +02:00
Louis Dureuil	9b7764575b	openai: don't pass apiKey when it is empty	2024-07-31 15:03:44 +02:00
Louis Dureuil	553440632e	Introduce Setting::some_or_not_set	2024-07-25 12:01:52 +02:00
Louis Dureuil	7a347966da	Allow explicit dimensions for ollama	2024-07-25 12:01:51 +02:00
Louis Dureuil	4654d51e05	Add custom headers for REST embedder	2024-07-25 12:01:51 +02:00
Louis Dureuil	4b74803dae	Change parameters in vector settings	2024-07-24 14:34:17 +02:00
Louis Dureuil	d731fa661b	ollama and openai use new EmbedderOptions	2024-07-24 14:34:17 +02:00
Louis Dureuil	a1beddd5d9	rest embedder: use json_template	2024-07-24 14:34:17 +02:00
Louis Dureuil	4109182ca4	Add json_template module	2024-07-24 14:34:12 +02:00
Louis Dureuil	1a297c048e	Error changes	2024-07-24 14:34:12 +02:00
Louis Dureuil	303e601b87	HuggingFace: Clearer error message when a model is not supported	2024-07-23 15:13:22 +02:00
Louis Dureuil	24240934f9	Improve errors when indexing documents with a user provided embedder	2024-07-16 13:39:01 +02:00
Louis Dureuil	f4c94ac57f	manual embedders: limit max size of errors to 250	2024-07-16 13:39:01 +02:00
Louis Dureuil	4087a88dbe	rest|ollama|openai: increase tries to 10 + randomize retry duration	2024-07-16 13:39:00 +02:00
Louis Dureuil	5adacf2f45	OpenAI: embed only the first MAX_TOKENS tokens	2024-07-16 13:39:00 +02:00
Louis Dureuil	65d0c32aa7	Allow overriding OpenAI's url	2024-07-16 13:39:00 +02:00
hanbings	0a40a98bb6	Make milli use edition 2021 (#4770) ... * Make milli use edition 2021 * Add lifetime annotations to milli. * Run cargo fmt	2024-07-09 17:25:39 +02:00
Tamo	ce08dc509b	add more tests and improve the location of the error	2024-06-27 11:51:45 +02:00
Tamo	1daaed163a	Make _vectors.:embedding.regenerate mandatory + tests + error messages	2024-06-27 11:04:58 +02:00
Louis Dureuil	e35ef31738	Small changes following review	2024-06-13 14:20:48 +02:00
Louis Dureuil	3bc8f81abc	user_provided => regenerate	2024-06-12 18:12:20 +02:00
Louis Dureuil	d0b05ae691	Add EmbedderAction to settings	2024-06-12 14:50:54 +02:00
Louis Dureuil	e9bf4eb100	Reformulate ParsedVectorsDiff in terms of VectorState	2024-06-12 14:11:44 +02:00
Louis Dureuil	b368105272	Add EmbedderConfigs::into_inner	2024-06-12 14:11:44 +02:00
Tamo	31a793d226	fix the regeneration of the embeddings in the search	2024-06-06 11:39:29 +02:00
Tamo	d85ab23b82	rename all occurences of user_defined to user_provided for consistency	2024-06-06 11:39:29 +02:00
Tamo	b7349910d9	implements mor review comments	2024-06-06 11:39:29 +02:00
Tamo	b867829ef1	remove useless dbg	2024-06-06 11:39:29 +02:00
Tamo	5d50850e12	always push the user defined vectors in arroy	2024-06-06 11:39:29 +02:00
Tamo	04f6523f3c	expose a new parameter to retrieve the embedders at search time	2024-06-06 11:36:11 +02:00
Tamo	84e498299b	Remove the vectors from the documents database	2024-06-06 11:36:11 +02:00
Louis Dureuil	d35278320e	Add support functions for accessing arroy writers and readers	2024-05-28 15:27:43 +02:00
Louis Dureuil	3412e7fbcf	"[]" is deserialized as 0 embedding rather than 1 embedding of dim 0	2024-05-22 12:25:21 +02:00
Louis Dureuil	16037e2169	Don't remove embedders that are not in the config from the document DB	2024-05-22 12:24:51 +02:00
Louis Dureuil	b17cb56dee	Test array of vectors	2024-05-20 14:44:10 +02:00
Louis Dureuil	52d9cb6e5a	Refactor vector indexing ... - use the parsed_vectors module - only parse `_vectors` once per document, instead of once per embedder per document	2024-05-20 10:36:17 +02:00
Louis Dureuil	98c811247e	Add parsed vectors module	2024-05-20 10:25:59 +02:00
Louis Dureuil	f4dd73ec8c	Destructure EmbedderOptions so we don't miss some options	2024-05-02 15:39:36 +02:00
meili-bors[bot]	c793b6ef6d	Merge #4600 ... 4600: Fix embedders api r=ManyTheFish a=ManyTheFish # Pull Request ## Related issue Fixes #4594 Fixes #4595 Co-authored-by: ManyTheFish <many@meilisearch.com>	2024-04-25 13:16:33 +00:00
Clément Renault	d4aeff92d0	Introduce the ThreadPoolNoAbort wrapper	2024-04-24 16:40:12 +02:00