Commit Graph

18 Commits

Author SHA1 Message Date
Tamo c5322df519
Revert "Revert "Merge remote-tracking branch 'origin/main' into release-v1.7.1"" 2024-03-20 10:08:28 +01:00
Tamo 567194b925 Revert "Merge remote-tracking branch 'origin/main' into release-v1.7.1"
This reverts commit bd74cce86a, reversing
changes made to d2f77e88bd.
2024-03-19 16:56:21 +01:00
meili-bors[bot] 5ed7b6a0b2
Merge #4456
4456: Add Ollama as an embeddings provider r=dureuill a=jakobklemm

# Pull Request

## Related issue
[Related Discord Thread](https://discord.com/channels/1006923006964154428/1211977150316683305)

## What does this PR do?
- Adds Ollama as a provider of Embeddings besides HuggingFace and OpenAI under the name `ollama`
- Adds the environment variable `MEILI_OLLAMA_URL` to set the embeddings URL of an Ollama instance with a default value of `http://localhost:11434/api/embeddings` if no variable is set
- Changes some of the structs and functions in `openai.rs` to be public so that they can be shared.
- Added more error variants for Ollama specific errors
- It uses the model `nomic-embed-text` as default, but any string value is allowed, however it won't automatically check if the model actually exists or is an embedding model

Tested against Ollama version `v0.1.27` and the `nomic-embed-text` model.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Co-authored-by: Jakob Klemm <jakob@jeykey.net>
Co-authored-by: Louis Dureuil <louis.dureuil@gmail.com>
2024-03-13 08:48:47 +00:00
Louis Dureuil 0c216048b5
Cap timeout duration 2024-03-05 12:19:25 +01:00
Louis Dureuil 36d17110d8
openai: Handle BAD_GETAWAY, be more resilient to failure 2024-03-05 12:18:54 +01:00
Jakob Klemm d3004d8040
Implemented Ollama as an embeddings provider
Initial prototype of Ollama embeddings actually working, error handlign / retries still missing.

Allow model to be any String and require dimensions parameter

Fixed rustfmt formatting issues

There were some formatting issues in the initial PR and this should not make the changes comply with the Rust style guidelines

Because I accidentally didn't follow the style guide for commits in my commit messages I squashed them into one to comply
2024-03-04 15:09:43 +01:00
Tamo e773dfa9ba
get rids of log in milli and add logs for the bucket sort 2024-02-08 15:04:05 +01:00
Louis Dureuil a1caac9bfb
Correct distribution shifts for new models 2024-02-07 15:09:16 +01:00
Louis Dureuil 32ee05ccef
Fix default dimensions for models 2024-02-07 11:52:09 +01:00
Louis Dureuil 74c180267e
pass dimensions only when defined 2024-02-07 11:52:08 +01:00
Louis Dureuil 7ae4013478
Make sure the overriden dimensions are always used when embedding 2024-02-07 11:51:44 +01:00
Gosti fb705116a6
feat: add new models and ability to override dimensions 2024-02-07 11:51:42 +01:00
Louis Dureuil fbf5f2a392
Don't use a runtime in extract_embedder, use it only for OpenAI 2024-02-01 10:33:27 +01:00
Louis Dureuil 393216bf30
Flatten embedders settings 2023-12-20 17:16:43 +01:00
Louis Dureuil 87bba98bd8
Various changes
- fixed seed for arroy
- check vector dimensions as soon as it is provided to search
- don't embed whitespace
2023-12-14 16:08:42 +01:00
Louis Dureuil a4536b1381
Small adjustments to respect the spec 2023-12-14 16:08:42 +01:00
Louis Dureuil 922a640188
WIP multi embedders
fixed template bugs
2023-12-14 16:08:41 +01:00
Louis Dureuil 13c2c6c16b
Small commit to add hybrid search and autoembedding 2023-12-14 16:07:48 +01:00