Commit Graph

9772 Commits

Author SHA1 Message Date
meili-bors[bot]
c2c1ba39ee
Merge #4826
4826: Update Charabia v0.9.0 r=dureuill a=ManyTheFish

# Pull Request

## Related Changelog
https://github.com/meilisearch/charabia/releases/tag/v0.9.0

## Notable Change for Meilisearch
Adds all math symbols from https://www.compart.com/en/unicode/category/Sm to the default separator list.



Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-07-25 14:08:38 +00:00
ManyTheFish
35567b2137 Update Charabia v0.9.0 2024-07-25 16:02:14 +02:00
meili-bors[bot]
00c97c7152
Merge #4818
4818: Custom headers and QoL improvements r=ManyTheFish a=dureuill

# Pull Request

## Related issue
Fixes #4734 
Depends on #4815 

## What does this PR do?
- Adds custom headers for rest embedders ([public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#41354652885242c899def07e36a66d49))
- Quality of life: allow specifying `dimensions` for `ollama` embedders ([public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#37218531431343dab3d2d3a9a1937e9d)). As for `rest` embedders, specifying `dimensions` disables the "test" embedding when the embedder is spawned.
- Improve error message again when indexing documents that don't have a vector for a user-provided vector
  1. Remove the contents of the document
  2. Display the docid of the first document that triggered the error
  3. Indicate how many documents in that chunk suffered from the same issue for that embedder


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-25 13:33:11 +00:00
Louis Dureuil
d4ea7cc2a9
fix clippy 👉👈 2024-07-25 12:10:32 +02:00
Louis Dureuil
8532fe8afc
Fix tests 2024-07-25 12:10:32 +02:00
Louis Dureuil
2413592bbf
Display docid when there are documents without manual embeddings for a manual embedder 2024-07-25 12:10:32 +02:00
Louis Dureuil
553440632e
Introduce Setting::some_or_not_set 2024-07-25 12:01:52 +02:00
Louis Dureuil
7a347966da
Allow explicit dimensions for ollama 2024-07-25 12:01:51 +02:00
Louis Dureuil
6c598fa06d
test custom headers 2024-07-25 12:01:51 +02:00
Louis Dureuil
8338df0dbe
Fix tests 2024-07-25 12:01:51 +02:00
Louis Dureuil
4654d51e05
Add custom headers for REST embedder 2024-07-25 12:01:51 +02:00
Louis Dureuil
22ef2d877f
Ensure test server has a single indexing thread 2024-07-25 12:01:51 +02:00
meili-bors[bot]
76bc2c18e8
Merge #4819
4819: Language settings r=dureuill a=ManyTheFish

# Pull Request

## Related issue
Fixes #4749 

## What does this PR do?
- [Implement localized search](c0c6955c0d)
- [Implement localized attributes settings](bde827b055)

## Related PRD

- [PRD](https://www.notion.so/meilisearch/Define-language-settings-to-impact-relevancy-bee62e18b7584c4f87d18a7654855329)
- [Public usage](https://www.notion.so/meilisearch/v1-10-Language-settings-usage-26c5d98b553349d9abacbe7aff698e4e)


Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-25 09:00:33 +00:00
Louis Dureuil
59115fd058
Fix tests 2024-07-25 10:52:57 +02:00
ManyTheFish
a918561ac1
Fix PR comments 2024-07-25 10:52:56 +02:00
ManyTheFish
70d71581ee
fix clippy 2024-07-25 10:52:56 +02:00
ManyTheFish
4fbe048cbf
Update Cargo.lock 2024-07-25 10:52:56 +02:00
ManyTheFish
e06fbcc607
Update snapshots 2024-07-25 10:52:56 +02:00
ManyTheFish
04fa44e7eb
Implement localized attributes settings 2024-07-25 10:51:27 +02:00
ManyTheFish
90c0a6db7d
Implement localized search 2024-07-25 10:51:27 +02:00
ManyTheFish
d82f8fd904
Add tests 2024-07-25 10:51:27 +02:00
ManyTheFish
cc02920f2b
Update charabia 2024-07-25 10:51:27 +02:00
meili-bors[bot]
c26bd68de5
Merge #4815
4815: Rest embedder api mk2 r=ManyTheFish a=dureuill

# Pull Request

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4756

- [x] [REST API parameter names and behavior are unclear](https://github.com/meilisearch/documentation/pull/2824#issuecomment-2124073720)
  - unclear names are removed. There remain only two parameters: `request`, a template of what Meilisearch's request to the embedding server should be, and `response`, a template of what the embedding server's response to Meilisearch should look like
- [x] [Bad error message or bad default value when we don't specify the `query` parameter](85d8455c11/meilisearch/tests/vector/rest.rs (L105-L140))
  - The replacement for `query`, which is `request`, is now a mandatory parameter. Omitting it will result in the following error message : "`.embedders.rest`: Missing field `request` (note: this field is mandatory for source rest)", which is clear
- [x] [Bad error message when both `pathToEmbeddings` and `embeddingObject` are missing](2141cb3b69/meilisearch/tests/vector/rest.rs (L142-L178))
  - These parameters no longer exist. Now, the point of extraction is given directly by the location of an `{{embedding}}` placeholder in the `response` parameter.
- [x] [Unexpected error when we don't specify both `pathToEmbeddings` and `embeddingObject` (only once should be required)](2141cb3b69/meilisearch/tests/vector/rest.rs (L180-L260))
  - These parameters no longer exist. Now, the point of extraction is given directly by the location of an `{{embedding}}` placeholder in the `response` parameter.
- [x] [Should not panic when the dimensions specified do not work with the model](2141cb3b69/meilisearch/tests/vector/rest.rs (L262-L299))
  - This no longer panics, instead returns "While embedding documents for embedder `rest`: runtime error: was expecting embeddings of dimension `2`, got embeddings of dimensions `3`"
- [x] [Be more flexible on the type of data that is accepted](https://github.com/meilisearch/meilisearch/issues/4757#issuecomment-2201948531)
  - [x] Always accept arrays of embeddings even if `inputType` is set to `text`
    - This is controlled by the repeat placeholder `"{..}"`, an array of embeddings can be configured even if the input is not in an array.
  - [x] Accept arrays of result at the root level and texts/array of text at the root level.
    -  doable with `request: "{{text}}"` and `response: "{{embedding}}"` or `response: ["{{embedding}}"]` (see test `vector::rest::server_raw`)

## What does this PR do?
- [See public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#8de842673ffa4a139210094a89c1ec3e)
- Add new `milli::vector::json_template` module to parse JSON templates with an injection placeholder and a repeat placeholder
- Change rest embedder to use two JSON templates
- Change ollama and openai embedders to use the new rest embedder
- Update settings
- Update and add tests

## Breaking change

> [!CAUTION]
> This PR is a breaking change to the REST embedder.
> Importing a dump containing a REST embedder configuration will fail in v1.10 with an error: "Error: unknown field `query`, expected one of `source`, `model`, `revision`, `apiKey`, `dimensions`, `documentTemplate`, `url`, `request`, `response`, `distribution` at line 1 column 752".

Upgrade procedure:

1. Remove any embedder with source "rest"
2. Create a dump
3. Import that dump in a v1.10
4. Re-add any removed embedder, using the new settings.

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Louis Dureuil <louis.dureuil@xinra.net>
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-24 16:32:52 +00:00
Clément Renault
80fdea9afc
Merge pull request #4823 from meilisearch/explicit-check-bench
Explicitly check permissions when receiving a slash command
2024-07-24 17:34:07 +02:00
Louis Dureuil
e3faacd160
Explicitly check permissions when receiving a slash command 2024-07-24 17:09:25 +02:00
Tamo
988552e178
add tests on the rest embedder 2024-07-24 14:34:17 +02:00
Louis Dureuil
0d8199f3b7
Change parameters in milli settings 2024-07-24 14:34:17 +02:00
Louis Dureuil
4b74803dae
Change parameters in vector settings 2024-07-24 14:34:17 +02:00
Louis Dureuil
d731fa661b
ollama and openai use new EmbedderOptions 2024-07-24 14:34:17 +02:00
Louis Dureuil
a1beddd5d9
rest embedder: use json_template 2024-07-24 14:34:17 +02:00
Louis Dureuil
4109182ca4
Add json_template module 2024-07-24 14:34:12 +02:00
Louis Dureuil
1a297c048e
Error changes 2024-07-24 14:34:12 +02:00
meili-bors[bot]
ecee0c922f
Merge #4822
4822: HuggingFace: Clearer error message when a model is not supported r=Kerollmops a=dureuill

# Pull Request

## Related issue
Context: <https://github.com/meilisearch/meilisearch/discussions/4820>

## What does this PR do?
- Improve error message when a model configuration cannot be loaded and its "architectures" field does not contain "BertModel"

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-23 14:09:47 +00:00
Louis Dureuil
303e601b87
HuggingFace: Clearer error message when a model is not supported 2024-07-23 15:13:22 +02:00
meili-bors[bot]
f6d2c59bca
Merge #4817
4817: Update version for the next release (v1.10.0) in Cargo.toml r=curquiza a=meili-bot

⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.

Co-authored-by: curquiza <curquiza@users.noreply.github.com>
2024-07-22 15:51:20 +00:00
curquiza
50b7093f8e Update version for the next release (v1.10.0) in Cargo.toml 2024-07-22 13:54:38 +00:00
meili-bors[bot]
48bc797dce
Merge #4812
4812: Allow `MEILI_NO_VERGEN` env var to skip vergen r=irevoire a=dureuill

- vergen checks the state of the `.git` directory to embed commit information into the `meilisearch` binary and the `cargo xtask bench` invocations.
- This check unfortunately results in too many recompilation of the `meilisearch` binary.
- This PR allows skipping vergen when the `MEILI_NO_VERGEN` variable is present in the environment

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-18 16:16:01 +00:00
Louis Dureuil
c6b33fd407
Allow MEILI_NO_VERGEN env var to skip vergen 2024-07-18 17:28:01 +02:00
meili-bors[bot]
6e9d0de8b7
Merge #4806
4806: Update rustls as much as possible r=Kerollmops a=irevoire

# Pull Request

## Related issue
Part of https://github.com/meilisearch/meilisearch/issues/4753

## What does this PR do?
- Update rustls as much as possible

## What is missing

In rustls-0.22.0 two structures we were using have been removed with no explanation or workaround
<img width="518" alt="image" src="https://github.com/user-attachments/assets/fa112db1-3400-4163-8819-7913f22d6b87">



Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-17 17:00:01 +00:00
Tamo
1bfb16386c Update rustls as much as possible 2024-07-17 18:21:26 +02:00
meili-bors[bot]
ea73615abf
Merge #4804
4804: Implements the experimental contains filter operator r=irevoire a=irevoire

# Pull Request
Related PRD: (private link) https://www.notion.so/meilisearch/Contains-Like-Filter-Operator-0d8ad53c6761466f913432eb1d843f1e
Public usage page: https://meilisearch.notion.site/Contains-filter-operator-usage-3e7421b0aacf45f48ab09abe259a1de6

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/3613

## What does this PR do?
- Extract the contains operator from this PR: https://github.com/meilisearch/meilisearch/pull/3751
- Gate it behind a feature flag
- Add tests


Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-17 15:47:11 +00:00
Tamo
02c61eabfa fix the range reported when the experimental feature has not been set 2024-07-17 16:54:33 +02:00
Tamo
56b60ec7a0 apply review comment 2024-07-17 16:13:40 +02:00
meili-bors[bot]
8f416e8f34
Merge #4805
4805: Log the time to index a batch of task r=Kerollmops a=irevoire

This was proposed by `@qdequele` in a private conversation and I think it’s a nice addition.

Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-17 11:45:39 +00:00
Tamo
cf760cbfb1 Log the time to index a batch of task 2024-07-17 11:56:57 +02:00
Tamo
2af9481804 Implements the experimental contains filter operator« 2024-07-17 11:13:37 +02:00
meili-bors[bot]
7a292b572a
Merge #4801
4801: AI quality-of-life improvements r=irevoire a=dureuill

# Pull Request

## Related issue
Fixes #4802 

## What does this PR do?
This PR implements several quality-of-life improvements described in the [public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#ece824a1814e47a0a986d786baff1be9)


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-17 09:00:47 +00:00
Louis Dureuil
8d6ac261ae
Add tests on various failure modes for embedders 2024-07-16 13:39:02 +02:00
Louis Dureuil
b4c8b01c88
Update existing snapshots 2024-07-16 13:39:01 +02:00
Louis Dureuil
24240934f9
Improve errors when indexing documents with a user provided embedder 2024-07-16 13:39:01 +02:00