Commit Graph

644 Commits

Author SHA1 Message Date
filip
f267bed352
remove a unnecessary comment
Co-authored-by: Tamo <irevoire@protonmail.ch>
2023-04-05 13:44:55 +02:00
Tamo
597d57bf1d Merge branch 'main' into bring-back-changes-v1.1.0 2023-04-05 11:32:14 +02:00
Filip Bachul
0fba08cd72 fmt 2023-04-03 20:18:26 +02:00
Filip Bachul
189d4c3b70 add geoPoint integration tests 2023-04-03 20:18:26 +02:00
Filip Bachul
52b4090286 update integration tests 2023-04-03 20:18:26 +02:00
bors[bot]
950f73b8bb
Merge #3623
3623: Update mini-dashboard to version v0.2.7 r=curquiza a=bidoubiwa

## Changes

* Retrieve the API Key from the url parameters (#416) `@qdequele`

## 🐛 Bug Fixes

* Fix show more button not displaying all fields (#419) `@bidoubiwa`

Thanks again to `@bidoubiwa,`     and `@qdequele!` 🎉


Co-authored-by: Charlotte Vermandel <charlottevermandel@gmail.com>
2023-03-30 08:31:29 +00:00
Charlotte Vermandel
e7153e0a97 Update mini-dashboard to version V0.2.7 2023-03-29 14:49:39 +02:00
ManyTheFish
6592746337 Fix other unrelated tests 2023-03-29 14:36:17 +02:00
Tamo
cf5145b542 Reduce the time to import a dump
With this commit, for a dump containing 1M tasks we went from 3m36s to import the task queue down to 1m02s
2023-03-29 14:27:40 +02:00
ManyTheFish
b744f33530 Add test 2023-03-29 12:01:52 +02:00
Tamo
a2b151e877 ensure that the task queue is correctly imported
reduce the size of the snapshots file
2023-03-21 14:41:46 +01:00
Clément Renault
ea016d97af
Implementing an IS EMPTY filter 2023-03-15 14:12:34 +01:00
Clément Renault
fa2ea4a379
Update the test to accept the new IS syntax 2023-03-14 10:31:27 +01:00
bors[bot]
fb1260ee88
Merge #3568 #3569
3568: CI: Fix `publish-aarch64` job that still uses ubuntu-18.04 r=Kerollmops a=curquiza

Fixes #3563 

Main change
- add the usage of the `ubuntu-18.04` container instead of the native `ubuntu-18.04` of GitHub actions: I had to install docker in the container.

Small additional changes
- remove useless `fail-fast` and unused/irrelevant matrix inputs (`build`, `linker`, `os`, `use-cross`...)
- Remove useless step in job

Proof of work with this CI triggered on this current branch: https://github.com/meilisearch/meilisearch/actions/runs/4366233882

3569: Enhance Japanese language detection r=dureuill a=ManyTheFish

# Pull Request

This PR is a prototype and can be tested by downloading [the dedicated docker image](https://hub.docker.com/layers/getmeili/meilisearch/prototype-better-language-detection-0/images/sha256-a12847de00e21a71ab797879fd09777dadcb0881f65b5f810e7d1ed434d116ef?context=explore):

```bash
$ docker pull getmeili/meilisearch:prototype-better-language-detection-0
```

## Context
Some Languages are harder to detect than others, this miss-detection leads to bad tokenization making some words or even documents completely unsearchable. Japanese is the main Language affected and can be detected as Chinese which has a completely different way of tokenization.

A [first iteration has been implemented for v1.1.0](https://github.com/meilisearch/meilisearch/pull/3347) but is an insufficient enhancement to make Japanese work. This first implementation was detecting the Language during the indexing to avoid bad detections during the search.
Unfortunately, some documents (shorter ones) can be wrongly detected as Chinese running bad tokenization for these documents and making possible the detection of Chinese during the search because it has been detected during the indexing.

For instance, a Japanese document `{"id": 1, "name": "東京スカパラダイスオーケストラ"}` is detected as Japanese during indexing, during the search the query `東京` will be detected as Japanese because only Japanese documents have been detected during indexing despite the fact that v1.0.2 would detect it as Chinese.
However if in the dataset there is at least one document containing a field with only Kanjis like:
_A document with only 1 field containing only Kanjis:_
```json
{
 "id":4,
 "name": "東京特許許可局"
}
```
_A document with 1 field containing only Kanjis and 1 field containing several Japanese characters:_
```json
{
 "id":105,
 "name": "東京特許許可局",
 "desc": "日経平均株価は26日 に約8カ月ぶりに2万4000円の心理的な節目を上回った。株高を支える材料のひとつは、自民党総裁選で3選を決めた安倍晋三首相の経済政策への期待だ。恩恵が見込まれるとされる人材サービスや建設株の一角が買われている。ただ思惑が先行して資金が集まっている面 は否めない。実際に政策効果を取り込む企業はどこか、なお未知数だ。"
}
```

Then, in both cases, the field `name` will be detected as Chinese during indexing allowing the search to detect Chinese in queries. Therefore,  the query `東京` will be detected as Chinese and only the two last documents will be retrieved by Meilisearch.

## Technical Approach

The current PR partially fixes these issues by:
1) Adding a check over potential miss-detections and rerunning the extraction of the document forcing the tokenization over the main Languages detected in it.
 >  1) run a first extraction allowing the tokenizer to detect any Language in any Script
 >  2) generate a distribution of tokens by Script and Languages (`script_language`)
 >  3) if for a Script we have a token distribution of one of the Language that is under the threshold, then we rerun the extraction forbidding the tokenizer to detect the marginal Languages
 >  4) the tokenizer will fall back on the other available Languages to tokenize the text. For example, if the Chinese were marginally detected compared to the Japanese on the CJ script, then the second extraction will force Japanese tokenization for CJ text in the document. however, the text on another script like Latin will not be impacted by this restriction.

2) Adding a filtering threshold during the search over Languages that have been marginally detected in documents

## Limits
This PR introduces 2 arbitrary thresholds:
1) during the indexing, a Language is considered miss-detected if the number of detected tokens of this Language is under 10% of the tokens detected in the same Script (Japanese and Chinese are 2 different Languages sharing the "same" script "CJK").
2) during the search, a Language is considered marginal if less than 5% of documents are detected as this Language.

This PR only partially fixes these issues:
-  the query `東京` now find Japanese documents if less than 5% of documents are detected as Chinese.
-  the document with the id `105` containing the Japanese field `desc` but the miss-detected field `name` is now completely detected and tokenized as Japanese and is found with the query `東京`.
-  the document with the id `4` no longer breaks the search Language detection but continues to be detected as a Chinese document and can't be found during the search.

## Related issue
Fixes #3565

## Possible future enhancements
- Change or contribute to the Library used to detect the Language
  - the related issue on Whatlang: https://github.com/greyblake/whatlang-rs/issues/122

Co-authored-by: curquiza <clementine@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2023-03-09 15:34:35 +00:00
bors[bot]
48a51e5cd6
Merge #3577
3577: Avoid fetching an LMDB value with an empty string r=ManyTheFish a=Kerollmops

# Pull Request

## Related issue
Fixes #3574 

## What does this PR do?
This PR fixes a bug where an empty key fetches an entry in the database. LMDB throws an error if an empty or too-long key is used to fetch an entry. This empty string seems to have been generated by the Charabia tokenizer.

Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-03-09 14:35:25 +00:00
Many the fish
dea101e3d9
Update meilisearch/src/routes/indexes/mod.rs
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-03-09 15:17:03 +01:00
Clément Renault
6da54d0cb6
Add a test to fix a diacritic issue 2023-03-09 14:57:38 +01:00
bors[bot]
667bb87e35
Merge #3541
3541: Add cache on the indexes stats r=dureuill a=irevoire

Fix https://github.com/meilisearch/meilisearch/issues/3540

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-03-09 13:32:52 +00:00
ManyTheFish
dff2715ef3 Try removing needless collect 2023-03-09 11:28:10 +01:00
Tamo
c5f22be6e1 add boolean support for csv documents 2023-03-09 11:12:49 +01:00
ManyTheFish
b4b859ec8c Fix typos 2023-03-09 10:58:35 +01:00
ManyTheFish
7e2fd82e41 Use Language allow list in the highlighter 2023-03-08 12:44:16 +01:00
ManyTheFish
3092cf0448 Fix clippy errors 2023-03-08 10:53:42 +01:00
Louis Dureuil
2f5b9fbbd8
Restore contribution of the index sizes to the db size
- the index size now contributes to the db size even if the index is not authorized
2023-03-07 14:05:27 +01:00
Tamo
d34faa8f9c put back the sleep as it was and fix the from 2023-03-06 18:09:09 +01:00
Tamo
e5d0bef6d8 update a comment 2023-03-06 17:04:24 +01:00
Louis Dureuil
076a3d371c Eagerly compute stats as fallback to the cache.
- Refactor all around to avoid spawning indexes more times than necessary
2023-03-06 16:57:31 +01:00
Tamo
fd5c48941a Add cache on the indexes stats 2023-03-06 16:57:31 +01:00
Tamo
e704728ee7 fix the snapshots permissions on unix system 2023-03-06 16:28:40 +01:00
bors[bot]
3d1046369c
Merge #3529
3529: Add an analytics on the geo bounding box feature r=ManyTheFish a=irevoire

Fixes #3527

[The specification of the geoBoundingBox](https://github.com/meilisearch/specifications/pull/223) feature has been updated and now introduces a new analytics to follow the usage of the geoBoundingBox feature in the search requests.

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-03-02 11:58:39 +00:00
bors[bot]
4f1ccbc495
Merge #3525
3525: Fix phrase search containing stop words r=ManyTheFish a=ManyTheFish

# Summary
A search with a phrase containing only stop words was returning an HTTP error 500,
this PR filters the phrase containing only stop words dropping them before the search starts, a query with a phrase containing only stop words now behaves like a placeholder search.

fixes https://github.com/meilisearch/meilisearch/issues/3521

related v1.0.2 PR on milli: https://github.com/meilisearch/milli/pull/779



Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-03-02 10:55:37 +00:00
Louis Dureuil
d4d4702f1b
Rephrase hint message 2023-02-27 13:46:16 +01:00
Tamo
7ae10abb6b fix the auth tests 2023-02-23 17:27:42 +01:00
Tamo
dc533584c6 Forbid the usage of the metrics route if your API key have a limitation on the indexes 2023-02-23 17:13:22 +01:00
bors[bot]
89ac1015f3
Merge #3524
3524: Update the metrics route r=irevoire a=irevoire

Fixes #3523

Make the metrics available by default without a feature flag.
+ Rename the cli-flag to `experimental-enable-metrics`.

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-02-23 15:11:10 +00:00
bors[bot]
ca25904c26
Merge #3331
3331: Limit the number of concurrently opened indexes r=dureuill a=dureuill

# Pull Request

## Related issue
Relevant to #1841, fixes #3382

## What does this PR do?

### User standpoint

- Limit the number of concurrently opened indexes (currently, the number of indexes that can be concurrently opened is computed at startup)
- When too many an index is opened, the least recently used one is closed and its virtual memory released.
- This allows a user to have an arbitrary number of indexes of an arbitrary size

### Implementation standpoint

- Added a LRU cache map in `index-scheduler::lru`. A more complete implementation  (eg with helper functions not used here) is available but would better fit a dedicated crate.
- Use the LRU cache map in the `IndexScheduler`. To simplify the lifecycle of indexes, they are never removed from the cache when they are in the middle of a resize or delete operation. To achieve this, an intermediate `Vec` stores the UUIDs of the indexes that are in the middle of such an operation.
- Upon creating the index scheduler object, compute the total virtual memory that is adressable by using a dichotomic search on the max size of an index. Use this as a base to compute the number of indexes that can be open with 2TiB per index. If the virtual memory address space is lower than 2TiB, then only allow for 1 index of a fraction of that size.

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-02-23 14:20:52 +00:00
Tamo
8a1b1a95f3 comment the right of the metrics 2023-02-23 13:59:01 +01:00
Tamo
8d47d2d018 update the auth api after the rebase 2023-02-23 13:15:51 +01:00
Tamo
750a2b6842 Update meilisearch/src/option.rs
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-02-23 12:26:22 +01:00
Tamo
bc7d4112d9 send the cli experimental feature in the analytics 2023-02-23 12:26:22 +01:00
Tamo
88a18677d0 rename the metrics cli flag 2023-02-23 12:26:22 +01:00
Tamo
68e30214ca remove the feature flag and reorganize the module slightly 2023-02-23 12:26:21 +01:00
bors[bot]
b985b96e4e
Merge #3530
3530: Fix highlighter bug r=Kerollmops a=ManyTheFish

# Pull Request

There was a highlighting issue on CJK's character, we were highlighting too many characters and these additional characters were duplicated after the highlight tag.

## Related issue
Fixes #3517 
Fixes #3526 

## What does this PR do?
- add a test showcasing the bug
- fix the bug by activating the char_map creation of the tokenizer during the highlighting process


Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-02-23 10:59:43 +00:00
Louis Dureuil
3db613ff77
Don't iterate all indexes manually 2023-02-23 11:29:09 +01:00
Louis Dureuil
5822764be9
Skip computing index budget in tests 2023-02-23 11:23:39 +01:00
Louis Dureuil
c63294f331
Switch to 2TiB default index size, updates documentation 2023-02-23 11:23:39 +01:00
Louis Dureuil
80b060f920
Use LRU cache 2023-02-23 11:23:39 +01:00
bors[bot]
f62703cd67
Merge #3534
3534: Update the csv error code from InvalidIndexCsvDelimiter to InvalidDocumentCsvDelimiter r=Kerollmops a=irevoire

Fixes #3533

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-02-23 07:05:12 +00:00
Tamo
76f82c880d update the csv error code from InvalidIndexCsvDelimiter to InvalidDocumentCsvDelimiter 2023-02-22 19:26:48 +01:00
ManyTheFish
28d6a4466d Make the tokenizer creating a char map during highlighting 2023-02-22 17:43:10 +01:00
Louis Dureuil
1ba2fae3ae
multi-search/authentication: Add authentication tests 2023-02-22 17:04:12 +01:00
Louis Dureuil
28d6ab78de
multi-search: Add multi search tests 2023-02-22 17:04:12 +01:00
Louis Dureuil
3ba5dfb6ec
multi-search: Add test server search method for multi search 2023-02-22 17:04:12 +01:00
Louis Dureuil
a23fbf6c7b
multi-search: Add search with an array of indexes 2023-02-22 17:04:12 +01:00
Louis Dureuil
596a98f7c6
multi-search: Add basic analytics 2023-02-22 16:37:18 +01:00
Louis Dureuil
14c4a222da
Authentication: AuthFilter::allow_index_creation both check that the index is authorized and the IndexCreate action 2023-02-22 16:37:13 +01:00
Louis Dureuil
690bb2e5cc
Authentication: Make allow_index_creation a private field 2023-02-22 16:35:52 +01:00
Louis Dureuil
42577403d8
Authentication: Directly pass the authfilter to the index scheduler 2023-02-22 16:35:52 +01:00
Louis Dureuil
c8c5944094
Authentication: is_index_authorized takes into account API key indexes even with a tenant token 2023-02-22 16:35:52 +01:00
Louis Dureuil
4b65851793
Authentication: Refactor authentication check to work for tenant token even without an index in URL
Callers need to manually check `is_index_authorized` when using the route without an index in URL
2023-02-22 16:35:51 +01:00
ManyTheFish
ad35edfa32 Add test 2023-02-22 15:47:15 +01:00
Tamo
033417e9cc add an analytics on the geo bounding box feature 2023-02-22 15:35:26 +01:00
bors[bot]
ac5a1e4c4b
Merge #3423
3423: Add min and max facet stats r=dureuill a=dureuill

# Pull Request

## Related issue
Fixes #3426

## What does this PR do?

### User standpoint

- When using a `facets` parameter in search, the facets that have numeric values are displayed in a new section of the response called `facetStats` that contains, per facet, the numeric min and max value of the hits returned by the search.

<details>
<summary>
Sample request/response
</summary>

```json
❯ curl \
  -X POST 'http://localhost:7700/indexes/meteorites/search?facets=mass' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "LL6", "facets":["mass", "recclass"], "limit": 5 }' | jsonxf
{
  "hits": [
    {
      "name": "Niger (LL6)",
      "id": "16975",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 3.3,
      "fall": "Fell"
    },
    {
      "name": "Appley Bridge",
      "id": "2318",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 15000,
      "fall": "Fell",
      "_geo": {
        "lat": 53.58333,
        "lng": -2.71667
      }
    },
    {
      "name": "Athens",
      "id": "4885",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 265,
      "fall": "Fell",
      "_geo": {
        "lat": 34.75,
        "lng": -87.0
      }
    },
    {
      "name": "Bandong",
      "id": "4935",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 11500,
      "fall": "Fell",
      "_geo": {
        "lat": -6.91667,
        "lng": 107.6
      }
    },
    {
      "name": "Benguerir",
      "id": "30443",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 25000,
      "fall": "Fell",
      "_geo": {
        "lat": 32.25,
        "lng": -8.15
      }
    }
  ],
  "query": "LL6",
  "processingTimeMs": 15,
  "limit": 5,
  "offset": 0,
  "estimatedTotalHits": 42,
  "facetDistribution": {
    "mass": {
      "110000": 1,
      "11500": 1,
      "1161": 1,
      "12000": 1,
      "1215.5": 1,
      "127000": 1,
      "15000": 1,
      "1676": 1,
      "1700": 1,
      "1710.5": 1,
      "18000": 1,
      "19000": 1,
      "220000": 1,
      "2220": 1,
      "22300": 1,
      "25000": 2,
      "265": 1,
      "271000": 1,
      "2840": 1,
      "3.3": 1,
      "3000": 1,
      "303": 1,
      "32000": 1,
      "34000": 1,
      "36.1": 1,
      "45000": 1,
      "460": 1,
      "478": 1,
      "483": 1,
      "5500": 2,
      "600": 1,
      "6000": 1,
      "67.8": 1,
      "678": 1,
      "680.5": 1,
      "6930": 1,
      "8": 1,
      "8300": 1,
      "840": 1,
      "8400": 1
    },
    "recclass": {
      "L/LL6": 3,
      "LL6": 39
    }
  },
  "facetStats": {
    "mass": {
      "min": 3.3,
      "max": 271000.0
    }
  }
}
```

</details>

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ ] Have you read the contributing guidelines?
- [ ] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-02-22 13:06:43 +00:00
ManyTheFish
6841f167b4 Add test 2023-02-21 18:02:52 +01:00
bors[bot]
39407885c2
Merge #3347
3347: Enhance language detection r=irevoire a=ManyTheFish

## Summary

Some completely unrelated Languages can share the same characters, in Meilisearch we detect the Languages using `whatlang`, which works well on large texts but fails on small search queries leading to a bad segmentation and normalization of the query.

This PR now stores the Languages detected during the indexing in order to reduce the Languages list that can be detected during the search.

## Detail

- Create a 19th database mapping the scripts and the Languages detected with the documents where the Language is detected
- Fill the newly created database during indexing
- Create an allow-list with this database and pass it to Charabia
- Add a test ensuring that a Japanese request containing kanjis only is detected as Japanese and not Chinese

## Related issues
Fixes #2403
Fixes #3513

Co-authored-by: f3r10 <frledesma@outlook.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2023-02-21 10:52:13 +00:00
bors[bot]
a3e41ba33e
Merge #3496
3496: Fix metrics feature r=irevoire a=james-2001

# Pull Request

## Related issue

Resolves: #3469
See also: #2763

## What does this PR do?
As reported the metrics feature was broken by still using and old reference to `meilisearch_auth::actions`. This commit switches to the new location, `meilisearch_types::keys::actions`.

The original issue was not *that* clear as to exactly what was broken, and the build logs have disappeared, but it seemed to just be this one line fix. If this is not the case and I've missed the mark let me know, and i'll head back to the drawing board.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Co-authored-by: James <james.a.may.2001@gmail.com>
2023-02-21 10:13:11 +00:00
James
ce807d760b Fix formatting issue on Opt struct
tab in enable_metrics_route to fix cargo fmt issues

Resolves: #3469
See also: #2763
2023-02-21 09:45:18 +00:00
James
5cff435bf6 Add feature flags to Opt structure
Resolves: #3469
See also: #2763
2023-02-21 07:41:41 +00:00
ManyTheFish
8aa808d51b Merge branch 'main' into enhance-language-detection 2023-02-20 18:14:34 +01:00
bors[bot]
1e9ac00800
Merge #3505
3505: Csv delimiter r=irevoire a=irevoire

Fixes https://github.com/meilisearch/meilisearch/issues/3442
Closes https://github.com/meilisearch/meilisearch/pull/2803
Specified in https://github.com/meilisearch/specifications/pull/221

This PR is a reimplementation of https://github.com/meilisearch/meilisearch/pull/2803, on the new engine. Thanks for your idea and initial PR `@MixusMinimax;` sorry I couldn’t update/merge your PR. Way too many changes happened on the engine in the meantime.

**Attention to reviewer**; I had to update deserr to implement the support of deserializing `char`s

-------

It introduces four new error messages;
- Invalid value in parameter csvDelimiter: expected a string of one character, but found an empty string
- Invalid value in parameter csvDelimiter: expected a string of one character, but found the following string of 5 characters: doggo
- csv delimiter must be an ascii character. Found: 🍰 
- The Content-Type application/json does not support the use of a csv delimiter. The csv delimiter can only be used with the Content-Type text/csv.

And one error code;
- `invalid_index_csv_delimiter`

The `invalid_content_type` error code is now also used when we encounter the `csvDelimiter` query parameter with a non-csv content type.

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-02-20 17:01:36 +00:00
ManyTheFish
23f4e82b53 Add test ensuring that Meilisearch works on kanji only requests 2023-02-20 15:43:29 +01:00
bors[bot]
a8f6f108e0
Merge #3515
3515: Consider null as a valid geo field r=irevoire a=irevoire

Fix #3497
Associated spec; https://github.com/meilisearch/specifications/pull/222

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-02-20 14:12:55 +00:00
Tamo
1479050f7a apply review suggestions 2023-02-20 14:53:37 +01:00
Louis Dureuil
74859ecd61
Add min and max facet stats 2023-02-20 13:52:27 +01:00
Charlotte Vermandel
dd120e0e16 Bump version of mini-dashboard to v0.2.6 2023-02-20 13:45:57 +01:00
Tamo
18796d6e6a Consider null as a valid geo object 2023-02-20 13:45:51 +01:00
bors[bot]
c91bfeaf15
Merge #3467
3467: Identify builds git tagged with `prototype-...` in CLI and analytics r=curquiza a=dureuill

# Pull Request

## What does this PR do?

- Parses the last git tag to extract a prototype name if:
  - Current build uses the prototype tag (not after the tag) precisely
  - The prototype tag name respects the following conditions:
    1. starts with `prototype-`
    2. ends with a number
    3. the hyphen-separated segment right before the number is not a number (required to reject commits after the tag).
- Display the prototype name in the launch summary in the CLI
- Send the prototype name to analytics if any
- Update prototypes instructions in CONTRIBUTING.md

|`VERGEN_GIT_SEMVER_LIGHTWEIGHT` value | Prototype |
|---|---|
| `Some("prototype-geo-bounding-box-0-139-gcde89018")` | `None` (does not end with a number) |
| `Some("prototype-geo-bounding-box-0-139-89018")` | `None` (before the last segment is a number) |
| `Some("prototype-geo-bounding-box-0")` | `Some("prototype-geo-bounding-box-0")` |
| `Some("prototype-geo-bounding-box")` | `None` (does not end with a number") |
| `Some("geo-bounding-box-0")` | `None` (does not start with "prototype") |
| `None` | `None` | 

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-02-20 09:27:51 +00:00
James
91048d209d Fix metrics feature
Metrics feature was relying on old references. Refactored with inspiration from the `get_stats` method in `meilisearch/src/routes/lib.rs`. `enable_metrics_routes` added to options in `segment_analytics`.

Resolves: #3469
See also: #2763
2023-02-17 20:11:57 +00:00
Tamo
f11c7d4b62 cargo run execute meilisearch by default 2023-02-16 18:03:45 +01:00
Tamo
e79f6f87f6 make cargo fmt&clippy happy 2023-02-16 18:00:40 +01:00
Tamo
5367d8f05a add two tests on the indexing of csvs 2023-02-16 17:37:11 +01:00
Tamo
52686da028 test various error on the document ressource 2023-02-16 17:37:10 +01:00
Tamo
8c074f5028 implements the csv delimiter without tests
Co-authored-by: Maxi Barmetler <maxi.barmetler@gmail.com>
2023-02-16 17:35:36 +01:00
Louis Dureuil
54240db495
Add note in code so one does not forget next time 2023-02-16 10:53:14 +01:00
Louis Dureuil
9bd1cfb3a3
Ignore -dirty flag 2023-02-16 10:53:14 +01:00
Louis Dureuil
f46cf46b8c
Add prototype to analytics if any 2023-02-16 10:53:14 +01:00
Louis Dureuil
c3a30a5a91
If using a prototype, display its name at Meilisearch startup 2023-02-16 10:53:14 +01:00
Tamo
74d1a67a99 Use the workspace inheritance feature of rust 1.64 2023-02-15 13:51:07 +01:00
Tamo
42a3cdca66
get rids of the unwrap_any function in favor of take_cf_content 2023-02-14 20:06:31 +01:00
Tamo
a43765d454
use the pre-defined deserr extractors 2023-02-14 20:05:30 +01:00
Tamo
769576fd94
get rids of the whole error_message module since it has been integrated into the last version of deserr 2023-02-14 20:05:27 +01:00
Tamo
8fb7b1d10f
bump deserr 2023-02-14 20:04:30 +01:00
James
62358bd31c Fix metrics feature
As reported the metrics feature was broken by still using and old reference to `meilisearch_auth::actions`. This commit switches to the new location, `meilisearch_types::keys::actions`.

Resolves: #3469
See also: #2763
2023-02-14 17:29:38 +00:00
Clément Renault
47748395dc
Update an authentication comment
Co-authored-by: Many the fish <many@meilisearch.com>
2023-02-13 17:20:08 +01:00
Clément Renault
764df24b7d
Make clippy happy (again) 2023-02-09 13:21:20 +01:00
Clément Renault
4570d5bf3a
Merge remote-tracking branch 'origin/main' into temp-wildcard 2023-02-09 13:14:05 +01:00
Kerollmops
c690c4fec4
Added and modified the current API Key and Tenant Token tests 2023-02-09 11:17:30 +01:00
Kerollmops
7b4b57ecc8
Fix the current tests 2023-02-08 14:54:05 +01:00
dependabot[bot]
5f56e6dd58
Bump tokio from 1.24.1 to 1.24.2
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.24.1 to 1.24.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-07 12:14:05 +00:00
bors[bot]
c88c3637b4
Merge #3461
3461: Bring v1 changes into main r=curquiza a=Kerollmops

Also bring back changes in milli (the remote repository) into main done during the pre-release

Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Philipp Ahlner <philipp@ahlner.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
2023-02-07 11:27:27 +00:00
Tamo
7a38fe624f
throw an error if the top left corner is found below the bottom right corner 2023-02-06 17:50:47 +01:00
Kerollmops
a015e232ab
Merge remote-tracking branch 'origin/release-v1.0.0' into bring-v1-changes 2023-02-06 16:41:10 +01:00
Guillaume Mourier
426d63b01b
Update insta test suite 2023-02-02 12:27:56 +01:00
Kerollmops
a36b1dbd70
Fix the tasks with the new patterns 2023-02-01 18:21:45 +01:00
Kerollmops
d563ed8a39
Making it work with index uid patterns 2023-02-01 17:51:30 +01:00
bors[bot]
20f8184c06
Merge #3441
3441: Fix import of dump v2 r=dureuill a=irevoire

# Pull Request
This bug was introduced because of a mistake we did earlier: We said the last version to export dump v2 was the v0.21.0 while it was the v0.22.0.
To fix the bug I updated our whole v2 reader to use the code from meilisearch v0.22.0.
Also:
- Import the bugged dump in the tests
- Test the import of this dump in the v2 reader and current reader

## Related issue
Fixes #3435


Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-31 13:23:57 +00:00
Tamo
4b7b2d6a90 fix the import of dump v2 generated by meilisearch v0.22.0 2023-01-31 13:03:28 +01:00
Louis Dureuil
924d5d4c11
clippy: remove needless lifetimes 2023-01-31 10:40:48 +01:00
Louis Dureuil
3296cf7ae6
clippy: remove needless lifetimes 2023-01-31 09:32:40 +01:00
Tamo
cac93f149e fix the tests after rebasing 2023-01-25 16:52:54 +01:00
Tamo
481df7a8b6 Update meilisearch/tests/documents/add_documents.rs
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-25 16:45:11 +01:00
Tamo
8356f109c1 bump milli to fix the last test 2023-01-25 16:45:11 +01:00
Tamo
934f2b3cb5 exhaustively test all the errors that can arise from a bad geo field 2023-01-25 16:45:11 +01:00
Tamo
a3f1b8fdb9 refactorize the test suite of the add_documents module to use snapshot tests when possible 2023-01-25 16:45:11 +01:00
bors[bot]
d963c2ce55
Merge #3419
3419: Test all the api key error codes r=dureuill a=irevoire

Partially fix #3325

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-25 15:09:19 +00:00
Kerollmops
184b8afd9e
Make it work in the CreateApiKey struct 2023-01-25 15:01:50 +01:00
Tamo
a858531574 apply review comments 2023-01-25 14:51:36 +01:00
Clément Renault
474d4ec498
Add tests for the index patterns 2023-01-25 14:22:16 +01:00
Tamo
04c4487660 udpate the analytics with the new stats method 2023-01-25 11:25:04 +01:00
Tamo
c92948b143 Compute the size of the auth-controller, index-scheduler and all update files in the global stats 2023-01-25 11:25:02 +01:00
bors[bot]
b3c2a4ae27
Merge #3412
3412: When adding documents, trying to update the primary-key now throw an error r=Kerollmops a=irevoire

While updating the test suite, I also noticed an issue with the indexed_documents value of failed tasks and had to update it. I also named a bunch of snapshots that had no name, sorry 😬

Fixes https://github.com/meilisearch/meilisearch/issues/3385
Fixes https://github.com/meilisearch/meilisearch/issues/3411

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-24 17:14:11 +00:00
Tamo
aa17a54feb
test all the api key error codes 2023-01-24 17:30:35 +01:00
Tamo
32364e9919 add tests on the index resource 2023-01-24 13:20:20 +01:00
bors[bot]
fea41ca788
Merge #3404
3404: Fix matching strategy error r=irevoire a=ManyTheFish

# Pull Request

## Related issue
Fixes #3391


Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-23 17:04:05 +00:00
bors[bot]
217504fff3
Merge #3406
3406: Master Key: Implements errors and warnings from the specification r=irevoire a=dureuill

<sub>Now in technicolor</sub>

# Pull Request

## What does this PR do?
- Uses `atty` and `termcolor` as dependency
- Use these dependencies to print colored background for warning messages
- Update messages to match https://github.com/meilisearch/specifications/pull/209

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ ] Have you read the contributing guidelines?
- [ ] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-23 16:39:18 +00:00
Tamo
5672118bfa
When adding documents, trying to update the primary-key now throw an error
While updating the test suite I also noticed an issue with the indexed_documents value of failed task and had to update it.
I also named a bunch of snapshots that had no name sorry 😬
2023-01-23 17:32:13 +01:00
Louis Dureuil
57682cbabe
Fix test url after #3398 2023-01-23 15:43:17 +01:00
ManyTheFish
5dd582918d
Add test 2023-01-23 15:40:42 +01:00
ManyTheFish
f0e6b9c0c5
Update deserr to 0.3.0 2023-01-23 14:13:04 +01:00
Louis Dureuil
56db54486c
Add tests 2023-01-23 14:00:30 +01:00
Louis Dureuil
a9b3f91467
Add missing space
Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>
2023-01-23 10:33:30 +01:00
Louis Dureuil
0de9a3ffe7
Implements errors and warnings from the specification
Now in technicolor
2023-01-19 18:04:45 +01:00
bors[bot]
8fb685f5aa
Merge #3401
3401: improve the error messages for the immutable fields r=dureuill a=irevoire

Fix https://github.com/meilisearch/meilisearch/issues/3400

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-19 15:52:50 +00:00
Tamo
e3742a38d4
improve the error messages for the immutable fields 2023-01-19 16:49:44 +01:00
Louis Dureuil
d2420f5c8f
Fix non insta tests 2023-01-19 16:10:05 +01:00
Louis Dureuil
72e2b220ed
Fix tests 2023-01-19 15:48:20 +01:00
bors[bot]
a1e9c44fe5
Merge #3389
3389: Return `invalid_search_facets` rather than `bad_request` when using facet on a non filterable attribute r=irevoire a=dureuill

# Pull Request

## Related issue

Fixes https://github.com/meilisearch/meilisearch/issues/3384

## What does the PR does

- title
- also adds a test

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-19 13:19:22 +00:00
Louis Dureuil
3d8ca62c35
InvalidFacetDistribution returns invalid_search_facet 2023-01-19 13:41:26 +01:00
Tamo
e8e7070cc6
improve the error message when no task filter are specified for the cancelation or deletion of tasks 2023-01-19 12:42:08 +01:00
Tamo
d0988e115f
fix the patch of description and name for the api-key 2023-01-18 19:07:26 +01:00
Tamo
5dcb920fb4
improve the tests 2023-01-18 18:27:00 +01:00
bors[bot]
6f7e0c431a
Merge #3341
3341: add functionnal + error tests on the swap_indexes route and fix a confusing error message r=loiclec a=irevoire

Fix https://github.com/meilisearch/meilisearch/issues/3340
Fix part of https://github.com/meilisearch/meilisearch/issues/3325
Fix https://github.com/meilisearch/meilisearch/issues/3381

Test both the functionality and the error codes

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
2023-01-18 16:32:22 +00:00
Tamo
00f6af6475
fix a wrong error message 2023-01-18 17:26:48 +01:00
bors[bot]
3e5b3df487
Merge #3370 #3373 #3375
3370: make the swap indexes not found errors return an IndexNotFound error-code r=irevoire a=irevoire

Fix https://github.com/meilisearch/meilisearch/issues/3368

3373: fix a wrong error code and add tests on the document resource r=irevoire a=irevoire

Fix https://github.com/meilisearch/meilisearch/issues/3371

3375: Avoid deleting all task invalid canceled by r=irevoire a=Kerollmops

Fixes #3369 by making sure that at least one `canceledBy` task filter parameter matches something.

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
2023-01-18 15:21:11 +00:00
Kerollmops
d3c796af38
Add a new test to check that invalid canceledBy works correctly 2023-01-18 15:50:46 +01:00
Kerollmops
182eea1f17
Introduce a canceledBy filter for the tests 2023-01-18 15:50:42 +01:00
Tamo
a4476c20f8
fix a wrong error code and add tests on the document resource 2023-01-18 15:28:02 +01:00
Tamo
57da80900d
make the swap indexes not found errors return an IndexNotFound error code 2023-01-18 14:16:00 +01:00
Tamo
0f727d079b fix the wrong error code on minWordSizeForTypos 2023-01-18 12:28:46 +01:00
Loïc Lecrenier
82bdb54537 Update the index swap tests after git rebase 2023-01-18 09:40:41 +01:00
Tamo
b6ec1f1c6d add functionnal + error tests on the swap_indexes route 2023-01-18 09:36:04 +01:00
Loïc Lecrenier
e225608337 Use invalid_index_uid error code in more places 2023-01-17 15:28:06 +01:00
Loïc Lecrenier
56e79fa850 Update task snapshot test and clean up details 2023-01-17 13:19:04 +01:00
Loïc Lecrenier
c71a8ea183 Update to latest milli and deserr 2023-01-17 13:10:38 +01:00
Loïc Lecrenier
b781f9a0f9 cargo fmt 2023-01-17 11:07:07 +01:00
Loïc Lecrenier
9194508a0f Refactor query parameter deserialisation logic 2023-01-17 11:07:07 +01:00
Loïc Lecrenier
766dd830ae Update deserr to latest version + add new error codes for missing fields
- missing_api_key_indexes
- missing_api_key_actions
- missing_api_key_expires_at

- missing_swap_indexes_indexes
2023-01-17 09:43:07 +01:00
Loïc Lecrenier
436ae4e466 Improve error messages generated by deserr
Split Json and Query Parameter error types
2023-01-17 09:43:07 +01:00
Kerollmops
cde62fcb5b
Merge remote-tracking branch 'origin/release-v1.0.0' into import-milli 2023-01-16 17:35:18 +01:00
Kerollmops
4fb47492e5
Make clippy happy 2023-01-16 16:35:58 +01:00
bors[bot]
a5c4fbbcea
Merge #3334
3334: Add specific error codes `immutable_...` r=irevoire a=loiclec

Add the following error codes:

When an immutable field of API key is sent to the `PATCH /keys` route: 
- `ImmutableApiKeyUid` 
- `ImmutableApiKeyKey`
- `ImmutableApiKeyActions`
- `ImmutableApiKeyIndexes`
- `ImmutableApiKeyExpiresAt`
- `ImmutableApiKeyCreatedAt`
- `ImmutableApiKeyUpdatedAt`

When an immutable field of Index is sent to the `PATCH /indexes/{uid}` route:
- `ImmutableIndexUid`
- `ImmutableIndexCreatedAt`
- `ImmutableIndexUpdatedAt`

Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-12 15:31:38 +00:00
Tamo
21b8cd53b7
reformat 2023-01-12 16:20:24 +01:00
Loïc Lecrenier
7f80b116bc
Add specific immutable_field error codes 2023-01-12 16:20:14 +01:00
Tamo
79c7f65c30
make a test more reliable 2023-01-12 15:39:28 +01:00
Tamo
a524dfb713
fix the analytics 2023-01-12 14:49:50 +01:00
Tamo
705fcaa3b8
reformat the imports 2023-01-12 14:09:15 +01:00
Loïc Lecrenier
a09b6a341d Move tasks route to deserr 2023-01-12 13:57:29 +01:00
Tamo
387874ea26
test the error codes on the task routes 2023-01-12 13:46:19 +01:00
Tamo
6d658f4c52
fix a wrong error code + update some error messages 2023-01-11 19:14:11 +01:00
Tamo
bf573885ea
integrate the latest version of milli 2023-01-11 19:08:39 +01:00
Tamo
a68ac3a1dc
reformat the headers 2023-01-11 19:08:39 +01:00
Tamo
b252c87197
add tests on the sub settings routes 2023-01-11 19:08:39 +01:00
Loïc Lecrenier
b0b7ad7caf
Apply review suggestions 2023-01-11 19:08:39 +01:00
Loïc Lecrenier
c91ffec72e
Update Cargo.toml 2023-01-11 19:08:39 +01:00
Loïc Lecrenier
1fc11264e8
Refactor deserr integration 2023-01-11 19:08:39 +01:00
bors[bot]
808e184069
Merge #3324
3324: Add a test on the search route for each possible error codes r=irevoire a=irevoire



Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-11 16:08:19 +00:00
Tamo
7a30d98264
fix a flaky test 2023-01-11 14:54:29 +01:00
Tamo
d0a85057a3 fix the bad filter test 2023-01-11 11:37:12 +01:00
bors[bot]
59704c000c
Merge #3326
3326: Test error codes on settings r=irevoire a=irevoire



Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-11 10:07:52 +00:00
bors[bot]
d80be0c28d
Merge #3322
3322: Update mini-dashboard to v0.2.5 r=curquiza a=mdubus



Co-authored-by: Morgane Dubus <30866152+mdubus@users.noreply.github.com>
2023-01-11 09:08:11 +00:00
Tamo
398c0c32cd
test all the error codes that can be throw in the settings 2023-01-10 18:19:27 +01:00
Tamo
d4157c0ce4
add a test on the search route for each possible error codes
snapshot the json directly instead of using the debug formatting
2023-01-10 17:59:24 +01:00
Morgane Dubus
f63fee5e97
Update Cargo.toml 2023-01-10 15:11:25 +01:00
Tamo
d308684395 remove two ununsed error codes + fix the sort error_code 2023-01-10 11:32:11 +01:00
Tamo
ff843881c5
remove the documentation of the query parameter extractor module 2023-01-09 15:14:48 +01:00
Loïc Lecrenier
ae08fba76e Remove forgotten comment 2023-01-09 13:45:03 +01:00
Loïc Lecrenier
af6d4b3031 Remove unused deserr extractor 2023-01-09 13:43:16 +01:00
Tamo
b03ee54fe0
makes clippy turbo-happy 2023-01-09 13:04:31 +01:00
Tamo
d17efb9ed6
use the published version of deserr 2023-01-09 12:51:10 +01:00
Loïc Lecrenier
9ab791bedc Update error codes on the api key routes 2023-01-09 12:30:25 +01:00
Loïc Lecrenier
96105a5e8d Update error codes on the documents/ routes 2023-01-09 12:30:25 +01:00
Tamo
e706628bb1 fix the error code of the swap index route 2023-01-06 14:48:25 +01:00
Tamo
3c630891bb
fix the error code for the swap index 2023-01-05 21:25:20 +01:00
Tamo
ce3e8794a2
fix the tests after the rebase 2023-01-05 20:52:26 +01:00
Tamo
50ce0409bc
Integrate deserr on the most important routes 2023-01-05 20:48:29 +01:00
bors[bot]
839b05c43d
Merge #3305
3305: Remove hidden but usable CLI arguments r=Kerollmops a=Kerollmops

`@curquiza` found out that we were exposing some internal CLI arguments: `nb-max-chunks` and `log-every-n`. In this PR I removed those two, the only two ones that I found. Those options shouldn't be accessible as non-documented in the documentation or the `--help` message.

Fixes https://github.com/meilisearch/meilisearch/issues/3307

Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-01-05 17:11:58 +00:00
Clément Renault
aa4b813237
Derive Default on IndexerOpts 2023-01-05 16:00:45 +01:00
Louis Dureuil
eb08a0fb0b
Remove --generate-master-key option 2023-01-05 14:55:24 +01:00
Clément Renault
cda529c07b
Remove hidden but usable CLI arguments 2023-01-05 14:25:41 +01:00
bors[bot]
c7a3992510
Merge #3303
3303: Update version for the next release (v1.0.0) in Cargo.toml files r=curquiza a=meili-bot

⚠️ This PR is automatically generated. Check the new version is the expected one before merging.

Co-authored-by: curquiza <curquiza@users.noreply.github.com>
2023-01-05 11:53:09 +00:00
curquiza
28408816ef Update version for the next release (v1.0.0) in Cargo.toml files 2023-01-05 11:45:15 +00:00
bors[bot]
0eaa8ca255
Merge #3266
3266: Improve the way we receive the documents payload- serde multiple ndjson fix r=curquiza a=jiangbo212

# Pull Request

## Related issue
Fixes #3037 

## Related PR
#3164 

## What does this PR do?
Sorry, This PR is mainly to fix the problems caused by my previously provided PR #3164. It causes multiple ndjson data deserialization failures
- Fix serde multiple ndjson data failures and add test to it
- Fix serde jsonarray error and againest serde it use `from_slice`. only use `from_slice` when serde error category is `data`, it indicate json data is a single json.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: jiangbo212 <peiyaoliukuan@126.com>
2023-01-05 11:30:29 +00:00
Loïc Lecrenier
2d74678b51 Replace underscores with hyphens in doc link to error code 2023-01-05 10:09:02 +01:00
bors[bot]
db7eaf23f4
Merge #3251
3251: Add a specific test on finite pagination placeolder search with disti… r=curquiza a=ManyTheFish

Add a specific test on finite pagination placeholder search with distinct attributes


related to https://github.com/meilisearch/milli/pull/743
related to https://github.com/meilisearch/meilisearch/issues/3200

poke `@curquiza` 

> note that the destination branch should be changed

Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-01-05 09:06:53 +00:00
bors[bot]
32f7cfa5cb
Merge #3295
3295: Adjust Master Key-related messages r=dureuill a=dureuill

# Pull Request

## Related issue
Follow up for #3272 

## What does this PR do?
- Consistently capitalize "master key" (instead of "Master Key" sometimes) (see https://github.com/meilisearch/specifications/pull/209#discussion_r1060081094)
- Clarify that the counted unit for master key length is bytes, not characters (see https://github.com/meilisearch/documentation/issues/2069#issuecomment-1368873167)

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-05 08:43:23 +00:00
bors[bot]
502d9e4b24
Merge #3278
3278: Remove `--max-index-size` and `--max-task-db-size` flags r=Kerollmops a=dureuill

# Pull Request

## Related issue
Fixes #3231 

## What does this PR do?
- Remove `--max-index-size` and `--max-task-db-size` flags from the CLI, config file and environment variable
- Set the size of all indexes to **500GiB** and the size of the task DB to **10GiB**.  Reviewers might want to review these values carefully.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-04 16:44:27 +00:00
Louis Dureuil
a85ff1f690
Fix documentation
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-01-04 17:20:03 +01:00
Louis Dureuil
233372abea
Remove --max-index-size and --max-task-db-size 2023-01-04 17:20:01 +01:00
bors[bot]
13d4ae264a
Merge #3269
3269: Simplify primary key inference r=dureuill a=dureuill

# Pull Request

## Related issue
Related to https://github.com/meilisearch/meilisearch/issues/3233

## What does this PR do?
- Integrates https://github.com/meilisearch/milli/pull/752 in meilisearch
- Remove `Serialize` and `Deserialize` from `error::Code` as it is unused.
- No longer filter on `milli` logs when `--log-level` is "info".
  - `milli` only has the newly-added inference log at the `info` level (from greping `info` in the codebase)
  - the default value for `--log-level` is "INFO" and not "info" since `v0.30` so the filter is not active by default.
- updates milli to v0.38.0

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-04 16:14:36 +00:00
Louis Dureuil
fcbd47281b
Fix tests 2023-01-04 14:24:20 +01:00
Louis Dureuil
5cb566b165
No longer filter out milli logs when --log-level is "info" 2023-01-04 14:24:20 +01:00
Louis Dureuil
c4aa5cc7d0
Merge --schedule-snapshot and --snapshot-interval-sec options 2023-01-04 14:13:54 +01:00
Loïc Lecrenier
d082ded7ad Explicitly restrict log level options to those that are documented
Fixes https://github.com/meilisearch/meilisearch/issues/3292
2023-01-04 09:40:24 +01:00
bors[bot]
947f08793a
Merge #3296
3296: Remove `--disable-auto-batching` CLI option r=gmourier a=loiclec

Fixes #3294 

The `index-scheduler` code is not modified, only the CLI options have changed.

Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
2023-01-03 16:57:14 +00:00
Louis Dureuil
17dac72464
Characters -> bytes 2023-01-03 15:31:02 +01:00
Loïc Lecrenier
b821c72459 Remove --disable-auto-batching CLI option 2023-01-03 15:01:04 +01:00
Louis Dureuil
7b2575c646
Master Key -> master key 2023-01-03 14:45:23 +01:00
bors[bot]
6425e06cf2
Merge #3274
3274: Reject master keys that are less than 16 bytes and add `--generate-master-key` CLI option r=irevoire a=dureuill

# Pull Request

## Related issue
Fix #3272 
Fix #3287

## What does this PR do?

### User standpoint

---

- Adds a `--generate-master-key` CLI flag to generate a fresh Master Key and exit.

<img width="1351" alt="Capture d’écran 2022-12-22 à 14 18 58" src="https://user-images.githubusercontent.com/41078892/209142778-eab52eeb-eaa8-409b-897a-c0d5728c8aaa.png">

---

(relevant fragment of the `--help` message)

<img width="1351" alt="Capture d’écran 2022-12-22 à 14 19 40" src="https://user-images.githubusercontent.com/41078892/209142891-ebfa2ed6-f231-4f76-a3ae-b7542c7aef04.png">

---

- When `meilisearch` is started in the `development` environment and no Master Key has been provided, then the binary prints a warning before starting.

<img width="1351" alt="Capture d’écran 2022-12-22 à 14 14 49" src="https://user-images.githubusercontent.com/41078892/209142158-54eba3b7-bf71-4f3f-8840-0600b13a1a9f.png">

---

- When `meilisearch` is started in the `development` environment and the provided Master Key is shorter than 16 bytes, then the binary prints a warning before starting.

<img width="1351" alt="Capture d’écran 2022-12-22 à 14 15 58" src="https://user-images.githubusercontent.com/41078892/209142295-0209fe47-c03b-424f-a73f-cee9b633137a.png">

---

- When `meilisearch` is started in the `production` environment, and no Master Key is provided, the error message is altered to generate a fresh Master Key.

<img width="1351" alt="Capture d’écran 2022-12-22 à 17 29 02" src="https://user-images.githubusercontent.com/41078892/209180540-0def5798-15db-47f0-a6ec-8cfa081dea77.png">


---

- When `meilisearch` is started in the `production` environment, and the provided Master Key is shorter than 16 bytes, then the binary exits with an error.

<img width="1351" alt="Capture d’écran 2022-12-22 à 17 28 47" src="https://user-images.githubusercontent.com/41078892/209180567-fa54fe33-fbc4-4b9f-b281-7dfb7b33af85.png">


---

This implements the solution B described here: https://github.com/meilisearch/product/discussions/538#discussioncomment-4391346 

### Implementation standpoint

- Add a new `meilisearch-auth::generate_master_key` function that uses a Cryptographic Random Number Generator (CRNG) to fill a vector of 32 bytes before encoding these bytes as base64

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
2023-01-02 16:00:40 +00:00
Tamo
1692f58b83
slightly update the message associated with the cli parameter + accept an env variable 2023-01-02 16:49:35 +01:00
Tamo
9ba4d0f921
update the error messages according to the spec 2023-01-02 16:43:23 +01:00
Louis Dureuil
3cba476a9f
Add --generate-master-key CLI option 2022-12-26 10:36:45 +01:00
Louis Dureuil
57e851d8a9
Check for key length 2022-12-26 10:36:45 +01:00
amab8901
9a39c4e40d Get date from IndexMetaData 2022-12-22 11:46:17 +01:00
amab8901
df176aaf01 Insert dump_reader.date() into create_raw_index(_) argument 2022-12-21 15:16:31 +01:00
amab8901
0893b175dc Merge branch 'main' into 2983-forward-date-to-milli 2022-12-21 14:31:19 +01:00
amab8901
d5978d11e1 Refactor 2022-12-21 14:28:00 +01:00
jiangbo212
2780e365e2 test update and ndjson serde use from_slice 2022-12-21 14:31:45 +08:00
jiangbo212
bf2a401a05 serde ndjson fix 2022-12-21 11:27:15 +08:00
Louis Dureuil
869d331680
Clippy fixes after updating Rust to v1.66 2022-12-19 14:17:12 +01:00
ManyTheFish
60ebf0ea0b Add a specific test on finite pagination placeolder search with distinct attributes 2022-12-15 17:28:20 +01:00
Louis Dureuil
ce84a59873
Re-apply some changes from #3132 2022-12-14 20:02:39 +01:00
Tamo
d66bb3a53f rename the two new functions 2022-12-14 17:27:43 +01:00
Tamo
6c0b8edab5 Fix typos
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2022-12-14 17:27:37 +01:00
Tamo
fbbc6eaeca Fix the import of dumps and snapshot.
Some flags were badly applied + the database wrongly deleted when they shouldn't
2022-12-14 17:27:28 +01:00
Kerollmops
7b2f2a4f9c
Do only one convertion to u64 2022-12-13 15:31:55 +01:00
Kerollmops
5d5615ef45
Rename the ReceivePayload error variant 2022-12-13 15:07:35 +01:00
jiangbo212
23c1b223b3 Merge branch 'fix-3037' of github.com:jiangbo212/meilisearch into fix-3037 2022-12-13 10:41:50 +08:00
jiangbo212
87ae0032bf review change 2022-12-13 10:41:43 +08:00
jiangbo212
7c24fea9f2 Merge branch 'main' into fix-3037 2022-12-13 05:16:03 +08:00
jiangbo212
fa46dfb7bb fmt fix 2022-12-12 22:02:56 +08:00
jiangbo212
169682d3ec Merge branch 'main' into fix-3037-new 2022-12-12 21:36:10 +08:00
Tamo
6f1c30b247 Fix the instance-uid in the data.ms
We were writing the instance-uid as bytes instead of string in the data.ms and thus we were unable to parse it later.
Also it was less practical for our user to retrieve it and send it to us.
2022-12-08 18:16:43 +01:00
Colby Allen
2262766494 chore: run fmt nightly on project 2022-12-08 08:31:15 -07:00
Colby Allen
ad2b1467da Renames meilisearch-http to meilisearch 2022-12-08 08:22:53 -07:00