Commit Graph

691 Commits

Author SHA1 Message Date
Clément Renault
d8b47b689e
Use the new read-txn-no-tls heed feature 2023-07-26 15:45:15 +02:00
Kerollmops
29ab54b259
Replace the hnsw crate by the instant-distance one 2023-07-25 12:37:35 +02:00
ManyTheFish
0497f93494 Update Charabia to the last version 2023-07-19 15:19:32 +02:00
meili-bors[bot]
2dfbb6813a
Merge #3913
3913: Expose a Puffin server to profile the indexing process r=Kerollmops a=Kerollmops

This PR exposes a puffin HTTP server to expose the internal timing it takes to index documents, delete documents, or update the settings of an index.

<img width="1752" alt="Capture d’écran 2023-07-10 à 18 44 58" src="https://github.com/meilisearch/meilisearch/assets/3610253/a3c7a6bf-db5b-42f4-8be1-c4e31c869843">

## To be done

 - [x] Move the puffin HTTP server under a feature flag.
 - [x] Use [the `puffin::set_scopes_on` function](https://docs.rs/puffin/latest/puffin/fn.set_scopes_on.html) to toggle it (by using the feature directly).
     When this function is called with `false`, [a call to `profile_scope!` talked 1-2ns](https://docs.rs/puffin/latest/puffin/fn.set_scopes_on.html).
 - [x] Create a _PROFILING.md_ file explaining how to use it.
   - [x] Explain that merging scopes on the interface is not always useful.
 - [x] Add more info on the number of batched tasks (using the `puffin::profile_scope!` macro data).
   - I added more info, but that's more continuous work when we consider we need more info here and there.
 - [x] Clean up some scopes, and don't touch too much code to inject puffin.
   - I am not sure that the _index_documents/mod.rs_ function is that complex with the addition of the scope.
 - [x] Think about what we consider frames. One indexation operation or the wall program. When must we stop the frame, then?
   - What we consider a frame is one single `IndexScheduler::tick` execution.
   - We can change that later.

Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2023-07-19 09:44:01 +00:00
Kerollmops
eef95de30e
First iteration on exposing puffin profiling 2023-07-18 17:38:13 +02:00
ManyTheFish
c106906f8f deactivate camelCase segmentation 2023-07-13 12:06:27 +02:00
Kerollmops
8ba1c8f88f
Update proc-macro2 to compile with the latest nightly 2023-07-12 11:47:27 +02:00
Kerollmops
e7f8daaf86
Update criterion to 0.5.1 to remove the atty dependency 2023-07-03 18:51:42 +02:00
Kerollmops
d1ff631df8
Replace the atty dependency with the is-terminal one 2023-07-03 18:51:42 +02:00
gillian-meilisearch
1d40452057 Update version for the next release (v1.3.0) in Cargo.toml 2023-07-03 08:32:21 +00:00
meili-bors[bot]
661d1f90dc
Merge #3866
3866: Update charabia v0.8.0 r=dureuill a=ManyTheFish

# Pull Request

Update Charabia:
- enhance Japanese segmentation
- enhance Latin Tokenization
  - words containing `_` are now properly segmented into several words
  - brackets `{([])}` are no more considered as context separators so word separated by brackets are now considered near together for the proximity ranking rule
- fixes #3815
- fixes #3778
- fixes [product#151](https://github.com/meilisearch/product/discussions/151)

> Important note: now the float numbers are segmented around the `.` so `3.22` is segmented as [`3`, `.`, `22`] but the middle dot isn't considered as a hard separator, which means that if we search `3.22` we find documents containing `3.22`

Co-authored-by: ManyTheFish <many@meilisearch.com>
2023-06-29 15:24:36 +00:00
ManyTheFish
e8dee3ca65 Update lock file 2023-06-29 17:02:24 +02:00
ManyTheFish
84845de9ef Update Charabia 2023-06-29 15:56:32 +02:00
Kerollmops
a385642ec3
Replace the BTreeMap by an IndexMap to return values in order 2023-06-29 14:33:31 +02:00
Kerollmops
737aec1705
Expose an _semanticSimilarity as a dot product in the documents 2023-06-27 12:32:41 +02:00
Kerollmops
c79e82c62a
Move back to the hnsw crate
This reverts commit 7a4b6c065482f988b01298642f4c18775503f92f.
2023-06-27 12:32:39 +02:00
Kerollmops
268a9ef416
Move to the hgg crate 2023-06-27 12:32:38 +02:00
Clément Renault
4571e512d2
Store the vectors in an HNSW in LMDB 2023-06-27 12:32:38 +02:00
Clément Renault
34349faeae
Create a new _vector extractor 2023-06-27 12:32:37 +02:00
meili-bors[bot]
45636d315c
Merge #3670
3670: Fix addition deletion bug r=irevoire a=irevoire

The first commit of this PR is a revert of https://github.com/meilisearch/meilisearch/pull/3667. It re-enable the auto-batching of addition and deletion of tasks. No new changes have been introduced outside of `milli`. So all the changes you see on the autobatcher have actually already been reviewed.

It fixes https://github.com/meilisearch/meilisearch/issues/3440.

### What was happening?

The issue was that the `external_documents_ids` generated in the `transform` were used in a very strange way that wasn’t compatible with the deletion of documents.
Instead of doing a clear merge between the external document IDs of the DB and the one returned by the transform + writing it on disk, we were doing some weird tricks with the soft-deleted to avoid writing the fst on disk as much as possible.
The new algorithm may be a bit slower but is way more straightforward and doesn’t change depending on if the soft deletion was used or not. Here is a list of the changes introduced:
1. We now do a clear distinction between the `new_external_documents_ids` coming from the transform and only held on RAM and the `external_documents_ids` coming from the DB.
2. The `new_external_documents_ids` (coming out of the transform) are now represented as an `fst`. We don't need to struggle with the hard, soft distinction + the soft_deleted => That's easier to understand
3. When indexing documents, we merge the `external_documents_ids` coming from the DB and the `new_external_documents_ids` coming from the transform.

### Other things introduced in this  PR

Since we constantly have to write small, very specialized fuzzers for this kind of bug, we decided to push the one used to reproduce this bug.
It's not perfect, but it's easy to improve in the future.
It'll also run for as long as possible on every merge on the main branch.

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Loïc Lecrenier <loic.lecrenier@icloud.com>
2023-06-19 09:09:30 +00:00
Tamo
6c6387d05e
move the fuzzer to its own crate 2023-05-29 12:27:39 +02:00
Tamo
4391cba6ca
fix the addition + deletion bug 2023-05-17 18:28:57 +02:00
Kerollmops
1a79fd0c3c
Use the new heed v0.12.6 2023-05-15 11:42:30 +02:00
Kerollmops
c4a40e7110
Use the writemap flag to reduce the memory usage 2023-05-15 10:15:33 +02:00
curquiza
3533d4f2bb Update version for the next release (v1.2.0) in Cargo.toml 2023-05-08 17:52:33 +00:00
Louis Dureuil
90bc230820
Merge remote-tracking branch 'origin/main' into search-refactor
Conflicts | resolution
----------|-----------
Cargo.lock | added mimalloc
Cargo.toml |  took origin/main version
milli/src/search/criteria/exactness.rs | deleted after checking it was only clippy changes
milli/src/search/query_tree.rs | deleted after checking it was only clippy changes
2023-05-03 12:19:06 +02:00
ManyTheFish
1bf2694604 Update cargo lock 2023-04-26 17:41:29 +02:00
Kerollmops
a109802d45
Upgrade the incompatible versions of the dependencies 2023-04-24 17:50:57 +02:00
Kerollmops
47b66e49b8
Upgrade the compatible versions of the dependencies 2023-04-24 17:50:52 +02:00
bors[bot]
654a3a9e19
Merge #3688
3688: Following release v1.1.1: bring back changes into `main` r=curquiza a=curquiza

`@meilisearch/engine-team` ensure the changes we bring to `main` are the ones you want

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: dureuill <dureuill@users.noreply.github.com>
2023-04-24 11:38:23 +00:00
dependabot[bot]
f0b4046c43
Bump h2 from 0.3.15 to 0.3.17
Bumps [h2](https://github.com/hyperium/h2) from 0.3.15 to 0.3.17.
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.3.15...v0.3.17)

---
updated-dependencies:
- dependency-name: h2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-13 17:03:48 +00:00
dureuill
cd45d21d6e Update version for the next release (v1.1.1) in Cargo.toml 2023-04-13 13:25:10 +00:00
Loïc Lecrenier
9259cdb12e Update Cargo.lock (was mistakenly changed during rebase) 2023-03-20 09:41:56 +01:00
Loïc Lecrenier
6c659dc12f Use MiMalloc in milli tests 2023-03-20 09:41:37 +01:00
curquiza
577e7126f9 Update version for the next release (v1.1.0) in Cargo.toml 2023-03-06 13:52:54 +00:00
Louis Dureuil
42577403d8
Authentication: Directly pass the authfilter to the index scheduler 2023-02-22 16:35:52 +01:00
bors[bot]
39407885c2
Merge #3347
3347: Enhance language detection r=irevoire a=ManyTheFish

## Summary

Some completely unrelated Languages can share the same characters, in Meilisearch we detect the Languages using `whatlang`, which works well on large texts but fails on small search queries leading to a bad segmentation and normalization of the query.

This PR now stores the Languages detected during the indexing in order to reduce the Languages list that can be detected during the search.

## Detail

- Create a 19th database mapping the scripts and the Languages detected with the documents where the Language is detected
- Fill the newly created database during indexing
- Create an allow-list with this database and pass it to Charabia
- Add a test ensuring that a Japanese request containing kanjis only is detected as Japanese and not Chinese

## Related issues
Fixes #2403
Fixes #3513

Co-authored-by: f3r10 <frledesma@outlook.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2023-02-21 10:52:13 +00:00
ManyTheFish
8aa808d51b Merge branch 'main' into enhance-language-detection 2023-02-20 18:14:34 +01:00
bors[bot]
1e9ac00800
Merge #3505
3505: Csv delimiter r=irevoire a=irevoire

Fixes https://github.com/meilisearch/meilisearch/issues/3442
Closes https://github.com/meilisearch/meilisearch/pull/2803
Specified in https://github.com/meilisearch/specifications/pull/221

This PR is a reimplementation of https://github.com/meilisearch/meilisearch/pull/2803, on the new engine. Thanks for your idea and initial PR `@MixusMinimax;` sorry I couldn’t update/merge your PR. Way too many changes happened on the engine in the meantime.

**Attention to reviewer**; I had to update deserr to implement the support of deserializing `char`s

-------

It introduces four new error messages;
- Invalid value in parameter csvDelimiter: expected a string of one character, but found an empty string
- Invalid value in parameter csvDelimiter: expected a string of one character, but found the following string of 5 characters: doggo
- csv delimiter must be an ascii character. Found: 🍰 
- The Content-Type application/json does not support the use of a csv delimiter. The csv delimiter can only be used with the Content-Type text/csv.

And one error code;
- `invalid_index_csv_delimiter`

The `invalid_content_type` error code is now also used when we encounter the `csvDelimiter` query parameter with a non-csv content type.

Co-authored-by: Tamo <tamo@meilisearch.com>
2023-02-20 17:01:36 +00:00
ManyTheFish
cb8d5f2d4b Update Charabia to 0.7.1 2023-02-20 14:00:31 +01:00
Tamo
8c074f5028 implements the csv delimiter without tests
Co-authored-by: Maxi Barmetler <maxi.barmetler@gmail.com>
2023-02-16 17:35:36 +01:00
Tamo
74d1a67a99 Use the workspace inheritance feature of rust 1.64 2023-02-15 13:51:07 +01:00
Tamo
a43765d454
use the pre-defined deserr extractors 2023-02-14 20:05:30 +01:00
Tamo
8fb7b1d10f
bump deserr 2023-02-14 20:04:30 +01:00
Clément Renault
4570d5bf3a
Merge remote-tracking branch 'origin/main' into temp-wildcard 2023-02-09 13:14:05 +01:00
dependabot[bot]
5f56e6dd58
Bump tokio from 1.24.1 to 1.24.2
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.24.1 to 1.24.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-07 12:14:05 +00:00
bors[bot]
c88c3637b4
Merge #3461
3461: Bring v1 changes into main r=curquiza a=Kerollmops

Also bring back changes in milli (the remote repository) into main done during the pre-release

Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Philipp Ahlner <philipp@ahlner.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
2023-02-07 11:27:27 +00:00
bors[bot]
821d92b5d0
Merge #3407
3407: Add Cargo feature for LMDB's POSIX semaphores r=dureuill a=GregoryConrad

See https://github.com/meilisearch/milli/pull/757

Co-authored-by: Gregory Conrad <gregorysconrad@gmail.com>
2023-02-07 08:25:20 +00:00
Kerollmops
fbec48f56e
Merge remote-tracking branch 'milli/main' into bring-v1-changes 2023-02-06 16:48:10 +01:00
Kerollmops
a377a49218
Make meiliserach depend on the local milli 2023-02-06 16:44:43 +01:00
Kerollmops
a015e232ab
Merge remote-tracking branch 'origin/release-v1.0.0' into bring-v1-changes 2023-02-06 16:41:10 +01:00
Kerollmops
d563ed8a39
Making it work with index uid patterns 2023-02-01 17:51:30 +01:00
ManyTheFish
f4569b04ad Update Charabia version 2023-02-01 15:26:26 +01:00
Louis Dureuil
231067a1c4
Bump milli to v0.41.1 2023-02-01 11:53:39 +01:00
Tamo
8356f109c1 bump milli to fix the last test 2023-01-25 16:45:11 +01:00
bors[bot]
217504fff3
Merge #3406
3406: Master Key: Implements errors and warnings from the specification r=irevoire a=dureuill

<sub>Now in technicolor</sub>

# Pull Request

## What does this PR do?
- Uses `atty` and `termcolor` as dependency
- Use these dependencies to print colored background for warning messages
- Update messages to match https://github.com/meilisearch/specifications/pull/209

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ ] Have you read the contributing guidelines?
- [ ] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2023-01-23 16:39:18 +00:00
Tamo
c79b6a1ee4
bump milli 2023-01-23 14:13:19 +01:00
dependabot[bot]
5f4497935f
Bump libgit2-sys from 0.14.1+1.5.0 to 0.14.2+1.5.1
Bumps [libgit2-sys](https://github.com/rust-lang/git2-rs) from 0.14.1+1.5.0 to 0.14.2+1.5.1.
- [Release notes](https://github.com/rust-lang/git2-rs/releases)
- [Commits](https://github.com/rust-lang/git2-rs/compare/0.14.1...libgit2-sys-0.14.2)

---
updated-dependencies:
- dependency-name: libgit2-sys
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-20 23:38:40 +00:00
Gregory Conrad
3f69dd6450 feat: add Cargo feature for LMDB's POSIX semaphores 2023-01-19 12:08:38 -05:00
Louis Dureuil
0de9a3ffe7
Implements errors and warnings from the specification
Now in technicolor
2023-01-19 18:04:45 +01:00
Loïc Lecrenier
c71a8ea183 Update to latest milli and deserr 2023-01-17 13:10:38 +01:00
Loïc Lecrenier
07b90dec08 Remove unused proptest dependency 2023-01-17 11:07:07 +01:00
Loïc Lecrenier
9194508a0f Refactor query parameter deserialisation logic 2023-01-17 11:07:07 +01:00
Loïc Lecrenier
766dd830ae Update deserr to latest version + add new error codes for missing fields
- missing_api_key_indexes
- missing_api_key_actions
- missing_api_key_expires_at

- missing_swap_indexes_indexes
2023-01-17 09:43:07 +01:00
Loïc Lecrenier
436ae4e466 Improve error messages generated by deserr
Split Json and Query Parameter error types
2023-01-17 09:43:07 +01:00
Kerollmops
507a7bad96
Use the local milli subcrate 2023-01-16 17:35:54 +01:00
Kerollmops
cde62fcb5b
Merge remote-tracking branch 'origin/release-v1.0.0' into import-milli 2023-01-16 17:35:18 +01:00
Kerollmops
03a82136dc
Remove the useless cli subcrate 2023-01-16 17:08:43 +01:00
Kerollmops
97005dd505
Bump the milli-imported crates to v1.0.0 2023-01-16 16:29:12 +01:00
Kerollmops
0cec352d2b
Merge remote-tracking branch 'milli/main' into import-milli 2023-01-16 16:20:22 +01:00
Tamo
bf573885ea
integrate the latest version of milli 2023-01-11 19:08:39 +01:00
Loïc Lecrenier
b0b7ad7caf
Apply review suggestions 2023-01-11 19:08:39 +01:00
Loïc Lecrenier
c91ffec72e
Update Cargo.toml 2023-01-11 19:08:39 +01:00
Loïc Lecrenier
1fc11264e8
Refactor deserr integration 2023-01-11 19:08:39 +01:00
Tamo
d17efb9ed6
use the published version of deserr 2023-01-09 12:51:10 +01:00
Tamo
50ce0409bc
Integrate deserr on the most important routes 2023-01-05 20:48:29 +01:00
curquiza
28408816ef Update version for the next release (v1.0.0) in Cargo.toml files 2023-01-05 11:45:15 +00:00
Loïc Lecrenier
2d74678b51 Replace underscores with hyphens in doc link to error code 2023-01-05 10:09:02 +01:00
Louis Dureuil
fcbd47281b
Fix tests 2023-01-04 14:24:20 +01:00
Louis Dureuil
0e98a71a24
Update milli to v0.38 2023-01-04 14:24:20 +01:00
Louis Dureuil
66e18eae79
auth: add generate_master_key function 2022-12-22 11:55:27 +01:00
Tamo
d8fb506c92
handle most io error instead of tagging everything as an internal 2022-12-19 20:50:40 +01:00
Kerollmops
60c3bac108 Bump milli to v0.37.3 2022-12-14 17:25:40 +01:00
jiangbo212
7c24fea9f2 Merge branch 'main' into fix-3037 2022-12-13 05:16:03 +08:00
curquiza
4631f4d97f Bump milli to v0.37.2 2022-12-08 18:16:48 +01:00
Colby Allen
ad2b1467da Renames meilisearch-http to meilisearch 2022-12-08 08:22:53 -07:00
curquiza
026cf223b3 Update version for the next release (v1.0.0) in Cargo.toml files 2022-12-08 12:20:17 +00:00
jiangbo212
717dd36547 Merge branch 'fix-3037' of github.com:jiangbo212/meilisearch into fix-3037 2022-12-07 22:54:16 +08:00
Kerollmops
1d5294d11a
Bump lmdb version 2022-12-07 15:29:56 +01:00
bors[bot]
2ed6017603
Merge #3204
3204: Bring back v0.30.1 changes to `main` r=curquiza a=curquiza

I was not able to just import `release-v0.30.1` to `main`, see:
<img width="1371" alt="Capture d’écran 2022-12-06 à 20 03 50" src="https://user-images.githubusercontent.com/20380692/206000844-b39b3063-7da2-475f-b3e4-1791c39a7c2f.png">

So I cherry-picked the commits.

⚠️ ⚠️ ⚠️ I had a git conflict here

<img width="730" alt="Capture d’écran 2022-12-06 à 20 09 04" src="https://user-images.githubusercontent.com/20380692/206001007-f56bc28f-c0b1-46a0-bb60-cce4e73b9584.png">


⚠️ ⚠️ ⚠️ Check out carefully how I fixed it


Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-12-07 11:08:37 +00:00
bors[bot]
9acac28574
Merge #3128
3128: Bumps cargo_toml version to most up to date r=curquiza a=colbsmcdolbs

# Pull Request

## Related issue
Fixes #3127

## What does this PR do?
- The README of this repository declares that one package is not up to date. In order to ensure Due Diligence, I have bumped the version number of the package. No test failures running on Windows.

## PR checklist
Please check if your PR fulfills the following requirements:
- [X] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [X] Have you read the contributing guidelines?
- [X] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Colby Allen <colbyjayallen@gmail.com>
2022-12-07 10:31:25 +00:00
Kerollmops
e4e4370a3c Clamp the databases size to the page size 2022-12-06 20:09:49 +01:00
Kerollmops
24c79b79f9 Bump milli to v0.37.1 2022-12-06 20:05:52 +01:00
curquiza
5db7c4057c Update version for the next release (v0.30.1) in Cargo.toml files 2022-12-06 20:05:46 +01:00
jiangbo212
7b08d700f7 requested changes fix 2022-12-03 18:52:20 +08:00
jiangbo212
9c28632498 Merge branch 'main' into fix-3037 2022-11-30 09:38:01 +08:00
jiangbo212
38982d13fe fix issue 3037 2022-11-30 00:03:22 +08:00
Clémentine Urquizar - curqui
457a473b72
Bring back release-v0.30.0 into release-v0.30.0-temp (final: into main) (#3145)
* Fix error code of the "duplicate index found" error

* Use the content of the ProcessingTasks in the tasks cancelation system

* Change the missing_filters error code into missing_task_filters

* WIP Introduce the invalid_task_uid error code

* Use more precise error codes/message for the task routes

+ Allow star operator in delete/cancel tasks
+ rename originalQuery to originalFilters
+ Display error/canceled_by in task view even when they are = null
+ Rename task filter fields by using their plural forms
+ Prepare an error code for canceledBy filter
+ Only return global tasks if the API key action `index.*` is there

* Add canceledBy task filter

* Update tests following task API changes

* Rename original_query to original_filters everywhere

* Update more insta-snap tests

* Make clippy happy

They're a happy clip now.

* Make rustfmt happy

>:-(

* Fix Index name parsing error message to fit the specification

* Bump milli version to 0.35.1

* Fix the new error messages

* fix the error messages and add tests

* rename the error codes for the sake of consistency

* refactor the way we send the cli informations + add the analytics for the config file and ssl usage

* Apply suggestions from code review

Co-authored-by: Clément Renault <clement@meilisearch.com>

* add a comment over the new infos structure

* reformat, sorry @kero

* Store analytics for the documents deletions

* Add analytics on all the settings

* Spawn threads with names

* Spawn rayon threads with names

* update the distinct attributes to the spec update

* update the analytics on the search route

* implements the analytics on the health and version routes

* Fix task details serialization

* Add the question mark to the task deletion query filter

* Add the question mark to the task cancelation query filter

* Fix tests

* add analytics on the task route

* Add all the missing fields of the new task query type
* Create a new analytics for the task deletion
* Create a new analytics for the task creation

* batch the tasks seen events

* Update the finite pagination analytics

* add the analytics of the swap-indexes route

* Stop removing the DB when failing to read it

* Rename originalFilters into originalFilters

* Rename matchedDocuments into providedIds

* Add `workflow_dispatch` to flaky.yml

* Bump grenad to 0.4.4

* Bump milli to version v0.37.0

* Don't multiply total memory returned by sysinfo anymore

sysinfo now returns bytes rather than KB

* Add a dispatch to the publish binaries workflow

* Fix publish release CI

* Don't use gold but the default linker

* Always display details for the indexDeletion task

* Fix the insta tests

* refactorize the whole test suite
1. Make a call to assert_internally_consistent automatically when snapshoting the scheduler. There is no point in snapshoting something broken and expect the dumb humans to notice.
2. Replace every possible call to assert_internally_consistent by a snapshot of the scheduler. It takes as many lines and ensure we never change something without noticing in any tests ever.
3. Name every snapshots: it's easier to debug when something goes wrong and easier to review in general.
4. Stop skipping breakpoints, it's too easy to miss something. Now you must explicitely show which path is the scheduler supposed to use.
5. Add a timeout on the channel.recv, it eases the process of writing tests, now when something file you get a failure instead of a deadlock.

* rebase on release-v0.30

* makes clippy happy

* update the snapshots after a rebase

* try to remove the flakyness of the failing test

* Add more analytics on the ranking rules positions

* Update the dump test to check for the dumpUid dumpCreation task details

* send the ranking rules as a string because amplitude is too dumb to process an array as a single value

* Display a null dumpUid until we computed the dump itself on disk

* Update tests

* Check if the master key is missing before returning an error

Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2022-11-28 16:27:41 +01:00
Colby Allen
6ecd486d1b Bumps cargo_toml version to most up to date 2022-11-23 16:27:54 -07:00
curquiza
68f80dbacf Update version for the next release (v0.30.0) in Cargo.toml files 2022-10-27 11:35:44 +00:00
Irevoire
6aa816d96a
use meili-snap in the dump 2022-10-27 11:35:22 +02:00
Irevoire
866a3676eb
reupload the test fix for the dump 2022-10-27 11:35:22 +02:00
Irevoire
953055e3d7
bump milli 2022-10-27 11:35:21 +02:00
Irevoire
8ebb49d1b1
bump milli 2022-10-27 11:35:19 +02:00
Kerollmops
942b7c338b
Compress the snapshot in a tarball 2022-10-27 11:35:15 +02:00
Kerollmops
4cafc63561
Reintroduce the versioning functions 2022-10-27 11:35:14 +02:00
Loïc Lecrenier
22cf0559fe
Implement task date filters
before/after enqueued/started/finished at
2022-10-27 11:34:14 +02:00
Kerollmops
b2c5bc67b7
Add more enum-iterator related stuff 2022-10-27 11:34:11 +02:00
Tamo
d1a6fb2971
bump enum-iter and fix a bunch of error messages 2022-10-27 11:34:09 +02:00
Tamo
78ce29f461
apply most style comments of the review 2022-10-27 11:34:08 +02:00
Tamo
a9eeb070b8
fix all the errors code and settings issues when importing a dump v2 2022-10-27 11:34:07 +02:00
Tamo
d976e680c5
first mostly working version 2022-10-27 11:34:06 +02:00
Loïc Lecrenier
9522b75454
Continue implementation of task deletion
1. Matched tasks are a roaring bitmap
2. Start implementation in meilisearch-http
3. Snapshots use meili-snap
4. Rename to TaskDeletion
2022-10-27 11:34:03 +02:00
Tamo
7034803712
move the API key in meilisearch_types 2022-10-27 11:34:02 +02:00
Tamo
b6c84e53ba
uncomment a task serialization test 2022-10-27 11:34:02 +02:00
Tamo
2f1eb78b1d
refactor the Task a little bit 2022-10-27 11:34:02 +02:00
Tamo
510ce9fc51
start moving a lot of task types to meilisearch_types 2022-10-27 11:34:01 +02:00
Tamo
fa4c1de019
store md5 instead of the whole snapshots 2022-10-27 11:34:01 +02:00
Loïc Lecrenier
3e4337c91f
Add meili-snap crate to make writing snapshot tests easier 2022-10-27 11:34:01 +02:00
Tamo
667c282e19
get rids of the index crate + the document_types crate 2022-10-27 11:34:00 +02:00
Loïc Lecrenier
4c55c30027
Add a DetailsView type and improve index scheduler snapshots
The DetailsView type is necessary because serde incorrectly
deserialises the `Details` type, so the database fails to correctly
decode Tasks
2022-10-27 11:33:59 +02:00
Tamo
47e0288747
rewrite the compat API to something more generic 2022-10-27 11:33:53 +02:00
Tamo
e845cc2b6f
fix the tests 2022-10-27 11:33:51 +02:00
Tamo
7bd6f63001
implement the dump reader v6 2022-10-27 11:33:51 +02:00
Tamo
87a6a337aa
write a dump exporter 2022-10-27 11:33:51 +02:00
Kerollmops
91e13c2824
Implement ErrorCode on the milli::Error type 2022-10-27 11:33:48 +02:00
Tamo
2c8f1a43e9
get rids of meilisearch-lib 2022-10-27 11:33:44 +02:00
Tamo
6f4dcc0c38
start implementing some logic to test the internal states of the scheduler 2022-10-27 11:33:43 +02:00
Tamo
84cd5cef0b
fix the tests 2022-10-27 11:33:43 +02:00
Irevoire
8d51c1f389
wip integrating the scheduler in meilisearch-http 2022-10-27 11:33:42 +02:00
Irevoire
250410495c
start integrating the index-scheduler in meilisearch-lib 2022-10-27 11:33:42 +02:00
Irevoire
8f0fd35358
add insta::json for later 2022-10-27 11:33:42 +02:00
Irevoire
8770e07397
I can index documents without meilisearch 2022-10-27 11:33:42 +02:00
Tamo
7d0c8a3379
test the register tasks 2022-10-27 11:33:40 +02:00
Tamo
fc098022c7
start integrating the index-scheduler in the meilisearch codebase 2022-10-27 11:33:40 +02:00
Tamo
366a344474
get rids of the horrendous spinlock in favor of synchronoise 2022-10-27 11:33:39 +02:00
Tamo
c97d51a624
add a bunch of tests 2022-10-27 11:33:39 +02:00
Tamo
94e29a9f5f
extract the index abstraction out of the index-scheduler in its own module 2022-10-27 11:33:38 +02:00
Tamo
76597fc382
import the update_file_store in the index-scheduler 2022-10-27 11:33:37 +02:00
Tamo
2afb381f95
get rids of nelson 2022-10-27 11:33:37 +02:00
Tamo
a9844bd4f6
move the update file store to another crate with as little dependencies as possible 2022-10-27 11:33:37 +02:00
Tamo
46b8ebcab4
fix the file store 2022-10-27 11:33:35 +02:00
Tamo
a7aa92df5f
fix most of the index module 2022-10-27 11:33:34 +02:00
Irevoire
d8b8e04ad1
wip porting the index back in the scheduler 2022-10-27 11:33:34 +02:00
Tamo
1a47949063
START THE REWRITE OF THE INDEX SCHEDULER: index & register has been implemented 2022-10-27 11:33:33 +02:00
ManyTheFish
4afed4de4f stabilize milli 2022-10-24 14:16:41 +02:00
ManyTheFish
c02ae4dfc0 Update roaring 2022-10-19 14:25:43 +02:00
ManyTheFish
062d17fbc0 Use a milli version that compute exhaustivelly the number of hits 2022-10-19 14:05:42 +02:00
bors[bot]
b0749407f3
Merge #2804
2804: Add environement variable `MEILI_CONFIG_FILE_PATH` to define the config file path r=Kerollmops a=choznerol

# Pull Request

## What does this PR do?
Fixes #2800

~This is a draft PR base on the code in #2745. I will `rebase` and mark it ready for review only after #2745 merge.~ Done rebase


## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?


## Demo

With `config.toml`, `config_copy1.toml` and `config_copy2.toml` present:
> <img width="692" alt="image" src="https://user-images.githubusercontent.com/12410942/192566891-6b3c9d26-736f-4e23-a09b-687fca1cb50d.png">

`MEILI_CONFIG_FILE_PATH` works:
> <img width="773" alt="image" src="https://user-images.githubusercontent.com/12410942/192567023-f751536e-992a-4e90-a176-cb19122248be.png">

`--config-file-path` still works:
> <img width="768" alt="image" src="https://user-images.githubusercontent.com/12410942/192567318-88c80b24-7873-4cec-8d08-16fe4d228055.png">

When both present, `--config-file-path` taks precedence:
> <img width="1214" alt="image" src="https://user-images.githubusercontent.com/12410942/192567477-8a7cffe1-96f0-42a9-a348-6dbec20dc1e7.png">



Co-authored-by: Lawrence Chou <choznerol@protonmail.com>
2022-10-17 10:40:05 +00:00
bors[bot]
5d895dd7da
Merge #2851
2851: Upgrade clap to 4.0 r=loiclec a=choznerol

# Pull Request

## Related issue
Fixes #2846 

This PR is draft based on #2847 to avoid conflict. I will rebase and mark as 'Ready for review' after #2847 is merged.

## What does this PR do?
1. Upgrade clap to the latest version or 4.0 (4.0.9 as of today) by following the [migrating instruction](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating) from [4.0 changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating)
2. Fix an `ArgGroup` typo that can only be caught after upgrading to 4.0 in 20a715e29ed17c5a76229c98fb31504ada873597

## Notable changes

### The `--help` message

The format, ordering and indentation of `--help` message was changed in 4.0. I recorded the output of `cargo run -- --help` before and after upgrade to 4.0 for reference.

<details>
<summary>diff</summary>

Output of `diff --ignore-all-space --text --unified --new-file help-message-before.txt help-message-after.txt`:

```diff
--- help-message-before.txt	2022-10-14 16:45:36.000000000 +0800
+++ help-message-after.txt	2022-10-14 16:36:53.000000000 +0800
`@@` -1,12 +1,8 `@@`
-meilisearch-http 0.29.1
+Usage: meilisearch [OPTIONS]
 
-USAGE:
-    meilisearch [OPTIONS]
-
-OPTIONS:
+Options:
         --config-file-path <CONFIG_FILE_PATH>
-            Set the path to a configuration file that should be used to setup the engine. Format
-            must be TOML
+          Set the path to a configuration file that should be used to setup the engine. Format must be TOML
 
         --db-path <DB_PATH>
             Designates the location where database files will be created and retrieved
`@@` -26,15 +22,14 `@@`
             [default: dumps/]
 
         --env <ENV>
-            Configures the instance's environment. Value must be either `production` or
-            `development`
+          Configures the instance's environment. Value must be either `production` or `development`
             
             [env: MEILI_ENV=]
             [default: development]
             [possible values: development, production]
 
     -h, --help
-            Print help information
+          Print help information (use `-h` for a summary)
 
         --http-addr <HTTP_ADDR>
             Sets the HTTP address and port Meilisearch will use
`@@` -43,63 +38,53 `@@`
             [default: 127.0.0.1:7700]
 
         --http-payload-size-limit <HTTP_PAYLOAD_SIZE_LIMIT>
-            Sets the maximum size of accepted payloads. Value must be given in bytes or explicitly
-            stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+          Sets the maximum size of accepted payloads. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
             
             [env: MEILI_HTTP_PAYLOAD_SIZE_LIMIT=]
             [default: 100000000]
 
         --ignore-dump-if-db-exists
-            Prevents a Meilisearch instance with an existing database from throwing an error when
-            using `--import-dump`. Instead, the dump will be ignored and Meilisearch will launch
-            using the existing database.
+          Prevents a Meilisearch instance with an existing database from throwing an error when using `--import-dump`. Instead, the dump will be ignored and Meilisearch will launch using the existing database.
             
             This option will trigger an error if `--import-dump` is not defined.
             
             [env: MEILI_IGNORE_DUMP_IF_DB_EXISTS=]
 
         --ignore-missing-dump
-            Prevents Meilisearch from throwing an error when `--import-dump` does not point to a
-            valid dump file. Instead, Meilisearch will start normally without importing any dump.
+          Prevents Meilisearch from throwing an error when `--import-dump` does not point to a valid dump file. Instead, Meilisearch will start normally without importing any dump.
             
             This option will trigger an error if `--import-dump` is not defined.
             
             [env: MEILI_IGNORE_MISSING_DUMP=]
 
         --ignore-missing-snapshot
-            Prevents a Meilisearch instance from throwing an error when `--import-snapshot` does not
-            point to a valid snapshot file.
+          Prevents a Meilisearch instance from throwing an error when `--import-snapshot` does not point to a valid snapshot file.
             
             This command will throw an error if `--import-snapshot` is not defined.
             
             [env: MEILI_IGNORE_MISSING_SNAPSHOT=]
 
         --ignore-snapshot-if-db-exists
-            Prevents a Meilisearch instance with an existing database from throwing an error when
-            using `--import-snapshot`. Instead, the snapshot will be ignored and Meilisearch will
-            launch using the existing database.
+          Prevents a Meilisearch instance with an existing database from throwing an error when using `--import-snapshot`. Instead, the snapshot will be ignored and Meilisearch will launch using the existing database.
             
             This command will throw an error if `--import-snapshot` is not defined.
             
             [env: MEILI_IGNORE_SNAPSHOT_IF_DB_EXISTS=]
 
         --import-dump <IMPORT_DUMP>
-            Imports the dump file located at the specified path. Path must point to a `.dump` file.
-            If a database already exists, Meilisearch will throw an error and abort launch
+          Imports the dump file located at the specified path. Path must point to a `.dump` file. If a database already exists, Meilisearch will throw an error and abort launch
             
             [env: MEILI_IMPORT_DUMP=]
 
         --import-snapshot <IMPORT_SNAPSHOT>
-            Launches Meilisearch after importing a previously-generated snapshot at the given
-            filepath
+          Launches Meilisearch after importing a previously-generated snapshot at the given filepath
             
             [env: MEILI_IMPORT_SNAPSHOT=]
 
         --log-level <LOG_LEVEL>
             Defines how much detail should be present in Meilisearch's logs.
             
-            Meilisearch currently supports five log levels, listed in order of increasing verbosity:
-            ERROR, WARN, INFO, DEBUG, TRACE.
+          Meilisearch currently supports five log levels, listed in order of increasing verbosity: ERROR, WARN, INFO, DEBUG, TRACE.
             
             [env: MEILI_LOG_LEVEL=]
             [default: INFO]
`@@` -110,31 +95,25 `@@`
             [env: MEILI_MASTER_KEY=]
 
         --max-index-size <MAX_INDEX_SIZE>
-            Sets the maximum size of the index. Value must be given in bytes or explicitly stating a
-            base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+          Sets the maximum size of the index. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
             
             [env: MEILI_MAX_INDEX_SIZE=]
             [default: 107374182400]
 
         --max-indexing-memory <MAX_INDEXING_MEMORY>
-            Sets the maximum amount of RAM Meilisearch can use when indexing. By default,
-            Meilisearch uses no more than two thirds of available memory
+          Sets the maximum amount of RAM Meilisearch can use when indexing. By default, Meilisearch uses no more than two thirds of available memory
             
             [env: MEILI_MAX_INDEXING_MEMORY=]
             [default: "21.33 TiB"]
 
         --max-indexing-threads <MAX_INDEXING_THREADS>
-            Sets the maximum number of threads Meilisearch can use during indexation. By default,
-            the indexer avoids using more than half of a machine's total processing units. This
-            ensures Meilisearch is always ready to perform searches, even while you are updating an
-            index
+          Sets the maximum number of threads Meilisearch can use during indexation. By default, the indexer avoids using more than half of a machine's total processing units. This ensures Meilisearch is always ready to perform searches, even while you are updating an index
             
             [env: MEILI_MAX_INDEXING_THREADS=]
             [default: 5]
 
         --max-task-db-size <MAX_TASK_DB_SIZE>
-            Sets the maximum size of the task database. Value must be given in bytes or explicitly
-            stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+          Sets the maximum size of the task database. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
             
             [env: MEILI_MAX_TASK_DB_SIZE=]
             [default: 107374182400]
```

- ~[help-message-before.txt](https://github.com/meilisearch/meilisearch/files/9715683/help-message-before.txt)~ [help-message-before.txt](https://github.com/meilisearch/meilisearch/files/9784156/help-message-before-2.txt)
- ~[help-message-after.txt](https://github.com/meilisearch/meilisearch/files/9715682/help-message-after.txt)~ [help-message-after.txt](https://github.com/meilisearch/meilisearch/files/9784091/help-message-after.txt)

</details>


## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Lawrence Chou <choznerol@protonmail.com>
2022-10-17 10:08:27 +00:00
Lawrence Chou
3c3ae3ff98
Impeove invalid config_file_path handling
1. Besides opt.config_file_path, also consider MEILI_CONFIG_FILE_PATH in the Err path because they are both user input.
2. Print out the incorrect file path in error message.
3. Add tests
https://github.com/meilisearch/meilisearch/pull/2804#discussion_r991999888
2022-10-12 12:04:48 +08:00
Andrey "MOU" Larionov
11b986a81d
Added support for specifying compression in tests
Refactored tests code to allow to specify compression (content-encoding) algorithm.

Added tests to verify what actix actually handle different content encodings properly.
2022-10-09 22:09:29 +02:00
Lawrence Chou
9e5ef8eb69
Upgrade clap to v4
Close #2846

4.0.0 changelog: 'https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#400---2022-09-28'

I followed the [Migrating steps](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating) and the only issue I encountered are:
1. The typo problem in previous commit "Fix clap ArgGroup typo"
2. I can't say I am 100% sure every [Subtle changes](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#breaking-changes) is fine for our use case, but at least after a quick read I didn't notice anything actionable.
2022-10-07 00:32:25 +08:00
bors[bot]
1b72eba1f3
Merge #2867
2867: Bring back `stable` into `main` r=Kerollmops a=curquiza

Following hotfix for v0.29.1

Co-authored-by: Loïc Lecrenier <loic@meilisearch.com>
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
2022-10-06 14:14:23 +00:00
Loïc Lecrenier
05af8f0e46 Update version for next release (v0.29.1) 2022-10-06 10:27:11 +02:00
Loïc Lecrenier
a2c91a87fe Upgrade dependencies
Except:
- clap stays on 3.0 because it is more complicated to upgrade
- enum_iterator goes up to 1.1.2 instead of 1.2 because of the vergen
dependency
2022-10-05 15:53:02 +02:00
Clémentine Urquizar - curqui
a7d2c9572e
Merge branch 'main' into stable 2022-10-04 14:20:10 +02:00
bors[bot]
fcedce8578
Merge #2745 #2789 #2814 #2826
2745: Config file support r=curquiza a=mlemesle

# Pull Request

## What does this PR do?
Fixes #2558

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


2789: Fix typos r=Kerollmops a=kianmeng

# Pull Request

## What does this PR do?

Found via `codespell -L crate,nam,hart`.

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


2814: Skip dashboard test if mini-dashboard feature is disabled r=Kerollmops a=jirutka

Fixes #2813

Fixes the following error:

    cargo test --no-default-features
    ...
    error: couldn't read target/debug/build/meilisearch-http-ec029d8c902cf2cb/out/generated.rs: No such file or directory (os error 2)
     --> meilisearch-http/tests/dashboard/mod.rs:8:9
      |
    8 |         include!(concat!(env!("OUT_DIR"), "/generated.rs"));
      |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |
      = note: this error originates in the macro `include` (in Nightly builds, run with -Z macro-backtrace for more info)

    error: could not compile `meilisearch-http` due to previous error

2826: Rename receivedDocumentIds into matchedDocuments r=Kerollmops a=Ugzuzg

# Pull Request

## What does this PR do?

Fixes #2799 
Changes DocumentDeletion task details response.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Tested with curl:
```
curl \
  -X POST 'http://localhost:7700/indexes/movies/documents/delete-batch' \
  -H 'Content-Type: application/json' \
  --data-binary '[
    23488,
    153738,
    437035,
    363869
  ]'

{"taskUid":1,"indexUid":"movies","status":"enqueued","type":"documentDeletion","enqueuedAt":"2022-10-01T20:06:37.105416054Z"}%

curl \
  -X GET 'http://localhost:7700/tasks/1'

{"uid":1,"indexUid":"movies","status":"succeeded","type":"documentDeletion","details":{"matchedDocuments":4,"deletedDocuments":2},"duration":"PT0.005708322S","enqueuedAt":"2022-10-01T20:06:37.105416054Z","startedAt":"2022-10-01T20:06:37.115562733Z","finishedAt":"2022-10-01T20:06:37.121271055Z"}
```

Co-authored-by: mlemesle <lemesle.martin@hotmail.fr>
Co-authored-by: Kian-Meng Ang <kianmeng@cpan.org>
Co-authored-by: Jakub Jirutka <jakub@jirutka.cz>
Co-authored-by: Jarasłaŭ Viktorčyk <ugzuzg@gmail.com>
2022-10-03 09:48:34 +00:00
Jakub Jirutka
5b57114771 Bump milli from 0.33.0 to 0.33.4 2022-09-14 20:52:11 +02:00
Clémentine Urquizar
268d59ccb1
Update milli version to v0.33.4 2022-09-13 18:01:09 +02:00
Clémentine Urquizar
aabd67a9fa
Update patch version to remove CVE 2022-09-12 14:36:45 +02:00
Kerollmops
441492f1c8
Bump milli to v0.33.3 2022-09-07 18:23:49 +02:00
mlemesle
403226a029 Add support for config file 2022-09-07 16:09:00 +02:00
ManyTheFish
50434d35d0 Update milli v0.33.2 2022-09-01 13:15:05 +02:00
evpeople
833ade80a6 cargo update 2022-08-31 13:58:53 +08:00
ManyTheFish
aff4b64265 Update dependencies 2022-08-23 16:39:39 +02:00
bors[bot]
0a2ef0037f
Merge #2689 #2690
2689: Use mimalloc as the global allocator r=Kerollmops a=loiclec

milli has switched its global allocator to mimalloc already, and we have seen some performance gains as a result. Furthermore, we can use mimalloc as the global allocator on all platforms whereas jemalloc was only activated on Linux. 

This PR brings mimalloc to Meilisearch as well. 

2690: Add LTO and codegen-units=1 to release compile options r=Kerollmops a=loiclec

This PR brings Meilisearch's release compile options in line with milli (see https://github.com/meilisearch/milli/pull/606 ). 

Adding LTO and codegen=units=1 will make compile times longer, but they also speed up the final binary significantly.

Co-authored-by: Loïc Lecrenier <loic@meilisearch.com>
2022-08-23 12:05:02 +00:00
Clémentine Urquizar
6fe3f285ce
Update version for next release (v0.29.0) 2022-08-23 13:39:56 +02:00
Loïc Lecrenier
e659c08ac4 Use mimalloc as the global allocator 2022-08-23 12:58:10 +02:00
mohandasspat
4bee0565e8 prometheus and grafana dashboards implemented 2022-08-22 13:21:22 +05:30
Clément Renault
9b2036ac05
Accept either an array of documents or a single document 2022-08-18 11:55:14 +02:00
bors[bot]
b5f91b91c3
Merge #2523
2523: Improve the tasks error reporting when processed in batches r=irevoire a=Kerollmops

This fixes #2478 by changing the behavior of the task handler when there is an error in a batch of document addition or update.

What changes is that when there is a user error in a task in a batch we now report this task as failed with the right error message but we continue to process the other tasks. A user error can be when a geo field is invalid, a document id is invalid, or missing.

fixes #2582, #2478

Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-08-16 14:15:30 +00:00
ManyTheFish
cf955a77db Fix(cli): Clamp databases max size to a multiple of system page size
fix #2659
2022-08-11 10:44:47 +02:00
Kerollmops
fe32097964 Update milli v0.32 2022-07-28 14:45:10 +02:00
Clémentine Urquizar
9cf6acb671
Fix highlight issue by updating milli to v0.31.2 2022-07-21 14:11:24 +04:00
Clémentine Urquizar
d01a3ab889
Update version for next release (v0.28.1) 2022-07-20 15:46:53 +04:00
ManyTheFish
a146fd45b9 Format API keys in hexa instead of base64 2022-07-05 16:14:18 +02:00
Irevoire
05ee2eff01
add more tests on the formatted route 2022-06-28 13:17:55 +02:00
bors[bot]
9e261b996f
Merge #2543
2543: fix all the array on the search get route and improve the tests r=curquiza a=irevoire

fix #2527

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-06-23 14:51:36 +00:00
Kerollmops
7feb15df28
Bump milli to 0.31.1 2022-06-23 10:47:48 +02:00
Tamo
c47369b502
fix all the array on the search get route and improve the tests 2022-06-22 14:33:44 +02:00
Clémentine Urquizar
32c8846514
Rollback 0.0.0 versionning 2022-06-22 12:20:12 +02:00
Clémentine Urquizar
7490383d4f
Update the not-released version in Cargo.toml files 2022-06-21 19:17:33 +02:00
Clémentine Urquizar
c6ed756dbc
Update script after review 2022-06-21 10:46:32 +02:00
Tamo
2063fbd985
chore: bump milli 2022-06-09 18:34:03 +02:00
pierre-l
36cb09eb25 Add a new meilisearch_types crate
Move `meilisearch_error` to `meilisearch_types::error`
Move `meilisearch_lib::index_resolver::IndexUid` to `meilisearch_types::index_uid`
Add a new `InvalidIndexUid` error in `meilisearch_types::index_uid`
2022-06-09 16:14:13 +02:00
bors[bot]
b9b32d65a8
Merge #2494
2494: Introduce the new faceting and pagination settings r=ManyTheFish a=Kerollmops

This PR introduces two new settings following the newly created spec https://github.com/meilisearch/specifications/pull/157:
 - The `faceting.max_values_per_facet` one describes the maximum number of values (each with a count) associated with a value in a facet distribution query.
 - The `pagination.limited_to` one describes the maximum number of documents that a search query can ever return.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-06-09 12:09:21 +00:00
Kerollmops
c924614527
Bump milli to 0.29.2 2022-06-09 10:54:28 +02:00
Clémentine Urquizar
fc4990b968
Update version for next release (v0.28.0) 2022-06-08 17:59:18 +02:00
bors[bot]
6171f17f1d
Merge #2468
2468: Update milli 0.29 r=Kerollmops a=ManyTheFish

- [x] Update milli to 0.29
- [x] Integrate charabia
- [x] Set disabled_words to default when Index::exact_words returns None
- [x] Fix ranking rules integration test

fixes #2375
fixes #2144
fixes #2417
fixes #2407

Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-06-08 14:29:20 +00:00
ManyTheFish
8d09772334 Update milli 2022-06-08 14:38:05 +02:00
ManyTheFish
987a7f8926 Wrap sha256 in HMAC instead of directly use sha256 2022-06-08 14:25:12 +02:00
Kerollmops
277a0a7967
Bump serde-cs to simplify our usage of the star_or function 2022-06-06 10:17:33 +02:00
bors[bot]
cf2d8de48a
Merge #2452
2452: Change http verbs r=ManyTheFish a=Kerollmops

This PR fixes #2419 by updating the HTTP verbs used to update the settings and every single setting parameter.

- [x] `PATCH /indexes/{indexUid}` instead of `PUT`
- [x] `PATCH /indexes/{indexUid}/settings`  instead of `POST`
- [x] `PATCH /indexes/{indexUid}/settings/typo-tolerance`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/displayed-attributes`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/distinct-attribute`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/filterable-attributes`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/ranking-rules`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/searchable-attributes`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/sortable-attributes`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/stop-words`  instead of `POST`
- [x] `PUT /indexes/{indexUid}/settings/synonyms`  instead of `POST`


Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-06-02 11:46:17 +00:00
bors[bot]
c9cd1738a5
Merge #2445
2445: Seek-based tasks list r=Kerollmops a=Kerollmops

This PR implements the seek-based pagination for the tasks list following [the spec](https://github.com/meilisearch/specifications/pull/115).

Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-06-02 10:25:54 +00:00
Kerollmops
bcb51905d7
Fix the authorization tests 2022-06-02 12:16:46 +02:00
bors[bot]
8b8c3e32f0
Merge #2450
2450: Bump the dependencies r=ManyTheFish a=Kerollmops

In order to use [the latest version of grenad](https://docs.rs/grenad) I bump the dependencies here. We also use the latest versions of all our other dependencies now.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-06-02 08:53:12 +00:00
Kerollmops
e769043576
Bump the dependencies 2022-06-01 17:59:05 +02:00
ManyTheFish
7652295d2c Encode key in base64 instead of hexa 2022-06-01 16:17:47 +02:00
ManyTheFish
96a5791e39 Add uid and name fields in keys 2022-06-01 16:07:44 +02:00