1711: MeiliSearch refactor introducing OBKV format r=MarinPostma a=MarinPostma
This PR refactor some multiple components of meilisearch, and introduce the obkv document format to meilisearch
- [x] Split meilisearch-http and meilisearch-lib
- [x] Replace `IndexActor` and `UuidResolver` with `IndexResolver`
- [x] Remove mentions to Actor
- [x] Remove Actor traits to simplify code
- [x] Integrate obkv document format
- [x] Remove `Data`
- [x] Restore all route
- [x] Replace `Box<dyn error>` with `anyhow::Error`
- [x] Introduce update file store
- [x] Update file store error handling
- [x] Fix dumps
- [x] Fix snapshots
- [x] Fix tests
- [x] Update module documentation
- [x] add csv suppport (feat `@ManyTheFish` #1729 )
- [x] add jsonl support
- [x] integrate geosearch (feat `@irevoire` #1725)
partially implements #1691 and #1690. The error handling is very basic now, I will finish it in the next pr.
Some unit tests have been disabled, I will re-enable them ASAP, but they need a bit more work.
close#1531
P.S: sorry for this monstrous PR :'(
Co-authored-by: mpostma <postma.marin@protonmail.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: many <maxime@meilisearch.com>
`jemallocator` has been abandoned for nearly two years, and `rustc`
itself moved to use `tikv-jemallocator` instead:
3965773ae7
Let's switch to a better maintained version.
1651: Use reset_sortable_fields r=Kerollmops a=shekhirin
Resolves https://github.com/meilisearch/MeiliSearch/issues/1635
1676: Add curl binary to final stage image r=curquiza a=ook
Reference: #1673
Changes: * add `curl` binary to final docker Melisearch image.
For metrics, docker funny layer management makes this add a shrink from 319MB to 315MB:
```
☁ MeiliSearch [feature/1673-add-curl-to-docker-image] ⚡ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
getmeili/meilisearch 0.22.0_ook_1673 938e239ad989 2 hours ago 315MB
getmeili/meilisearch latest 258fa3aa1230 6 days ago 319MB
```
1684: bump dependencies r=MarinPostma a=MarinPostma
Bump meilisearch dependencies.
We still depend on custom patch that have been upgraded along the way.
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Thomas Lecavelier <thomas@followanalytics.com>
Co-authored-by: mpostma <postma.marin@protonmail.com>
1682: Change the format of custom ranking rules when importing old dumps r=curquiza a=Kerollmops
This PR changes the format of the custom ranking rules from `asc(price)` to `title:asc` as the format changed between v0.21 and v0.22. The dumps are now correctly importing the custom ranking rules.
This PR also change the previous default ranking rules (without sort) to the new default ranking rules (with the new sort).
Co-authored-by: Kerollmops <clement@meilisearch.com>
1652: Remove dependabot r=MarinPostma a=curquiza
Fixes#1649
Dependabot for vulnerability and security updates is still activated.
1654: Add Script for Windows r=MarinPostma a=singh08prashant
fixes#1570
changes:
1. added script for detecting windows os running git bash
2. appended `.exe` to `$release_file` for windows as listed [here](https://github.com/meilisearch/MeiliSearch/releases/)
3. removed global `$BINARY_NAME='meilisearch'` as windows require `.exe` file
1657: Bring vergen hotfix from `stable` to `main` r=MarinPostma a=curquiza
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: singh08prashant <singh08prashant@gmail.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
1540: Update milli to version 0.8.1 r=curquiza a=curquiza
Integrates this fix into MeiliSearch https://github.com/meilisearch/milli/pull/296
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
1539: Use serdeval for validating json format. r=curquiza a=MarinPostma
uses [serdeval](https://github.com/MarinPostma/serdeval) to validate that the json payload is valid json, and in the correct format.
fix#1535
Co-authored-by: mpostma <postma.marin@protonmail.com>
1498: Show the filterable and not the faceted attributes in the settings r=Kerollmops a=Kerollmops
Fixes#1497
Co-authored-by: Clément Renault <clement@meilisearch.com>
1478: refactor routes r=irevoire a=MarinPostma
refactor the route directory, so the module tree follows the route structure
Co-authored-by: mpostma <postma.marin@protonmail.com>
1457: Hotfix highlight on emojis panic r=Kerollmops a=ManyTheFish
When the highlight bound is in the middle of a character
or if we are out of bounds, we highlight the complete matching word.
note: we should enhance the tokenizer and the Highlighter to match char indices.
Fix#1368
Co-authored-by: many <maxime@meilisearch.com>
When the highlight bound is in the middle of a character
or if we are out of bounds, we highlight the complete matching word.
note: we should enhance the tokenizer and the Highlighter to match char indices.
Fix#1368
259: Run rustfmt one the whole project and add it to the CI r=curquiza a=irevoire
Since there is currently no other PR modifying the code, I think it's a good time to reformat everything and add rustfmt to the ci.
Co-authored-by: Tamo <tamo@meilisearch.com>
258: Use rustls instead of openssl r=curquiza a=irevoire
I also removed all the `default-features` of reqwest since we are only using the JSON one.
Fix#255
Co-authored-by: Tamo <tamo@meilisearch.com>
246: Stop logging the no space left on device error r=curquiza a=irevoire
closes#208
@qdequele what do you think of that?
Are there any other errors we need to ignore?
As you can see in the code, once we are in `Sentry` the error has already been converted to a `String` so the only thing we can do to see if we need to send the error or not is to match the `String` against our error message.
If we have a lot of other logs we want to ignore I would suggest prefixing all the logs with something like:
```
User error: No space left on device
```
So in Sentry, we could just check if the log start by `User error:` and ignore all these errors at once
Co-authored-by: Tamo <tamo@meilisearch.com>
249: Use half of the computer threads for the indexing process by default r=Kerollmops a=irevoire
closes#241
By default, we use only half of the CPU threads when indexing documents; this allows the user to use the search while indexing. Also, the machine will not appear unresponsive when indexing a large batch of documents.
On the special case where a user only has one core, we use it entirely 😄
Co-authored-by: Tamo <tamo@meilisearch.com>
240: Rework error messages r=irevoire a=MarinPostma
Simplify the error messages, and make them more compliant with legacy Meilisearch.
Basically, stop composing the messages, and simply forward the message of inner errors.
Co-authored-by: marin postma <postma.marin@protonmail.com>
232: Fix payload size limit r=MarinPostma a=MarinPostma
Fix#223
This was due to the fact that Payload ignores the limit payload size limit. I fixed it by implementing my own `Payload` extractor that checks that the size of the payload is not too large.
I also refactored the `create_app` a bit.
Co-authored-by: marin postma <postma.marin@protonmail.com>
227: improve mini dashboard routing r=MarinPostma a=MarinPostma
The dependency we use to statically serve the mini-dashboard used globing to serve the mini-dashboard files. This caused all unfound routes to be caught by the "/" serving the dashboard assets. This fix makes it so that the assets have a dedicated route, and any unfound route is caught by the default service and return a 404.
Co-authored-by: marin postma <postma.marin@protonmail.com>
226: Make facetsDistribution name iso r=MarinPostma a=curquiza
Even if there is an English mistake in `facets_distribution` (because of the `s`) @gmourier asked me to keep the typo: the name of `facetsDistribution` might change completely in the future, he wants to avoid two breakings.
@gmourier can you confirm before we merge this PR?
Sorry I left this update in the code (I'm confused because no issues was open to update `facetsDistribution`), there might have been a confusion with `fieldsDistribution` that has been renamed into `fieldDistribution`. Sorry!
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
213: Implement all the CLI options r=MarinPostma a=irevoire
closes#206
And I looked into #204, I fixed some default values and tried to test as many options as possible, and I think the cli is already mostly working.
If someone knows any issues about it, I would like to hear more 🙂
Co-authored-by: Tamo <tamo@meilisearch.com>
211: fix index deletion race condition r=MarinPostma a=MarinPostma
Make update store block if the currently processing update is from an index we are trying to delete. This ensure that no write to the index can occur after it has been deleted.
218: Update milli version to v0.5.0 r=MarinPostma a=curquiza
Co-authored-by: marin postma <postma.marin@protonmail.com>
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
210: Error handling r=MarinPostma a=MarinPostma
This pr implements the error handling for meilisearch.
Rather than grouping errors by types, this implementation groups them by scope, each scope enclosing errors from a scope further down, or new errors within this scope. This makes the tracking of the origins of errors easier , and error handling easier at the module level.
All errors that are eventually returned to the user implement the `Into<ResponseError>` trait. `ReponseError` in turn implements the `ErrorCode` trait from `meilisearch-error`.
Some new errors have been introduced with the new engine for which we haven't defined error codes yet. It has been decided with @gmourier that those would return the `internal-error` code until the correct error code is specified.
Co-authored-by: marin postma <postma.marin@protonmail.com>
