2851: Upgrade clap to 4.0 r=loiclec a=choznerol
# Pull Request
## Related issue
Fixes#2846
This PR is draft based on #2847 to avoid conflict. I will rebase and mark as 'Ready for review' after #2847 is merged.
## What does this PR do?
1. Upgrade clap to the latest version or 4.0 (4.0.9 as of today) by following the [migrating instruction](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating) from [4.0 changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating)
2. Fix an `ArgGroup` typo that can only be caught after upgrading to 4.0 in 20a715e29ed17c5a76229c98fb31504ada873597
## Notable changes
### The `--help` message
The format, ordering and indentation of `--help` message was changed in 4.0. I recorded the output of `cargo run -- --help` before and after upgrade to 4.0 for reference.
<details>
<summary>diff</summary>
Output of `diff --ignore-all-space --text --unified --new-file help-message-before.txt help-message-after.txt`:
```diff
--- help-message-before.txt 2022-10-14 16:45:36.000000000 +0800
+++ help-message-after.txt 2022-10-14 16:36:53.000000000 +0800
`@@` -1,12 +1,8 `@@`
-meilisearch-http 0.29.1
+Usage: meilisearch [OPTIONS]
-USAGE:
- meilisearch [OPTIONS]
-
-OPTIONS:
+Options:
--config-file-path <CONFIG_FILE_PATH>
- Set the path to a configuration file that should be used to setup the engine. Format
- must be TOML
+ Set the path to a configuration file that should be used to setup the engine. Format must be TOML
--db-path <DB_PATH>
Designates the location where database files will be created and retrieved
`@@` -26,15 +22,14 `@@`
[default: dumps/]
--env <ENV>
- Configures the instance's environment. Value must be either `production` or
- `development`
+ Configures the instance's environment. Value must be either `production` or `development`
[env: MEILI_ENV=]
[default: development]
[possible values: development, production]
-h, --help
- Print help information
+ Print help information (use `-h` for a summary)
--http-addr <HTTP_ADDR>
Sets the HTTP address and port Meilisearch will use
`@@` -43,63 +38,53 `@@`
[default: 127.0.0.1:7700]
--http-payload-size-limit <HTTP_PAYLOAD_SIZE_LIMIT>
- Sets the maximum size of accepted payloads. Value must be given in bytes or explicitly
- stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+ Sets the maximum size of accepted payloads. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
[env: MEILI_HTTP_PAYLOAD_SIZE_LIMIT=]
[default: 100000000]
--ignore-dump-if-db-exists
- Prevents a Meilisearch instance with an existing database from throwing an error when
- using `--import-dump`. Instead, the dump will be ignored and Meilisearch will launch
- using the existing database.
+ Prevents a Meilisearch instance with an existing database from throwing an error when using `--import-dump`. Instead, the dump will be ignored and Meilisearch will launch using the existing database.
This option will trigger an error if `--import-dump` is not defined.
[env: MEILI_IGNORE_DUMP_IF_DB_EXISTS=]
--ignore-missing-dump
- Prevents Meilisearch from throwing an error when `--import-dump` does not point to a
- valid dump file. Instead, Meilisearch will start normally without importing any dump.
+ Prevents Meilisearch from throwing an error when `--import-dump` does not point to a valid dump file. Instead, Meilisearch will start normally without importing any dump.
This option will trigger an error if `--import-dump` is not defined.
[env: MEILI_IGNORE_MISSING_DUMP=]
--ignore-missing-snapshot
- Prevents a Meilisearch instance from throwing an error when `--import-snapshot` does not
- point to a valid snapshot file.
+ Prevents a Meilisearch instance from throwing an error when `--import-snapshot` does not point to a valid snapshot file.
This command will throw an error if `--import-snapshot` is not defined.
[env: MEILI_IGNORE_MISSING_SNAPSHOT=]
--ignore-snapshot-if-db-exists
- Prevents a Meilisearch instance with an existing database from throwing an error when
- using `--import-snapshot`. Instead, the snapshot will be ignored and Meilisearch will
- launch using the existing database.
+ Prevents a Meilisearch instance with an existing database from throwing an error when using `--import-snapshot`. Instead, the snapshot will be ignored and Meilisearch will launch using the existing database.
This command will throw an error if `--import-snapshot` is not defined.
[env: MEILI_IGNORE_SNAPSHOT_IF_DB_EXISTS=]
--import-dump <IMPORT_DUMP>
- Imports the dump file located at the specified path. Path must point to a `.dump` file.
- If a database already exists, Meilisearch will throw an error and abort launch
+ Imports the dump file located at the specified path. Path must point to a `.dump` file. If a database already exists, Meilisearch will throw an error and abort launch
[env: MEILI_IMPORT_DUMP=]
--import-snapshot <IMPORT_SNAPSHOT>
- Launches Meilisearch after importing a previously-generated snapshot at the given
- filepath
+ Launches Meilisearch after importing a previously-generated snapshot at the given filepath
[env: MEILI_IMPORT_SNAPSHOT=]
--log-level <LOG_LEVEL>
Defines how much detail should be present in Meilisearch's logs.
- Meilisearch currently supports five log levels, listed in order of increasing verbosity:
- ERROR, WARN, INFO, DEBUG, TRACE.
+ Meilisearch currently supports five log levels, listed in order of increasing verbosity: ERROR, WARN, INFO, DEBUG, TRACE.
[env: MEILI_LOG_LEVEL=]
[default: INFO]
`@@` -110,31 +95,25 `@@`
[env: MEILI_MASTER_KEY=]
--max-index-size <MAX_INDEX_SIZE>
- Sets the maximum size of the index. Value must be given in bytes or explicitly stating a
- base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+ Sets the maximum size of the index. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
[env: MEILI_MAX_INDEX_SIZE=]
[default: 107374182400]
--max-indexing-memory <MAX_INDEXING_MEMORY>
- Sets the maximum amount of RAM Meilisearch can use when indexing. By default,
- Meilisearch uses no more than two thirds of available memory
+ Sets the maximum amount of RAM Meilisearch can use when indexing. By default, Meilisearch uses no more than two thirds of available memory
[env: MEILI_MAX_INDEXING_MEMORY=]
[default: "21.33 TiB"]
--max-indexing-threads <MAX_INDEXING_THREADS>
- Sets the maximum number of threads Meilisearch can use during indexation. By default,
- the indexer avoids using more than half of a machine's total processing units. This
- ensures Meilisearch is always ready to perform searches, even while you are updating an
- index
+ Sets the maximum number of threads Meilisearch can use during indexation. By default, the indexer avoids using more than half of a machine's total processing units. This ensures Meilisearch is always ready to perform searches, even while you are updating an index
[env: MEILI_MAX_INDEXING_THREADS=]
[default: 5]
--max-task-db-size <MAX_TASK_DB_SIZE>
- Sets the maximum size of the task database. Value must be given in bytes or explicitly
- stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+ Sets the maximum size of the task database. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
[env: MEILI_MAX_TASK_DB_SIZE=]
[default: 107374182400]
```
- ~[help-message-before.txt](https://github.com/meilisearch/meilisearch/files/9715683/help-message-before.txt)~ [help-message-before.txt](https://github.com/meilisearch/meilisearch/files/9784156/help-message-before-2.txt)
- ~[help-message-after.txt](https://github.com/meilisearch/meilisearch/files/9715682/help-message-after.txt)~ [help-message-after.txt](https://github.com/meilisearch/meilisearch/files/9784091/help-message-after.txt)
</details>
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Lawrence Chou <choznerol@protonmail.com>
The 'Tests on windows-latest' now failed with error message below
---- option::test::test_meilli_config_file_path_invalid stdout ----
thread 'option::test::test_meilli_config_file_path_invalid' panicked at 'assertion failed:
left: `"unable to open or read the \"../configgg.toml\" configuration file: The system cannot find the file specified. (os error 2)."`,
right: `"unable to open or read the \"../configgg.toml\" configuration file: No such file or directory (os error 2)."`', meilisearch-http\src\option.rs:555:17
https://github.com/meilisearch/meilisearch/actions/runs/3231941308/jobs/5291998750
2876: Full support for compressed (Gzip, Brotli, Zlib) API requests r=Kerollmops a=mou
# Pull Request
## Related issue
Fixes#2802
## What does this PR do?
- Adds missed content-encoding support for streamed requests (documents)
- Adds additional tests to validate content-encoding support is in place
- Adds new tests to validate content-encoding support for previously existing code (built-in actix functionality for unmarshaling JSON payloads)
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Andrey "MOU" Larionov <anlarionov@gmail.com>
Actix provides different content encodings out of the box, but only if we use built-in content wrappers and containers. This patch wraps its own Payload implementation with an actix decoder, which enables request compression support.
2861: Change default bind address to localhost r=Kerollmops a=Fall1ngStar
# Pull Request
## Related issue
Fixes#2782
## What does this PR do?
- Change the default bind address to `localhost` so that it can be accessed with IPv6
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Fall1ngStar <fall1ngstar.public@gmail.com>
Close#2800
This is an alternative to the `--config-file-path` option. If both `--config-file-path` and `MEILI_CONFIG_FILE_PATH` are present, `--config-file-path` takes precedence according to the "Priority order" section of #2558.
Not sure why but the compiler didn't catch this until clap is upgraded to v4.
Follwoing are the error from 'cargo test':
running 2 tests
test routes::indexes::search::test::test_fix_sort_query_parameters ... ok
test option::test::test_valid_opt ... FAILED
failures:
---- option::test::test_valid_opt stdout ----
thread 'option::test::test_valid_opt' panicked at 'Command meilisearch-http: Argument or group 'import-snapshot' specified in 'requires*' for 'ignore_missing_snapshot' does not exist', /Users/ychou/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-4.0.9/src/builder/debug_asserts.rs:152:13
note: run with environment variable to display a backtrace
failures:
option::test::test_valid_opt
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
2839: Update internal CLI documentation r=curquiza a=jeertmans
# Pull Request
## Related issue
Fixes#2810
## What does this PR do?
- Make internal CLI documentation match the online one
## Remarks
- Scope is limited to `meilisearch/meilisearch-http/src/option.rs`, not the flattened structs: should I also take care of them?
- Could not find online docs for `enable_metrics_route`
- Max column width wrapping was done by hand, so may not be perfect
- I removed the links from the internal doc: should I put them back?
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Co-authored-by: Jérome Eertmans <jeertmans@icloud.com>
2841: Bail if config file contains 'config_file_path' r=Kerollmops a=arriven
# Pull Request
## Related issue
Fixes#2801
## What does this PR do?
- Return an error if config file contains 'config_file_path'
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Co-authored-by: arriven <20084245+Arriven@users.noreply.github.com>
2745: Config file support r=curquiza a=mlemesle
# Pull Request
## What does this PR do?
Fixes#2558
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
2789: Fix typos r=Kerollmops a=kianmeng
# Pull Request
## What does this PR do?
Found via `codespell -L crate,nam,hart`.
## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
2814: Skip dashboard test if mini-dashboard feature is disabled r=Kerollmops a=jirutka
Fixes#2813
Fixes the following error:
cargo test --no-default-features
...
error: couldn't read target/debug/build/meilisearch-http-ec029d8c902cf2cb/out/generated.rs: No such file or directory (os error 2)
--> meilisearch-http/tests/dashboard/mod.rs:8:9
|
8 | include!(concat!(env!("OUT_DIR"), "/generated.rs"));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in the macro `include` (in Nightly builds, run with -Z macro-backtrace for more info)
error: could not compile `meilisearch-http` due to previous error
2826: Rename receivedDocumentIds into matchedDocuments r=Kerollmops a=Ugzuzg
# Pull Request
## What does this PR do?
Fixes#2799
Changes DocumentDeletion task details response.
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Tested with curl:
```
curl \
-X POST 'http://localhost:7700/indexes/movies/documents/delete-batch' \
-H 'Content-Type: application/json' \
--data-binary '[
23488,
153738,
437035,
363869
]'
{"taskUid":1,"indexUid":"movies","status":"enqueued","type":"documentDeletion","enqueuedAt":"2022-10-01T20:06:37.105416054Z"}%
curl \
-X GET 'http://localhost:7700/tasks/1'
{"uid":1,"indexUid":"movies","status":"succeeded","type":"documentDeletion","details":{"matchedDocuments":4,"deletedDocuments":2},"duration":"PT0.005708322S","enqueuedAt":"2022-10-01T20:06:37.105416054Z","startedAt":"2022-10-01T20:06:37.115562733Z","finishedAt":"2022-10-01T20:06:37.121271055Z"}
```
Co-authored-by: mlemesle <lemesle.martin@hotmail.fr>
Co-authored-by: Kian-Meng Ang <kianmeng@cpan.org>
Co-authored-by: Jakub Jirutka <jakub@jirutka.cz>
Co-authored-by: Jarasłaŭ Viktorčyk <ugzuzg@gmail.com>
* Add some tests
* Disallow index creation when API key doesn't havec explicitelly the right on the creating index
* Fix lazy index creation with `indexes.*` action
2504: New README 🌟 r=curquiza a=curquiza
⚠️ Please do not only look at the Markdown but also how the GitHub renders the README 😇👉👉 [Rendered](https://github.com/meilisearch/meilisearch/blob/new-readme/README.md) 👈👈
2697: Accept an environment variable to enable the metrics route r=ManyTheFish a=Kerollmops
With the PR Meilisearch is able to accept the `MEILI_ENABLE_METRICS_ROUTE` environment variable to enable the newly introduces metrics route.
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2674: Add analytics on the stats routes r=ManyTheFish a=irevoire
# Pull Request
## What does this PR do?
Implements https://github.com/meilisearch/specifications/pull/169
## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue?
- [ ] Have you read the contributing guidelines?
- [ ] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Irevoire <tamo@meilisearch.com>
2664: 🐞 fix: Support https in print_launch_resume r=irevoire a=evpeople
fix#2660
# Pull Request
## What does this PR do?
Fixes#2660
<!-- Please link the issue you're trying to fix with this PR, if none then please create an issue first. -->
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: evpeople <hangcaihui@gmail.com>
2523: Improve the tasks error reporting when processed in batches r=irevoire a=Kerollmops
This fixes#2478 by changing the behavior of the task handler when there is an error in a batch of document addition or update.
What changes is that when there is a user error in a task in a batch we now report this task as failed with the right error message but we continue to process the other tasks. A user error can be when a geo field is invalid, a document id is invalid, or missing.
fixes#2582, #2478
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Move `meilisearch_error` to `meilisearch_types::error`
Move `meilisearch_lib::index_resolver::IndexUid` to `meilisearch_types::index_uid`
Add a new `InvalidIndexUid` error in `meilisearch_types::index_uid`
2494: Introduce the new faceting and pagination settings r=ManyTheFish a=Kerollmops
This PR introduces two new settings following the newly created spec https://github.com/meilisearch/specifications/pull/157:
- The `faceting.max_values_per_facet` one describes the maximum number of values (each with a count) associated with a value in a facet distribution query.
- The `pagination.limited_to` one describes the maximum number of documents that a search query can ever return.
Co-authored-by: Kerollmops <clement@meilisearch.com>
2460: Create custom error types for `TaskType`, `TaskStatus`, and `IndexUid` r=Kerollmops a=walterbm
# Pull Request
## What does this PR do?
Fixes#2443 by making the following changes:
- Add custom `TaskTypeError` for `TaskType::from_str`
- Add custom `TaskStatusError` for `TaskStatus::from_str`
- Add custom `IndexUidFormatError` for `IndexUid::from_str`
- Implement `From<IndexUidFormatError> for IndexResolverError` to convert between errors
- Replace all usages of `IndexUid::new` with `IndexUid::from_str`
- **NOTE** I am relatively new to Rust and I struggled a lot with this final part. This PR ended up with a messy error conversion which does not seem ideal. Please let me know if you have any suggestions for how to make this better and I'll be happy to make any updates!
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: walter <walter.beller.morales@gmail.com>
2471: Remove the connection keep-alive timeout r=MarinPostma a=Thearas
# Pull Request
## What does this PR do?
Fixes <https://github.com/meilisearch/meilisearch-go/issues/221>.
Meilisearch has a default connection keep-alive timeout for 5s, which means it will close the connections with idle time >= 5s.
This PR set actix-web keep-alive config to `KeepAlive::Os`, let the client and system to decide when to close the connection.
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Co-authored-by: Thearas <thearas850@gmail.com>
2454: Unify the pagination of the index and documents route behind a common type r=curquiza a=irevoire
`@MarinPostma` wdyt of keeping the `auto_paginate_sized` until we implement the pagination on every route that needs it just to see if it could be useful to something else
Co-authored-by: Tamo <tamo@meilisearch.com>
2445: Seek-based tasks list r=Kerollmops a=Kerollmops
This PR implements the seek-based pagination for the tasks list following [the spec](https://github.com/meilisearch/specifications/pull/115).
Co-authored-by: Kerollmops <clement@meilisearch.com>
2446: rename Succeded to Succeeded r=irevoire a=MarinPostma
this pr renames `TaskEvent::Succeded` to `TaskEvent::Succeeded` and apply the migration to the dumps
Co-authored-by: ad hoc <postma.marin@protonmail.com>
- Return Documents API resources on `/documents` in an array in the the results field.
- Add limit, offset and total in the response body.
- Rename `attributesToRetrieve` into `fields` (only for the `/documents` endpoints, not for the `/search` ones).
- The `displayedAttributes` settings does not impact anymore the displayed fields returned in the `/documents` endpoints. These settings only impacts the `/search` endpoint.
Fix#2372
2410: Make dump a task r=Kerollmops a=MarinPostma
This PR transforms the dump task into a proper task.
The `GET /dumps/:dump_uid` is removed.
Some changes were made to make this work, and a bit a refactoring was necessary.
- The `dump_actor` module has been renamed do `dumps` and moved to the root
- There isn't a `DumpActor` anymore, and the dump process is handled by the `DumpHandler`.
- The `TaskPerformer` is renamed to `BatchHandler`
- The `BatchHandler` trait no longer has a `perform_job` method, but instead has a `accept` method returning whether a handler can proccess a batch
- The scheduler now accept a list of `BatchHandler`, and iterates trhough them until it finds one to accept the current batch.
- `Job` doesn't exist anymore, and everything in now inside of the `BatchContent` enum.
- The `Vec<TaskId>` from `Batch` is replaced with a `BatchContent` enum which hints at the content.
- The Scheduler is slightly modified to accept batch, and prioritize them before regular tasks.
- The `TaskList` are not identified by a `String` representing the index uid anymore, but by a `TaskListIdentifier` which also works for dumps which are not targeting any specific indexes.
- The `GET /dump/:dump_id` no longer exists
- `DumpActorError` is renamed to `DumpError`
close#2410
Co-authored-by: ad hoc <postma.marin@protonmail.com>
