Commit Graph

94 Commits

Author SHA1 Message Date
Louis Dureuil
e35db5e59b
Rename dumps-dir to dump-dir in CLI
Also rename the associated environment variable
2022-12-01 15:46:54 +01:00
Clémentine Urquizar - curqui
457a473b72
Bring back release-v0.30.0 into release-v0.30.0-temp (final: into main) (#3145)
* Fix error code of the "duplicate index found" error

* Use the content of the ProcessingTasks in the tasks cancelation system

* Change the missing_filters error code into missing_task_filters

* WIP Introduce the invalid_task_uid error code

* Use more precise error codes/message for the task routes

+ Allow star operator in delete/cancel tasks
+ rename originalQuery to originalFilters
+ Display error/canceled_by in task view even when they are = null
+ Rename task filter fields by using their plural forms
+ Prepare an error code for canceledBy filter
+ Only return global tasks if the API key action `index.*` is there

* Add canceledBy task filter

* Update tests following task API changes

* Rename original_query to original_filters everywhere

* Update more insta-snap tests

* Make clippy happy

They're a happy clip now.

* Make rustfmt happy

>:-(

* Fix Index name parsing error message to fit the specification

* Bump milli version to 0.35.1

* Fix the new error messages

* fix the error messages and add tests

* rename the error codes for the sake of consistency

* refactor the way we send the cli informations + add the analytics for the config file and ssl usage

* Apply suggestions from code review

Co-authored-by: Clément Renault <clement@meilisearch.com>

* add a comment over the new infos structure

* reformat, sorry @kero

* Store analytics for the documents deletions

* Add analytics on all the settings

* Spawn threads with names

* Spawn rayon threads with names

* update the distinct attributes to the spec update

* update the analytics on the search route

* implements the analytics on the health and version routes

* Fix task details serialization

* Add the question mark to the task deletion query filter

* Add the question mark to the task cancelation query filter

* Fix tests

* add analytics on the task route

* Add all the missing fields of the new task query type
* Create a new analytics for the task deletion
* Create a new analytics for the task creation

* batch the tasks seen events

* Update the finite pagination analytics

* add the analytics of the swap-indexes route

* Stop removing the DB when failing to read it

* Rename originalFilters into originalFilters

* Rename matchedDocuments into providedIds

* Add `workflow_dispatch` to flaky.yml

* Bump grenad to 0.4.4

* Bump milli to version v0.37.0

* Don't multiply total memory returned by sysinfo anymore

sysinfo now returns bytes rather than KB

* Add a dispatch to the publish binaries workflow

* Fix publish release CI

* Don't use gold but the default linker

* Always display details for the indexDeletion task

* Fix the insta tests

* refactorize the whole test suite
1. Make a call to assert_internally_consistent automatically when snapshoting the scheduler. There is no point in snapshoting something broken and expect the dumb humans to notice.
2. Replace every possible call to assert_internally_consistent by a snapshot of the scheduler. It takes as many lines and ensure we never change something without noticing in any tests ever.
3. Name every snapshots: it's easier to debug when something goes wrong and easier to review in general.
4. Stop skipping breakpoints, it's too easy to miss something. Now you must explicitely show which path is the scheduler supposed to use.
5. Add a timeout on the channel.recv, it eases the process of writing tests, now when something file you get a failure instead of a deadlock.

* rebase on release-v0.30

* makes clippy happy

* update the snapshots after a rebase

* try to remove the flakyness of the failing test

* Add more analytics on the ranking rules positions

* Update the dump test to check for the dumpUid dumpCreation task details

* send the ranking rules as a string because amplitude is too dumb to process an array as a single value

* Display a null dumpUid until we computed the dump itself on disk

* Update tests

* Check if the master key is missing before returning an error

Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2022-11-28 16:27:41 +01:00
Kerollmops
2f577b6fcd
Patch the IndexScheduler in meilisearch-http to use the options struct 2022-10-27 11:35:16 +02:00
Kerollmops
4736e00253
Handle the CLI options related to snapshots 2022-10-27 11:35:15 +02:00
Kerollmops
942b7c338b
Compress the snapshot in a tarball 2022-10-27 11:35:15 +02:00
Kerollmops
4cafc63561
Reintroduce the versioning functions 2022-10-27 11:35:14 +02:00
Kerollmops
89e127e4f4
Declare the auth path in the index scheduler 2022-10-27 11:35:14 +02:00
Kerollmops
c063f154fb
Add the snapshots directory path to the IndexScheduler 2022-10-27 11:35:14 +02:00
Irevoire
8a23e707c1
fix the task view and forward the task db size 2022-10-27 11:35:09 +02:00
Irevoire
e9055f5572
fix clippy 2022-10-27 11:35:08 +02:00
Irevoire
ca4234b445
fix the deletion of the data.ms in case of failure 2022-10-27 11:35:06 +02:00
Clément Renault
80b2e70ee7
Introduce a rustfmt file 2022-10-27 11:35:05 +02:00
Kerollmops
725158b454
Introduce the core algorithm of task cancelation 2022-10-27 11:34:11 +02:00
Tamo
634eb52926
extract the create_app function for the tests 2022-10-27 11:34:09 +02:00
Tamo
6bd6321226
dump the content of the dump tasks instead of recreating at import time with wrong API keys 2022-10-27 11:34:08 +02:00
Tamo
9fe24fbff2
get rids of the useless Seek before creating a grenad reader 2022-10-27 11:34:08 +02:00
Tamo
78ce29f461
apply most style comments of the review 2022-10-27 11:34:08 +02:00
Tamo
e0221fc0a3
fix a synchronization bug while importing tasks 2022-10-27 11:34:07 +02:00
Tamo
554600dfd8
fix the deletion of the data.ms in case of errors 2022-10-27 11:34:07 +02:00
Tamo
d976e680c5
first mostly working version 2022-10-27 11:34:06 +02:00
Tamo
b7f9c94f4a
write the dump export 2022-10-27 11:34:05 +02:00
Tamo
667c282e19
get rids of the index crate + the document_types crate 2022-10-27 11:34:00 +02:00
Kerollmops
91e13c2824
Implement ErrorCode on the milli::Error type 2022-10-27 11:33:48 +02:00
Tamo
fba9aa214a
remove the create_app macro 2022-10-27 11:33:45 +02:00
Tamo
2c8f1a43e9
get rids of meilisearch-lib 2022-10-27 11:33:44 +02:00
Irevoire
8d51c1f389
wip integrating the scheduler in meilisearch-http 2022-10-27 11:33:42 +02:00
Tamo
6aa3ad6b6c
move prometheus behind a feature flag 2022-08-29 14:36:59 +02:00
mohandasspat
2f92169e48 clippy issue in metrics fixed 2022-08-22 13:21:22 +05:30
mohandasspat
2b8f3c26ec Changed prometheus metrics feature as optional 2022-08-22 13:21:22 +05:30
mohandasspat
25fecf9360 clippy & rustfmt fixed 2022-08-22 13:21:22 +05:30
mohandasspat
4bee0565e8 prometheus and grafana dashboards implemented 2022-08-22 13:21:22 +05:30
ManyTheFish
ae174c2cca Fix task serialization 2022-08-11 13:35:35 +02:00
ManyTheFish
e6f03f82df Fix clippy warnings 2022-07-28 15:56:22 +02:00
ManyTheFish
58d2aad309 Change binary option and add env var support 2022-07-28 15:13:49 +02:00
Kerollmops
e3426d5b7a Improve the tasks error reporting 2022-07-28 15:12:54 +02:00
Kerollmops
aff8cd1774
Make clippy happy 2022-07-04 13:36:56 +02:00
pierre-l
36cb09eb25 Add a new meilisearch_types crate
Move `meilisearch_error` to `meilisearch_types::error`
Move `meilisearch_lib::index_resolver::IndexUid` to `meilisearch_types::index_uid`
Add a new `InvalidIndexUid` error in `meilisearch_types::index_uid`
2022-06-09 16:14:13 +02:00
Kerollmops
3684c822f1
Add indexUid filtering on the /tasks route 2022-05-31 11:33:20 +02:00
ad hoc
62ce8e0bda
chore(http): rename auto batching cli option 2022-03-07 15:19:19 +01:00
Rob Ede
3b2e467ca6
update actix-web dependency to 4.0 2022-02-28 19:03:37 +01:00
mpostma
c9a236b0af
feat(lib): auto-batching 2022-02-01 18:06:20 +01:00
bors[bot]
c8bb54cd94
Merge #2098
2098: feat(dump): Provide the same cli options as the snapshots r=MarinPostma a=irevoire

Add two cli options for the dump:
- `--ignore-missing-dump`
- `--ignore-dump-if-db-exists`

Fix #2087

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-01-26 14:32:23 +00:00
Tamo
bad4bed439
feat(dump): Provide the same cli options as the snapshots
Add two cli options for the dump:
- `--ignore-missing-dump`
- `--ignore-dump-if-db-exists`

Fix #2087
2022-01-26 14:34:06 +01:00
Rob Ede
9f5fee404b
chore(all): update actix-web dependency to 4.0.0-beta.21 2022-01-21 20:44:17 +00:00
Clémentine Urquizar
595250c93e
Allow any header for CORS 2022-01-05 15:38:47 +01:00
ManyTheFish
450b81ca13 Bug(CORS): Add missing allowed headers
fix #2040
2022-01-03 13:41:12 +01:00
many
ee7970f603 feat(auth): Extend API keys
- Add API keys in snapshots
- Add API keys in dumps
- Rename action indexes.add to indexes.create
- fix QA #1979

fix #1979
fix #1995
fix #2001
fix #2003
related to #1890
2021-12-14 17:33:39 +01:00
many
ffefd0caf2
feat(auth): API keys
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0085-api-keys.md

- Add tests on API keys management route (meilisearch-http/tests/auth/api_keys.rs)
- Add tests checking authorizations on each meilisearch routes (meilisearch-http/tests/auth/authorization.rs)
- Implement API keys management routes (meilisearch-http/src/routes/api_key.rs)
- Create module to manage API keys and authorizations (meilisearch-auth)
- Reimplement GuardedData to extend authorizations (meilisearch-http/src/extractors/authentication/mod.rs)
- Change X-MEILI-API-KEY by Authorization Bearer (meilisearch-http/src/extractors/authentication/mod.rs)
- Change meilisearch routes to fit to the new authorization feature (meilisearch-http/src/routes/)

- close #1867
2021-12-06 09:52:41 +01:00
Marin Postma
a30e02c18c feat(all): Task store
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0060-refashion-updates-apis.md

linked PR:

- #1889
- #1891
- #1892
- #1902
- #1906
- #1911
- #1914
- #1915
- #1916
- #1918
- #1924
- #1925
- #1926
- #1930
- #1936
- #1937
- #1942
- #1944
- #1945
- #1946
- #1947
- #1950
- #1951
- #1957
- #1959
- #1960
- #1961
- #1962
- #1964

- https://github.com/meilisearch/milli/pull/414
- https://github.com/meilisearch/milli/pull/409
- https://github.com/meilisearch/milli/pull/406
- https://github.com/meilisearch/milli/pull/418

- close #1687
- close #1786
- close #1940
- close #1948
- close #1949
- close #1932
- close #1956
2021-12-02 20:14:42 +01:00
marin postma
d65f055030
pass anaytics into Arc instead of static ref 2021-10-29 17:25:55 +02:00