Commit Graph

873 Commits

Author SHA1 Message Date
tamo
c3552cecdf
WIP rebase on main 2021-05-10 20:24:18 +02:00
tamo
0f94ef8abc
WIP: dump 2021-05-10 20:24:18 +02:00
tamo
0275b36fb0
[WIP] rebase on main 2021-05-10 20:24:14 +02:00
tamo
1b5fc61eb6
[WIP] rebase on main 2021-05-10 20:23:12 +02:00
tamo
0fee81678e
[WIP] rebase on main 2021-05-10 20:22:18 +02:00
tamo
c4d898a265
split the dumps between v1 and v2 2021-05-10 20:20:57 +02:00
tamo
e389c088eb
WIP: rebasing on master 2021-05-10 20:20:57 +02:00
Marin Postma
0cc79d414f
add test 2021-05-10 18:34:25 +02:00
Marin Postma
8d11b368d1
implement check 2021-05-10 18:22:41 +02:00
Marin Postma
706643dfed
type setting struct 2021-05-10 17:30:09 +02:00
Marin Postma
b192cb9c1f
enable string syntax for the filters 2021-05-06 12:48:31 +02:00
Marin Postma
ec7eb7798f
remove facet setting 2021-05-04 22:36:31 +02:00
Marin Postma
a717925caa
remove filters, rename facet_filters to filter 2021-05-04 18:20:56 +02:00
Clémentine Urquizar
eb03a3ccb1
Upgrade Milli and Tokenizer 2021-05-04 17:56:19 +02:00
Marin Postma
928fb34eff
bump milli and fix tests 2021-05-04 15:10:22 +02:00
bors[bot]
1e6b40a24b
Merge #172
172: Fix cors authentication issue r=MarinPostma a=MarinPostma

The error was due to the middleware returning an error, instead of a response containing the error.

close #110


Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-05-03 08:38:42 +00:00
Marin Postma
78217bcf18
Fix cors authentication issue 2021-04-29 16:28:12 +02:00
Morgane Dubus
3d5fba94c2 Update mini-dashboard with version 0.1.2 2021-04-29 15:22:41 +02:00
bors[bot]
8bc7dd8b03
Merge #143
143: Shared update store r=irevoire a=MarinPostma

This PR changes the updates process so that only one instance of an update store is shared among indexes.

This allows updates to always be processed sequentially without additional synchronization, and fixes the bug where all the first pending update for each index were reported as processing whereas only one was.

EDIT:

I ended having to rewrite the whole `UpdateStore` to allow updates being really queued and processed sequentially in the ordered they were added. For that purpose I created a `pending_queue` that orders the updates by a global update id.

To find the next `update_id` to use, both globally and for each index, I have created another database that contains the next id to use.

Finally, all updates that have been processed (with success or otherwise) are all stores in an `updates` database.

The layout for the keys of these databases are such that it is easy to iterate over the elements for a particular index, and greatly reduces the amount of code to do so, compared to the former implementation.

I have also simplified the locking mechanism for the update store, thanks to the StateLock data structure, that allow both an arbitrary number of readers and a single writer to concurrently access the state. The current state can be either Idle, Processing, or Snapshotting. When an update or snapshotting is ongoing, the process holds the state lock until it is done processing its task. When it is done, it sets bask the state to Idle.

I have made other small improvements here and there, and have let some other for work, such as:
- When creating an update file to hold a request's content, it would be preferable to first create a temporary file, and then atomically persist it when we have written to it. This would simplify the case when there is no data to be written to the file, since we wouldn't have to take care about cleaning after ourselves.
- The logic for content validation must be factored.
- Some more tests related to error handling in the process_pending_update function.
- The issue #159

close #114


Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-04-27 18:41:55 +00:00
Marin Postma
a961f0ce75
fix clippy warnings 2021-04-27 18:28:46 +02:00
Marin Postma
bb79a15c04
reenable ranking rules route 2021-04-27 15:29:00 +02:00
Marin Postma
4fe2a13c71
rewrite update store 2021-04-27 15:20:52 +02:00
Marin Postma
51829ad85e
review fixes 2021-04-27 15:10:57 +02:00
Marin Postma
c78f351300
fix tests 2021-04-27 15:10:57 +02:00
Marin Postma
ee675eadf1
fix stats 2021-04-27 15:10:55 +02:00
Marin Postma
33830d5ecf
fix snapshots 2021-04-27 15:09:55 +02:00
Marin Postma
2b154524bb
fix filtered out pending update 2021-04-27 15:09:23 +02:00
Marin Postma
b626d02ffe
simplify index actor run loop 2021-04-27 15:09:22 +02:00
Marin Postma
9ce68d11a7
single update store instance 2021-04-27 15:09:21 +02:00
Marin Postma
5a38f13cae
multi_index udpate store 2021-04-27 15:07:13 +02:00
Clémentine Urquizar
1ba46f8f77
Disable clippy rule 2021-04-27 12:43:00 +02:00
Clémentine Urquizar
f80ea24d2b
Add tests on every platform and fix clippy errors 2021-04-27 12:42:59 +02:00
Marin Postma
5014f74649
put mini-dashboard in out-dir 2021-04-27 09:32:17 +02:00
Clémentine Urquizar
f3b6bf55a6
Update version for the next release (alpha4) 2021-04-26 19:05:16 +02:00
Marin Postma
c2461e5066
review fixes 2021-04-26 10:20:46 +02:00
Marin Postma
e4bd1bc5ce
update actix-web-static-file rev 2021-04-22 11:42:41 +02:00
Marin Postma
90f57c1329
update CI & Dockerfile 2021-04-22 11:22:09 +02:00
Marin Postma
6af769af20
bump mini-dashboard 2021-04-22 10:45:05 +02:00
Marin Postma
6bcf20c70e
serve static site 2021-04-22 10:26:54 +02:00
Marin Postma
bb79695e44
load mini-dashboard assets 2021-04-22 10:26:54 +02:00
Marin Postma
ea5517bc8c
add mini-dashboard feature 2021-04-22 10:26:54 +02:00
Clémentine Urquizar
a72d2f66cd
use <em> tags instead of <mark> tags for highlighting 2021-04-21 19:14:55 +02:00
Marin Postma
dd2914873b
fix document fields order 2021-04-20 21:30:30 +02:00
Marin Postma
d9a29cae60
fix ignored displayed attributes 2021-04-20 21:23:35 +02:00
Marin Postma
7a737d2bd3
support wildcard 2021-04-20 21:23:35 +02:00
Marin Postma
881b099c8e
add tests 2021-04-20 21:23:34 +02:00
Marin Postma
c6bb36efa5
implement _formated 2021-04-20 21:23:28 +02:00
Marin Postma
526a05565e
add SearchHit structure 2021-04-20 21:22:48 +02:00
Clémentine Urquizar
b8e535579f
Update version for the next release (alpha3) 2021-04-20 16:11:07 +02:00
Marin Postma
f8c338e3a7
add test for dedicated distinct route 2021-04-20 15:49:17 +02:00
Marin Postma
6c470cf687
enable distinct-attribute setting route 2021-04-20 11:34:18 +02:00
Marin Postma
ec63e13896
bump actix 2021-04-20 11:29:32 +02:00
Marin Postma
1746132c7d
add test set/reset distinct attribute 2021-04-20 11:29:08 +02:00
mpostma
ec230c2835
enable distinct 2021-04-20 11:29:06 +02:00
Marin Postma
45665245dc
bump actix 2021-04-20 11:07:23 +02:00
bors[bot]
94c5c5843b
Merge #149
149: Handle star in attributes_to_retrieve r=MarinPostma a=curquiza

Closes #147

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-19 17:29:21 +00:00
Clémentine Urquizar
8eceba98d3
Handle star in attributes_to_retrieve 2021-04-19 18:20:19 +02:00
Clémentine Urquizar
2c380731b9
Update milli version to v0.1.1 2021-04-19 16:03:39 +02:00
bors[bot]
7ce74f95a2
Merge #146
146: Remove another unused legacy file r=MarinPostma a=irevoire

When doing #135 I missed an old useless file in the scr/routes directory

Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-15 18:05:28 +00:00
tamo
ec3a08ea0c
remove another unused legacy file 2021-04-15 14:44:43 +02:00
Clémentine Urquizar
b0717b75d9
Update tokenizer to v0.2.1 2021-04-14 19:06:18 +02:00
Alexey Shekhirin
f87afbc558
fix(http): commit date & SHA in startup message 2021-04-13 20:16:18 +03:00
Alexey Shekhirin
9eaf048a06
fix(http): use BTreeMap instead of HashMap to preserve stats order 2021-04-13 11:59:07 +03:00
Alexey Shekhirin
adfdb99abc
feat(http): calculate updates' and uuids' dbs size 2021-04-09 15:59:12 +03:00
Alexey Shekhirin
ae1655586c
fixes after review 2021-04-09 14:40:48 +03:00
Alexey Shekhirin
698a1ea582
feat(http): store processing as RwLock<Option<Uuid>> in index_actor 2021-04-09 14:34:43 +03:00
Alexey Shekhirin
87412f63ef
feat(http): implement is_indexing for stats 2021-04-09 14:34:42 +03:00
Alexey Shekhirin
09d9a29176
test(http): server & index stats 2021-04-09 14:34:42 +03:00
Alexey Shekhirin
dd9eae8c26
feat(http): stats route 2021-04-09 14:34:42 +03:00
bors[bot]
a1d04fbff5
Merge #136
136: Rename update status "pending" into "enqueued" r=curquiza a=curquiza

Closes #107 

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-08 16:46:12 +00:00
Alexey Shekhirin
51ba1bd7d3
fix(http, index): init analyzer with optional stop words
Next release

update tokenizer
2021-04-08 17:16:13 +03:00
bors[bot]
f881e8691e
Merge #135
135: Add stop words r=curquiza a=irevoire

closes #21 

Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-08 11:29:00 +00:00
Clémentine Urquizar
cb23775d18
Rename pending into enqueued 2021-04-07 19:46:36 +02:00
bors[bot]
0344cf5874
Merge #122
122: Update display r=MarinPostma a=curquiza



Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-07 12:33:25 +00:00
tamo
dcd60a5b45
add more tests for the stop_words 2021-04-06 18:29:38 +02:00
tamo
b1962c8e02
remove legacy files from meilisearch that have been replaced by a macro in routes/settings/mod.rs 2021-04-06 16:29:04 +02:00
tamo
40ef9a3c6a
push a first implementation of the stop_words 2021-04-06 16:29:04 +02:00
Clémentine Urquizar
4ee6ce7871
Next release 2021-04-01 17:16:16 +02:00
bors[bot]
6cb8052d3d
Merge #104
104: Update all the response format (issue #64) r=MarinPostma a=irevoire

closes #64 

Co-authored-by: Irevoire <tamo@meilisearch.com>
Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-01 14:22:57 +00:00
tamo
73973e2b9e
fix more settings routes 2021-04-01 15:50:45 +02:00
Marin Postma
248e9b3808
Merge remote-tracking branch 'origin/main' into snapshots 2021-04-01 15:10:33 +02:00
tamo
79c63049d7
update the settings routes 2021-04-01 11:52:26 +02:00
Irevoire
96cffeab1e
update all the response format to be ISO with meilisearch, see #64 2021-04-01 11:43:03 +02:00
Marin Postma
d8af4a7202
ignore snapshot test (#130) 2021-03-31 20:07:52 +02:00
Alexey Shekhirin
3d51db5929 fix(ci, http): commit_sha and commit_date in docker builds
chore(ci): cache dependencies in Docker build
2021-03-31 13:56:28 +03:00
tamo
0f2143e7fd
remove the now useless dead_code flags 2021-03-26 14:15:12 +01:00
Clémentine Urquizar
b9f79c8df0
Update display 2021-03-26 12:12:55 +01:00
mpostma
d7c077cffb
atomic snapshot import 2021-03-25 14:48:51 +01:00
mpostma
7d6ec7f3d3
resolve merge 2021-03-25 14:21:05 +01:00
mpostma
f3dc853be3
Merge remote-tracking branch 'origin/main' into snapshots 2021-03-25 13:45:07 +01:00
mpostma
48507460b2
add snapshot tests 2021-03-25 12:02:10 +01:00
mpostma
d029464de8
fix snapshot path 2021-03-25 10:23:31 +01:00
mpostma
79d09705d8
perform snapshot on startup 2021-03-25 09:35:15 +01:00
bors[bot]
868658f3d8
Merge #109
109: Make updates atomic r=curquiza a=MarinPostma

Until now, the index_uid->uuid mapping was done before the update was written to disk in the case of automatic index creation. This was an issue when the update failed, and the index would still exists in the uuid resolver.

This is fixed by this pr, by first creating the update with an uuid if the index does not exist, and then register this uuid to the uuid resolver.

This is preliminary work to the implementation of snapshots (#19).

This pr also changes the `resolve` method on the `UuidResolver` to `get` to make it clearer.


The `create_uuid` method may be bound to disappear when the index name resolution is handled by a remote machine.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-24 12:24:32 +00:00
bors[bot]
fe87477238
Merge #115
115: Add the exhaustiveNbHits in search response body (returns always false) r=curquiza a=irevoire

closes #103 

Co-authored-by: tamo <irevoire@protonmail.ch>
Co-authored-by: Irevoire <irevoire@protonmail.ch>
2021-03-24 12:16:53 +00:00
mpostma
d892a2643e
fix clippy 2021-03-24 12:38:59 +01:00
mpostma
83ffdc888a
remove bad file name test 2021-03-24 12:38:59 +01:00
mpostma
4041d9dc48
format code 2021-03-24 12:38:59 +01:00
mpostma
1f16c8d224
integration test snapshot 2021-03-24 12:38:59 +01:00
mpostma
06f9dae0f3
remove prints 2021-03-24 12:38:59 +01:00
mpostma
48d5f88c1a
fix snapshot dir already exists 2021-03-24 12:38:59 +01:00
mpostma
eb53ed4cc1
load snapshot 2021-03-24 12:38:59 +01:00
mpostma
46293546f3
add tests and mocks 2021-03-24 12:38:59 +01:00
mpostma
3cc3637e2d
refactor for tests 2021-03-24 12:38:56 +01:00
mpostma
1f51fc8baf
create indexes snapshots concurrently 2021-03-24 12:38:12 +01:00
mpostma
e9da191b7d
fix snapshot bugs 2021-03-24 12:38:12 +01:00
mpostma
d73fbdef2e
remove from snapshot 2021-03-24 12:38:12 +01:00
mpostma
44dcfe29aa
clean snapshot creation 2021-03-24 12:38:12 +01:00
mpostma
a85e7abb0c
fix snapshot creation 2021-03-24 12:38:12 +01:00
mpostma
4847884165
restore snapshots 2021-03-24 12:38:12 +01:00
mpostma
7f6a54cb12
add lock to prevent snapshot during update 2021-03-24 12:38:12 +01:00
mpostma
520f7c09ba
sequential index snapshot 2021-03-24 12:38:12 +01:00
mpostma
35a7b800eb
snapshot indexes 2021-03-24 12:38:12 +01:00
mpostma
c966b1dd94
use options to schedule snapshot 2021-03-24 12:38:11 +01:00
mpostma
ee838be41b
implement snapshot scheduler 2021-03-24 12:38:11 +01:00
Irevoire
127e944866
Update meilisearch-http/src/index/search.rs
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 19:13:22 +01:00
Irevoire
cc81aca6a4
Update meilisearch-http/src/index/search.rs
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-03-23 10:47:19 +01:00
Irevoire
46d7cedb18
Update meilisearch-http/src/index/search.rs
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 10:46:59 +01:00
mpostma
5f33672f0e
change payload send to use stream methods 2021-03-22 19:49:21 +01:00
mpostma
b690f1103a
fix typos 2021-03-22 19:25:56 +01:00
tamo
91089db444
add the exhaustive nb hits to be ISO, currently it's always set to false 2021-03-22 18:41:33 +01:00
bors[bot]
2f418ee767
Merge #108
108: use write senders for updates r=MarinPostma a=MarinPostma

 Use write senders to send updates to the `IndexActor`, so updates are performed sequentially on all indexes.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-22 14:18:43 +00:00
Irevoire
7ecefe37da
fix root route 2021-03-19 11:34:54 +01:00
mpostma
147756750b
create uuid on successful update addition
also change resolve to get in uuid resolver
2021-03-18 09:09:26 +01:00
mpostma
8b99860e85
use write sender for updates 2021-03-18 08:32:05 +01:00
Irevoire
6b4ea7f594
ensure the reset_settings also return a 202 2021-03-17 15:09:13 +01:00
Irevoire
c8b05712fa
return 202 on settings update / reset 2021-03-17 14:44:32 +01:00
Clémentine Urquizar
8a52979ffa
Update Cargo.toml 2021-03-16 19:54:34 +01:00
bors[bot]
ca3b343b1f
Merge #96
96: Check json payload on document addition r=curquiza a=MarinPostma

Check if the json payload in updates is valid. It uses a json validator to avoid allocation, and only serializes the json in case of error, to return a pretty message.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 17:20:44 +00:00
bors[bot]
f8ea081df5
Merge #98
98: replace body with json r=curquiza a=MarinPostma



Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 17:12:30 +00:00
tamo
233c1e304d
use json instead of body when crafting the request 2021-03-16 17:45:59 +01:00
tamo
a268d0e283
return a 200 on health check 2021-03-16 17:42:01 +01:00
bors[bot]
764ced8b5c
Merge #88
88: restore name field in index meta r=MarinPostma a=MarinPostma

Makes the IndexMetadata payload iso with legacy meilisearch and closes #67 


Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 15:50:08 +00:00
mpostma
3c25ab0d50
replace body with json 2021-03-16 16:46:07 +01:00
bors[bot]
c6dbd81823
Merge #90
90: restore version route r=MarinPostma a=MarinPostma

close #74


Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 13:53:23 +00:00
mpostma
204c743bcc
add json payload check on document addition 2021-03-16 14:28:13 +01:00
bors[bot]
2138f54954
Merge #89
89: delete index returns 204 instead of 200 r=curquiza a=MarinPostma

 close #63

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 13:01:32 +00:00
mpostma
f4cf96915a
remove guard on add documetn route 2021-03-16 12:04:32 +01:00
mpostma
6a742ee62c
restore version route 2021-03-15 19:11:27 +01:00
mpostma
58fab035bb
delete index returns 204 instead of 200 2021-03-15 18:44:33 +01:00
mpostma
07bb1e2c4e
fix tests 2021-03-15 18:38:13 +01:00
mpostma
94bd14ede3
add name to index_metadata 2021-03-15 18:35:16 +01:00
mpostma
dd324807f9
last review edits + fmt 2021-03-15 18:11:10 +01:00
mpostma
c29b86849b
use actix cors git dependency 2021-03-15 17:40:20 +01:00
mpostma
abbea59732
fix clippy warnings 2021-03-15 16:52:05 +01:00
mpostma
01479dcf99
rename name to uid in code 2021-03-15 14:43:47 +01:00
mpostma
0c80d891c0
clean Cargo.toml 2021-03-15 14:29:30 +01:00
mpostma
f727dcc8c6
update milli 2021-03-15 14:26:59 +01:00
mpostma
55fadd7f87
change facetedAttributes to attributesForFaceting 2021-03-15 13:53:50 +01:00
mpostma
fcf1d4e922
fix displayed attributes in search 2021-03-15 12:20:33 +01:00
mpostma
c079f60346
fixup! fix displayed attributes in document retrieval 2021-03-15 11:01:14 +01:00
mpostma
77c0a0fba5
add test get document displayed attributes 2021-03-15 10:36:12 +01:00
mpostma
adc71a70ce
fix displayed attributes in document retrieval 2021-03-15 10:17:41 +01:00
mpostma
99c89cf2ba
use options max db sizes 2021-03-13 10:09:10 +01:00
mpostma
49b74b587a
enable jemalloc only on linux 2021-03-12 17:47:40 +01:00
mpostma
2ee2e6a9b2
clean project 2021-03-12 14:57:24 +01:00
mpostma
c4846dafca
implement update index 2021-03-12 14:48:43 +01:00
mpostma
77d5dd452f
remove open_or_create 2021-03-12 14:16:54 +01:00
mpostma
e4d45b0500
fix various bugs 2021-03-12 00:37:43 +01:00
mpostma
7d9637861f
fix add primary key on index creation 2021-03-11 22:55:29 +01:00
mpostma
271c8ba991
change index name to uid 2021-03-11 22:47:29 +01:00
mpostma
8617bcf8bd
add ranking rules 2021-03-11 22:39:16 +01:00
mpostma
66b64c1f80
correct error on settings delete unexisting index 2021-03-11 22:33:31 +01:00
mpostma
30dd790884
handle badly formatted index uid 2021-03-11 22:23:48 +01:00
mpostma
40b3451a4e
fix unexisting update store + race conditions 2021-03-11 22:11:58 +01:00
mpostma
3f68460d6c
fix update dedup 2021-03-11 20:58:51 +01:00
mpostma
79a4bc8129
use meta from milli 2021-03-11 19:40:18 +01:00
mpostma
1fad72e019
fix test bug with tempdir 2021-03-11 17:59:47 +01:00
mpostma
2ae90f9c5d
lazy load update store 2021-03-11 14:23:11 +01:00
mpostma
53cf500e36
uuid resolver hard state 2021-03-10 18:04:20 +01:00
mpostma
a56e8c1a0c
fix tests 2021-03-10 14:47:04 +01:00
mpostma
0cd8869349
update relevant changes from master 2021-03-10 14:43:10 +01:00
mpostma
5ecf514d28
restructure project 2021-03-10 13:46:49 +01:00