49a0e8aa19
use a RwLock instead of a Mutex
2021-05-24 18:19:34 +02:00
912f0286b3
remove the dump_inner trickery
2021-05-24 18:06:20 +02:00
dcf29e1081
fix the error handling in case there is a panic while creating a dump
2021-05-24 17:33:42 +02:00
529f7962f4
handle parallel requests for the dump actor
2021-05-24 15:42:12 +02:00
8a11c6c429
Implements the legacy behaviour of the dump
...
When asked if a dump exists we check if it's the current dump, and if
it's not then we check on the filesystem for any file matching our
`uid.dump`
2021-05-24 12:35:46 +02:00
4cbf866821
merge with main
2021-05-12 18:12:37 +02:00
e0e23636c6
fix the serializer + reformat the file
2021-05-12 17:04:24 +02:00
295f496e8a
atomic index dump load
2021-05-12 16:21:37 +02:00
6d837e3e07
the route to create a dump must return a 202
2021-05-11 17:34:34 +02:00
1b671d4302
fix-snapshot
2021-05-11 13:57:18 +02:00
c30b32e173
add the criterion attribute when importing dumps from the v1
2021-05-11 13:21:36 +02:00
9e798fea75
fix the import of dump without unprocessing updates
2021-05-11 13:03:47 +02:00
384afb3455
fix the way we return the settings
2021-05-11 11:47:04 +02:00
92a7c8cd17
make clippy happy
2021-05-11 00:27:22 +02:00
8b7735c20a
move the import of the updates in the v2 and ignore the v1 for now
2021-05-11 00:20:55 +02:00
7d748fa384
integrate the new Settings in the dumps
2021-05-10 20:48:06 +02:00
d767990424
fix the import of the updates in the dump
2021-05-10 20:25:12 +02:00
ef438852cd
fix the v1
2021-05-10 20:25:12 +02:00
40ced3ff8d
first working version
2021-05-10 20:25:12 +02:00
5f5402a3ab
provide a way to access the internal content path of all processing State
2021-05-10 20:25:12 +02:00
26dcb9e66d
bump milli version and fix a performance issue for large dumps
2021-05-10 20:25:12 +02:00
956012da95
fix dump lock
2021-05-10 20:25:12 +02:00
24192fc550
fix tests
2021-05-10 20:25:12 +02:00
efca63f9ce
[WIP] rebase on main
2021-05-10 20:25:09 +02:00
c3552cecdf
WIP rebase on main
2021-05-10 20:24:18 +02:00
0f94ef8abc
WIP: dump
2021-05-10 20:24:18 +02:00
0275b36fb0
[WIP] rebase on main
2021-05-10 20:24:14 +02:00
1b5fc61eb6
[WIP] rebase on main
2021-05-10 20:23:12 +02:00
0fee81678e
[WIP] rebase on main
2021-05-10 20:22:18 +02:00
c4d898a265
split the dumps between v1 and v2
2021-05-10 20:20:57 +02:00
e389c088eb
WIP: rebasing on master
2021-05-10 20:20:57 +02:00
0cc79d414f
add test
2021-05-10 18:34:25 +02:00
8d11b368d1
implement check
2021-05-10 18:22:41 +02:00
706643dfed
type setting struct
2021-05-10 17:30:09 +02:00
b192cb9c1f
enable string syntax for the filters
2021-05-06 12:48:31 +02:00
ec7eb7798f
remove facet setting
2021-05-04 22:36:31 +02:00
a717925caa
remove filters, rename facet_filters to filter
2021-05-04 18:20:56 +02:00
eb03a3ccb1
Upgrade Milli and Tokenizer
2021-05-04 17:56:19 +02:00
928fb34eff
bump milli and fix tests
2021-05-04 15:10:22 +02:00
1e6b40a24b
Merge #172
...
172: Fix cors authentication issue r=MarinPostma a=MarinPostma
The error was due to the middleware returning an error, instead of a response containing the error.
close #110
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-05-03 08:38:42 +00:00
78217bcf18
Fix cors authentication issue
2021-04-29 16:28:12 +02:00
3d5fba94c2
Update mini-dashboard with version 0.1.2
2021-04-29 15:22:41 +02:00
8bc7dd8b03
Merge #143
...
143: Shared update store r=irevoire a=MarinPostma
This PR changes the updates process so that only one instance of an update store is shared among indexes.
This allows updates to always be processed sequentially without additional synchronization, and fixes the bug where all the first pending update for each index were reported as processing whereas only one was.
EDIT:
I ended having to rewrite the whole `UpdateStore` to allow updates being really queued and processed sequentially in the ordered they were added. For that purpose I created a `pending_queue` that orders the updates by a global update id.
To find the next `update_id` to use, both globally and for each index, I have created another database that contains the next id to use.
Finally, all updates that have been processed (with success or otherwise) are all stores in an `updates` database.
The layout for the keys of these databases are such that it is easy to iterate over the elements for a particular index, and greatly reduces the amount of code to do so, compared to the former implementation.
I have also simplified the locking mechanism for the update store, thanks to the StateLock data structure, that allow both an arbitrary number of readers and a single writer to concurrently access the state. The current state can be either Idle, Processing, or Snapshotting. When an update or snapshotting is ongoing, the process holds the state lock until it is done processing its task. When it is done, it sets bask the state to Idle.
I have made other small improvements here and there, and have let some other for work, such as:
- When creating an update file to hold a request's content, it would be preferable to first create a temporary file, and then atomically persist it when we have written to it. This would simplify the case when there is no data to be written to the file, since we wouldn't have to take care about cleaning after ourselves.
- The logic for content validation must be factored.
- Some more tests related to error handling in the process_pending_update function.
- The issue #159
close #114
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-04-27 18:41:55 +00:00
a961f0ce75
fix clippy warnings
2021-04-27 18:28:46 +02:00
bb79a15c04
reenable ranking rules route
2021-04-27 15:29:00 +02:00
4fe2a13c71
rewrite update store
2021-04-27 15:20:52 +02:00
51829ad85e
review fixes
2021-04-27 15:10:57 +02:00
c78f351300
fix tests
2021-04-27 15:10:57 +02:00
ee675eadf1
fix stats
2021-04-27 15:10:55 +02:00
33830d5ecf
fix snapshots
2021-04-27 15:09:55 +02:00
2b154524bb
fix filtered out pending update
2021-04-27 15:09:23 +02:00
b626d02ffe
simplify index actor run loop
2021-04-27 15:09:22 +02:00
9ce68d11a7
single update store instance
2021-04-27 15:09:21 +02:00
5a38f13cae
multi_index udpate store
2021-04-27 15:07:13 +02:00
1ba46f8f77
Disable clippy rule
2021-04-27 12:43:00 +02:00
f80ea24d2b
Add tests on every platform and fix clippy errors
2021-04-27 12:42:59 +02:00
5014f74649
put mini-dashboard in out-dir
2021-04-27 09:32:17 +02:00
f3b6bf55a6
Update version for the next release (alpha4)
2021-04-26 19:05:16 +02:00
c2461e5066
review fixes
2021-04-26 10:20:46 +02:00
e4bd1bc5ce
update actix-web-static-file rev
2021-04-22 11:42:41 +02:00
90f57c1329
update CI & Dockerfile
2021-04-22 11:22:09 +02:00
6af769af20
bump mini-dashboard
2021-04-22 10:45:05 +02:00
6bcf20c70e
serve static site
2021-04-22 10:26:54 +02:00
bb79695e44
load mini-dashboard assets
2021-04-22 10:26:54 +02:00
ea5517bc8c
add mini-dashboard feature
2021-04-22 10:26:54 +02:00
a72d2f66cd
use <em> tags instead of <mark> tags for highlighting
2021-04-21 19:14:55 +02:00
dd2914873b
fix document fields order
2021-04-20 21:30:30 +02:00
d9a29cae60
fix ignored displayed attributes
2021-04-20 21:23:35 +02:00
7a737d2bd3
support wildcard
2021-04-20 21:23:35 +02:00
881b099c8e
add tests
2021-04-20 21:23:34 +02:00
c6bb36efa5
implement _formated
2021-04-20 21:23:28 +02:00
526a05565e
add SearchHit structure
2021-04-20 21:22:48 +02:00
b8e535579f
Update version for the next release (alpha3)
2021-04-20 16:11:07 +02:00
f8c338e3a7
add test for dedicated distinct route
2021-04-20 15:49:17 +02:00
6c470cf687
enable distinct-attribute setting route
2021-04-20 11:34:18 +02:00
ec63e13896
bump actix
2021-04-20 11:29:32 +02:00
1746132c7d
add test set/reset distinct attribute
2021-04-20 11:29:08 +02:00
ec230c2835
enable distinct
2021-04-20 11:29:06 +02:00
45665245dc
bump actix
2021-04-20 11:07:23 +02:00
94c5c5843b
Merge #149
...
149: Handle star in attributes_to_retrieve r=MarinPostma a=curquiza
Closes #147
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-19 17:29:21 +00:00
8eceba98d3
Handle star in attributes_to_retrieve
2021-04-19 18:20:19 +02:00
2c380731b9
Update milli version to v0.1.1
2021-04-19 16:03:39 +02:00
7ce74f95a2
Merge #146
...
146: Remove another unused legacy file r=MarinPostma a=irevoire
When doing #135 I missed an old useless file in the scr/routes directory
Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-15 18:05:28 +00:00
ec3a08ea0c
remove another unused legacy file
2021-04-15 14:44:43 +02:00
b0717b75d9
Update tokenizer to v0.2.1
2021-04-14 19:06:18 +02:00
f87afbc558
fix(http): commit date & SHA in startup message
2021-04-13 20:16:18 +03:00
9eaf048a06
fix(http): use BTreeMap instead of HashMap to preserve stats order
2021-04-13 11:59:07 +03:00
adfdb99abc
feat(http): calculate updates' and uuids' dbs size
2021-04-09 15:59:12 +03:00
ae1655586c
fixes after review
2021-04-09 14:40:48 +03:00
698a1ea582
feat(http): store processing as RwLock<Option<Uuid>> in index_actor
2021-04-09 14:34:43 +03:00
87412f63ef
feat(http): implement is_indexing for stats
2021-04-09 14:34:42 +03:00
09d9a29176
test(http): server & index stats
2021-04-09 14:34:42 +03:00
dd9eae8c26
feat(http): stats route
2021-04-09 14:34:42 +03:00
a1d04fbff5
Merge #136
...
136: Rename update status "pending" into "enqueued" r=curquiza a=curquiza
Closes #107
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-08 16:46:12 +00:00
51ba1bd7d3
fix(http, index): init analyzer with optional stop words
...
Next release
update tokenizer
2021-04-08 17:16:13 +03:00
f881e8691e
Merge #135
...
135: Add stop words r=curquiza a=irevoire
closes #21
Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-08 11:29:00 +00:00
cb23775d18
Rename pending into enqueued
2021-04-07 19:46:36 +02:00
0344cf5874
Merge #122
...
122: Update display r=MarinPostma a=curquiza
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-07 12:33:25 +00:00
dcd60a5b45
add more tests for the stop_words
2021-04-06 18:29:38 +02:00
b1962c8e02
remove legacy files from meilisearch that have been replaced by a macro in routes/settings/mod.rs
2021-04-06 16:29:04 +02:00
40ef9a3c6a
push a first implementation of the stop_words
2021-04-06 16:29:04 +02:00
4ee6ce7871
Next release
2021-04-01 17:16:16 +02:00
6cb8052d3d
Merge #104
...
104: Update all the response format (issue #64 ) r=MarinPostma a=irevoire
closes #64
Co-authored-by: Irevoire <tamo@meilisearch.com>
Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-01 14:22:57 +00:00
73973e2b9e
fix more settings routes
2021-04-01 15:50:45 +02:00
248e9b3808
Merge remote-tracking branch 'origin/main' into snapshots
2021-04-01 15:10:33 +02:00
79c63049d7
update the settings routes
2021-04-01 11:52:26 +02:00
96cffeab1e
update all the response format to be ISO with meilisearch, see #64
2021-04-01 11:43:03 +02:00
d8af4a7202
ignore snapshot test ( #130 )
2021-03-31 20:07:52 +02:00
3d51db5929
fix(ci, http): commit_sha and commit_date in docker builds
...
chore(ci): cache dependencies in Docker build
2021-03-31 13:56:28 +03:00
0f2143e7fd
remove the now useless dead_code flags
2021-03-26 14:15:12 +01:00
b9f79c8df0
Update display
2021-03-26 12:12:55 +01:00
d7c077cffb
atomic snapshot import
2021-03-25 14:48:51 +01:00
7d6ec7f3d3
resolve merge
2021-03-25 14:21:05 +01:00
f3dc853be3
Merge remote-tracking branch 'origin/main' into snapshots
2021-03-25 13:45:07 +01:00
48507460b2
add snapshot tests
2021-03-25 12:02:10 +01:00
d029464de8
fix snapshot path
2021-03-25 10:23:31 +01:00
79d09705d8
perform snapshot on startup
2021-03-25 09:35:15 +01:00
868658f3d8
Merge #109
...
109: Make updates atomic r=curquiza a=MarinPostma
Until now, the index_uid->uuid mapping was done before the update was written to disk in the case of automatic index creation. This was an issue when the update failed, and the index would still exists in the uuid resolver.
This is fixed by this pr, by first creating the update with an uuid if the index does not exist, and then register this uuid to the uuid resolver.
This is preliminary work to the implementation of snapshots (#19 ).
This pr also changes the `resolve` method on the `UuidResolver` to `get` to make it clearer.
The `create_uuid` method may be bound to disappear when the index name resolution is handled by a remote machine.
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-24 12:24:32 +00:00
fe87477238
Merge #115
...
115: Add the exhaustiveNbHits in search response body (returns always false) r=curquiza a=irevoire
closes #103
Co-authored-by: tamo <irevoire@protonmail.ch>
Co-authored-by: Irevoire <irevoire@protonmail.ch>
2021-03-24 12:16:53 +00:00
d892a2643e
fix clippy
2021-03-24 12:38:59 +01:00
83ffdc888a
remove bad file name test
2021-03-24 12:38:59 +01:00
4041d9dc48
format code
2021-03-24 12:38:59 +01:00
1f16c8d224
integration test snapshot
2021-03-24 12:38:59 +01:00
06f9dae0f3
remove prints
2021-03-24 12:38:59 +01:00
48d5f88c1a
fix snapshot dir already exists
2021-03-24 12:38:59 +01:00
eb53ed4cc1
load snapshot
2021-03-24 12:38:59 +01:00
46293546f3
add tests and mocks
2021-03-24 12:38:59 +01:00
3cc3637e2d
refactor for tests
2021-03-24 12:38:56 +01:00
1f51fc8baf
create indexes snapshots concurrently
2021-03-24 12:38:12 +01:00
e9da191b7d
fix snapshot bugs
2021-03-24 12:38:12 +01:00
d73fbdef2e
remove from snapshot
2021-03-24 12:38:12 +01:00
44dcfe29aa
clean snapshot creation
2021-03-24 12:38:12 +01:00
a85e7abb0c
fix snapshot creation
2021-03-24 12:38:12 +01:00
4847884165
restore snapshots
2021-03-24 12:38:12 +01:00
7f6a54cb12
add lock to prevent snapshot during update
2021-03-24 12:38:12 +01:00
520f7c09ba
sequential index snapshot
2021-03-24 12:38:12 +01:00
35a7b800eb
snapshot indexes
2021-03-24 12:38:12 +01:00
c966b1dd94
use options to schedule snapshot
2021-03-24 12:38:11 +01:00
ee838be41b
implement snapshot scheduler
2021-03-24 12:38:11 +01:00
127e944866
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 19:13:22 +01:00
cc81aca6a4
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-03-23 10:47:19 +01:00
46d7cedb18
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 10:46:59 +01:00
5f33672f0e
change payload send to use stream methods
2021-03-22 19:49:21 +01:00
b690f1103a
fix typos
2021-03-22 19:25:56 +01:00
91089db444
add the exhaustive nb hits to be ISO, currently it's always set to false
2021-03-22 18:41:33 +01:00
2f418ee767
Merge #108
...
108: use write senders for updates r=MarinPostma a=MarinPostma
Use write senders to send updates to the `IndexActor`, so updates are performed sequentially on all indexes.
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-22 14:18:43 +00:00
7ecefe37da
fix root route
2021-03-19 11:34:54 +01:00
147756750b
create uuid on successful update addition
...
also change resolve to get in uuid resolver
2021-03-18 09:09:26 +01:00
8b99860e85
use write sender for updates
2021-03-18 08:32:05 +01:00
6b4ea7f594
ensure the reset_settings also return a 202
2021-03-17 15:09:13 +01:00
c8b05712fa
return 202 on settings update / reset
2021-03-17 14:44:32 +01:00
8a52979ffa
Update Cargo.toml
2021-03-16 19:54:34 +01:00
ca3b343b1f
Merge #96
...
96: Check json payload on document addition r=curquiza a=MarinPostma
Check if the json payload in updates is valid. It uses a json validator to avoid allocation, and only serializes the json in case of error, to return a pretty message.
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 17:20:44 +00:00
f8ea081df5
Merge #98
...
98: replace body with json r=curquiza a=MarinPostma
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 17:12:30 +00:00
233c1e304d
use json instead of body when crafting the request
2021-03-16 17:45:59 +01:00
a268d0e283
return a 200 on health check
2021-03-16 17:42:01 +01:00
764ced8b5c
Merge #88
...
88: restore name field in index meta r=MarinPostma a=MarinPostma
Makes the IndexMetadata payload iso with legacy meilisearch and closes #67
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 15:50:08 +00:00
3c25ab0d50
replace body with json
2021-03-16 16:46:07 +01:00
c6dbd81823
Merge #90
...
90: restore version route r=MarinPostma a=MarinPostma
close #74
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 13:53:23 +00:00
204c743bcc
add json payload check on document addition
2021-03-16 14:28:13 +01:00
2138f54954
Merge #89
...
89: delete index returns 204 instead of 200 r=curquiza a=MarinPostma
close #63
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 13:01:32 +00:00
f4cf96915a
remove guard on add documetn route
2021-03-16 12:04:32 +01:00
6a742ee62c
restore version route
2021-03-15 19:11:27 +01:00
58fab035bb
delete index returns 204 instead of 200
2021-03-15 18:44:33 +01:00
07bb1e2c4e
fix tests
2021-03-15 18:38:13 +01:00
94bd14ede3
add name to index_metadata
2021-03-15 18:35:16 +01:00
dd324807f9
last review edits + fmt
2021-03-15 18:11:10 +01:00
c29b86849b
use actix cors git dependency
2021-03-15 17:40:20 +01:00
abbea59732
fix clippy warnings
2021-03-15 16:52:05 +01:00
01479dcf99
rename name to uid in code
2021-03-15 14:43:47 +01:00
0c80d891c0
clean Cargo.toml
2021-03-15 14:29:30 +01:00
f727dcc8c6
update milli
2021-03-15 14:26:59 +01:00
55fadd7f87
change facetedAttributes to attributesForFaceting
2021-03-15 13:53:50 +01:00
fcf1d4e922
fix displayed attributes in search
2021-03-15 12:20:33 +01:00
c079f60346
fixup! fix displayed attributes in document retrieval
2021-03-15 11:01:14 +01:00
77c0a0fba5
add test get document displayed attributes
2021-03-15 10:36:12 +01:00
adc71a70ce
fix displayed attributes in document retrieval
2021-03-15 10:17:41 +01:00
99c89cf2ba
use options max db sizes
2021-03-13 10:09:10 +01:00
49b74b587a
enable jemalloc only on linux
2021-03-12 17:47:40 +01:00
2ee2e6a9b2
clean project
2021-03-12 14:57:24 +01:00
c4846dafca
implement update index
2021-03-12 14:48:43 +01:00
77d5dd452f
remove open_or_create
2021-03-12 14:16:54 +01:00
e4d45b0500
fix various bugs
2021-03-12 00:37:43 +01:00
7d9637861f
fix add primary key on index creation
2021-03-11 22:55:29 +01:00
271c8ba991
change index name to uid
2021-03-11 22:47:29 +01:00
8617bcf8bd
add ranking rules
2021-03-11 22:39:16 +01:00
66b64c1f80
correct error on settings delete unexisting index
2021-03-11 22:33:31 +01:00
30dd790884
handle badly formatted index uid
2021-03-11 22:23:48 +01:00
40b3451a4e
fix unexisting update store + race conditions
2021-03-11 22:11:58 +01:00
3f68460d6c
fix update dedup
2021-03-11 20:58:51 +01:00
79a4bc8129
use meta from milli
2021-03-11 19:40:18 +01:00
1fad72e019
fix test bug with tempdir
2021-03-11 17:59:47 +01:00
2ae90f9c5d
lazy load update store
2021-03-11 14:23:11 +01:00
53cf500e36
uuid resolver hard state
2021-03-10 18:04:20 +01:00
a56e8c1a0c
fix tests
2021-03-10 14:47:04 +01:00
0cd8869349
update relevant changes from master
2021-03-10 14:43:10 +01:00
5ecf514d28
restructure project
2021-03-10 13:46:49 +01:00
tamo