Commit Graph

142 Commits

Author SHA1 Message Date
tamo
8b7735c20a
move the import of the updates in the v2 and ignore the v1 for now 2021-05-11 00:20:55 +02:00
tamo
7d748fa384
integrate the new Settings in the dumps 2021-05-10 20:48:06 +02:00
tamo
d767990424
fix the import of the updates in the dump 2021-05-10 20:25:12 +02:00
tamo
ef438852cd
fix the v1 2021-05-10 20:25:12 +02:00
tamo
40ced3ff8d
first working version 2021-05-10 20:25:12 +02:00
tamo
5f5402a3ab
provide a way to access the internal content path of all processing State 2021-05-10 20:25:12 +02:00
tamo
26dcb9e66d
bump milli version and fix a performance issue for large dumps 2021-05-10 20:25:12 +02:00
Marin Postma
956012da95
fix dump lock 2021-05-10 20:25:12 +02:00
tamo
24192fc550
fix tests 2021-05-10 20:25:12 +02:00
tamo
efca63f9ce
[WIP] rebase on main 2021-05-10 20:25:09 +02:00
tamo
c3552cecdf
WIP rebase on main 2021-05-10 20:24:18 +02:00
tamo
0f94ef8abc
WIP: dump 2021-05-10 20:24:18 +02:00
tamo
0275b36fb0
[WIP] rebase on main 2021-05-10 20:24:14 +02:00
tamo
1b5fc61eb6
[WIP] rebase on main 2021-05-10 20:23:12 +02:00
tamo
0fee81678e
[WIP] rebase on main 2021-05-10 20:22:18 +02:00
tamo
c4d898a265
split the dumps between v1 and v2 2021-05-10 20:20:57 +02:00
tamo
e389c088eb
WIP: rebasing on master 2021-05-10 20:20:57 +02:00
Marin Postma
706643dfed
type setting struct 2021-05-10 17:30:09 +02:00
Marin Postma
ec7eb7798f
remove facet setting 2021-05-04 22:36:31 +02:00
Marin Postma
a961f0ce75
fix clippy warnings 2021-04-27 18:28:46 +02:00
Marin Postma
4fe2a13c71
rewrite update store 2021-04-27 15:20:52 +02:00
Marin Postma
51829ad85e
review fixes 2021-04-27 15:10:57 +02:00
Marin Postma
c78f351300
fix tests 2021-04-27 15:10:57 +02:00
Marin Postma
ee675eadf1
fix stats 2021-04-27 15:10:55 +02:00
Marin Postma
33830d5ecf
fix snapshots 2021-04-27 15:09:55 +02:00
Marin Postma
2b154524bb
fix filtered out pending update 2021-04-27 15:09:23 +02:00
Marin Postma
b626d02ffe
simplify index actor run loop 2021-04-27 15:09:22 +02:00
Marin Postma
9ce68d11a7
single update store instance 2021-04-27 15:09:21 +02:00
Marin Postma
5a38f13cae
multi_index udpate store 2021-04-27 15:07:13 +02:00
Clémentine Urquizar
1ba46f8f77
Disable clippy rule 2021-04-27 12:43:00 +02:00
Clémentine Urquizar
f80ea24d2b
Add tests on every platform and fix clippy errors 2021-04-27 12:42:59 +02:00
Alexey Shekhirin
adfdb99abc
feat(http): calculate updates' and uuids' dbs size 2021-04-09 15:59:12 +03:00
Alexey Shekhirin
ae1655586c
fixes after review 2021-04-09 14:40:48 +03:00
Alexey Shekhirin
698a1ea582
feat(http): store processing as RwLock<Option<Uuid>> in index_actor 2021-04-09 14:34:43 +03:00
Alexey Shekhirin
87412f63ef
feat(http): implement is_indexing for stats 2021-04-09 14:34:42 +03:00
Alexey Shekhirin
dd9eae8c26
feat(http): stats route 2021-04-09 14:34:42 +03:00
Clémentine Urquizar
cb23775d18
Rename pending into enqueued 2021-04-07 19:46:36 +02:00
tamo
0f2143e7fd
remove the now useless dead_code flags 2021-03-26 14:15:12 +01:00
mpostma
d7c077cffb
atomic snapshot import 2021-03-25 14:48:51 +01:00
mpostma
7d6ec7f3d3
resolve merge 2021-03-25 14:21:05 +01:00
mpostma
f3dc853be3
Merge remote-tracking branch 'origin/main' into snapshots 2021-03-25 13:45:07 +01:00
mpostma
48507460b2
add snapshot tests 2021-03-25 12:02:10 +01:00
mpostma
d029464de8
fix snapshot path 2021-03-25 10:23:31 +01:00
mpostma
79d09705d8
perform snapshot on startup 2021-03-25 09:35:15 +01:00
bors[bot]
868658f3d8
Merge #109
109: Make updates atomic r=curquiza a=MarinPostma

Until now, the index_uid->uuid mapping was done before the update was written to disk in the case of automatic index creation. This was an issue when the update failed, and the index would still exists in the uuid resolver.

This is fixed by this pr, by first creating the update with an uuid if the index does not exist, and then register this uuid to the uuid resolver.

This is preliminary work to the implementation of snapshots (#19).

This pr also changes the `resolve` method on the `UuidResolver` to `get` to make it clearer.


The `create_uuid` method may be bound to disappear when the index name resolution is handled by a remote machine.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-24 12:24:32 +00:00
mpostma
d892a2643e
fix clippy 2021-03-24 12:38:59 +01:00
mpostma
83ffdc888a
remove bad file name test 2021-03-24 12:38:59 +01:00
mpostma
4041d9dc48
format code 2021-03-24 12:38:59 +01:00
mpostma
1f16c8d224
integration test snapshot 2021-03-24 12:38:59 +01:00
mpostma
06f9dae0f3
remove prints 2021-03-24 12:38:59 +01:00
mpostma
48d5f88c1a
fix snapshot dir already exists 2021-03-24 12:38:59 +01:00
mpostma
eb53ed4cc1
load snapshot 2021-03-24 12:38:59 +01:00
mpostma
46293546f3
add tests and mocks 2021-03-24 12:38:59 +01:00
mpostma
3cc3637e2d
refactor for tests 2021-03-24 12:38:56 +01:00
mpostma
1f51fc8baf
create indexes snapshots concurrently 2021-03-24 12:38:12 +01:00
mpostma
e9da191b7d
fix snapshot bugs 2021-03-24 12:38:12 +01:00
mpostma
d73fbdef2e
remove from snapshot 2021-03-24 12:38:12 +01:00
mpostma
44dcfe29aa
clean snapshot creation 2021-03-24 12:38:12 +01:00
mpostma
a85e7abb0c
fix snapshot creation 2021-03-24 12:38:12 +01:00
mpostma
4847884165
restore snapshots 2021-03-24 12:38:12 +01:00
mpostma
7f6a54cb12
add lock to prevent snapshot during update 2021-03-24 12:38:12 +01:00
mpostma
520f7c09ba
sequential index snapshot 2021-03-24 12:38:12 +01:00
mpostma
35a7b800eb
snapshot indexes 2021-03-24 12:38:12 +01:00
mpostma
c966b1dd94
use options to schedule snapshot 2021-03-24 12:38:11 +01:00
mpostma
ee838be41b
implement snapshot scheduler 2021-03-24 12:38:11 +01:00
mpostma
5f33672f0e
change payload send to use stream methods 2021-03-22 19:49:21 +01:00
mpostma
b690f1103a
fix typos 2021-03-22 19:25:56 +01:00
mpostma
147756750b
create uuid on successful update addition
also change resolve to get in uuid resolver
2021-03-18 09:09:26 +01:00
mpostma
8b99860e85
use write sender for updates 2021-03-18 08:32:05 +01:00
bors[bot]
ca3b343b1f
Merge #96
96: Check json payload on document addition r=curquiza a=MarinPostma

Check if the json payload in updates is valid. It uses a json validator to avoid allocation, and only serializes the json in case of error, to return a pretty message.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 17:20:44 +00:00
mpostma
204c743bcc
add json payload check on document addition 2021-03-16 14:28:13 +01:00
mpostma
94bd14ede3
add name to index_metadata 2021-03-15 18:35:16 +01:00
mpostma
dd324807f9
last review edits + fmt 2021-03-15 18:11:10 +01:00
mpostma
abbea59732
fix clippy warnings 2021-03-15 16:52:05 +01:00
mpostma
01479dcf99
rename name to uid in code 2021-03-15 14:43:47 +01:00
mpostma
99c89cf2ba
use options max db sizes 2021-03-13 10:09:10 +01:00
mpostma
49b74b587a
enable jemalloc only on linux 2021-03-12 17:47:40 +01:00
mpostma
2ee2e6a9b2
clean project 2021-03-12 14:57:24 +01:00
mpostma
c4846dafca
implement update index 2021-03-12 14:48:43 +01:00
mpostma
77d5dd452f
remove open_or_create 2021-03-12 14:16:54 +01:00
mpostma
e4d45b0500
fix various bugs 2021-03-12 00:37:43 +01:00
mpostma
7d9637861f
fix add primary key on index creation 2021-03-11 22:55:29 +01:00
mpostma
271c8ba991
change index name to uid 2021-03-11 22:47:29 +01:00
mpostma
66b64c1f80
correct error on settings delete unexisting index 2021-03-11 22:33:31 +01:00
mpostma
30dd790884
handle badly formatted index uid 2021-03-11 22:23:48 +01:00
mpostma
40b3451a4e
fix unexisting update store + race conditions 2021-03-11 22:11:58 +01:00
mpostma
3f68460d6c
fix update dedup 2021-03-11 20:58:51 +01:00
mpostma
79a4bc8129
use meta from milli 2021-03-11 19:40:18 +01:00
mpostma
1fad72e019
fix test bug with tempdir 2021-03-11 17:59:47 +01:00
mpostma
2ae90f9c5d
lazy load update store 2021-03-11 14:23:11 +01:00
mpostma
53cf500e36
uuid resolver hard state 2021-03-10 18:04:20 +01:00
mpostma
5ecf514d28
restructure project 2021-03-10 13:46:49 +01:00