Kerollmops
|
725158b454
|
Introduce the core algorithm of task cancelation
|
2022-10-27 11:34:11 +02:00 |
|
Kerollmops
|
b2c5bc67b7
|
Add more enum-iterator related stuff
|
2022-10-27 11:34:11 +02:00 |
|
Kerollmops
|
591527a99d
|
Prefer using TaskDeletion in the dumps
|
2022-10-27 11:34:11 +02:00 |
|
Kerollmops
|
1ca9a67c49
|
Introduce the task cancelation task type
|
2022-10-27 11:34:11 +02:00 |
|
Kerollmops
|
703ba7a1fb
|
Introduce the ProcessingTasks struct
|
2022-10-27 11:34:10 +02:00 |
|
Loïc Lecrenier
|
ea60d35c71
|
Delete a task's persisted data when appropriate
|
2022-10-27 11:34:10 +02:00 |
|
Tamo
|
9e85f050b2
|
fix the tests
|
2022-10-27 11:34:09 +02:00 |
|
Tamo
|
2f748480a1
|
share the rtxn between the access to the tasks and to the indexes
|
2022-10-27 11:34:09 +02:00 |
|
Tamo
|
6bd6321226
|
dump the content of the dump tasks instead of recreating at import time with wrong API keys
|
2022-10-27 11:34:08 +02:00 |
|
Tamo
|
655705eb2b
|
remove useless todo
|
2022-10-27 11:34:08 +02:00 |
|
Tamo
|
83f3c5ec57
|
flush the dump-writer only once everything has been inserted
|
2022-10-27 11:34:08 +02:00 |
|
Tamo
|
78ce29f461
|
apply most style comments of the review
|
2022-10-27 11:34:08 +02:00 |
|
Tamo
|
d0e91555d1
|
rebase on index-scheduler
|
2022-10-27 11:34:08 +02:00 |
|
Tamo
|
e0221fc0a3
|
fix a synchronization bug while importing tasks
|
2022-10-27 11:34:07 +02:00 |
|
Tamo
|
ba150f2127
|
commit after creating an index
|
2022-10-27 11:34:07 +02:00 |
|
Tamo
|
e9295c03ce
|
the index-scheduler needs to wake-up after importing a dump
|
2022-10-27 11:34:06 +02:00 |
|
Tamo
|
d481669b7e
|
fix the content_file import
|
2022-10-27 11:34:06 +02:00 |
|
Tamo
|
dd506e5d87
|
stop dumping the current dumping task as enqueued so it's not looping for ever
|
2022-10-27 11:34:06 +02:00 |
|
Tamo
|
d976e680c5
|
first mostly working version
|
2022-10-27 11:34:06 +02:00 |
|
Tamo
|
c051166bcc
|
update the API a little bit
|
2022-10-27 11:34:05 +02:00 |
|
Tamo
|
72a906ae75
|
fix the tests
|
2022-10-27 11:34:05 +02:00 |
|
Tamo
|
b7f9c94f4a
|
write the dump export
|
2022-10-27 11:34:05 +02:00 |
|
Loïc Lecrenier
|
8954b1bd1d
|
Fix number of deleted tasks details after duplicate task deletion
|
2022-10-27 11:34:05 +02:00 |
|
Loïc Lecrenier
|
8defad6c38
|
Add task deletion tests where the same task is deleted twice
|
2022-10-27 11:34:05 +02:00 |
|
Loïc Lecrenier
|
f32b973945
|
Return an error when calling DELETE /tasks with an empty query
|
2022-10-27 11:34:04 +02:00 |
|
Loïc Lecrenier
|
fbd2be2ec8
|
Apply suggested changes from PR review
|
2022-10-27 11:34:04 +02:00 |
|
Loïc Lecrenier
|
441417447e
|
Avoid creating two read txn at the same time
|
2022-10-27 11:34:04 +02:00 |
|
Loïc Lecrenier
|
8c6aeaada5
|
Update snapshot tests following git rebase that fixes a bug
|
2022-10-27 11:34:04 +02:00 |
|
Loïc Lecrenier
|
8bb0fcd144
|
Finish first draft of the DELETE /tasks route
|
2022-10-27 11:34:04 +02:00 |
|
Loïc Lecrenier
|
9522b75454
|
Continue implementation of task deletion
1. Matched tasks are a roaring bitmap
2. Start implementation in meilisearch-http
3. Snapshots use meili-snap
4. Rename to TaskDeletion
|
2022-10-27 11:34:03 +02:00 |
|
Kerollmops
|
e4d461ecba
|
Make sure that we do not batch tasks from different indexes
|
2022-10-27 11:34:03 +02:00 |
|
Kerollmops
|
b029369653
|
Add a test to check different indexes autobatching
|
2022-10-27 11:34:03 +02:00 |
|
Kerollmops
|
408d00136c
|
Extract index creation rights and simplify the autobatcher rules
|
2022-10-27 11:34:03 +02:00 |
|
Kerollmops
|
2c24c7d403
|
Fix invalid import of tasks types
|
2022-10-27 11:34:03 +02:00 |
|
Tamo
|
7034803712
|
move the API key in meilisearch_types
|
2022-10-27 11:34:02 +02:00 |
|
Tamo
|
c192146fbe
|
remove an unused file
|
2022-10-27 11:34:02 +02:00 |
|
Tamo
|
2f1eb78b1d
|
refactor the Task a little bit
|
2022-10-27 11:34:02 +02:00 |
|
Tamo
|
510ce9fc51
|
start moving a lot of task types to meilisearch_types
|
2022-10-27 11:34:01 +02:00 |
|
Tamo
|
0af00f6b32
|
fix all the import and comment most of the dump v6
|
2022-10-27 11:34:01 +02:00 |
|
Tamo
|
667c282e19
|
get rids of the index crate + the document_types crate
|
2022-10-27 11:34:00 +02:00 |
|
Loïc Lecrenier
|
9a74ea0943
|
Fix compiler errors related autobatching option of the index scheduler
|
2022-10-27 11:34:00 +02:00 |
|
Loïc Lecrenier
|
eabac9676b
|
Fix typo and remove useless code in tests
|
2022-10-27 11:34:00 +02:00 |
|
Loïc Lecrenier
|
ab4e649221
|
Apply suggestions from code review
Co-authored-by: Tamo <tamo@meilisearch.com>
|
2022-10-27 11:34:00 +02:00 |
|
Loïc Lecrenier
|
568199fc0d
|
Add more task deletion tests
|
2022-10-27 11:33:59 +02:00 |
|
Loïc Lecrenier
|
13a72f8757
|
Use more complete snapshot tests for the index scheduler
|
2022-10-27 11:33:59 +02:00 |
|
Loïc Lecrenier
|
4c55c30027
|
Add a DetailsView type and improve index scheduler snapshots
The DetailsView type is necessary because serde incorrectly
deserialises the `Details` type, so the database fails to correctly
decode Tasks
|
2022-10-27 11:33:59 +02:00 |
|
Loïc Lecrenier
|
dc81992eb2
|
Implement TaskDeletion in the index scheduler
|
2022-10-27 11:33:59 +02:00 |
|
Kerollmops
|
fe84f2648b
|
Allow a user to disable the auto batching system
|
2022-10-27 11:33:59 +02:00 |
|
Kerollmops
|
e2a766acb5
|
Add a test to check that it works without autobatching
|
2022-10-27 11:33:58 +02:00 |
|
Kerollmops
|
db9d1b18ca
|
Remove the IndexScheduler::notify method
|
2022-10-27 11:33:58 +02:00 |
|
Kerollmops
|
19c6f8303f
|
Make sure that the index-scheduler tick loop is rerun after processing
|
2022-10-27 11:33:58 +02:00 |
|
Kerollmops
|
b311eb3bed
|
Add a test that verifies that sending multiple tasks works
|
2022-10-27 11:33:58 +02:00 |
|
Tamo
|
0972587cfc
|
start writting the compat layer between v5 and v6
|
2022-10-27 11:33:52 +02:00 |
|
Tamo
|
e845cc2b6f
|
fix the tests
|
2022-10-27 11:33:51 +02:00 |
|
Clément Renault
|
123f47dbc4
|
Create the index only if the task has the rights to do so
|
2022-10-27 11:33:50 +02:00 |
|
Clément Renault
|
068a4b2884
|
Correctly batch tasks with different index creation rights
|
2022-10-27 11:33:50 +02:00 |
|
Clément Renault
|
87212cfd20
|
Use a ControlFlow in the autobatcher function
|
2022-10-27 11:33:50 +02:00 |
|
Kerollmops
|
f1b1cfdbcc
|
IndexDeletion operation have ClearAll details
|
2022-10-27 11:33:50 +02:00 |
|
Kerollmops
|
a083c9e452
|
Only mark the first clear document with the amount of cleared documents
|
2022-10-27 11:33:50 +02:00 |
|
Kerollmops
|
b24b13b036
|
Let the tick function set the Failed status itself
|
2022-10-27 11:33:50 +02:00 |
|
Kerollmops
|
566c15fb74
|
Fill an IndexDeletion task with the number of documents removed
|
2022-10-27 11:33:49 +02:00 |
|
Kerollmops
|
6b3b05fb73
|
Panic if we encountered a wring KindWithContent type
|
2022-10-27 11:33:49 +02:00 |
|
Kerollmops
|
36e5efde0d
|
Update the tasks statuses
|
2022-10-27 11:33:49 +02:00 |
|
Kerollmops
|
2fbdd104b8
|
Implement the IndexDeletion batch operation
|
2022-10-27 11:33:49 +02:00 |
|
Kerollmops
|
da363a92ac
|
Implement the IndexUpdate batch operation
|
2022-10-27 11:33:49 +02:00 |
|
Kerollmops
|
0543cba6eb
|
Implement the IndexCreate batch operation
|
2022-10-27 11:33:48 +02:00 |
|
Kerollmops
|
cf6084151b
|
Make sure that meilisearch-http works without index wrapper
|
2022-10-27 11:33:48 +02:00 |
|
Kerollmops
|
d76634a36c
|
Remove the Index wrapper and use milli::Index directly
|
2022-10-27 11:33:48 +02:00 |
|
Kerollmops
|
9e8242c57d
|
Remove the IndexRename operation
|
2022-10-27 11:33:48 +02:00 |
|
Kerollmops
|
5fa214abb1
|
Move the IndexScheduler to the root of the index-scheduler crate
|
2022-10-27 11:33:47 +02:00 |
|
Kerollmops
|
9a9e98fb77
|
Add a TODO about the index creation
|
2022-10-27 11:33:47 +02:00 |
|
Kerollmops
|
5d21c790ef
|
Make clippy happy
|
2022-10-27 11:33:47 +02:00 |
|
Kerollmops
|
31de33d5ee
|
Implement a recursive indexation for the index-related operations
|
2022-10-27 11:33:47 +02:00 |
|
Kerollmops
|
07286fcc79
|
Implement the SettingsAndDocumentImport batch operation
|
2022-10-27 11:33:47 +02:00 |
|
Kerollmops
|
f68906f5dc
|
Merge both DocumentAddition/Update into one DocumentImport variant
|
2022-10-27 11:33:46 +02:00 |
|
Kerollmops
|
5174c78f87
|
Implement the DocumentClear batch operation
|
2022-10-27 11:33:46 +02:00 |
|
Kerollmops
|
025bb5f616
|
Implement the DocumentClearAndSettings batch operation
|
2022-10-27 11:33:46 +02:00 |
|
Kerollmops
|
41ec737e73
|
Implement the Settings batch operation
|
2022-10-27 11:33:46 +02:00 |
|
Kerollmops
|
7b4a913704
|
Implement the DocumentUpdate batch operation
|
2022-10-27 11:33:46 +02:00 |
|
Kerollmops
|
a6a1043abb
|
Implement the DocumentDeletion batch operation
|
2022-10-27 11:33:46 +02:00 |
|
Tamo
|
7a0f17c912
|
remove an old unworking part of the batch execution
|
2022-10-27 11:33:45 +02:00 |
|
Tamo
|
c2899fe9b2
|
bring back the IndexMeta and IndexStats in meilisearch-http
|
2022-10-27 11:33:45 +02:00 |
|
Tamo
|
2c8f1a43e9
|
get rids of meilisearch-lib
|
2022-10-27 11:33:44 +02:00 |
|
Tamo
|
0ba1c46e19
|
fix a deadlock
|
2022-10-27 11:33:44 +02:00 |
|
Tamo
|
22bfb5a7a0
|
remove Clone from the IndexScheduler
|
2022-10-27 11:33:44 +02:00 |
|
Tamo
|
d8d3499aec
|
remove a bunch of comments
|
2022-10-27 11:33:44 +02:00 |
|
Tamo
|
64e132ce53
|
move as many fields as possible out of the IndexScheduler
|
2022-10-27 11:33:44 +02:00 |
|
Tamo
|
9e1f38ec7c
|
move the test function in the test module
|
2022-10-27 11:33:44 +02:00 |
|
Tamo
|
6f4dcc0c38
|
start implementing some logic to test the internal states of the scheduler
|
2022-10-27 11:33:43 +02:00 |
|
Tamo
|
84cd5cef0b
|
fix the tests
|
2022-10-27 11:33:43 +02:00 |
|
Tamo
|
ae86a8ccd6
|
slightly refactor the autobatching tests
|
2022-10-27 11:33:43 +02:00 |
|
Tamo
|
ce2dfecc03
|
connect the new scheduler to meilisearch-http officially.
I can index documents and do search
|
2022-10-27 11:33:43 +02:00 |
|
Tamo
|
cb4feabca2
|
implements the get_tasks
|
2022-10-27 11:33:43 +02:00 |
|
Tamo
|
19154e48fe
|
fix all compilation errors
|
2022-10-27 11:33:42 +02:00 |
|
Irevoire
|
8d51c1f389
|
wip integrating the scheduler in meilisearch-http
|
2022-10-27 11:33:42 +02:00 |
|
Irevoire
|
250410495c
|
start integrating the index-scheduler in meilisearch-lib
|
2022-10-27 11:33:42 +02:00 |
|
Irevoire
|
8f0fd35358
|
add insta::json for later
|
2022-10-27 11:33:42 +02:00 |
|
Irevoire
|
8770e07397
|
I can index documents without meilisearch
|
2022-10-27 11:33:42 +02:00 |
|
Tamo
|
edd8344dc9
|
wip
|
2022-10-27 11:33:42 +02:00 |
|
Tamo
|
e547552702
|
create the end Batch type for all Index* operations
|
2022-10-27 11:33:41 +02:00 |
|
Tamo
|
925971809a
|
create the end Batch type for all Document* operation
|
2022-10-27 11:33:41 +02:00 |
|
Tamo
|
1ea9c0b4c0
|
write most of the run loop
|
2022-10-27 11:33:41 +02:00 |
|
Tamo
|
9ff0fe952e
|
split the run function in two
|
2022-10-27 11:33:41 +02:00 |
|
Tamo
|
a8b18b2c96
|
fix the register test
|
2022-10-27 11:33:40 +02:00 |
|
Tamo
|
5436b996ab
|
reduce the size of the snapshots
|
2022-10-27 11:33:40 +02:00 |
|
Tamo
|
7d0c8a3379
|
test the register tasks
|
2022-10-27 11:33:40 +02:00 |
|
Tamo
|
fc098022c7
|
start integrating the index-scheduler in the meilisearch codebase
|
2022-10-27 11:33:40 +02:00 |
|
Tamo
|
b816535e33
|
greatly reduce the number of warnings
|
2022-10-27 11:33:40 +02:00 |
|
Tamo
|
38e4ffe73c
|
fix smol typo
|
2022-10-27 11:33:40 +02:00 |
|
Tamo
|
366a344474
|
get rids of the horrendous spinlock in favor of synchronoise
|
2022-10-27 11:33:39 +02:00 |
|
Tamo
|
7b6673dc1d
|
implement the index swap in the index mapper
|
2022-10-27 11:33:39 +02:00 |
|
Tamo
|
03aca2e452
|
move the index mapping logic in another structure
|
2022-10-27 11:33:39 +02:00 |
|
Tamo
|
4129783019
|
migrate the index handling code in a different file + implements the create index
|
2022-10-27 11:33:39 +02:00 |
|
Tamo
|
1804416afa
|
reintroduce the uuid mapping for the indexes
|
2022-10-27 11:33:39 +02:00 |
|
Tamo
|
c97d51a624
|
add a bunch of tests
|
2022-10-27 11:33:39 +02:00 |
|
Tamo
|
803f2157af
|
split the DocumentAdditionOrUpdate in two tasks; DocumentAddition and DocumentUpdate
|
2022-10-27 11:33:38 +02:00 |
|
Tamo
|
b7c5b71a53
|
starts importing the real tasks
|
2022-10-27 11:33:38 +02:00 |
|
Tamo
|
5cc8f96237
|
get rids of the auto-generated mains
|
2022-10-27 11:33:38 +02:00 |
|
Tamo
|
94e29a9f5f
|
extract the index abstraction out of the index-scheduler in its own module
|
2022-10-27 11:33:38 +02:00 |
|
Tamo
|
48138c21a9
|
rename the update-file-store to file-store since it can store any kind of file
|
2022-10-27 11:33:38 +02:00 |
|
Tamo
|
76597fc382
|
import the update_file_store in the index-scheduler
|
2022-10-27 11:33:37 +02:00 |
|
Tamo
|
a0588d6b94
|
finishes the global skelton of the auto-batcher
|
2022-10-27 11:33:37 +02:00 |
|
Tamo
|
b3c9b128d9
|
polish the global structure of the batch creation
|
2022-10-27 11:33:37 +02:00 |
|
Irevoire
|
448f44f631
|
move the autobatcher logic to another file
|
2022-10-27 11:33:36 +02:00 |
|
Tamo
|
f638774764
|
add the document format file
|
2022-10-27 11:33:36 +02:00 |
|
Tamo
|
516860f342
|
fix the create_new_batch method
|
2022-10-27 11:33:36 +02:00 |
|
Tamo
|
6b9689a1c0
|
fix the whole batchKind thingy
|
2022-10-27 11:33:36 +02:00 |
|
Tamo
|
af0f5d6c0c
|
implements most operations
|
2022-10-27 11:33:36 +02:00 |
|
Tamo
|
5a7fcf2688
|
fix a few typos
|
2022-10-27 11:33:35 +02:00 |
|
Tamo
|
30d2b24689
|
implements the index deletion, creation and swap
|
2022-10-27 11:33:35 +02:00 |
|
Tamo
|
72b2e68de4
|
makes the updates getters smoother to uses
|
2022-10-27 11:33:35 +02:00 |
|
Tamo
|
7879189c6b
|
make the project compile again
|
2022-10-27 11:33:35 +02:00 |
|
Tamo
|
46b8ebcab4
|
fix the file store
|
2022-10-27 11:33:35 +02:00 |
|
Tamo
|
fa742f60e8
|
make the file store entirely synchronous, including the file deletion
|
2022-10-27 11:33:35 +02:00 |
|
Tamo
|
a7aa92df5f
|
fix most of the index module
|
2022-10-27 11:33:34 +02:00 |
|
Irevoire
|
d8b8e04ad1
|
wip porting the index back in the scheduler
|
2022-10-27 11:33:34 +02:00 |
|
Irevoire
|
fe330e1be9
|
add a little bit of documentation
|
2022-10-27 11:33:34 +02:00 |
|
Tamo
|
2c4e5ce8be
|
implements the filter query
|
2022-10-27 11:33:34 +02:00 |
|
Tamo
|
705af94fd7
|
add the task to the index db in the register task
|
2022-10-27 11:33:34 +02:00 |
|
Tamo
|
ed745591e1
|
split the scheduler into multiples files
|
2022-10-27 11:33:34 +02:00 |
|
Tamo
|
22d24dba56
|
implement the get_batch method
|
2022-10-27 11:33:33 +02:00 |
|
Tamo
|
1a47949063
|
START THE REWRITE OF THE INDEX SCHEDULER: index & register has been implemented
|
2022-10-27 11:33:33 +02:00 |
|