347 Commits

Author	SHA1	Message	Date
Loïc Lecrenier	4d25c159e6	Apply code review suggestions	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	e9cd6cbbee	Revert implementation of get_status to query only the database	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	424202d773	Pause the index scheduler for one second when a fatal error occurs	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	4a35eb9849	Fix (hopefully) queries that include processing tasks	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	493a8cff31	Adjust task details correctly following index swap	2022-10-27 11:35:10 +02:00
Loïc Lecrenier	4de445d386	Start testing unexpected errors and panics in index scheduler	2022-10-27 11:35:10 +02:00
Loïc Lecrenier	e3848b5f28	Add assert method to verify validity of index scheduler state	2022-10-27 11:35:10 +02:00
Irevoire	ecf4e43b3d	rename the dumpExport to dumpCreation	2022-10-27 11:35:10 +02:00
Irevoire	8a23e707c1	fix the task view and forward the task db size	2022-10-27 11:35:09 +02:00
Irevoire	eb4bdde432	fix clippy	2022-10-27 11:35:09 +02:00
Irevoire	1d04ce611d	remove ununsed function	2022-10-27 11:35:08 +02:00
Irevoire	e9055f5572	fix clippy	2022-10-27 11:35:08 +02:00
Irevoire	874499a2d2	fix all the snapshots	2022-10-27 11:35:08 +02:00
Irevoire	ecdcbf350f	update all the snapshots with the new kind name	2022-10-27 11:35:08 +02:00
Irevoire	c8ee453b6c	fix the autobatched document deletion	2022-10-27 11:35:07 +02:00
Irevoire	a8de5368e5	fix the index creation in case an index already exists	2022-10-27 11:35:07 +02:00
Irevoire	9bb2e3c790	fix the failed document addition with a primary key	2022-10-27 11:35:07 +02:00
Irevoire	8d1408c65e	fix the import of the dumpv4&v5 when there is no instance-uid + rename the Kind+KindWithContent+Details variant for the DocumentImport and the Setting	2022-10-27 11:35:05 +02:00
Irevoire	131fe30934	fix the error messages and the index stats	2022-10-27 11:35:05 +02:00
Irevoire	50386921df	fix the index creation	2022-10-27 11:35:05 +02:00
Clément Renault	32cfac0cfd	Sort the TOML dependencies	2022-10-27 11:35:05 +02:00
Clément Renault	80b2e70ee7	Introduce a rustfmt file	2022-10-27 11:35:05 +02:00
Clément Renault	2a7ef3b352	Reapply #2830	2022-10-27 11:34:18 +02:00
Clément Renault	72ec4ce96b	Fix allow_index_creation useless field	2022-10-27 11:34:17 +02:00
Clément Renault	75857bf476	Fix the insta tests	2022-10-27 11:34:17 +02:00
Irevoire	0bbf80186f	push the snapshot files	2022-10-27 11:34:17 +02:00
Irevoire	b6a0abea9f	fix the index deletion when the index doesn’t exists but would be created by one of the autobatched tasks	2022-10-27 11:34:16 +02:00
Irevoire	5303bbffab	fix the last rule about merging the allow_index_creation	2022-10-27 11:34:16 +02:00
Irevoire	fc944c39a5	simplify the code A LOT and create less false positive	2022-10-27 11:34:16 +02:00
Irevoire	a1d4cc673d	add a whole new batch of tests around the index already exists / allow_index_creation	2022-10-27 11:34:16 +02:00
Irevoire	28d9f2c041	fix all the snapshot tests	2022-10-27 11:34:16 +02:00
Irevoire	d9218578e3	it probably works but it's also horrendous	2022-10-27 11:34:16 +02:00
Loïc Lecrenier	11fee30f47	Apply review suggestions and stop using rtxn.commit	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	17cd2a4aa0	Implement POST /indexes-swap	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	28bd8b6c6b	Remove key from index_tasks database when the value is empty	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	169f386418	Add some documentation to the index scheduler	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	bdb17954d2	Fix bug where assert used != instead of == ... And update snapshot tests.	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	23b01a58df	cargo fmt	2022-10-27 11:34:14 +02:00
Loïc Lecrenier	ec3391808d	Fix date parsing for task queries ... Use rfc3339 or YYYY-MM-DD. Add a day to the parsed date when it is an excluded lower bound and the YYYY-MM-DD was used. Also the Query type does not need to be serialisable anymore	2022-10-27 11:34:14 +02:00
Loïc Lecrenier	10a547df4f	Apply suggestions from code review ... Co-authored-by: Clément Renault <clement@meilisearch.com> Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Tamo <tamo@meilisearch.com> Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Tamo <tamo@meilisearch.com> Apply code review suggestion Co-authored-by: Clément Renault <clement@meilisearch.com>	2022-10-27 11:34:14 +02:00
Loïc Lecrenier	22cf0559fe	Implement task date filters ... before/after enqueued/started/finished at	2022-10-27 11:34:14 +02:00
Irevoire	5765883600	fix the auto-generated details	2022-10-27 11:34:14 +02:00
Tamo	cff003c928	remove the unused variants from the autobatcher	2022-10-27 11:34:14 +02:00
Tamo	ab8f1c2865	fix a bunch of snapshot tests	2022-10-27 11:34:13 +02:00
Kerollmops	50b8b9df6a	Delete the tasks content file once the transaction has been successfully committed	2022-10-27 11:34:13 +02:00
Kerollmops	ec0a5a9f01	Remove the useless r#union thing	2022-10-27 11:34:13 +02:00
Kerollmops	6460b78e08	Clean up the delete_persisted_task_data function	2022-10-27 11:34:13 +02:00
Kerollmops	d21651c968	Throw the error if we can't register the tasks in the store	2022-10-27 11:34:13 +02:00
Kerollmops	6e904d0997	Introduce a ProcessingTasks constructor	2022-10-27 11:34:12 +02:00
Kerollmops	b373d19831	Extract the must_stop flag out of the RwLock	2022-10-27 11:34:12 +02:00
Kerollmops	3cbfacb616	Prefer using an u64 instead of a usize in some places	2022-10-27 11:34:12 +02:00
Kerollmops	79c4275bfc	Delete the persisted data when we cancel a task	2022-10-27 11:34:12 +02:00
Kerollmops	c2ec4a089b	Put the original URL query in the tasks details	2022-10-27 11:34:12 +02:00
Kerollmops	751e9bac3b	Add the tasks cancel route to cancel tasks	2022-10-27 11:34:11 +02:00
Kerollmops	290945e258	Update the canceledBy and finishedAt fields	2022-10-27 11:34:11 +02:00
Kerollmops	725158b454	Introduce the core algorithm of task cancelation	2022-10-27 11:34:11 +02:00
Kerollmops	b2c5bc67b7	Add more enum-iterator related stuff	2022-10-27 11:34:11 +02:00
Kerollmops	591527a99d	Prefer using TaskDeletion in the dumps	2022-10-27 11:34:11 +02:00
Kerollmops	1ca9a67c49	Introduce the task cancelation task type	2022-10-27 11:34:11 +02:00
Kerollmops	703ba7a1fb	Introduce the ProcessingTasks struct	2022-10-27 11:34:10 +02:00
Loïc Lecrenier	ea60d35c71	Delete a task's persisted data when appropriate	2022-10-27 11:34:10 +02:00
Tamo	9e85f050b2	fix the tests	2022-10-27 11:34:09 +02:00
Tamo	2f748480a1	share the rtxn between the access to the tasks and to the indexes	2022-10-27 11:34:09 +02:00
Tamo	6bd6321226	dump the content of the dump tasks instead of recreating at import time with wrong API keys	2022-10-27 11:34:08 +02:00
Tamo	655705eb2b	remove useless todo	2022-10-27 11:34:08 +02:00
Tamo	83f3c5ec57	flush the dump-writer only once everything has been inserted	2022-10-27 11:34:08 +02:00
Tamo	78ce29f461	apply most style comments of the review	2022-10-27 11:34:08 +02:00
Tamo	d0e91555d1	rebase on index-scheduler	2022-10-27 11:34:08 +02:00
Tamo	e0221fc0a3	fix a synchronization bug while importing tasks	2022-10-27 11:34:07 +02:00
Tamo	ba150f2127	commit after creating an index	2022-10-27 11:34:07 +02:00
Tamo	e9295c03ce	the index-scheduler needs to wake-up after importing a dump	2022-10-27 11:34:06 +02:00
Tamo	d481669b7e	fix the content_file import	2022-10-27 11:34:06 +02:00
Tamo	dd506e5d87	stop dumping the current dumping task as enqueued so it's not looping for ever	2022-10-27 11:34:06 +02:00
Tamo	d976e680c5	first mostly working version	2022-10-27 11:34:06 +02:00
Tamo	c051166bcc	update the API a little bit	2022-10-27 11:34:05 +02:00
Tamo	72a906ae75	fix the tests	2022-10-27 11:34:05 +02:00
Tamo	b7f9c94f4a	write the dump export	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	8954b1bd1d	Fix number of deleted tasks details after duplicate task deletion	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	8defad6c38	Add task deletion tests where the same task is deleted twice	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	f32b973945	Return an error when calling DELETE /tasks with an empty query	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	fbd2be2ec8	Apply suggested changes from PR review	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	441417447e	Avoid creating two read txn at the same time	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	8c6aeaada5	Update snapshot tests following git rebase that fixes a bug	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	8bb0fcd144	Finish first draft of the DELETE /tasks route	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	9522b75454	Continue implementation of task deletion ... 1. Matched tasks are a roaring bitmap 2. Start implementation in meilisearch-http 3. Snapshots use meili-snap 4. Rename to TaskDeletion	2022-10-27 11:34:03 +02:00
Kerollmops	e4d461ecba	Make sure that we do not batch tasks from different indexes	2022-10-27 11:34:03 +02:00
Kerollmops	b029369653	Add a test to check different indexes autobatching	2022-10-27 11:34:03 +02:00
Kerollmops	408d00136c	Extract index creation rights and simplify the autobatcher rules	2022-10-27 11:34:03 +02:00
Kerollmops	2c24c7d403	Fix invalid import of tasks types	2022-10-27 11:34:03 +02:00
Tamo	7034803712	move the API key in meilisearch_types	2022-10-27 11:34:02 +02:00
Tamo	c192146fbe	remove an unused file	2022-10-27 11:34:02 +02:00
Tamo	2f1eb78b1d	refactor the Task a little bit	2022-10-27 11:34:02 +02:00
Tamo	510ce9fc51	start moving a lot of task types to meilisearch_types	2022-10-27 11:34:01 +02:00
Tamo	0af00f6b32	fix all the import and comment most of the dump v6	2022-10-27 11:34:01 +02:00
Tamo	667c282e19	get rids of the index crate + the document_types crate	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	9a74ea0943	Fix compiler errors related autobatching option of the index scheduler	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	eabac9676b	Fix typo and remove useless code in tests	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	ab4e649221	Apply suggestions from code review ... Co-authored-by: Tamo <tamo@meilisearch.com>	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	568199fc0d	Add more task deletion tests	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	13a72f8757	Use more complete snapshot tests for the index scheduler	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	4c55c30027	Add a DetailsView type and improve index scheduler snapshots ... The DetailsView type is necessary because serde incorrectly deserialises the `Details` type, so the database fails to correctly decode Tasks	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	dc81992eb2	Implement TaskDeletion in the index scheduler	2022-10-27 11:33:59 +02:00
Kerollmops	fe84f2648b	Allow a user to disable the auto batching system	2022-10-27 11:33:59 +02:00
Kerollmops	e2a766acb5	Add a test to check that it works without autobatching	2022-10-27 11:33:58 +02:00
Kerollmops	db9d1b18ca	Remove the IndexScheduler::notify method	2022-10-27 11:33:58 +02:00
Kerollmops	19c6f8303f	Make sure that the index-scheduler tick loop is rerun after processing	2022-10-27 11:33:58 +02:00
Kerollmops	b311eb3bed	Add a test that verifies that sending multiple tasks works	2022-10-27 11:33:58 +02:00
Tamo	0972587cfc	start writting the compat layer between v5 and v6	2022-10-27 11:33:52 +02:00
Tamo	e845cc2b6f	fix the tests	2022-10-27 11:33:51 +02:00
Clément Renault	123f47dbc4	Create the index only if the task has the rights to do so	2022-10-27 11:33:50 +02:00
Clément Renault	068a4b2884	Correctly batch tasks with different index creation rights	2022-10-27 11:33:50 +02:00
Clément Renault	87212cfd20	Use a ControlFlow in the autobatcher function	2022-10-27 11:33:50 +02:00
Kerollmops	f1b1cfdbcc	IndexDeletion operation have ClearAll details	2022-10-27 11:33:50 +02:00
Kerollmops	a083c9e452	Only mark the first clear document with the amount of cleared documents	2022-10-27 11:33:50 +02:00
Kerollmops	b24b13b036	Let the tick function set the Failed status itself	2022-10-27 11:33:50 +02:00
Kerollmops	566c15fb74	Fill an IndexDeletion task with the number of documents removed	2022-10-27 11:33:49 +02:00
Kerollmops	6b3b05fb73	Panic if we encountered a wring KindWithContent type	2022-10-27 11:33:49 +02:00
Kerollmops	36e5efde0d	Update the tasks statuses	2022-10-27 11:33:49 +02:00
Kerollmops	2fbdd104b8	Implement the IndexDeletion batch operation	2022-10-27 11:33:49 +02:00
Kerollmops	da363a92ac	Implement the IndexUpdate batch operation	2022-10-27 11:33:49 +02:00
Kerollmops	0543cba6eb	Implement the IndexCreate batch operation	2022-10-27 11:33:48 +02:00
Kerollmops	cf6084151b	Make sure that meilisearch-http works without index wrapper	2022-10-27 11:33:48 +02:00
Kerollmops	d76634a36c	Remove the Index wrapper and use milli::Index directly	2022-10-27 11:33:48 +02:00
Kerollmops	9e8242c57d	Remove the IndexRename operation	2022-10-27 11:33:48 +02:00
Kerollmops	5fa214abb1	Move the IndexScheduler to the root of the index-scheduler crate	2022-10-27 11:33:47 +02:00
Kerollmops	9a9e98fb77	Add a TODO about the index creation	2022-10-27 11:33:47 +02:00
Kerollmops	5d21c790ef	Make clippy happy	2022-10-27 11:33:47 +02:00
Kerollmops	31de33d5ee	Implement a recursive indexation for the index-related operations	2022-10-27 11:33:47 +02:00
Kerollmops	07286fcc79	Implement the SettingsAndDocumentImport batch operation	2022-10-27 11:33:47 +02:00
Kerollmops	f68906f5dc	Merge both DocumentAddition/Update into one DocumentImport variant	2022-10-27 11:33:46 +02:00
Kerollmops	5174c78f87	Implement the DocumentClear batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	025bb5f616	Implement the DocumentClearAndSettings batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	41ec737e73	Implement the Settings batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	7b4a913704	Implement the DocumentUpdate batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	a6a1043abb	Implement the DocumentDeletion batch operation	2022-10-27 11:33:46 +02:00
Tamo	7a0f17c912	remove an old unworking part of the batch execution	2022-10-27 11:33:45 +02:00
Tamo	c2899fe9b2	bring back the IndexMeta and IndexStats in meilisearch-http	2022-10-27 11:33:45 +02:00
Tamo	2c8f1a43e9	get rids of meilisearch-lib	2022-10-27 11:33:44 +02:00
Tamo	0ba1c46e19	fix a deadlock	2022-10-27 11:33:44 +02:00
Tamo	22bfb5a7a0	remove Clone from the IndexScheduler	2022-10-27 11:33:44 +02:00
Tamo	d8d3499aec	remove a bunch of comments	2022-10-27 11:33:44 +02:00
Tamo	64e132ce53	move as many fields as possible out of the IndexScheduler	2022-10-27 11:33:44 +02:00
Tamo	9e1f38ec7c	move the test function in the test module	2022-10-27 11:33:44 +02:00
Tamo	6f4dcc0c38	start implementing some logic to test the internal states of the scheduler	2022-10-27 11:33:43 +02:00
Tamo	84cd5cef0b	fix the tests	2022-10-27 11:33:43 +02:00
Tamo	ae86a8ccd6	slightly refactor the autobatching tests	2022-10-27 11:33:43 +02:00
Tamo	ce2dfecc03	connect the new scheduler to meilisearch-http officially. ... I can index documents and do search	2022-10-27 11:33:43 +02:00
Tamo	cb4feabca2	implements the get_tasks	2022-10-27 11:33:43 +02:00
Tamo	19154e48fe	fix all compilation errors	2022-10-27 11:33:42 +02:00
Irevoire	8d51c1f389	wip integrating the scheduler in meilisearch-http	2022-10-27 11:33:42 +02:00
Irevoire	250410495c	start integrating the index-scheduler in meilisearch-lib	2022-10-27 11:33:42 +02:00
Irevoire	8f0fd35358	add insta::json for later	2022-10-27 11:33:42 +02:00
Irevoire	8770e07397	I can index documents without meilisearch	2022-10-27 11:33:42 +02:00
Tamo	edd8344dc9	wip	2022-10-27 11:33:42 +02:00
Tamo	e547552702	create the end Batch type for all Index* operations	2022-10-27 11:33:41 +02:00
Tamo	925971809a	create the end Batch type for all Document* operation	2022-10-27 11:33:41 +02:00
Tamo	1ea9c0b4c0	write most of the run loop	2022-10-27 11:33:41 +02:00
Tamo	9ff0fe952e	split the run function in two	2022-10-27 11:33:41 +02:00
Tamo	a8b18b2c96	fix the register test	2022-10-27 11:33:40 +02:00
Tamo	5436b996ab	reduce the size of the snapshots	2022-10-27 11:33:40 +02:00
Tamo	7d0c8a3379	test the register tasks	2022-10-27 11:33:40 +02:00
Tamo	fc098022c7	start integrating the index-scheduler in the meilisearch codebase	2022-10-27 11:33:40 +02:00
Tamo	b816535e33	greatly reduce the number of warnings	2022-10-27 11:33:40 +02:00
Tamo	38e4ffe73c	fix smol typo	2022-10-27 11:33:40 +02:00
Tamo	366a344474	get rids of the horrendous spinlock in favor of synchronoise	2022-10-27 11:33:39 +02:00
Tamo	7b6673dc1d	implement the index swap in the index mapper	2022-10-27 11:33:39 +02:00
Tamo	03aca2e452	move the index mapping logic in another structure	2022-10-27 11:33:39 +02:00
Tamo	4129783019	migrate the index handling code in a different file + implements the create index	2022-10-27 11:33:39 +02:00
Tamo	1804416afa	reintroduce the uuid mapping for the indexes	2022-10-27 11:33:39 +02:00
Tamo	c97d51a624	add a bunch of tests	2022-10-27 11:33:39 +02:00
Tamo	803f2157af	split the DocumentAdditionOrUpdate in two tasks; DocumentAddition and DocumentUpdate	2022-10-27 11:33:38 +02:00
Tamo	b7c5b71a53	starts importing the real tasks	2022-10-27 11:33:38 +02:00
Tamo	5cc8f96237	get rids of the auto-generated mains	2022-10-27 11:33:38 +02:00
Tamo	94e29a9f5f	extract the index abstraction out of the index-scheduler in its own module	2022-10-27 11:33:38 +02:00
Tamo	48138c21a9	rename the update-file-store to file-store since it can store any kind of file	2022-10-27 11:33:38 +02:00
Tamo	76597fc382	import the update_file_store in the index-scheduler	2022-10-27 11:33:37 +02:00
Tamo	a0588d6b94	finishes the global skelton of the auto-batcher	2022-10-27 11:33:37 +02:00
Tamo	b3c9b128d9	polish the global structure of the batch creation	2022-10-27 11:33:37 +02:00
Irevoire	448f44f631	move the autobatcher logic to another file	2022-10-27 11:33:36 +02:00
Tamo	f638774764	add the document format file	2022-10-27 11:33:36 +02:00
Tamo	516860f342	fix the create_new_batch method	2022-10-27 11:33:36 +02:00
Tamo	6b9689a1c0	fix the whole batchKind thingy	2022-10-27 11:33:36 +02:00
Tamo	af0f5d6c0c	implements most operations	2022-10-27 11:33:36 +02:00
Tamo	5a7fcf2688	fix a few typos	2022-10-27 11:33:35 +02:00
Tamo	30d2b24689	implements the index deletion, creation and swap	2022-10-27 11:33:35 +02:00
Tamo	72b2e68de4	makes the updates getters smoother to uses	2022-10-27 11:33:35 +02:00
Tamo	7879189c6b	make the project compile again	2022-10-27 11:33:35 +02:00
Tamo	46b8ebcab4	fix the file store	2022-10-27 11:33:35 +02:00
Tamo	fa742f60e8	make the file store entirely synchronous, including the file deletion	2022-10-27 11:33:35 +02:00
Tamo	a7aa92df5f	fix most of the index module	2022-10-27 11:33:34 +02:00
Irevoire	d8b8e04ad1	wip porting the index back in the scheduler	2022-10-27 11:33:34 +02:00
Irevoire	fe330e1be9	add a little bit of documentation	2022-10-27 11:33:34 +02:00
Tamo	2c4e5ce8be	implements the filter query	2022-10-27 11:33:34 +02:00
Tamo	705af94fd7	add the task to the index db in the register task	2022-10-27 11:33:34 +02:00
Tamo	ed745591e1	split the scheduler into multiples files	2022-10-27 11:33:34 +02:00
Tamo	22d24dba56	implement the get_batch method	2022-10-27 11:33:33 +02:00
Tamo	1a47949063	START THE REWRITE OF THE INDEX SCHEDULER: index & register has been implemented	2022-10-27 11:33:33 +02:00