273 Commits

Author	SHA1	Message	Date
Irevoire	a3fc0d3bd9	Fix the last regression	2022-10-27 11:35:17 +02:00
Kerollmops	2f577b6fcd	Patch the IndexScheduler in meilisearch-http to use the options struct	2022-10-27 11:35:16 +02:00
Kerollmops	71b50853dc	Introduce an options struct to create the IndexScheduler	2022-10-27 11:35:16 +02:00
Kerollmops	e35fe33712	Fix some bugs with files	2022-10-27 11:35:15 +02:00
Kerollmops	942b7c338b	Compress the snapshot in a tarball	2022-10-27 11:35:15 +02:00
Kerollmops	4cafc63561	Reintroduce the versioning functions	2022-10-27 11:35:14 +02:00
Kerollmops	89e127e4f4	Declare the auth path in the index scheduler	2022-10-27 11:35:14 +02:00
Kerollmops	eec43ec953	Implement a first version of the snapshots	2022-10-27 11:35:14 +02:00
Kerollmops	c063f154fb	Add the snapshots directory path to the IndexScheduler	2022-10-27 11:35:14 +02:00
Kerollmops	e0548e42e7	Rename the Snapshot task into SnapshotCreation	2022-10-27 11:35:14 +02:00
Kerollmops	4d43a9f5b1	Rename the index-scheduler module into insta_snapshot	2022-10-27 11:35:14 +02:00
Loïc Lecrenier	6db90ba6cc	Make sure that we don't delete or cancel future tasks ... This should already have been the case before, but there is no harm in adding another check.	2022-10-27 11:35:13 +02:00
Irevoire	61f0940f8c	fix an issue with the dates	2022-10-27 11:35:13 +02:00
Irevoire	241300d2d8	add more naive tests around the document addition + remove the old unused snapshot files	2022-10-27 11:35:13 +02:00
Irevoire	570b2d1167	add some naive document addition tests	2022-10-27 11:35:12 +02:00
Loïc Lecrenier	d92425658e	Add index scheduler tests for task cancelation	2022-10-27 11:35:12 +02:00
Loïc Lecrenier	16fac10074	Fix crash when batching an index swap task containing 0 swaps	2022-10-27 11:35:12 +02:00
Irevoire	7ed3f00b1e	reformat	2022-10-27 11:35:12 +02:00
Irevoire	7e52f1effb	remove a lot of unecessary clone and ref	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	4d25c159e6	Apply code review suggestions	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	424202d773	Pause the index scheduler for one second when a fatal error occurs	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	4a35eb9849	Fix (hopefully) queries that include processing tasks	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	493a8cff31	Adjust task details correctly following index swap	2022-10-27 11:35:10 +02:00
Loïc Lecrenier	4de445d386	Start testing unexpected errors and panics in index scheduler	2022-10-27 11:35:10 +02:00
Loïc Lecrenier	e3848b5f28	Add assert method to verify validity of index scheduler state	2022-10-27 11:35:10 +02:00
Irevoire	ecf4e43b3d	rename the dumpExport to dumpCreation	2022-10-27 11:35:10 +02:00
Irevoire	8a23e707c1	fix the task view and forward the task db size	2022-10-27 11:35:09 +02:00
Irevoire	1d04ce611d	remove ununsed function	2022-10-27 11:35:08 +02:00
Irevoire	e9055f5572	fix clippy	2022-10-27 11:35:08 +02:00
Irevoire	8d1408c65e	fix the import of the dumpv4&v5 when there is no instance-uid + rename the Kind+KindWithContent+Details variant for the DocumentImport and the Setting	2022-10-27 11:35:05 +02:00
Clément Renault	80b2e70ee7	Introduce a rustfmt file	2022-10-27 11:35:05 +02:00
Irevoire	b6a0abea9f	fix the index deletion when the index doesn’t exists but would be created by one of the autobatched tasks	2022-10-27 11:34:16 +02:00
Loïc Lecrenier	11fee30f47	Apply review suggestions and stop using rtxn.commit	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	17cd2a4aa0	Implement POST /indexes-swap	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	169f386418	Add some documentation to the index scheduler	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	ec3391808d	Fix date parsing for task queries ... Use rfc3339 or YYYY-MM-DD. Add a day to the parsed date when it is an excluded lower bound and the YYYY-MM-DD was used. Also the Query type does not need to be serialisable anymore	2022-10-27 11:34:14 +02:00
Loïc Lecrenier	22cf0559fe	Implement task date filters ... before/after enqueued/started/finished at	2022-10-27 11:34:14 +02:00
Irevoire	5765883600	fix the auto-generated details	2022-10-27 11:34:14 +02:00
Kerollmops	6460b78e08	Clean up the delete_persisted_task_data function	2022-10-27 11:34:13 +02:00
Kerollmops	d21651c968	Throw the error if we can't register the tasks in the store	2022-10-27 11:34:13 +02:00
Kerollmops	6e904d0997	Introduce a ProcessingTasks constructor	2022-10-27 11:34:12 +02:00
Kerollmops	b373d19831	Extract the must_stop flag out of the RwLock	2022-10-27 11:34:12 +02:00
Kerollmops	751e9bac3b	Add the tasks cancel route to cancel tasks	2022-10-27 11:34:11 +02:00
Kerollmops	290945e258	Update the canceledBy and finishedAt fields	2022-10-27 11:34:11 +02:00
Kerollmops	b2c5bc67b7	Add more enum-iterator related stuff	2022-10-27 11:34:11 +02:00
Kerollmops	591527a99d	Prefer using TaskDeletion in the dumps	2022-10-27 11:34:11 +02:00
Kerollmops	1ca9a67c49	Introduce the task cancelation task type	2022-10-27 11:34:11 +02:00
Kerollmops	703ba7a1fb	Introduce the ProcessingTasks struct	2022-10-27 11:34:10 +02:00
Loïc Lecrenier	ea60d35c71	Delete a task's persisted data when appropriate	2022-10-27 11:34:10 +02:00
Tamo	9e85f050b2	fix the tests	2022-10-27 11:34:09 +02:00
Tamo	6bd6321226	dump the content of the dump tasks instead of recreating at import time with wrong API keys	2022-10-27 11:34:08 +02:00
Tamo	655705eb2b	remove useless todo	2022-10-27 11:34:08 +02:00
Tamo	78ce29f461	apply most style comments of the review	2022-10-27 11:34:08 +02:00
Tamo	d0e91555d1	rebase on index-scheduler	2022-10-27 11:34:08 +02:00
Tamo	e0221fc0a3	fix a synchronization bug while importing tasks	2022-10-27 11:34:07 +02:00
Tamo	ba150f2127	commit after creating an index	2022-10-27 11:34:07 +02:00
Tamo	e9295c03ce	the index-scheduler needs to wake-up after importing a dump	2022-10-27 11:34:06 +02:00
Tamo	d481669b7e	fix the content_file import	2022-10-27 11:34:06 +02:00
Tamo	d976e680c5	first mostly working version	2022-10-27 11:34:06 +02:00
Tamo	c051166bcc	update the API a little bit	2022-10-27 11:34:05 +02:00
Tamo	b7f9c94f4a	write the dump export	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	8954b1bd1d	Fix number of deleted tasks details after duplicate task deletion	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	8defad6c38	Add task deletion tests where the same task is deleted twice	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	f32b973945	Return an error when calling DELETE /tasks with an empty query	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	441417447e	Avoid creating two read txn at the same time	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	8c6aeaada5	Update snapshot tests following git rebase that fixes a bug	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	8bb0fcd144	Finish first draft of the DELETE /tasks route	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	9522b75454	Continue implementation of task deletion ... 1. Matched tasks are a roaring bitmap 2. Start implementation in meilisearch-http 3. Snapshots use meili-snap 4. Rename to TaskDeletion	2022-10-27 11:34:03 +02:00
Kerollmops	e4d461ecba	Make sure that we do not batch tasks from different indexes	2022-10-27 11:34:03 +02:00
Kerollmops	b029369653	Add a test to check different indexes autobatching	2022-10-27 11:34:03 +02:00
Tamo	c192146fbe	remove an unused file	2022-10-27 11:34:02 +02:00
Tamo	2f1eb78b1d	refactor the Task a little bit	2022-10-27 11:34:02 +02:00
Tamo	510ce9fc51	start moving a lot of task types to meilisearch_types	2022-10-27 11:34:01 +02:00
Tamo	667c282e19	get rids of the index crate + the document_types crate	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	9a74ea0943	Fix compiler errors related autobatching option of the index scheduler	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	eabac9676b	Fix typo and remove useless code in tests	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	568199fc0d	Add more task deletion tests	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	13a72f8757	Use more complete snapshot tests for the index scheduler	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	4c55c30027	Add a DetailsView type and improve index scheduler snapshots ... The DetailsView type is necessary because serde incorrectly deserialises the `Details` type, so the database fails to correctly decode Tasks	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	dc81992eb2	Implement TaskDeletion in the index scheduler	2022-10-27 11:33:59 +02:00
Kerollmops	e2a766acb5	Add a test to check that it works without autobatching	2022-10-27 11:33:58 +02:00
Kerollmops	db9d1b18ca	Remove the IndexScheduler::notify method	2022-10-27 11:33:58 +02:00
Kerollmops	19c6f8303f	Make sure that the index-scheduler tick loop is rerun after processing	2022-10-27 11:33:58 +02:00
Kerollmops	b311eb3bed	Add a test that verifies that sending multiple tasks works	2022-10-27 11:33:58 +02:00
Tamo	0972587cfc	start writting the compat layer between v5 and v6	2022-10-27 11:33:52 +02:00
Clément Renault	068a4b2884	Correctly batch tasks with different index creation rights	2022-10-27 11:33:50 +02:00
Kerollmops	b24b13b036	Let the tick function set the Failed status itself	2022-10-27 11:33:50 +02:00
Kerollmops	36e5efde0d	Update the tasks statuses	2022-10-27 11:33:49 +02:00
Kerollmops	cf6084151b	Make sure that meilisearch-http works without index wrapper	2022-10-27 11:33:48 +02:00
Kerollmops	d76634a36c	Remove the Index wrapper and use milli::Index directly	2022-10-27 11:33:48 +02:00
Kerollmops	5fa214abb1	Move the IndexScheduler to the root of the index-scheduler crate	2022-10-27 11:33:47 +02:00
Tamo	2c8f1a43e9	get rids of meilisearch-lib	2022-10-27 11:33:44 +02:00
Tamo	6f4dcc0c38	start implementing some logic to test the internal states of the scheduler	2022-10-27 11:33:43 +02:00
Irevoire	8d51c1f389	wip integrating the scheduler in meilisearch-http	2022-10-27 11:33:42 +02:00
Irevoire	250410495c	start integrating the index-scheduler in meilisearch-lib	2022-10-27 11:33:42 +02:00
Irevoire	8770e07397	I can index documents without meilisearch	2022-10-27 11:33:42 +02:00
Tamo	7d0c8a3379	test the register tasks	2022-10-27 11:33:40 +02:00
Tamo	fc098022c7	start integrating the index-scheduler in the meilisearch codebase	2022-10-27 11:33:40 +02:00
Tamo	b816535e33	greatly reduce the number of warnings	2022-10-27 11:33:40 +02:00
Tamo	366a344474	get rids of the horrendous spinlock in favor of synchronoise	2022-10-27 11:33:39 +02:00
Tamo	03aca2e452	move the index mapping logic in another structure	2022-10-27 11:33:39 +02:00
Tamo	4129783019	migrate the index handling code in a different file + implements the create index	2022-10-27 11:33:39 +02:00
Tamo	1804416afa	reintroduce the uuid mapping for the indexes	2022-10-27 11:33:39 +02:00
Tamo	94e29a9f5f	extract the index abstraction out of the index-scheduler in its own module	2022-10-27 11:33:38 +02:00
Tamo	48138c21a9	rename the update-file-store to file-store since it can store any kind of file	2022-10-27 11:33:38 +02:00
Tamo	76597fc382	import the update_file_store in the index-scheduler	2022-10-27 11:33:37 +02:00
Tamo	b3c9b128d9	polish the global structure of the batch creation	2022-10-27 11:33:37 +02:00
Irevoire	448f44f631	move the autobatcher logic to another file	2022-10-27 11:33:36 +02:00
Tamo	6b9689a1c0	fix the whole batchKind thingy	2022-10-27 11:33:36 +02:00
Tamo	af0f5d6c0c	implements most operations	2022-10-27 11:33:36 +02:00
Tamo	5a7fcf2688	fix a few typos	2022-10-27 11:33:35 +02:00
Tamo	30d2b24689	implements the index deletion, creation and swap	2022-10-27 11:33:35 +02:00
Tamo	72b2e68de4	makes the updates getters smoother to uses	2022-10-27 11:33:35 +02:00
Tamo	7879189c6b	make the project compile again	2022-10-27 11:33:35 +02:00
Tamo	46b8ebcab4	fix the file store	2022-10-27 11:33:35 +02:00
Tamo	a7aa92df5f	fix most of the index module	2022-10-27 11:33:34 +02:00
Irevoire	d8b8e04ad1	wip porting the index back in the scheduler	2022-10-27 11:33:34 +02:00
Irevoire	fe330e1be9	add a little bit of documentation	2022-10-27 11:33:34 +02:00
Tamo	2c4e5ce8be	implements the filter query	2022-10-27 11:33:34 +02:00
Tamo	705af94fd7	add the task to the index db in the register task	2022-10-27 11:33:34 +02:00
Tamo	ed745591e1	split the scheduler into multiples files	2022-10-27 11:33:34 +02:00
Tamo	22d24dba56	implement the get_batch method	2022-10-27 11:33:33 +02:00
Tamo	1a47949063	START THE REWRITE OF THE INDEX SCHEDULER: index & register has been implemented	2022-10-27 11:33:33 +02:00