8440 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Tamo	e4adf7fd6d	fix the path of the deps	2023-11-07 16:17:20 +01:00
Tamo	1585b3ed2f	re-implement the snapshot import at startup	2023-11-07 14:15:57 +01:00
Tamo	c48f72e6b8	fix a few warnings	2023-11-06 16:32:49 +01:00
Tamo	41178e59fc	fix the leader election	2023-11-06 15:00:14 +01:00
Tamo	d1bc7ec58a	fix all the bugs on the snapshots export/import pipeline	2023-11-06 12:34:48 +01:00
Tamo	c5ec817f52	WIP fix a lot of bugs: The follower do not wake up when a new snapshot is available	2023-11-02 18:21:17 +01:00
Tamo	d0a3582a56	no deadlock on start	2023-11-02 16:45:33 +01:00
Tamo	0c18962b13	it compiles, now tokio is complaining that we block the main thread (and is right), I need to patch the zookeeper client to not rely on tokio anymore	2023-11-02 12:55:16 +01:00
Tamo	366144146b	WIT: it compiles but the processing of tasks and loading of snapshots is still not implemented	2023-11-02 10:47:54 +01:00
Tamo	03b510945b	ICE: use a git version of my crate so rust team can pull the repo	2023-10-31 17:59:59 +01:00
Tamo	c3a8d4b7fb	ICE	2023-10-31 17:39:10 +01:00
Tamo	c573261ac4	WIP: start updating the zookeeper client => leader election is missing	2023-10-31 13:04:32 +01:00
Clément Renault	8f04353b7d	bump strois to a temporary version while we wait for rusty_s3 to do a new release again	2023-10-17 11:49:18 +02:00
Clément Renault	b9983a48c7	No more create the S3 Bucket	2023-10-17 10:26:03 +02:00
Tamo	b22f1260bf	bump strois to a temporary version while we wait for rusty_s3 to do a new release	2023-10-13 11:04:12 +02:00
Tamo	dfb84f80da	bump strois version	2023-10-10 19:25:12 +02:00
Tamo	98b67f217a	move to our new S3 lib	2023-09-28 11:24:18 +02:00
Tamo	6325cda74f	bump charabia	2023-09-21 15:18:44 +02:00
Tamo	c71ba72f73	fix build in release mode	2023-09-21 11:05:57 +02:00
Tamo	ecd36b15f0	exposes all the s3 arguments	2023-09-13 18:17:56 +02:00
Clément Renault	8a2e8a887f	Load the latest snapshot when we start the engine	2023-09-12 18:08:24 +02:00
Clément Renault	309c33a418	Fix again the dots	2023-09-12 17:55:01 +02:00
Clément Renault	9b01506cee	Move the load snapshot step into a function	2023-09-12 16:05:02 +02:00
Clément Renault	f37fdceb15	Use slashes instead of dots for the s3 paths separators	2023-09-12 15:46:15 +02:00
Clément Renault	f544cfa444	Remove tasks and content file on the s3	2023-09-12 15:19:45 +02:00
Clément Renault	c158d03337	Fix internal error	2023-09-12 14:46:13 +02:00
Tamo	b7109c0fd2	start a script to run everything	2023-09-12 11:34:59 +02:00
Kerollmops	a53a0fdb77	Store content files into the S3	2023-09-11 18:17:22 +02:00
Clément Renault	719fdd701b	Fix and crash when the tasks path is unknown	2023-09-07 11:31:18 +02:00
Kerollmops	01c13c98ac	Mastering minio	2023-09-06 17:54:21 +02:00
Tamo	5b89276fcc	starts using s3	2023-09-05 19:25:09 +02:00
Kerollmops	41697c4d65	Introduce the zk-tasks folder	2023-09-04 18:24:34 +02:00
Kerollmops	7d85753573	Make the snapshot download work	2023-09-04 17:38:56 +02:00
Kerollmops	76657af1f9	Add the options into the IndexScheduler	2023-09-04 16:38:05 +02:00
Tamo	966cbdab69	make the tests compile again	2023-09-04 15:39:54 +02:00
Clément Renault	0c68b9ed4c	WIP making the final snapshot swap	2023-08-31 15:56:42 +02:00
Clément Renault	d7233ecdb8	Make things to compile again	2023-08-31 14:55:14 +02:00
Clément Renault	95a011af13	Wrap the IndexScheduler fields into an inner struct	2023-08-31 10:36:33 +02:00
Clément Renault	e257710961	WIP fix the tests	2023-08-30 18:03:24 +02:00
Clément Renault	9dd4423054	Fix the watcher ordering of the auth/ node	2023-08-30 17:51:22 +02:00
Clément Renault	8c3ad57ef9	React to changes towards the cluster members	2023-08-30 17:40:12 +02:00
Clément Renault	2d1434da81	Keep the ZK flow when enqueuing tasks	2023-08-30 17:15:15 +02:00
Clément Renault	c488a4a351	Fixup a lot of small issues on the ZK config	2023-08-30 16:42:55 +02:00
Kerollmops	0c7d7c68bc	WIP moving to the sync zookeeper API	2023-08-30 15:06:12 +02:00
Tamo	854745c670	wip: starts working on importing the snapshots	2023-08-16 18:41:05 +02:00
Tamo	777eebb759	starts creating snapshot, the import is still missing	2023-08-10 15:00:25 +02:00
Tamo	61ccfaf9bc	wake up after registering a task	2023-08-10 09:39:39 +02:00
Tamo	f0c4d36ff7	implement the deletion of tasks after processing a batch ... add a lot of comments and logs	2023-08-10 09:36:43 +02:00
Tamo	8c20d6e2fe	fix the leader election	2023-08-09 17:23:13 +02:00
ManyTheFish	8e437ed76c	Start leader election and task processing (WIP)	2023-08-09 16:52:38 +02:00