Louis Dureuil
e1aa534389
Wait 10 seconds in case of irrecoverable error
2025-04-24 16:06:19 +02:00
DanasFi
fd0623c085
Fix typo in function to get size until task queue stops
2025-04-17 10:48:56 +02:00
DanasFi
eeb33b913c
Corrected metric for task queue total size
2025-04-17 10:46:26 +02:00
DanasFi
3d93efc6aa
Added metric to check task queue size until stop
2025-04-17 10:46:25 +02:00
Tamo
b025f1bcf1
Merge branch 'main' into release-v1.14.0-tmp
2025-04-14 12:35:47 +02:00
Tamo
796a325972
Fix typos
...
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-04-03 15:53:42 +02:00
Tamo
1db550ec7f
make meilisearch accept cancelation tasks even when the disk is full
2025-04-03 15:47:56 +02:00
Clément Renault
4d90e3d2ec
Make Cargo and Clippy happy
2025-04-01 11:26:34 +02:00
Louis Dureuil
e2d372823a
Disable the cache by default and make it experimental
2025-03-13 17:22:51 +01:00
Kerollmops
5e6abcf50c
Prefer using WithoutTls for the auth env
2025-03-13 11:29:00 +01:00
Kerollmops
55ca2c4481
Avoid opening the Auth environment multiple times
2025-03-13 11:07:49 +01:00
Kerollmops
1d499ed9b2
Use the new arroy upgrade method to move from 0.4 to 0.5
2025-03-13 11:07:44 +01:00
Kerollmops
3bc62f0549
WIP: Still need to introduce a Env::copy_to_path method
2025-03-13 11:07:39 +01:00
Tamo
628119e31e
fix the dumpless upgrade potential corruption when upgrading from the v1.12
2025-02-05 18:08:50 +01:00
Louis Dureuil
4918b9ffb6
Network stored in DB
2025-02-05 15:03:15 +01:00
Tamo
b9e9fc376a
add the version in the index-scheduler
2025-01-23 16:51:22 +01:00
Tamo
e41ebd3047
expose the number of database in the index-scheduler and rewrite the lib.rs to use the value provided in the options instead of a magic number
2025-01-23 16:51:21 +01:00
Tamo
3ef7a478cd
move the version check to the task queue
2025-01-23 16:48:32 +01:00
Tamo
d3654906bf
Add the new tasks with most of the job done
2025-01-23 16:48:32 +01:00
meili-bors[bot]
42854c0bca
Merge #5223
...
5223: Limit batched tasks total size r=curquiza a=Kerollmops
Introduce a new engine parameter (env and config, too) to limit the maximum payload size processed by the engine in batches. You can [review the Discussion and usage on GitHub](https://github.com/orgs/meilisearch/discussions/801 ).
Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-01-09 16:13:17 +00:00
Clément Renault
d0bdff7b7b
Make the batched tasks size limit effectively work
2025-01-09 12:06:28 +01:00
Clément Renault
71e5605daa
Make clippy happy
2025-01-08 18:24:39 +01:00
Tamo
cb82b0798a
Split the index-scheduler in ~500 loc modules
2025-01-06 14:08:26 +01:00
Tamo
4eae92f411
fix list indexes
2024-12-26 18:48:25 +01:00
Tamo
d12364c1e0
fix the tests
2024-12-11 18:30:48 +01:00
Tamo
ab75f53efd
update all snapshots
2024-12-11 16:25:02 +01:00
Tamo
df9b68f8ed
inital implementation of the progress
2024-12-11 16:25:01 +01:00
airycanon
b75f1f4c17
fix tests
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/index-scheduler/src/snapshots/lib.rs/fail_in_process_batch_for_document_deletion/after_removing_the_documents.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fifth_task_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fourth_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/third_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key_batch_wrong_key/second_and_third_tasks_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/all_other_tasks_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/third_task_succeeds.snap
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
# crates/meilisearch/tests/vector/mod.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
2024-12-06 02:03:02 +08:00
airycanon
95ed079761
attach index name in errors
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
2024-12-06 01:12:13 +08:00
meili-bors[bot]
26be5e0733
Merge #5123
...
5123: Fix batch details r=dureuill a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5079
Fixes https://github.com/meilisearch/meilisearch/issues/5112
## What does this PR do?
- Make the processing tasks actually processing in the stats of the batch instead of enqueued
- Stop counting one extra task for all non-prioritized batches in the stats
- Add a test
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-05 15:21:55 +00:00
Tamo
cb0c3a5aad
stop adding one enqueued tasks to all unprioritized batches
2024-12-04 15:48:28 +01:00
Tamo
cbcf6c9ba3
make the processing tasks as processing in a batch
2024-12-04 14:48:48 +01:00
Tamo
bf742d81cf
add a test
2024-12-04 14:47:02 +01:00
Tamo
71d53f413f
increase the margin allowed to delete task
2024-12-03 11:07:03 +01:00
meili-bors[bot]
054622bd16
Merge #5094
...
5094: Implement a bbqueue channel between the extractors and the writer r=dureuill a=Kerollmops
This PR switches from a bounded crossbeam channel only with allocated entries for the communication between the extractors and the writer to a [BBQueue](https://github.com/jamesmunns/bbqueue )-based system with a Single Producer Single Consumer kind of Circular/Ring Buffers channel.
- [x] Implement the BBQueue channel system...
- [x] with a crossbeam channel to wake up the receiver.
- [x] Manage the BBQueue allocated memory dynamically.
- [x] Support content that doesn't fit in the bbqueues.
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-12-03 08:00:55 +00:00
Tamo
beeb31ce41
Update crates/index-scheduler/src/lib.rs
2024-12-02 15:32:16 +01:00
Tamo
a439fa3e1a
While spamming the batches route we could see a processing batch becoming missing and then finished, this commit ensures the batches goes from processing to finished directly
2024-12-02 12:02:16 +01:00
Clément Renault
e1e76f39d0
Clean up dependencies
2024-11-27 14:30:34 +01:00
ManyTheFish
36962b943b
First batch of PR comment
2024-11-21 16:38:11 +01:00
Louis Dureuil
1f9692cd04
Increase map size for tests
2024-11-20 17:52:21 +01:00
Louis Dureuil
6e6acfcf1b
Merge branch 'main' into indexer-edition-2024
2024-11-20 16:59:58 +01:00
Tamo
83d1f858c1
Update crates/index-scheduler/src/lib.rs
...
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-11-20 14:36:05 +01:00
Tamo
a7ac590e9e
implements the reverse query parameter for the batches
2024-11-20 13:29:52 +01:00
Tamo
8ad68dd708
stop leaking the update files of the canceled tasks
2024-11-20 13:17:54 +01:00
Tamo
7e379b3d14
remove useless prints
2024-11-20 12:27:12 +01:00
Tamo
e145d71a62
implements the two last TODOs
2024-11-20 10:51:06 +01:00
Tamo
b906e3ed70
improve the way we access the mutex
2024-11-20 10:51:06 +01:00
Tamo
4abcd9c04e
add some stats on the batches
2024-11-20 10:51:06 +01:00
Tamo
1fcb9526f5
fix the task cancelation
2024-11-20 10:51:06 +01:00
Tamo
15eefa4fcc
fixes a lot of small issue, the test about the cancellation is still failing
2024-11-20 10:51:05 +01:00
