Tamo
33921747b7
stop skipping empty tasks when adding documents
2024-12-30 17:48:25 +01:00
Tamo
ad4dc70720
rename the ComputingTheChanges to ComputingDocumentChanges in the edit document progress
2024-12-11 18:09:54 +01:00
Tamo
5d682b4700
rename the ComputingTheChanges to ComputingDocumentChanges
2024-12-11 18:08:45 +01:00
Tamo
85577e70cd
reuse the enqueued
2024-12-11 18:05:34 +01:00
Tamo
786b0fabea
implement the progress for almost all the tasks
2024-12-11 16:26:08 +01:00
Tamo
26733c705d
add progress for the task deletion and task cancelation
2024-12-11 16:25:02 +01:00
Tamo
df9b68f8ed
inital implementation of the progress
2024-12-11 16:25:01 +01:00
Kerollmops
bb00e70087
Reintroduce the document addition logs
2024-12-11 10:39:04 +01:00
airycanon
b75f1f4c17
fix tests
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/index-scheduler/src/snapshots/lib.rs/fail_in_process_batch_for_document_deletion/after_removing_the_documents.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fifth_task_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fourth_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/third_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key_batch_wrong_key/second_and_third_tasks_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/all_other_tasks_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/third_task_succeeds.snap
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
# crates/meilisearch/tests/vector/mod.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
2024-12-06 02:03:02 +08:00
Tamo
95975944d7
fix the dumps missing the empty swap index tasks
2024-12-05 14:23:38 +01:00
Tamo
cb0c3a5aad
stop adding one enqueued tasks to all unprioritized batches
2024-12-04 15:48:28 +01:00
meili-bors[bot]
054622bd16
Merge #5094
...
5094: Implement a bbqueue channel between the extractors and the writer r=dureuill a=Kerollmops
This PR switches from a bounded crossbeam channel only with allocated entries for the communication between the extractors and the writer to a [BBQueue](https://github.com/jamesmunns/bbqueue )-based system with a Single Producer Single Consumer kind of Circular/Ring Buffers channel.
- [x] Implement the BBQueue channel system...
- [x] with a crossbeam channel to wake up the receiver.
- [x] Manage the BBQueue allocated memory dynamically.
- [x] Support content that doesn't fit in the bbqueues.
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-12-03 08:00:55 +00:00
Louis Dureuil
057143214d
Fix warnings
2024-12-02 14:42:31 +01:00
Louis Dureuil
d78f4666a0
Fix autobatching of documents and settings
2024-12-02 12:25:01 +01:00
Clément Renault
98d4a2909e
Fix the way we spawn the rayon threadpool
2024-11-27 16:05:44 +01:00
Louis Dureuil
fa15be5bc4
Add span around commit
2024-11-26 09:45:48 +01:00
Louis Dureuil
221e547e86
Slight changes
2024-11-21 16:47:44 +01:00
ManyTheFish
36962b943b
First batch of PR comment
2024-11-21 16:38:11 +01:00
Many the fish
ff38f29981
Update crates/index-scheduler/src/batch.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-21 14:18:39 +01:00
Louis Dureuil
6e6acfcf1b
Merge branch 'main' into indexer-edition-2024
2024-11-20 16:59:58 +01:00
Louis Dureuil
ff9c92c409
rename documents -> substep
2024-11-20 15:12:02 +01:00
Clément Renault
8380ddbdcd
Fix progress of into_changes
2024-11-20 15:10:09 +01:00
Louis Dureuil
84600a10d1
Add MSP to document_update.into_changes()
2024-11-20 14:53:37 +01:00
Tamo
ec06879d28
apply review changes
2024-11-20 14:40:36 +01:00
Tamo
bdb51a85fe
now that the task cancelation shares their started at with all the tasks of their batch we don't need the trick of retrieving the previous batch anymore
2024-11-20 10:51:07 +01:00
Tamo
62646af7b9
implements the automatic batch deletion
2024-11-20 10:51:06 +01:00
Tamo
1fcb9526f5
fix the task cancelation
2024-11-20 10:51:06 +01:00
Tamo
15eefa4fcc
fixes a lot of small issue, the test about the cancellation is still failing
2024-11-20 10:51:05 +01:00
Tamo
6062914654
add the batch_id to the tasks
2024-11-20 10:42:54 +01:00
Louis Dureuil
bfefaf71c2
Progress displayed in logs
2024-11-19 09:32:52 +01:00
Louis Dureuil
5f93651cef
fixes
2024-11-18 16:23:11 +01:00
Clément Renault
5b4c06c24c
Plug the grenad max memory parameter
2024-11-18 11:28:04 +01:00
Clément Renault
677d7293f5
Fix a lot of primary key related tests
2024-11-18 10:59:05 +01:00
Clément Renault
bd31ea2174
Check for at least one valid task after setting their statuses
2024-11-18 10:59:05 +01:00
Clément Renault
83865d2ebd
Expose intermediate errors when processing batches
2024-11-18 10:59:05 +01:00
Clément Renault
9e8367f1e6
Move the rayon thread pool outside the extract method
2024-11-14 10:40:32 +01:00
Louis Dureuil
8a314ab81d
Fix primary key fid order
2024-11-08 00:05:12 +01:00
Louis Dureuil
ee03743355
Merge branch 'indexer-edition-2024' into indexer-edition-2024-doc-chunks
2024-11-06 15:50:53 +01:00
ManyTheFish
10feeb88f2
Merge branch 'main' into indexer-edition-2024
2024-11-06 15:19:18 +01:00
Clément Renault
9c1e54a2c8
Move crates under a sub folder to clean up the code
2024-10-21 08:18:43 +02:00
