meili-bors[bot]
054622bd16
Merge #5094
...
5094: Implement a bbqueue channel between the extractors and the writer r=dureuill a=Kerollmops
This PR switches from a bounded crossbeam channel only with allocated entries for the communication between the extractors and the writer to a [BBQueue](https://github.com/jamesmunns/bbqueue )-based system with a Single Producer Single Consumer kind of Circular/Ring Buffers channel.
- [x] Implement the BBQueue channel system...
- [x] with a crossbeam channel to wake up the receiver.
- [x] Manage the BBQueue allocated memory dynamically.
- [x] Support content that doesn't fit in the bbqueues.
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-12-03 08:00:55 +00:00
Louis Dureuil
e905a72d73
remove mimalloc on Windows
2024-12-02 18:13:56 +01:00
meili-bors[bot]
2e879c1df8
Merge #5109
...
5109: Fix autobatch r=dureuill a=dureuill
Fixes most SDK tests and flaky failures
Changes:
- Make sure that the settings are not autobatched with document operations, as the new indexer no longer supports this operating mode
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-02 16:30:51 +00:00
Louis Dureuil
d040aff101
Stop allocating 1GiB for documents
2024-12-02 16:30:14 +01:00
Tamo
beeb31ce41
Update crates/index-scheduler/src/lib.rs
2024-12-02 15:32:16 +01:00
Louis Dureuil
057143214d
Fix warnings
2024-12-02 14:42:31 +01:00
Louis Dureuil
6a1d26a60c
Update autobatching tests
2024-12-02 14:15:15 +01:00
Louis Dureuil
d78f4666a0
Fix autobatching of documents and settings
2024-12-02 12:25:01 +01:00
Tamo
a439fa3e1a
While spamming the batches route we could see a processing batch becoming missing and then finished, this commit ensures the batches goes from processing to finished directly
2024-12-02 12:02:16 +01:00
Clément Renault
767259be7e
Prefer returning a abort indexation rather than throwing a panic
2024-12-02 11:53:42 +01:00
Clément Renault
e9f34fb4b1
Make the frame consumer pulling fair
2024-12-02 11:49:01 +01:00
Clément Renault
d5c07ef7b3
Manage key length conversion error correctly
2024-12-02 11:03:00 +01:00
Clément Renault
5e218f3f4d
Remove a sync_all (mark my words)
2024-12-02 11:03:00 +01:00
Clément Renault
bcab61ab1d
Do spurious wake ups on the receiver side
2024-12-02 11:03:00 +01:00
Clément Renault
263c5a348e
Move the spin looping for BBQueue frames into a dedicated function
2024-12-02 10:33:49 +01:00
Clément Renault
be7d2fbe63
Move the EntryHeader up in the file and document the safety related to the size
2024-12-02 10:19:11 +01:00
Clément Renault
f7f9a131e4
Improve copying bytes into aligned memory area
2024-12-02 10:15:58 +01:00
Clément Renault
5df5eb2db2
Clarify a method name
2024-12-02 10:10:48 +01:00
Clément Renault
30eb0e5b5b
Rename recv and read methods to recv_action and recv_frame
2024-12-02 10:08:01 +01:00
Clément Renault
5b860cb989
Fix english in the doc
2024-12-02 10:06:35 +01:00
Clément Renault
76d0623b11
Reduce the number of unwraps
2024-12-02 10:05:06 +01:00
Clément Renault
db4eaf4d2d
Rename serialize_into into serialize_into_writer
2024-12-02 10:03:27 +01:00
Clément Renault
13f21206a6
Call the serialize_into_writer method from the serialize_into one
2024-12-02 10:03:01 +01:00
Clément Renault
14ee7aa84c
Make sure the BBQueue is at least 50 MiB
2024-11-28 18:02:48 +01:00
Clément Renault
8a35cd1743
Adjust the BBQueue buffers to use 2% instead of 10%
2024-11-28 16:00:15 +01:00
meili-bors[bot]
8d33af1dff
Merge #5102
...
5102: Update mini-dashboard to v0.2.16 version r=curquiza a=curquiza
Fixes https://github.com/meilisearch/meilisearch/issues/5093
Fixes this bug: https://github.com/meilisearch/mini-dashboard/issues/563
Co-authored-by: curquiza <clementine@meilisearch.com>
2024-11-28 14:57:27 +00:00
Clément Renault
3c7ac093d3
Take the BBQueue capacity into account in the max memory
2024-11-28 15:43:14 +01:00
Clément Renault
b57dd5c58e
Remove the Vector variant and use the Vectors
2024-11-28 15:20:43 +01:00
ManyTheFish
90b428a8c3
Apply change requests
2024-11-28 15:16:13 +01:00
Clément Renault
096a28656e
Fix a bug around deleting all the vectors of a doc
2024-11-28 15:15:06 +01:00
curquiza
3dc87f5baa
Update mini-dashboard to v0.2.16 version
2024-11-28 14:33:05 +01:00
Clément Renault
cc4bd54669
Correctly construct the Embeddings struct
2024-11-28 13:53:25 +01:00
ManyTheFish
5383f41bba
Polish test_setting_routes!
2024-11-28 12:04:21 +01:00
Clément Renault
58eab9a018
Send large payload through crossbeam
2024-11-28 12:01:06 +01:00
ManyTheFish
9f36ffcbdb
Polish make_setting_routes!
2024-11-28 11:44:09 +01:00
ManyTheFish
68c4717e21
Change the settings tests and macros to avoid oversights
2024-11-28 11:34:35 +01:00
Clément Renault
5c488e20cc
Send the geo rtree through crossbeam channel
2024-11-27 18:03:45 +01:00
Clément Renault
da650f834e
Plug the NoPanicThreadPool in the tests and benchmarks
2024-11-27 17:04:49 +01:00
Clément Renault
e83534a430
Fix the indexer::index to correctly use the rayon::ThreadPool
2024-11-27 16:27:43 +01:00
Clément Renault
98d4a2909e
Fix the way we spawn the rayon threadpool
2024-11-27 16:05:44 +01:00
Clément Renault
a514ce472a
Make clippy happy
2024-11-27 14:59:04 +01:00
Clément Renault
cc63802115
Modify and return the IndexEmbeddings to write them later
2024-11-27 14:58:03 +01:00
Clément Renault
acec45ad7c
Send a WakeUp when writing data in the BBQueue buffers
2024-11-27 14:33:23 +01:00
Clément Renault
08d6413365
Fix result types
2024-11-27 14:32:42 +01:00
Clément Renault
70802eb7c7
Fix most issues with the lifetimes
2024-11-27 14:32:42 +01:00
Clément Renault
6ac5b3b136
Finish most of the channels types
2024-11-27 14:32:26 +01:00
Clément Renault
e1e76f39d0
Clean up dependencies
2024-11-27 14:30:34 +01:00
Clément Renault
2094ce8a9a
Move the arroy building after the writing loop
2024-11-27 14:30:33 +01:00
Clément Renault
8442db8101
Implement mostly all senders
2024-11-27 14:16:35 +01:00
Clément Renault
79671c9faa
Implement a first version of the bbqueue channels
2024-11-27 14:15:00 +01:00
