ManyTheFish
9a33628331
Implement Incremental document database stats computing
2025-02-17 16:36:33 +01:00
meili-bors[bot]
0c3e7fe963
Merge #5316
...
5316: Fix the dumpless upgrade corruption r=dureuill a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5280
## What does this PR do?
- Add a test that ensure we write the version in the index-scheduler even if we have a bug while writing the VERSION file
- Do what was described in the issue
Co-authored-by: Tamo <tamo@meilisearch.com>
2025-02-10 09:53:57 +00:00
Tamo
45f843ccb9
fmt
2025-02-10 10:46:42 +01:00
Kerollmops
2b0e17ede0
Make sure arroy is using the rayon thread-pool
2025-02-06 15:28:10 +01:00
Kerollmops
cb1b7513af
Log the memory metrics only once
2025-01-29 15:21:52 +01:00
Kerollmops
db032079d8
Show indexation allocated memory
2025-01-29 14:21:02 +01:00
Louis Dureuil
f5a4a1c8b2
Give more RAM to bbqueue.
...
- bbqueue buffers used to have (5% * 2%) / num_threads
- they now have 5% / num_threads
2025-01-24 12:18:32 +01:00
Louis Dureuil
4275833bab
Rename compute.rs to post_process.rs
2025-01-07 15:31:20 +01:00
Louis Dureuil
de7f8c4406
refactor indexer mod
2025-01-07 15:29:02 +01:00
Tamo
fa885e75b4
rename the send_progress in progress
2024-12-11 18:13:12 +01:00
Tamo
867e6a8f1d
rename the send_progress field to progress since it s not sending anything
2024-12-11 16:25:01 +01:00
Tamo
df9b68f8ed
inital implementation of the progress
2024-12-11 16:25:01 +01:00
Kerollmops
a751972c57
Prefer using a stable than a random hash builder
2024-12-10 14:25:53 +01:00
Kerollmops
89637bcaaf
Use bumparaw-collections in Meilisearch/milli
2024-12-10 11:52:20 +01:00
Kerollmops
f5dd8dfc3e
Rollback max memory usage changes
2024-12-09 10:26:30 +01:00
Louis Dureuil
3a11e39c01
Force max_memory to a min of 100MiB
2024-12-04 17:53:30 +01:00
Kerollmops
8ecb726683
Fix the minimun BBQueue channel threshold
2024-12-03 15:49:11 +01:00
Clément Renault
d5c07ef7b3
Manage key length conversion error correctly
2024-12-02 11:03:00 +01:00
Clément Renault
30eb0e5b5b
Rename recv and read methods to recv_action and recv_frame
2024-12-02 10:08:01 +01:00
Clément Renault
14ee7aa84c
Make sure the BBQueue is at least 50 MiB
2024-11-28 18:02:48 +01:00
Clément Renault
8a35cd1743
Adjust the BBQueue buffers to use 2% instead of 10%
2024-11-28 16:00:15 +01:00
Clément Renault
3c7ac093d3
Take the BBQueue capacity into account in the max memory
2024-11-28 15:43:14 +01:00
Clément Renault
b57dd5c58e
Remove the Vector variant and use the Vectors
2024-11-28 15:20:43 +01:00
Clément Renault
096a28656e
Fix a bug around deleting all the vectors of a doc
2024-11-28 15:15:06 +01:00
Clément Renault
cc4bd54669
Correctly construct the Embeddings struct
2024-11-28 13:53:25 +01:00
Clément Renault
58eab9a018
Send large payload through crossbeam
2024-11-28 12:01:06 +01:00
Clément Renault
e83534a430
Fix the indexer::index to correctly use the rayon::ThreadPool
2024-11-27 16:27:43 +01:00
Clément Renault
98d4a2909e
Fix the way we spawn the rayon threadpool
2024-11-27 16:05:44 +01:00
Clément Renault
cc63802115
Modify and return the IndexEmbeddings to write them later
2024-11-27 14:58:03 +01:00
Clément Renault
acec45ad7c
Send a WakeUp when writing data in the BBQueue buffers
2024-11-27 14:33:23 +01:00
Clément Renault
70802eb7c7
Fix most issues with the lifetimes
2024-11-27 14:32:42 +01:00
Clément Renault
6ac5b3b136
Finish most of the channels types
2024-11-27 14:32:26 +01:00
Clément Renault
2094ce8a9a
Move the arroy building after the writing loop
2024-11-27 14:30:33 +01:00
Clément Renault
8442db8101
Implement mostly all senders
2024-11-27 14:16:35 +01:00
meili-bors[bot]
a2f64f6552
Merge #5095
...
5095: Span to measure the part of db writes that is after the merge/extraction r=curquiza a=dureuill
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-27 11:10:00 +00:00
meili-bors[bot]
d0b2c0a523
Merge #5091
...
5091: Settings opt out r=Kerollmops a=ManyTheFish
# Pull Request
Related PRD: https://www.notion.so/meilisearch/API-usage-Settings-to-opt-out-indexing-features-fff4b06b651f8108ade3f858aeb16b14?pvs=4
## Related issue
Fixes #4979
- [x] Add setting opt-out
- [x] Add analytics
- [x] Add tests
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2024-11-26 15:50:28 +00:00
Louis Dureuil
8f57b4fdf4
Span to measure the part of db writes that is after the merge/extraction
2024-11-26 14:46:36 +01:00
Louis Dureuil
aa460819a7
Add more precise spans
2024-11-26 09:45:36 +01:00
ManyTheFish
d66dc363ed
Test and implement settings opt-out
2024-11-25 18:23:22 +01:00
Clément Renault
25aac45fc7
Expose better error messages
2024-11-25 15:54:43 +01:00
Clément Renault
5727e00374
Remove useless geo skipped
2024-11-21 16:47:08 +01:00
Clément Renault
9b60843831
Remove commented lines
2024-11-21 16:47:07 +01:00
ManyTheFish
94b260fd25
Remove orphan span
2024-11-21 12:12:07 +01:00
Clément Renault
ab2c83f868
Use the disk less when computing prefixes
2024-11-21 10:45:37 +01:00
Louis Dureuil
e0864f1b21
Separate side effect and debug asserts
2024-11-20 16:25:17 +01:00
Clément Renault
a38344acb3
Replace eprintlns by tracing
2024-11-20 15:29:51 +01:00
Louis Dureuil
651c30899e
Allow fetching embedders from inside tests
2024-11-20 13:16:36 +01:00
Louis Dureuil
c782c09208
Move step to a dedicated mod and replace it with an enum
2024-11-18 18:22:13 +01:00
Louis Dureuil
75943a5a9b
Add TODO to remember replacing steps with an enum
2024-11-18 17:40:51 +01:00
Louis Dureuil
04c38220ca
Move MostlySend, ThreadLocal, FullySend to their own commit
2024-11-18 16:43:05 +01:00
