meili-bors[bot]
796acd1aee
Merge #5288
...
5288: Improve AI logging r=dureuill a=Kerollmops
This PR fixes #5285 and brings the changes from #5233 to simplify debugging indexation and search performance issues related to AI. The following texts can be found in the logs to debug and understand performance issues:
- `embed_one: search` represents the time we spent waiting for the embedding generation, i.e., OpenAI, local HuggingFace, Ollama.
- `filtered_universe: search::universe` the time spent filtering the documents.
- ~`next_bucket: search::vector_sort` is the time spent finding the nearest neighbors (ANNs) in the vector store (arroy), locally~ was being triggered too many times.
- `indexing::vectors` is the time arroy spends indexing the new vectors for a batch.
- `documents::extract vectors` and `documents::merge vectors` to see the time spent generating and writing the embeddings.
Co-authored-by: Kerollmops <clement@meilisearch.com>
2025-02-04 10:20:45 +00:00
Kerollmops
424c5bde40
Move the embedding computation and extraction log to debug
2025-01-29 16:40:36 +01:00
Kerollmops
cb1b7513af
Log the memory metrics only once
2025-01-29 15:21:52 +01:00
Clément Renault
a9d0f4a002
Improve english comments
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-29 15:16:40 +01:00
Kerollmops
db032079d8
Show indexation allocated memory
2025-01-29 14:21:02 +01:00
Clément Renault
a00796c46a
Improve the naming in the log message
2025-01-29 14:21:02 +01:00
Kerollmops
6112bd8caa
Display the channel congestion
2025-01-29 14:21:02 +01:00
Kerollmops
cec88cfc29
Measure the bbqueue congestion
2025-01-29 14:21:02 +01:00
Kerollmops
4a5923a55e
log the time arroy took to insert embeddings
2025-01-27 14:22:17 +01:00
Clément Renault
9b579069df
Comment the max grant of the bbqueue
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-24 12:18:32 +01:00
Louis Dureuil
f5a4a1c8b2
Give more RAM to bbqueue.
...
- bbqueue buffers used to have (5% * 2%) / num_threads
- they now have 5% / num_threads
2025-01-24 12:18:32 +01:00
Kerollmops
5ab4cdb1f3
Reduce the maximum grant possible we can store in the BBQueue
2025-01-24 12:18:32 +01:00
Tamo
787472453d
write the version of the index while upgrading it
2025-01-23 16:51:24 +01:00
Tamo
c27c923439
introduce a trait to upgrade the indexes
2025-01-23 16:51:23 +01:00
Tamo
41eeffd88d
fmt
2025-01-23 16:51:20 +01:00
Tamo
20ac59c946
fix the field distribution when upgrading from the v1_12
2025-01-23 16:51:19 +01:00
Tamo
cfc1e193b6
update the test with the stats
2025-01-23 16:51:19 +01:00
Tamo
0cc25c7e4c
add a large test importing a data.ms from the v1.12.0
2025-01-23 16:51:18 +01:00
Tamo
3ef7a478cd
move the version check to the task queue
2025-01-23 16:48:32 +01:00
Tamo
d3654906bf
Add the new tasks with most of the job done
2025-01-23 16:48:32 +01:00
Louis Dureuil
d6063079af
Unify facet strings by their normalized value
2025-01-22 15:50:42 +01:00
Louis Dureuil
a6470a0c37
Improve error log
2025-01-22 15:50:41 +01:00
Louis Dureuil
8a54f14b8e
Demote panic to error log
2025-01-22 15:49:24 +01:00
Kerollmops
63c8cbae5b
Improve the panic message when deleting an unknown entry
2025-01-14 10:31:44 +01:00
Louis Dureuil
72ded27e98
Update after review
2025-01-14 10:24:50 +01:00
Louis Dureuil
4070895a21
Add support to upgrade to v1.12.3 in meilitool
2025-01-14 10:24:27 +01:00
Louis Dureuil
a21711f473
Fix test
2025-01-14 10:23:59 +01:00
Louis Dureuil
f0ec8cbffe
Add currently failing test
2025-01-14 10:23:15 +01:00
meili-bors[bot]
247eaed872
Merge #5221
...
5221: Merge bitmaps by using `Extend::extend` r=Kerollmops a=Kerollmops
This PR tries to speed up the merging of bitmaps by using [the new `Extend::extend` implementation](https://github.com/RoaringBitmap/roaring-rs/pull/306 ).
Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-01-13 13:43:28 +00:00
meili-bors[bot]
cc4aca78c4
Merge #5220
...
5220: Merge back changes of v1.12.2 in main r=dureuill a=dureuill
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: dureuill <dureuill@users.noreply.github.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-13 10:54:36 +00:00
Clément Renault
00a03742ff
Prefer using extend when merging bitmaps than unions (less allocations)
2025-01-09 10:42:38 +01:00
Louis Dureuil
d11e359244
When spilling on the next fid, no longer ignore children
2025-01-09 10:36:38 +01:00
Louis Dureuil
09d45439c7
Check valid_facet_value as part of a filter of the iterator
2025-01-09 10:36:38 +01:00
Louis Dureuil
5d92da0c73
No longer ignore the first child without parent
2025-01-09 10:36:38 +01:00
Louis Dureuil
677bb39e73
Modernize valid_lmdb_key
2025-01-09 10:36:38 +01:00
Louis Dureuil
85ea77de0b
Switch to an iterative algorithm for find_changed_parents
2025-01-09 10:36:38 +01:00
Louis Dureuil
03317be0bd
Update after review
2025-01-09 10:36:38 +01:00
Louis Dureuil
4aa7c8f7b1
Remove unused FacetFieldIdOperation
2025-01-09 10:36:37 +01:00
Louis Dureuil
ce57a342a3
center groups
2025-01-09 10:36:37 +01:00
Louis Dureuil
1cc6cd78e0
Fix uselessly deep stack trace
2025-01-09 10:36:37 +01:00
Louis Dureuil
c204afdc79
Update snapshot
2025-01-09 10:36:37 +01:00
Louis Dureuil
c14967eeac
Use new incremental facet indexing and enable sanity checks in debug
2025-01-09 10:36:35 +01:00
Louis Dureuil
f38db86120
Add new incremental facet indexing
2025-01-09 10:24:36 +01:00
Louis Dureuil
50b155fa2d
add valid_facet_value utility function
2025-01-09 10:24:36 +01:00
Louis Dureuil
a533c8e041
Add sanity checks for facet values
2025-01-09 10:24:36 +01:00
Tamo
908adee6fc
Fix the addition of empty payload
2025-01-09 10:24:36 +01:00
Clément Renault
71e5605daa
Make clippy happy
2025-01-08 18:24:39 +01:00
Clément Renault
68333424c6
Remove a useless script test
2025-01-08 15:59:43 +01:00
Clément Renault
5e8144b0e1
Remove fuzzing feature
2025-01-08 15:59:03 +01:00
Louis Dureuil
4275833bab
Rename compute.rs to post_process.rs
2025-01-07 15:31:20 +01:00
