ManyTheFish
9f3663e768
Implement Incremental document database stats computing
2025-02-26 17:01:35 +01:00
ManyTheFish
d9642ec916
Use checked_div in average computation
2025-02-26 17:01:34 +01:00
ManyTheFish
818e8b0237
Fix zero division
2025-02-26 17:01:31 +01:00
ManyTheFish
4f77a7fba5
fix clippy
2025-02-26 17:01:29 +01:00
ManyTheFish
058f08dff5
fix snapshots
2025-02-26 17:01:26 +01:00
ManyTheFish
9a6c1730aa
Add document database stats
2025-02-26 17:01:25 +01:00
Strift
91a8a97045
Bump
2025-02-26 17:01:24 +01:00
ManyTheFish
15788773af
Check the exact_word database when computing zero typo query
2025-02-26 17:01:22 +01:00
Kerollmops
025b9b79bb
Update the snapshots
2025-02-26 17:01:21 +01:00
Kerollmops
1c60b17a37
Update version for the next release (v1.13.1) in Cargo.toml
2025-02-26 17:01:19 +01:00
Louis Dureuil
3b2cd54b9d
tests: add a check to know if a Value has an uid
2025-02-25 17:24:45 +01:00
Tamo
0833cb7d34
Mention openAPI in CONTRIBUTING.md
2025-02-25 12:01:26 +01:00
meili-bors[bot]
b0d4f9590f
Merge #5364
...
5364: Rename `callTrace` into `progressTrace` r=Kerollmops a=Kerollmops
Rename the `callTrace` field into a `progressTrace`.
Co-authored-by: Kerollmops <clement@meilisearch.com>
2025-02-25 09:34:13 +00:00
Kerollmops
dfce20be21
Rename callTrace into progressTrace
2025-02-25 10:09:03 +01:00
Louis Dureuil
24fe6cd205
Fix multiple embeddings in hf
2025-02-24 16:24:04 +01:00
Louis Dureuil
e374b095a2
Fix tests
2025-02-24 14:11:26 +01:00
Louis Dureuil
9f3e4801b1
Refactor settings validation and introduce SubEmbedderSettings
2025-02-24 13:58:26 +01:00
Louis Dureuil
b85180fedb
Error types
2025-02-24 13:58:26 +01:00
Louis Dureuil
3cdcc54a9e
analytics
2025-02-24 13:58:26 +01:00
Louis Dureuil
294cf39cad
Integrate composite embedder
2025-02-24 13:58:26 +01:00
Louis Dureuil
4a2643daa2
Rename embed_one to embed_search and embed_chunks* to embed_index*
2025-02-24 13:58:26 +01:00
Louis Dureuil
8d2d9066ba
Add composite embedder
2025-02-24 13:58:26 +01:00
Louis Dureuil
526476e168
Move settings test to its own file
2025-02-24 13:58:26 +01:00
meili-bors[bot]
ea7bae9a71
Merge #5356
...
5356: Display the internal indexing steps with timings on the `/batches` route r=irevoire a=Kerollmops
This PR computes the durations of each step, stores them in a map, and prints them (for now).
```
"callTrace": {
"processing tasks > retrieving config": "185.38µs",
"processing tasks > computing document changes > preparing update file > payload": "23.11ms",
"processing tasks > computing document changes > preparing update file": "23.26ms",
"processing tasks > computing document changes": "24.06ms",
"processing tasks > indexing > extracting documents > document": "15.13ms",
"processing tasks > indexing > extracting documents": "15.13ms",
"processing tasks > indexing > extracting facets > document": "5.70ms",
"processing tasks > indexing > extracting facets": "5.72ms",
"processing tasks > indexing > extracting words > document": "597.24ms",
"processing tasks > indexing > extracting words": "597.25ms",
"processing tasks > indexing > extracting word proximity > document": "1.14s",
"processing tasks > indexing > extracting word proximity": "1.15s",
"processing tasks > indexing > tail writing to database": "430.91ms",
"processing tasks > indexing > waiting for extractors": "52.54µs",
"processing tasks > indexing > writing embeddings to database": "47.79µs",
"processing tasks > indexing > post-processing facets": "476.04µs",
"processing tasks > indexing > post-processing words": "97.82ms",
"processing tasks > indexing > finalizing": "67.41ms",
"processing tasks > indexing": "2.40s",
"processing tasks": "2.43s",
"writing tasks to disk > task": "37.71µs",
"writing tasks to disk": "67.13µs"
},
"writeChannelCongestion": {
"attempts": 2608482,
"blocking_attempts": 0,
"blocking_ratio": 0.0
}
```
## To Do
- [x] Update the batches PRD + delivery + tracking issue.
- [x] Store that in the batches to be visible from the `/batches` route.
- [x] Display the writer's congestion.
- [x] Display the info back in the logs too.
- [ ] (optional) Compute the size of each database by [using LMDB](https://docs.rs/heed/latest/heed/struct.DatabaseStat.html ).
- [x] Push them in reverse order so that "processing task" is after the other sub-steps.
Co-authored-by: Kerollmops <clement@meilisearch.com>
2025-02-20 17:38:50 +00:00
Kerollmops
76fd5d92d7
Clarify the tail writing to database
2025-02-20 17:35:23 +01:00
Kerollmops
245a55722a
Remove commented code
2025-02-20 16:48:18 +01:00
Kerollmops
434fad5327
Fix insta tests again
2025-02-20 16:41:48 +01:00
Kerollmops
243a5fa6a8
Log the call trace and congestion
2025-02-20 14:17:34 +01:00
Kerollmops
9d314ace09
Fix the insta tests
2025-02-20 11:51:58 +01:00
Kerollmops
1b1172ad16
Fix dump tests
2025-02-20 10:44:53 +01:00
Kerollmops
1d99c8465c
Hide the batch stats to make insta pass
2025-02-20 10:16:54 +01:00
Kerollmops
05cc8c650c
Expose the write channel congestion in the batches
2025-02-19 15:47:54 +01:00
Louis Dureuil
14e1459bf5
Document settings
2025-02-19 15:06:22 +01:00
Louis Dureuil
589bf30ec6
make clippy happy
2025-02-19 11:38:07 +01:00
Louis Dureuil
b367c71ad2
fixup test
2025-02-19 11:31:17 +01:00
Kerollmops
3ff1de0a21
Expose the call trace in the batch stats
2025-02-19 11:24:11 +01:00
Louis Dureuil
1005a60fb8
Fixup dump settings
2025-02-19 11:03:48 +01:00
Kerollmops
e9add14189
Reorder steps
2025-02-18 19:26:41 +01:00
Kerollmops
4a058a080e
Simplify the name generation
2025-02-18 18:48:44 +01:00
Kerollmops
11a11fc870
Accumulate step durations from the progress system
2025-02-18 18:33:19 +01:00
Louis Dureuil
cd0dfa3f1b
Fix test cases
2025-02-18 17:21:52 +01:00
Louis Dureuil
7b4ce468a6
Allow overriding pooling method
2025-02-18 17:12:23 +01:00
Louis Dureuil
11759c4be4
Support pooling
2025-02-18 16:10:51 +01:00
meili-bors[bot]
0f1aeb8eaa
Merge #5351
...
5351: Bring back v1.13.0 changes into main r=irevoire a=Kerollmops
This PR brings back the changes made in v1.13 into the main branch.
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Clémentine <clementine@meilisearch.com>
Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-02-18 08:05:02 +00:00
meili-bors[bot]
5e7803632d
Merge #5342
...
5342: Fix workload sha r=dureuill a=ManyTheFish
The dataset shasum was wrong for some workloads making the `/bench workloads/*.json` crash
Co-authored-by: ManyTheFish <many@meilisearch.com>
2025-02-12 16:27:09 +00:00
meili-bors[bot]
885710a07b
Merge #5341
...
5341: Embeddings stats r=ManyTheFish a=ManyTheFish
# Pull Request
## Related issue
Fixes #5321
## What does this PR do?
- Add embedding stats
- force dumpless upgrade to recompute stats
- add tests
Co-authored-by: ManyTheFish <many@meilisearch.com>
2025-02-12 15:46:37 +00:00
ManyTheFish
c55fdad2c3
Fix dumpless upgrade target version
2025-02-12 16:35:05 +01:00
ManyTheFish
1caad4c4b0
Add multiple embeddings for the same embedder in tests
2025-02-12 16:13:34 +01:00
ManyTheFish
8419ed52a1
fix clippy
2025-02-12 14:38:51 +01:00
ManyTheFish
a65c52cc97
Convert dump test into snapshots
2025-02-12 14:14:10 +01:00
