Tamo
75d5cea624
use a with_capacity while allocating the progress view
2024-12-11 18:17:33 +01:00
Tamo
ab9213fa94
ensure we never write the progress to the db
2024-12-11 18:16:20 +01:00
Tamo
45d5d4bf40
make the progressview public
2024-12-11 18:15:33 +01:00
Tamo
fa885e75b4
rename the send_progress in progress
2024-12-11 18:13:12 +01:00
Tamo
29fc77ee5b
remove usuless print
2024-12-11 18:11:19 +01:00
Tamo
ad4dc70720
rename the ComputingTheChanges to ComputingDocumentChanges in the edit document progress
2024-12-11 18:09:54 +01:00
Tamo
5d682b4700
rename the ComputingTheChanges to ComputingDocumentChanges
2024-12-11 18:08:45 +01:00
Tamo
f1beb60204
make the progress use payload instead of documents
2024-12-11 18:07:45 +01:00
Tamo
85577e70cd
reuse the enqueued
2024-12-11 18:05:34 +01:00
Tamo
c5536c37b5
rename the atomic::name to unit_name
2024-12-11 18:03:06 +01:00
Tamo
9245c89cfe
move the macros to milli
2024-12-11 18:00:46 +01:00
Tamo
1f54dfa883
update the macro to look more like an enum
2024-12-11 16:26:09 +01:00
Tamo
786b0fabea
implement the progress for almost all the tasks
2024-12-11 16:26:08 +01:00
Tamo
26733c705d
add progress for the task deletion and task cancelation
2024-12-11 16:25:02 +01:00
Tamo
ab75f53efd
update all snapshots
2024-12-11 16:25:02 +01:00
Tamo
867e6a8f1d
rename the send_progress field to progress since it s not sending anything
2024-12-11 16:25:01 +01:00
Tamo
6f4823fc97
make the number of document in the document tasks more incremental
2024-12-11 16:25:01 +01:00
Tamo
df9b68f8ed
inital implementation of the progress
2024-12-11 16:25:01 +01:00
meili-bors[bot]
8c19cb0a0b
Merge #5146
...
5146: Offline upgrade v1.12 r=irevoire a=ManyTheFish
# Pull Request
## Related issue
Fixes #4978
## What does this PR do?
- add v1_11_to_v1_12 function to upgrade Meilisearch from v1.11 to v1.12
- Convert the update files from OBKV to ndjson format
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2024-12-11 13:39:14 +00:00
Many the fish
5c492031d9
Update crates/meilitool/src/upgrade/v1_12.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-11 14:34:18 +01:00
meili-bors[bot]
fb1caa4724
Merge #5148
...
5148: Do not duplicate NDJson data when unecessary r=dureuill a=Kerollmops
This PR improves the NDJSON support. Usually, we save all of the user's document content into a temporary file, validate its content, and then convert everything into NDJSON in the file store (update files in the tasks).
It is a waste of time when users are already sending NDJSON. So, this PR removes the last copy and directly stores the user content in the file store, validating it from the file store. If an issue arises, the file will not persist and will be dropped/deleted instead.
Related to #5078 .
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-11 13:00:50 +00:00
Kerollmops
5622b9607d
Wrap the read NDJSON pass into a tokio blocking
2024-12-11 12:18:36 +01:00
Kerollmops
01bcc601be
Use a nonrandom hasher when decoding JSON
2024-12-11 12:04:29 +01:00
Kerollmops
93fbdc06d3
Use a nonrandom hasher when decoding NDJSON
2024-12-11 12:03:09 +01:00
Kerollmops
69c931334f
Fix the error messages categorization with invalid NDJson
2024-12-11 12:02:48 +01:00
Kerollmops
d683f5980c
Do not duplicate NDJson when unecessary
2024-12-11 12:02:48 +01:00
meili-bors[bot]
f8ba112f66
Merge #5150
...
5150: Reintroduce the Document Addition Logs r=dureuill a=Kerollmops
This PR reintroduces lost tracing logs showing some information about the number of indexed documents.
Related to #5078 . Resolves [this comment](https://github.com/meilisearch/meilisearch/pull/4900/files?show-deleted-files=true&show-viewed-files=true&file-filters%5B%5D=#r1852158338 ) and [this other one](https://github.com/meilisearch/meilisearch/pull/4900/files?show-deleted-files=true&show-viewed-files=true&file-filters%5B%5D=#r1852159073 ).
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-11 10:48:48 +00:00
ManyTheFish
c614d0dd35
Add context when returning an error
2024-12-11 10:55:39 +01:00
ManyTheFish
479607e5dd
Convert update files from OBKV to ndjson
2024-12-11 10:55:39 +01:00
Kerollmops
bb00e70087
Reintroduce the document addition logs
2024-12-11 10:39:04 +01:00
meili-bors[bot]
e974be9518
Merge #5145
...
5145: Use bumparaw-collections in Meilisearch/milli r=dureuill a=Kerollmops
This PR is related to #5078 . It uses the now published bumparaw-collections and (soon) makes the `RawMap` hasher nonrandom.
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-10 15:51:01 +00:00
Kerollmops
aeb6b74725
Make sure we use an FxHashBuilder on the Value
2024-12-10 15:52:22 +01:00
Kerollmops
a751972c57
Prefer using a stable than a random hash builder
2024-12-10 14:25:53 +01:00
Kerollmops
6b269795d2
Update bumparaw-collections to 0.1.2
2024-12-10 14:25:13 +01:00
Kerollmops
89637bcaaf
Use bumparaw-collections in Meilisearch/milli
2024-12-10 11:52:20 +01:00
meili-bors[bot]
1995040846
Merge #5142
...
5142: Try merge optimisation r=dureuill a=ManyTheFish
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-12-09 14:48:26 +00:00
ManyTheFish
07f42e8057
Do not index a filed count when no word is counted
2024-12-09 15:45:12 +01:00
ManyTheFish
71f59749dc
Reduce union impact in merging
2024-12-09 15:44:06 +01:00
meili-bors[bot]
3b0b9967f6
Merge #5141
...
5141: Use the right amount of max memory and not impact the settings r=curquiza a=Kerollmops
Fixes #5132 . Related to #5125 .
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-09 10:40:46 +00:00
meili-bors[bot]
123b54a178
Merge #5056
...
5056: Attach index name in error message r=irevoire a=airycanon
# Pull Request
## Related issue
Fixes #4392
## What does this PR do?
- ...
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: airycanon <airycanon@airycanon.me>
2024-12-09 09:59:12 +00:00
Kerollmops
f5dd8dfc3e
Rollback max memory usage changes
2024-12-09 10:26:30 +01:00
Kerollmops
bcfed70888
Revert "Merge #5125 "
...
This reverts commit 9a9383643f9a6b5ee9ab2ace3e9d63b920d94a53, reversing
changes made to cac355bfa7e72ca3c5c02cacb4f2fcd3f2dd336e.
2024-12-09 10:08:02 +01:00
meili-bors[bot]
503ef3bbc9
Merge #5138
...
5138: Allow xtask bench to proceed without a commit message r=Kerollmops a=dureuill
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-09 09:00:12 +00:00
Louis Dureuil
08f2c696b0
Allow xtask bench to proceed without a commit message
2024-12-09 09:36:59 +01:00
airycanon
b75f1f4c17
fix tests
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/index-scheduler/src/snapshots/lib.rs/fail_in_process_batch_for_document_deletion/after_removing_the_documents.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fifth_task_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fourth_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/third_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key_batch_wrong_key/second_and_third_tasks_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/all_other_tasks_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/third_task_succeeds.snap
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
# crates/meilisearch/tests/vector/mod.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
2024-12-06 02:03:02 +08:00
airycanon
95ed079761
attach index name in errors
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
2024-12-06 01:12:13 +08:00
meili-bors[bot]
4a082683df
Merge #5131
...
5131: Ignore documents whose selected fields didn't change r=dureuill a=dureuill
Attempts to improve the new indexer performance by ignoring documents whose selected fields didn't change:
- Add `Update::has_changed_for_fields` function
- Ignore documents whose searchable attributes didn't change for word docids and word pair proximity extraction
- Ignore documents whose faceted attributes didn't change for facet extraction
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-05 16:04:16 +00:00
meili-bors[bot]
26be5e0733
Merge #5123
...
5123: Fix batch details r=dureuill a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5079
Fixes https://github.com/meilisearch/meilisearch/issues/5112
## What does this PR do?
- Make the processing tasks actually processing in the stats of the batch instead of enqueued
- Stop counting one extra task for all non-prioritized batches in the stats
- Add a test
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-05 15:21:55 +00:00
Louis Dureuil
bd5110a2fe
Fix clippy warnings
2024-12-05 16:13:07 +01:00
Louis Dureuil
fa8b9acdf6
Ignore documents that didn't change in facets
2024-12-05 16:12:52 +01:00
