10900 Commits

Author SHA1 Message Date
Louis Dureuil
8ad92f7eb0
Fix tests 2025-01-30 15:13:19 +01:00
Louis Dureuil
9951f5c8b3
Multi search reads header to know if it is being proxied 2025-01-30 15:13:19 +01:00
Louis Dureuil
f4f372e113
Proxy search requests 2025-01-30 15:13:19 +01:00
Louis Dureuil
481d890f53
Allow rebuilding a SearchQueryWithIndex from its components 2025-01-30 15:13:19 +01:00
Louis Dureuil
3647c9d226
Add WeightedScoreValues to be able to compare remote scores 2025-01-30 15:13:18 +01:00
Louis Dureuil
ce9b5e0029
Make search types serialize and deserialize so that reading from a proxy is possible 2025-01-30 15:13:18 +01:00
Louis Dureuil
b232bf935b
Route network 2025-01-30 15:13:18 +01:00
Louis Dureuil
b02c2b9a6f
Add new permissions 2025-01-30 11:33:41 +01:00
Louis Dureuil
4d99f22b3d
New error codes 2025-01-30 11:33:41 +01:00
Louis Dureuil
de04d5eddd
Support network in dumps 2025-01-30 11:33:41 +01:00
Louis Dureuil
b563ddfc11
Network stored in DB 2025-01-30 11:31:24 +01:00
Louis Dureuil
8f737df0d0
Network types 2025-01-30 11:31:24 +01:00
Louis Dureuil
3998a850f7
Add new experimental feature 2025-01-30 11:31:24 +01:00
meili-bors[bot]
42257eec53
Merge #5272
5272: Fix Batches Deletion and flaky tests r=irevoire a=Kerollmops

- This issue fixes #5263 by removing the batches from the date and time databases.
- It also introduces a new `enqueued_at` field in the batch object to quickly retrieve them in the `batches.enqueued_at` database
- Finally, it probably fixes all the flaky tests of the batches: https://github.com/meilisearch/meilisearch/issues/5256

Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
2025-01-28 16:14:11 +00:00
Tamo
1beda3b9af
fix another flaky test 2025-01-28 16:53:50 +01:00
Tamo
8676e94f5c
fix the flaky tests 2025-01-28 16:53:50 +01:00
Tamo
ef47a0d820
apply review comment 2025-01-28 16:53:50 +01:00
Tamo
e0f0da57e2
make sure the batches we snapshots actually all contains an enqueued_at 2025-01-28 16:53:50 +01:00
Tamo
485e3127c7
use the remove_n_tasks_datetime_earlier_than function when updating batches 2025-01-28 16:53:50 +01:00
Tamo
58f90b70c7
store the enqueued at to eases the batch deletion 2025-01-28 16:53:50 +01:00
Tamo
508db9020d
update the snapshots 2025-01-28 16:53:50 +01:00
Kerollmops
6ff37c6fc4
Fix the insta snapshots 2025-01-28 16:53:50 +01:00
Kerollmops
f21ae1f5d1
Remove the batch id from the date time databases 2025-01-28 16:53:50 +01:00
meili-bors[bot]
483c52f07b
Merge #5289
5289: Fix workload files after removing the vectorStore experimental feature r=Kerollmops a=dureuill

Running the bench [currently fails](https://github.com/meilisearch/meilisearch/actions/runs/12990029453) on embedding-related workloads, due to the call to `/experimental-features` that is used to enable the vector store:

In v1.13, `vectorStore` is no longer an experimental feature, so trying to enable it causes a 400

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-28 10:28:21 +00:00
Louis Dureuil
f88f415a00
Fix workload files after removing the vectorStore experimental feature 2025-01-27 14:39:28 +01:00
meili-bors[bot]
df9d10ac44
Merge #5284
5284: Fix [5281] Removed CouldNotUpgrade from error file  r=irevoire a=manojks1999

# Pull Request

## Related issue
Fixes #5281

## What does this PR do?
- ...

## PR checklist
Please check if your PR fulfills the following requirements:
- [ * ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ * ] Have you read the contributing guidelines?
- [ * ] Have you made sure that the title is accurate and descriptive of the changes?


Co-authored-by: manojks1999 <9743manoj@gmail.com>
v1.13.0-rc.0
2025-01-27 09:26:39 +00:00
manojks1999
528d9d6d8b Removed CouldNotUpgrade from error file 2025-01-26 21:04:57 +05:30
meili-bors[bot]
022205af90
Merge #5279
5279: Bring back changes from v1.12.7 into main r=dureuill a=Kerollmops

This PR brings back v1.12.7 into main.

Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-01-24 11:48:46 +00:00
Louis Dureuil
50280bf02b
Support offline upgrade up to v1.12.7 2025-01-24 12:25:33 +01:00
Clément Renault
9b579069df
Comment the max grant of the bbqueue
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-24 12:18:32 +01:00
Louis Dureuil
f5a4a1c8b2
Give more RAM to bbqueue.
- bbqueue buffers used to have (5% * 2%) / num_threads
- they now have 5% / num_threads
2025-01-24 12:18:32 +01:00
Kerollmops
5ab4cdb1f3
Reduce the maximum grant possible we can store in the BBQueue 2025-01-24 12:18:32 +01:00
meili-bors[bot]
1f54f07f72
Merge #5264
5264: Dumpless upgrade r=dureuill a=irevoire

# Pull Request
Usage: https://meilisearch.notion.site/Dumpless-upgrade-fff4b06b651f81f1acafe24d4687b3f7?pvs=74

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5162

## What does this PR do?
- Implement the dumpless upgrade with multiple hooks:
  - In meilisearch directly before the task queue has been opened
  - In the index-scheduler while processing the task
  - In milli while upgrading the indexes
- There is no hook at search/query time to handle the old version of a database. That's left to the next person upgrading a database
- A new special type of task (`upgradeDatabase`) that can be retried has been introduced
- A new experimental cli flag has been introduced
- The version has been upgraded to the v1.13.0 in this PR otherwise it was a lot of useless work to test the dumpless upgrade
- Multiple tests have been introduced

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Update the issue template we use for features, mentioning what we should do in case of a database upgrade
- [ ] The experimental feature discussion should be opened and updated in the PR
- [ ] Update the PRD
    - [ ] Add the new error codes
    - [ ] Add the task details
    - [ ] Add the telemetry

## Notes

The new tests introduced are not _that_ slow
![image](https://github.com/user-attachments/assets/c5884540-482f-41eb-97ef-fc995c62d666)



Co-authored-by: Tamo <tamo@meilisearch.com>
2025-01-23 16:22:37 +00:00
Louis Dureuil
73d8a4eace
Remove db.snapshot 2025-01-23 17:21:42 +01:00
Louis Dureuil
c1e5897076
Do not assume v1.12 when there is no index-scheduler version 2025-01-23 17:16:53 +01:00
Louis Dureuil
718a98fbbf
remove : char from filenames 2025-01-23 17:08:35 +01:00
Louis Dureuil
86bf231d29
Change to meilitool after rebase 2025-01-23 16:59:32 +01:00
Louis Dureuil
182c3f4b80
Write assumed version to the index-scheduler version db when it is missing 2025-01-23 16:51:25 +01:00
Tamo
c1eba66443
introduce a corruption in the v1.12 data.ms field distribution 2025-01-23 16:51:24 +01:00
Tamo
7197ced673
fix the bad index version on opening 2025-01-23 16:51:24 +01:00
Tamo
4f21ee6c66
update the data.ms snapshot 2025-01-23 16:51:24 +01:00
Tamo
787472453d
write the version of the index while upgrading it 2025-01-23 16:51:24 +01:00
Tamo
8f65f35de9
rewrite part of the index-scheduler upgrade test 2025-01-23 16:51:23 +01:00
Tamo
c27c923439
introduce a trait to upgrade the indexes 2025-01-23 16:51:23 +01:00
Tamo
fd5649091d
add the upgradeTo field in the details 2025-01-23 16:51:23 +01:00
Tamo
9a57736773
fix the early exit when rewriting a batch 2025-01-23 16:51:23 +01:00
Tamo
7740997ea8
reintroduce the unrecoverable error and use it where its supposed to be used 2025-01-23 16:51:22 +01:00
Tamo
7eb23f73ba
add the version to the index-scheduler snapshots + fix a bug when opening an index scheduler for the first time 2025-01-23 16:51:22 +01:00
Tamo
b9e9fc376a
add the version in the index-scheduler 2025-01-23 16:51:22 +01:00
Tamo
27bf2f1298
remove the empty progress made for the upgrade database 2025-01-23 16:51:22 +01:00