130 Commits

Author SHA1 Message Date
Tamo
27bf2f1298
remove the empty progress made for the upgrade database 2025-01-23 16:51:22 +01:00
Tamo
d4d82fbd0c
commit the index wtxn before the index-scheduler wtxn 2025-01-23 16:51:21 +01:00
Tamo
eda09a54da
improve the index-scheduler tests 2025-01-23 16:51:21 +01:00
Tamo
b132d70413
fix the details in all cases 2025-01-23 16:51:21 +01:00
Tamo
e41ebd3047
expose the number of database in the index-scheduler and rewrite the lib.rs to use the value provided in the options instead of a magic number 2025-01-23 16:51:21 +01:00
Tamo
705d31e8bd
apply all the comments changes 2025-01-23 16:51:21 +01:00
Tamo
7d95950ce6
fix warning 2025-01-23 16:51:21 +01:00
Tamo
41eeffd88d
fmt 2025-01-23 16:51:20 +01:00
Tamo
1eb9fe8562
remove warnings 2025-01-23 16:51:20 +01:00
Tamo
bac7a1623a
fix the upgrade test 2025-01-23 16:51:19 +01:00
Tamo
5458850d21
write a test ensuring the index-scheduler is effectively down when the upgrade task fail and try to process it when it restarts. There is a bug when deleting this task 2025-01-23 16:51:19 +01:00
Tamo
cfc1e193b6
update the test with the stats 2025-01-23 16:51:19 +01:00
Tamo
0cc25c7e4c
add a large test importing a data.ms from the v1.12.0 2025-01-23 16:51:18 +01:00
Tamo
102681e384
starts adding tests and fix the starts of meilisearch 2025-01-23 16:51:18 +01:00
Tamo
3ef7a478cd
move the version check to the task queue 2025-01-23 16:48:32 +01:00
Tamo
e70ac35e02
fix bugs after rebase 2025-01-23 16:48:32 +01:00
Tamo
d3654906bf
Add the new tasks with most of the job done 2025-01-23 16:48:32 +01:00
Louis Dureuil
50fca8fc70
Create update files in new format 2025-01-22 15:51:21 +01:00
Louis Dureuil
d95384a636
Remove batch ids on export 2025-01-22 15:50:42 +01:00
Louis Dureuil
2cf57d584e
Handle empty payloads 2025-01-22 15:50:42 +01:00
Kerollmops
59242b9c4f
Fix warnings 2025-01-22 15:50:42 +01:00
Louis Dureuil
6a6212d4e1
Fix warnings 2025-01-22 15:50:42 +01:00
ManyTheFish
a8006a3750
Change format of update file when importing dump 2025-01-22 15:50:41 +01:00
Louis Dureuil
805531c90d
Do not explode on missing content file if the task has no docs 2025-01-22 15:50:41 +01:00
meili-bors[bot]
c85146524b
Merge #5232
5232: Stabilize vector store feature r=Kerollmops a=dureuill

# Pull Request

## Related issue
Fixes #4733 

## What does this PR do?
- `vectorStore` feature can no longer be set or get from `/experimental-features`
- That feature has been removed, and there is no longer any check for its activation
- Always display `embedders` in the settings, even if empty
- Always hide `_vectors` in documents, unless `retrieveVectors: true`
- Make error codes consistent with the usual nomenclature
- Update tests as needed


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-16 11:50:21 +00:00
Tamo
445e5aff02
fix the corruption 2025-01-15 12:38:40 +01:00
Tamo
234d0c360f
Add a test reproducing the issue 2025-01-15 12:29:56 +01:00
Louis Dureuil
87ea080c10
Fully remove vector store feature 2025-01-14 13:55:34 +01:00
Louis Dureuil
29eeb84ce3
Add --experimental-disable-vector-store CLI flag 2025-01-14 13:53:53 +01:00
Louis Dureuil
d78951feb7
vectorStore stabilization
- `vectorStore` feature is always enabled
- `vectorStore` can no longer be set in the `/experimental-features` PATCH route
- `vectorStore` status is no longer returned in the `/experimental-features` GET route
2025-01-14 13:53:53 +01:00
meili-bors[bot]
cc4aca78c4
Merge #5220
5220: Merge back changes of v1.12.2 in main r=dureuill a=dureuill



Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: dureuill <dureuill@users.noreply.github.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-13 10:54:36 +00:00
meili-bors[bot]
42854c0bca
Merge #5223
5223: Limit batched tasks total size r=curquiza a=Kerollmops

Introduce a new engine parameter (env and config, too) to limit the maximum payload size processed by the engine in batches. You can [review the Discussion and usage on GitHub](https://github.com/orgs/meilisearch/discussions/801).

Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-01-09 16:13:17 +00:00
Clément Renault
d0bdff7b7b
Make the batched tasks size limit effectively work 2025-01-09 12:06:28 +01:00
Tamo
647a10bf18
stop skipping empty tasks when adding documents 2025-01-09 10:24:34 +01:00
Clément Renault
71e5605daa
Make clippy happy 2025-01-08 18:24:39 +01:00
Tamo
99f5e09a79
fix the tests 2025-01-07 16:42:53 +01:00
Tamo
dd128656cb
fix all the tests 2025-01-07 16:28:12 +01:00
Tamo
4456df5a46
fix some tests 2025-01-07 16:28:11 +01:00
Tamo
43bb02e7b4
split the autobatcher in two 2025-01-07 15:02:03 +01:00
Tamo
cb82b0798a
Split the index-scheduler in ~500 loc modules 2025-01-06 14:08:26 +01:00
meili-bors[bot]
baeefa4817
Merge #5166
5166: fix list indexes r=dureuill a=irevoire

# Pull Request

### Smol benchmark on a meilisearch with 1009 indexes:

**Before** this PR on my computer, it was taking 5.5s to call the `GET /indexes` route on a cold computer where all the indexes were closed.
**After** this PR it takes 0.009s to call the route on the first 20 indexes, and 0.176 for the last 20 indexes (retrieving the first or last indexes on main has no impact on performances).

If my computations are right, that's between 61111.1% and 3125% faster on this test 😂 

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4694

## What does this PR do?
- Add the primary key to the cache we already have in the index-mapper
- Provide a new route to retrieve the paginated indexes straight from the cache without opening them
- Fix a bug where the cache was not computed when loading a dump and was forcing us to open the indexes to compute their stats on the fly

## Is it breaking?

Since the field I added is an `Option` I think we should consider it as non-breaking and let it update itself automatically on the next operation of this index.
I also tested to run my patch over a DB generated on release-v1.12.0 and it works. The importing a dump also works.

Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-31 10:55:22 +00:00
Gnosnay
525e67ba93 Fix the format and linter error 2024-12-28 20:35:55 +08:00
Gnosnay
44eb153619 Replace hardcoded string with constants 2024-12-28 20:35:55 +08:00
Tamo
4eae92f411
fix list indexes 2024-12-26 18:48:25 +01:00
meili-bors[bot]
1fc90fbacb
Merge #5147
5147: Batch progress r=dureuill a=irevoire

# Pull Request

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5068

## What does this PR do?
- ...

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ ] Have you read the contributing guidelines?
- [ ] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-12 09:15:54 +00:00
Tamo
0d0c18f519
rename the Step::name into Step::current_step 2024-12-11 18:41:03 +01:00
Tamo
d12364c1e0
fix the tests 2024-12-11 18:30:48 +01:00
Tamo
8cd3a1aa57
fmt 2024-12-11 18:18:40 +01:00
Tamo
29fc77ee5b
remove usuless print 2024-12-11 18:11:19 +01:00
Tamo
ad4dc70720
rename the ComputingTheChanges to ComputingDocumentChanges in the edit document progress 2024-12-11 18:09:54 +01:00