124 Commits

Author SHA1 Message Date
Tamo
27103263f7
expose the number of database in the index-scheduler and rewrite the lib.rs to use the value provided in the options instead of a magic number 2025-01-22 15:27:47 +01:00
Tamo
ad18b02d5e
apply all the comments changes 2025-01-22 15:27:47 +01:00
Tamo
13a951a8b4
fix warning 2025-01-21 17:09:14 +01:00
Tamo
7d5649649b
fmt 2025-01-21 16:54:29 +01:00
Tamo
ef4e2bb709
remove warnings 2025-01-21 16:54:29 +01:00
Tamo
1af895bdfb
fix the upgrade test 2025-01-21 16:54:29 +01:00
Tamo
87a0367524
write a test ensuring the index-scheduler is effectively down when the upgrade task fail and try to process it when it restarts. There is a bug when deleting this task 2025-01-21 16:54:29 +01:00
Tamo
653ae387e4
update the test with the stats 2025-01-21 16:54:29 +01:00
Tamo
d76c752c7e
add a large test importing a data.ms from the v1.12.0 2025-01-21 16:54:29 +01:00
Tamo
7f8ee6657a
starts adding tests and fix the starts of meilisearch 2025-01-21 16:54:29 +01:00
Tamo
acebf68261
move the version check to the task queue 2025-01-21 16:54:29 +01:00
Tamo
002f332ce5
fix bugs after rebase 2025-01-21 16:54:29 +01:00
Tamo
c21e6f7469
Add the new tasks with most of the job done 2025-01-21 16:54:29 +01:00
meili-bors[bot]
c85146524b
Merge #5232
5232: Stabilize vector store feature r=Kerollmops a=dureuill

# Pull Request

## Related issue
Fixes #4733 

## What does this PR do?
- `vectorStore` feature can no longer be set or get from `/experimental-features`
- That feature has been removed, and there is no longer any check for its activation
- Always display `embedders` in the settings, even if empty
- Always hide `_vectors` in documents, unless `retrieveVectors: true`
- Make error codes consistent with the usual nomenclature
- Update tests as needed


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-16 11:50:21 +00:00
Tamo
445e5aff02
fix the corruption 2025-01-15 12:38:40 +01:00
Tamo
234d0c360f
Add a test reproducing the issue 2025-01-15 12:29:56 +01:00
Louis Dureuil
87ea080c10
Fully remove vector store feature 2025-01-14 13:55:34 +01:00
Louis Dureuil
29eeb84ce3
Add --experimental-disable-vector-store CLI flag 2025-01-14 13:53:53 +01:00
Louis Dureuil
d78951feb7
vectorStore stabilization
- `vectorStore` feature is always enabled
- `vectorStore` can no longer be set in the `/experimental-features` PATCH route
- `vectorStore` status is no longer returned in the `/experimental-features` GET route
2025-01-14 13:53:53 +01:00
meili-bors[bot]
cc4aca78c4
Merge #5220
5220: Merge back changes of v1.12.2 in main r=dureuill a=dureuill



Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: dureuill <dureuill@users.noreply.github.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-13 10:54:36 +00:00
meili-bors[bot]
42854c0bca
Merge #5223
5223: Limit batched tasks total size r=curquiza a=Kerollmops

Introduce a new engine parameter (env and config, too) to limit the maximum payload size processed by the engine in batches. You can [review the Discussion and usage on GitHub](https://github.com/orgs/meilisearch/discussions/801).

Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-01-09 16:13:17 +00:00
Clément Renault
d0bdff7b7b
Make the batched tasks size limit effectively work 2025-01-09 12:06:28 +01:00
Tamo
647a10bf18
stop skipping empty tasks when adding documents 2025-01-09 10:24:34 +01:00
Clément Renault
71e5605daa
Make clippy happy 2025-01-08 18:24:39 +01:00
Clément Renault
091f989b72
Upgrade incompatible dependencies 2025-01-08 15:58:03 +01:00
Clément Renault
48a9ad4c17
Fix insta to 1.39 2025-01-08 15:18:08 +01:00
Clément Renault
b997039a91
Upgrade compatible dependencies 2025-01-08 13:52:14 +01:00
Tamo
99f5e09a79
fix the tests 2025-01-07 16:42:53 +01:00
Tamo
dd128656cb
fix all the tests 2025-01-07 16:28:12 +01:00
Tamo
4456df5a46
fix some tests 2025-01-07 16:28:11 +01:00
Tamo
43bb02e7b4
split the autobatcher in two 2025-01-07 15:02:03 +01:00
Tamo
cb82b0798a
Split the index-scheduler in ~500 loc modules 2025-01-06 14:08:26 +01:00
meili-bors[bot]
baeefa4817
Merge #5166
5166: fix list indexes r=dureuill a=irevoire

# Pull Request

### Smol benchmark on a meilisearch with 1009 indexes:

**Before** this PR on my computer, it was taking 5.5s to call the `GET /indexes` route on a cold computer where all the indexes were closed.
**After** this PR it takes 0.009s to call the route on the first 20 indexes, and 0.176 for the last 20 indexes (retrieving the first or last indexes on main has no impact on performances).

If my computations are right, that's between 61111.1% and 3125% faster on this test 😂 

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4694

## What does this PR do?
- Add the primary key to the cache we already have in the index-mapper
- Provide a new route to retrieve the paginated indexes straight from the cache without opening them
- Fix a bug where the cache was not computed when loading a dump and was forcing us to open the indexes to compute their stats on the fly

## Is it breaking?

Since the field I added is an `Option` I think we should consider it as non-breaking and let it update itself automatically on the next operation of this index.
I also tested to run my patch over a DB generated on release-v1.12.0 and it works. The importing a dump also works.

Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-31 10:55:22 +00:00
Gnosnay
525e67ba93 Fix the format and linter error 2024-12-28 20:35:55 +08:00
Gnosnay
44eb153619 Replace hardcoded string with constants 2024-12-28 20:35:55 +08:00
Tamo
4eae92f411
fix list indexes 2024-12-26 18:48:25 +01:00
meili-bors[bot]
1fc90fbacb
Merge #5147
5147: Batch progress r=dureuill a=irevoire

# Pull Request

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5068

## What does this PR do?
- ...

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ ] Have you read the contributing guidelines?
- [ ] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-12 09:15:54 +00:00
Tamo
0d0c18f519
rename the Step::name into Step::current_step 2024-12-11 18:41:03 +01:00
Tamo
d12364c1e0
fix the tests 2024-12-11 18:30:48 +01:00
Tamo
8cd3a1aa57
fmt 2024-12-11 18:18:40 +01:00
Tamo
29fc77ee5b
remove usuless print 2024-12-11 18:11:19 +01:00
Tamo
ad4dc70720
rename the ComputingTheChanges to ComputingDocumentChanges in the edit document progress 2024-12-11 18:09:54 +01:00
Tamo
5d682b4700
rename the ComputingTheChanges to ComputingDocumentChanges 2024-12-11 18:08:45 +01:00
Tamo
85577e70cd
reuse the enqueued 2024-12-11 18:05:34 +01:00
Tamo
9245c89cfe
move the macros to milli 2024-12-11 18:00:46 +01:00
meili-bors[bot]
eaabc1af2f
Merge #5144
5144: Exactly 512 bytes docid fails r=Kerollmops a=dureuill

# Pull Request

## Related issue
Fixes #5050 

## What does this PR do?
- Return a user error rather than an internal one for docids of exactly 512 bytes
- Fix up error message to indicate that exactly 512 bytes long docids are not supported.
- Fix up error message to reflect that index uids are actually limited to 400 bytes in length

## Impact

- Impacts docs: 
    - update [this paragraph](https://www.meilisearch.com/docs/learn/resources/known_limitations#length-of-primary-key-values) to say 511 bytes instead of 512 

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-11 15:41:05 +00:00
Tamo
1f54dfa883
update the macro to look more like an enum 2024-12-11 16:26:09 +01:00
Tamo
786b0fabea
implement the progress for almost all the tasks 2024-12-11 16:26:08 +01:00
Tamo
26733c705d
add progress for the task deletion and task cancelation 2024-12-11 16:25:02 +01:00
Tamo
ab75f53efd
update all snapshots 2024-12-11 16:25:02 +01:00