Tamo
ab75f53efd
update all snapshots
2024-12-11 16:25:02 +01:00
Kerollmops
5622b9607d
Wrap the read NDJSON pass into a tokio blocking
2024-12-11 12:18:36 +01:00
Kerollmops
69c931334f
Fix the error messages categorization with invalid NDJson
2024-12-11 12:02:48 +01:00
Kerollmops
d683f5980c
Do not duplicate NDJson when unecessary
2024-12-11 12:02:48 +01:00
meili-bors[bot]
3b0b9967f6
Merge #5141
...
5141: Use the right amount of max memory and not impact the settings r=curquiza a=Kerollmops
Fixes #5132 . Related to #5125 .
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-09 10:40:46 +00:00
Kerollmops
bcfed70888
Revert "Merge #5125 "
...
This reverts commit 9a9383643f9a6b5ee9ab2ace3e9d63b920d94a53, reversing
changes made to cac355bfa7e72ca3c5c02cacb4f2fcd3f2dd336e.
2024-12-09 10:08:02 +01:00
airycanon
b75f1f4c17
fix tests
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/index-scheduler/src/snapshots/lib.rs/fail_in_process_batch_for_document_deletion/after_removing_the_documents.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fifth_task_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fourth_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/third_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key_batch_wrong_key/second_and_third_tasks_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/all_other_tasks_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/third_task_succeeds.snap
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
# crates/meilisearch/tests/vector/mod.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
2024-12-06 02:03:02 +08:00
airycanon
95ed079761
attach index name in errors
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
2024-12-06 01:12:13 +08:00
meili-bors[bot]
26be5e0733
Merge #5123
...
5123: Fix batch details r=dureuill a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5079
Fixes https://github.com/meilisearch/meilisearch/issues/5112
## What does this PR do?
- Make the processing tasks actually processing in the stats of the batch instead of enqueued
- Stop counting one extra task for all non-prioritized batches in the stats
- Add a test
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-05 15:21:55 +00:00
meili-bors[bot]
1537323eb9
Merge #5119
...
5119: Settings opt out error msg r=Kerollmops a=ManyTheFish
# Pull Request
## Related issue
PRD: https://meilisearch.notion.site/API-usage-Settings-to-opt-out-indexing-features-fff4b06b651f8108ade3f858aeb16b14?pvs=4
## What does this PR do?
Add a new error code and message when the user tries a facet search on an index where the facet search is disabled:
```json
{
"message": "The facet search is disabled for this index",
"code": "facet_search_disabled",
"type": "invalid_request",
"link": "https://docs.meilisearch.com/errors#invalid_facet_search_disabled "
}
```
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-12-05 13:51:11 +00:00
ManyTheFish
a0a3b55700
Change error code
2024-12-05 14:48:29 +01:00
Tamo
214b51de87
try to fix the snapshot on demand flaky test
2024-12-05 14:45:54 +01:00
Kerollmops
9020a50df8
Change the default max memory usage to 5% of the total memory
2024-12-05 10:14:46 +01:00
ManyTheFish
fc1df5793c
fix tests
2024-12-04 14:35:20 +01:00
ManyTheFish
953a82ca04
Add new error message
2024-12-04 11:15:29 +01:00
meili-bors[bot]
054622bd16
Merge #5094
...
5094: Implement a bbqueue channel between the extractors and the writer r=dureuill a=Kerollmops
This PR switches from a bounded crossbeam channel only with allocated entries for the communication between the extractors and the writer to a [BBQueue](https://github.com/jamesmunns/bbqueue )-based system with a Single Producer Single Consumer kind of Circular/Ring Buffers channel.
- [x] Implement the BBQueue channel system...
- [x] with a crossbeam channel to wake up the receiver.
- [x] Manage the BBQueue allocated memory dynamically.
- [x] Support content that doesn't fit in the bbqueues.
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-12-03 08:00:55 +00:00
Louis Dureuil
e905a72d73
remove mimalloc on Windows
2024-12-02 18:13:56 +01:00
Tamo
a439fa3e1a
While spamming the batches route we could see a processing batch becoming missing and then finished, this commit ensures the batches goes from processing to finished directly
2024-12-02 12:02:16 +01:00
meili-bors[bot]
8d33af1dff
Merge #5102
...
5102: Update mini-dashboard to v0.2.16 version r=curquiza a=curquiza
Fixes https://github.com/meilisearch/meilisearch/issues/5093
Fixes this bug: https://github.com/meilisearch/mini-dashboard/issues/563
Co-authored-by: curquiza <clementine@meilisearch.com>
2024-11-28 14:57:27 +00:00
ManyTheFish
90b428a8c3
Apply change requests
2024-11-28 15:16:13 +01:00
curquiza
3dc87f5baa
Update mini-dashboard to v0.2.16 version
2024-11-28 14:33:05 +01:00
ManyTheFish
5383f41bba
Polish test_setting_routes!
2024-11-28 12:04:21 +01:00
ManyTheFish
9f36ffcbdb
Polish make_setting_routes!
2024-11-28 11:44:09 +01:00
ManyTheFish
68c4717e21
Change the settings tests and macros to avoid oversights
2024-11-28 11:34:35 +01:00
Clément Renault
e1e76f39d0
Clean up dependencies
2024-11-27 14:30:34 +01:00
meili-bors[bot]
d0b2c0a523
Merge #5091
...
5091: Settings opt out r=Kerollmops a=ManyTheFish
# Pull Request
Related PRD: https://www.notion.so/meilisearch/API-usage-Settings-to-opt-out-indexing-features-fff4b06b651f8108ade3f858aeb16b14?pvs=4
## Related issue
Fixes #4979
- [x] Add setting opt-out
- [x] Add analytics
- [x] Add tests
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2024-11-26 15:50:28 +00:00
ManyTheFish
2e896f30a5
Fix PR comments
2024-11-26 16:06:33 +01:00
meili-bors[bot]
e241f91285
Merge #5062
...
5062: Fix bugs for v1.12 r=Kerollmops a=ManyTheFish
# Pull Request
## Related issue
Fixes #4984
Fixes https://github.com/meilisearch/meilisearch/issues/4974
Fixes [SDK test](https://github.com/meilisearch/meilisearch/actions/runs/11886701996/job/33118278794 )
## What does this PR do?
- add 3 tests
- fix bugs
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-11-26 08:10:50 +00:00
ManyTheFish
d66dc363ed
Test and implement settings opt-out
2024-11-25 18:23:22 +01:00
Clément Renault
5606679c53
Use the obkv and grenad crates.io versions
2024-11-25 16:24:59 +01:00
meili-bors[bot]
98a785b0d7
Merge #5080
...
5080: Fix getting a single batch through the GET route r=Kerollmops a=dureuill
# Pull Request
## Related issue
Fixes a bug where getting a single batch does not work
Related to #5070
fix by `@Kerollmops`
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-21 17:08:46 +00:00
Louis Dureuil
ba7500998e
Fix getting a single batch through the GET route
2024-11-21 17:59:31 +01:00
meili-bors[bot]
19e6f675b3
Merge #4900
...
4900: Indexer edition 2024 r=Kerollmops a=dureuill
This PR is implementing the indexer edition 2024, largely inspired by [the ideas from this blog post](https://blog.kerollmops.com/meilisearch-is-too-slow ).
Fixes https://github.com/meilisearch/meilisearch/issues/4985
## Features
- Stream-first approach to reading documents.
- Minimum disk write operations.
- RAM usage-first approach to avoid modifying common bitmaps on disk but in memory.
- Reduced LMDB fragmentation by writing entries only once...
- ...computing the final version of the entries in parallel...
- ...and storing them in write-optimized data structures before sending them to the BTree (LMDB).
- Indexing in multiple transactions to improve large dataset support (dumps).
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-11-21 16:19:10 +00:00
ManyTheFish
36962b943b
First batch of PR comment
2024-11-21 16:38:11 +01:00
Tamo
1e694ae432
improve the count of the number of tasks in a batch
2024-11-20 17:48:26 +01:00
Louis Dureuil
bda2b41d11
update snaps after merge
2024-11-20 17:08:30 +01:00
Louis Dureuil
6e6acfcf1b
Merge branch 'main' into indexer-edition-2024
2024-11-20 16:59:58 +01:00
ManyTheFish
4d616f8794
Parse every attributes and filter before tokenization
2024-11-20 15:15:25 +01:00
ManyTheFish
35bbe1c2a2
Add failing test on settings changes
2024-11-20 14:48:12 +01:00
Tamo
a7ac590e9e
implements the reverse query parameter for the batches
2024-11-20 13:29:52 +01:00
Clément Renault
32d0e50a75
Fix all the benchmark compilation errors
2024-11-20 13:16:32 +01:00
Clément Renault
aba8a0e9e0
Fix some tests but not all of them
2024-11-20 13:16:31 +01:00
Tamo
7e379b3d14
remove useless prints
2024-11-20 12:27:12 +01:00
Tamo
56eacd221f
update the tests after the rebase
2024-11-20 10:54:38 +01:00
Tamo
b906e3ed70
improve the way we access the mutex
2024-11-20 10:51:06 +01:00
Tamo
4abcd9c04e
add some stats on the batches
2024-11-20 10:51:06 +01:00
Tamo
5d10c2312b
remove unused file
2024-11-20 10:51:06 +01:00
Tamo
f1d38581e5
add the front end tests on the batches routes
2024-11-20 10:51:06 +01:00
Tamo
d489f5635f
add the mapping between the task and batches
2024-11-20 10:49:23 +01:00
Tamo
a1251c3c83
Implements the get all batches route with filters working
2024-11-20 10:42:55 +01:00
