4796: Generate vectors in dumps r=dureuill a=dureuill
# Pull Request
## What does this PR do?
1. Add an Index::embeddings method to compute the embeddings of a document
2. Write generated vectors in dumps
3. Remove generated vectors when importing dumps
4. Cherry pick the `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION` workaround so that the older CI can still build
## Manual Tests (TODO)
- [ ] Import a dump from a v1.8.3 into a v1.8.4 successfully
- [x] Import a dump from a v1.8.4 into a v1.8.4 successfully
- [x] Import a dump from a v1.8.4 into a v1.9.0 successfully
- [x] generated vectors are not regenerated
- [x] user provided vectors are still available
- [x] generated vectors still have the correct value
- [x] updating a document with generated vectors attempts to regenerate
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
4709: Update version for the next release (v1.8.3) in Cargo.toml r=dureuill a=meili-bot
⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.
Co-authored-by: dureuill <dureuill@users.noreply.github.com>
4707: Only spawn thread pool once r=irevoire a=dureuill
# Pull Request
## Related issue
Fixes#4692
## What does this PR do?
- There was a rayon thread pool of 40 threads that would be spawned multiple times per indexing operation.
- Perhaps due to the sheer number of spawned threads, or to a leak in rayon thread pools, the system was unable to reclaim all the spawned threads at a sufficient rate.
- As a result, the stack for the threads would accumulate and consume virtual memory, and eventually physical memory too.
- Fortunately, the pool can actually be created once and then always reused. This PR performs this change.
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
4681: Fix concurrency issue r=irevoire a=dureuill
# Pull Request
## Related issue
Fixes#4654
## What does this PR do?
- Asynchronously drop permits
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
4688: Update version for the next release (v1.8.2) in Cargo.toml r=dureuill a=meili-bot
⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.
Co-authored-by: dureuill <dureuill@users.noreply.github.com>
4648: Update version for the next release (v1.8.1) in Cargo.toml r=ManyTheFish a=meili-bot
⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.
Co-authored-by: ManyTheFish <ManyTheFish@users.noreply.github.com>
4642: Index the _geo fields when changing the setting while there is already documents in the DB r=ManyTheFish a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4640
Fixes https://github.com/meilisearch/meilisearch/issues/4628
## What does this PR do?
- Add an integration test that first indexes the document and then changes the settings
- Fix `extract_geo_point` by detecting if the `_geo` field has been faceted in this setting change and index all documents
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
4617: Destructure `EmbedderOptions` so we don't miss some options r=dureuill a=dureuill
# Pull Request
## Related issue
#4595 was caused by the code not destructuring the embedder options.
## What does this PR do?
This PR adds the missing `url` parameter for ollama, and makes sure similar issue cannot happen in the future
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
4597: Fix embeddings settings update r=ManyTheFish a=ManyTheFish
# Pull Request
- add some conditions reducing the work done when changing the settings
- add some benchmarks on embedders
## Related issue
Fixes#4585
Co-authored-by: ManyTheFish <many@meilisearch.com>
4593: Stop crashing when panic occurs in thread pool r=ManyTheFish a=Kerollmops
This PR fixes#4362 by introducing a new boolean to catch panics in the rayon thread pool. The boolean is read after performing the operations in rayon, and the indexation process is stopped. This first version doesn't expose the panic message but marks the task as failed.
The current implementation exposes a `ThreadPoolNoAbort` wrapper. The `rayon::ThreadPool` has been wrapped to check that nothing went wrong after running the `ThreadPool::install` function. An atomic boolean and some `store/load` logic make the system work efficiently.
Before, Meilisearch was completely crashing...
<img width="1563" alt="Capture d’écran 2024-04-22 à 15 49 02" src="https://github.com/meilisearch/meilisearch/assets/3610253/ce114917-a881-4fbb-85df-c195fcf0c7cb">
Now, it handles the panics correctly and marks the task as failed.
<img width="1558" alt="Capture d’écran 2024-04-22 à 15 42 14" src="https://github.com/meilisearch/meilisearch/assets/3610253/8bd031ef-5e8f-4a12-a91e-c823597a2344">
Co-authored-by: Clément Renault <clement@meilisearch.com>
4576: increase the default search time budget from 150ms to 1.5s r=ManyTheFish a=irevoire
# Pull Request
## Related issue
Fixes#4575
## What does this PR do?
- increase the default search time budget from 150ms to 1.5s
Co-authored-by: Tamo <tamo@meilisearch.com>
4581: Always show facet numbers in alpha order in the facet distribution r=ManyTheFish a=Kerollmops
This PR fixes#4559 by making sure that the number facets (facets that come from numbers from the documents) are always displayed in alpha order, even when there is a small amount to display.
The issue was due to some algorithms executed when the number of facet values to display was small. We can see that now, facet values are always displayed correctly.
```json
"facetDistribution": {
"release_year": {
"2010": 1,
"2011": 1,
"2012": 1,
"2013": 1,
"2014": 1,
"2015": 1,
"2016": 1,
"2017": 1,
"2018": 1,
"2019": 19,
"2020": 1,
"2021": 1,
"2022": 1,
"2023": 1,
"2024": 1,
"2025": 1
}
}
```
Co-authored-by: Clément Renault <clement@meilisearch.com>
4580: Update the search logs r=Kerollmops a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4579
## What does this PR do?
- Update the debug implementation of the search query and search results so it’s way smaller and doesn’t display useless information
Co-authored-by: Tamo <tamo@meilisearch.com>
