Commit Graph

7468 Commits

Author SHA1 Message Date
curquiza
be300138e4 Add release check when starting latest CI 2022-12-07 12:22:44 +01:00
bors[bot]
2ed6017603
Merge #3204
3204: Bring back v0.30.1 changes to `main` r=curquiza a=curquiza

I was not able to just import `release-v0.30.1` to `main`, see:
<img width="1371" alt="Capture d’écran 2022-12-06 à 20 03 50" src="https://user-images.githubusercontent.com/20380692/206000844-b39b3063-7da2-475f-b3e4-1791c39a7c2f.png">

So I cherry-picked the commits.

⚠️ ⚠️ ⚠️ I had a git conflict here

<img width="730" alt="Capture d’écran 2022-12-06 à 20 09 04" src="https://user-images.githubusercontent.com/20380692/206001007-f56bc28f-c0b1-46a0-bb60-cce4e73b9584.png">


⚠️ ⚠️ ⚠️ Check out carefully how I fixed it


Co-authored-by: curquiza <curquiza@users.noreply.github.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-12-07 11:08:37 +00:00
Kerollmops
c1337f9e08
Update dump snap to new version 2022-12-07 11:48:29 +01:00
bors[bot]
9acac28574
Merge #3128
3128: Bumps cargo_toml version to most up to date r=curquiza a=colbsmcdolbs

# Pull Request

## Related issue
Fixes #3127

## What does this PR do?
- The README of this repository declares that one package is not up to date. In order to ensure Due Diligence, I have bumped the version number of the package. No test failures running on Windows.

## PR checklist
Please check if your PR fulfills the following requirements:
- [X] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [X] Have you read the contributing guidelines?
- [X] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Colby Allen <colbyjayallen@gmail.com>
2022-12-07 10:31:25 +00:00
jiangbo212
cb1d184904 fmt fix 2022-12-07 17:04:24 +08:00
jiangbo212
2841b09789
Merge branch 'meilisearch:main' into fix-3037 2022-12-07 16:30:21 +08:00
jiangbo212
35f3dd68b6 error change and tokio file use change 2022-12-07 16:20:36 +08:00
Kerollmops
f1de3aa75a Make the tests use MB to trigger page size issues 2022-12-06 20:10:10 +01:00
Kerollmops
e4e4370a3c Clamp the databases size to the page size 2022-12-06 20:09:49 +01:00
Kerollmops
24c79b79f9 Bump milli to v0.37.1 2022-12-06 20:05:52 +01:00
curquiza
5db7c4057c Update version for the next release (v0.30.1) in Cargo.toml files 2022-12-06 20:05:46 +01:00
Tamo
f53bdc4320
update the contributing.md 2022-12-06 17:41:05 +01:00
bors[bot]
0a301b5f88
Merge #723
723: Fix bug in handling of soft deleted documents when updating settings r=Kerollmops a=loiclec

# Pull Request

## Related issue
Fixes (partially, until merged into meilisearch) https://github.com/meilisearch/meilisearch/issues/3021

## What does this PR do?
This PR fixes the bug where a `missing key in documents database` internal error message could appear when indexing documents.

When updating the settings, before clearing the database and before creating the transform output, we now modify the `ExternalDocumentsIds` structure to get rid of all references to soft deleted document ids in its FSTs.

It used to be that updating the settings would clear the soft-deleted document ids, but keep the original `ExternalDocumentsIds` structure. As a consequence of this, when processing a future document addition, we could wrongly believe that a document was being replaced when, in fact, it was a completely new document. See the tests `bug_3021_first`, `bug_3021_second`, and `bug_3021` for a minimal test case that would have reproduced the issue.
 
We need to take special care to:
- evaluate how users should update to v0.30.1 (containing this fix): dump? reimporting all documents from scratch?
- understand IF/HOW this bug could have caused duplicate documents to be returned 
- and evaluate the correctness of the fix, of course :)


Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
2022-12-06 14:37:38 +00:00
Loïc Lecrenier
a993b68684 Cargo fmt >:-( 2022-12-06 15:22:10 +01:00
Loïc Lecrenier
80c7a00567 Fix compilation error in tests of settings update 2022-12-06 15:19:26 +01:00
Loïc Lecrenier
67d8cec209 Fix bug in handling of soft deleted documents when updating settings 2022-12-06 15:09:19 +01:00
bors[bot]
2867d2e91a
Merge #3190
3190: Fix the dump date-import of the dumpv4 r=irevoire a=irevoire

# Pull Request
After merging https://github.com/meilisearch/meilisearch/pull/3012 I realized that the tests on the date of the dump-v4 were still ignored, thus, I fixed them and then noticed #3012 wasn't working properly.

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/2987 a second time


`@funilrys` since you wrote most of the code you might be interested, but don't feel obligated to review this code. 
Someone from the team will double-check it works 😁 

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-12-06 10:47:00 +00:00
bors[bot]
2a846aaae7
Merge #719
719: Add more members of `filter_parser` to `milli::` & `From<&str>` implementation for `Token` r=Kerollmops a=GregoryConrad

## What does this PR do?
The current `milli::Filter` and `milli::FilterCondition` APIs require working with some members of `filter_parser` directly that `milli::` does *not* re-export to its users (at least when not parsing input using `parse`). Also, using `filter_parser` does not make sense when using milli from an embedded context where there is no query to parse.

Instead of reworking `milli::Filter` and `milli::FilterCondition`, this PR adds two non-breaking changes that ease the use of milli:
- Re-exports more members of the dependent version of `filter_parser` in `milli`
- Implements `From<&str>` for `filter_parser::Token`
  - This will also allow some basic tests that need to create a `Token` from a string to avoid some boilerplate.

In conjunction, both of these will allow milli users to easily create a `Token` from a `&str` without needing to add `filter_parser` as an extra dependency.

Note: I wanted to use `FromStr` for the `From` implementation; however, it requires returning a `Result` which is not needed for the conversion. Thus, I just left it as `From<&str>`.

Co-authored-by: Gregory Conrad <gregorysconrad@gmail.com>
2022-12-06 10:36:00 +00:00
bors[bot]
1458a12531
Merge #3197
3197: Revert "Upgrade alpine 3.16 to 3.17" r=irevoire a=curquiza

Reverts meilisearch/meilisearch#3189

Because `rust:alpine3.17` does not exist, and our scheduled CI failed: https://github.com/meilisearch/meilisearch/actions/runs/3626327181

`@ivanionut` for your information, I'm sorry I should have better checked before accepting the PR, this is my bad


Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
2022-12-06 10:25:11 +00:00
Clémentine Urquizar - curqui
cbb8d0f97b
Revert "Upgrade alpine 3.16 to 3.17" 2022-12-06 11:09:57 +01:00
Tamo
bef81065f9 return the same time in case we didn't found a created or updated at 2022-12-06 11:03:23 +01:00
Tamo
180511795b
Update dump/src/reader/v4/mod.rs fix typo
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2022-12-06 10:53:43 +01:00
bors[bot]
3bef6e6690
Merge #3175
3175: Rename dump command from --dumps-dir to --dump-dir r=dureuill a=dureuill

# Pull Request

## Related issue
Fixes #3132 

## What does this PR do?
- Rename the dump commands, env variables and default config

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2022-12-06 09:49:42 +00:00
bors[bot]
d6eacb2aac
Merge #722
722: Geosearch for zero radius r=irevoire a=amab8901

# Pull Request

## Related issue
Fixes #3167 (https://github.com/meilisearch/meilisearch/issues/3167)

## What does this PR do?
- allows Geosearch with zero radius to return the specified location when the coordinates match perfectly (instead of returning nothing). See link for more details.
- new attempt on https://github.com/meilisearch/milli/pull/713

## PR checklist
Please check if your PR fulfills the following requirements:
- [ X ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ X ] Have you read the contributing guidelines?
- [ X ] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: amab8901 <amab8901@protonmail.com>
Co-authored-by: Tamo <irevoire@protonmail.ch>
2022-12-05 19:57:08 +00:00
Tamo
212dbfa3b5
Update milli/src/search/facet/filter.rs 2022-12-05 20:56:21 +01:00
amab8901
456da5de9c Geosearch for zero radius 2022-12-05 20:11:46 +01:00
bors[bot]
46e26ab550
Merge #720
720: Make soft deletion optional in document addition and deletion + add lots of tests r=irevoire a=loiclec

# Pull Request

## What does this PR do?
When debugging recent issues, I created a few unit tests in the hopes reproducing the bugs I was looking for. In the end, I didn't find any, but I thought it would still be good to keep those tests. 

More importantly, I added a field to the `DeleteDocuments` and `IndexDocuments` builders, called `disable_soft_deletion`. If set to `true`, the indexing/deletion will never add documents to the `soft_deleted_documents_ids` and instead perform a real deletion of the documents from the databases.

For the new tests, I have:
- Improved the insta-snapshot format of the `external_documents_ids` structure
- Added more tests for the facet DB indexing, deletion, and search algorithms, making sure to test them when the facet DB contains strings (instead of numbers) as well.
- Added more tests for the incremental indexing of the prefix proximity databases. For example, to see if documents are replaced correctly and if common prefixes are deleted correctly.
- Added tests that mix soft deletion and hard deletion, including when processing batches of document updates. 


Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
2022-12-05 18:26:01 +00:00
bors[bot]
9b23885e85
Merge #3188
3188: re-enable the dump test on the dates r=irevoire a=irevoire

I just noticed that we have the real date in the dump-v1 contrarily to the dump-v2/3/4/5, thus we can ensure it doesn't change unexpectedly 👍 

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-12-05 18:10:22 +00:00
bors[bot]
8b46093117
Merge #3189
3189: Upgrade alpine 3.16 to 3.17 r=curquiza a=ivanionut

Upgrade alpine 3.16 to 3.17

Co-authored-by: Ivan Ionut <ivan.ionut@gmail.com>
2022-12-05 17:39:10 +00:00
Tamo
9c89e3dadc
uncomment more test for the dump v4 2022-12-05 18:15:29 +01:00
Tamo
b0cf431614
Fix the dump date-import of the dumpv4 2022-12-05 18:08:35 +01:00
Ivan Ionut
afe520a67e Upgrade alpine 3.16 to 3.17 2022-12-05 17:49:15 +01:00
Tamo
688911ed34
re-enable the dump test on the dates 2022-12-05 17:05:37 +01:00
bors[bot]
776af129bf
Merge #3012
3012: Extract the dates out of the dumpv4. r=irevoire a=funilrys

Hi there, 

please review this PR that tries to fix #2987. I'm still learning Rust and I found that #2987 is an excellent way for me to read and learn what others do with Rust. So please excuse my semantics ...

Stay safe and healthy.

---

# Pull Request

This patch possibly fixes #2987.

This patch introduces a way to fill the IndexMetadata.created_at and IndexMetadata.updated_at keys from the tasks events. This is done by reading the creation date of the first event (created_at) and the creation date of the last event (updated_at).


## Related issue
Fixes #2987

## What does this PR do?
- Extract the dates out of the dumpv4.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: funilrys <contact@funilrys.com>
2022-12-05 15:57:07 +00:00
Louis Dureuil
492fd2829a
use a consistent dump directory name in tests
changed from 'dump' to 'dumps' to be consistent with the default settings

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-12-05 16:56:28 +01:00
bors[bot]
ffa6d1ed4e
Merge #3186
3186: Update mini-dashboard to v0.2.4 r=curquiza a=mdubus



Co-authored-by: Morgane Dubus <30866152+mdubus@users.noreply.github.com>
2022-12-05 14:45:27 +00:00
Morgane Dubus
293efb7485
Update Cargo.toml 2022-12-05 14:54:01 +01:00
Loïc Lecrenier
cda4ba2bb6 Add document import tests 2022-12-05 12:02:49 +01:00
Loïc Lecrenier
ae59d37b75 Improve insta-snap of the external document ids 2022-12-05 10:51:02 +01:00
Loïc Lecrenier
f2cf981641 Add more tests and allow disabling of soft-deletion outside of tests
Also allow disabling soft-deletion in the IndexDocumentsConfig
2022-12-05 10:51:01 +01:00
jiangbo212
6766712840 fmt fix 2022-12-04 23:05:34 +08:00
jiangbo212
980776b646 test fail fix 2022-12-04 22:31:23 +08:00
jiangbo212
6bdd37beb8 tokio file write update 2022-12-04 18:25:06 +08:00
Gregory Conrad
50954d31fa feat: Re-export Span and Token to milli:: 2022-12-03 13:37:33 -05:00
Gregory Conrad
1b5b5778c1 feat: Add From<&str> implementation for Token 2022-12-03 13:13:41 -05:00
funilrys
8b6eba4f0b
Apply fmt. 2022-12-03 17:47:02 +01:00
funilrys
e510ace179
fixup! Re-open tasks queue. 2022-12-03 17:41:33 +01:00
funilrys
f056fc118f
Re-open tasks queue.
Indeed, before this patch, I was (probably) breaking every usage
of the tasks BufReader. This patch solves the issue by reopening
the the tasks file every time its needed.
2022-12-03 17:29:41 +01:00
jiangbo212
5a770ffe47 test fail fix 2022-12-03 22:48:38 +08:00
jiangbo212
7b08d700f7 requested changes fix 2022-12-03 18:52:20 +08:00