Commit Graph

258 Commits

Author SHA1 Message Date
mpostma
d263f762bf feat(http): accept empty document additions
wip
2022-01-13 12:46:56 +01:00
Irevoire
dfaeb19566
fix(dump): Fix the import of dumps when there is no data.ms 2022-01-13 12:30:58 +01:00
bors[bot]
d0aa5f747c
Merge #2067
2067: chore(all): fix rust edition r=irevoire a=MarinPostma

I hadn't correctly set the rust edition in my previous pr, and cargo was returning a warning. This time I followed this guide: https://doc.rust-lang.org/edition-guide/editions/transitioning-an-existing-project-to-a-new-edition.html


Co-authored-by: mpostma <postma.marin@protonmail.com>
2022-01-12 13:32:42 +00:00
mpostma
f6d53e03f1 chore(http): migrate from structopt to clap3 2022-01-12 14:07:19 +01:00
mpostma
3ecebd15ee chore(all): fix rust edition 2022-01-12 11:14:50 +01:00
bors[bot]
5d48f72ade
Merge #2065
2065: MeiliSearch v0.25.0: `stable` -> `main` r=curquiza a=curquiza



Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: many <maxime@meilisearch.com>
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
Co-authored-by: Maxime Legendre <maximelegendre@MacBook-Pro-de-Maxime.local>
Co-authored-by: Maxime Legendre <maximelegendre@mbp-de-maxime.home>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-01-11 16:30:22 +00:00
bors[bot]
1818026a84
Merge #2057
2057: fix(dump): Uncompress the dump IN the data.ms  r=irevoire a=irevoire

When loading a dump with docker, we had two problems.
After creating a tempdirectory, uncompressing and re-indexing the dump:
1. We try to `move` the new “data.ms” onto the currently present
   one. The problem is that if the `data.ms` is a mount point because
   that's what peoples do with docker usually. We can't override
   a mount point, and thus we were throwing an error.
2. The tempdir is created in `/tmp`, which is usually quite small AND may not
   be on the same partition as the `data.ms`. This means when we tried to move
   the dump over the `data.ms`, it was also failing because we can't move data
   between two partitions.
------------------
1 was fixed by deleting the *content* of the `data.ms` and moving the *content*
of the tempdir *inside* the `data.ms`. If someone tries to create volumes inside
the `data.ms` that's his problem, not ours.
2 was fixed by creating the tempdir *inside* of the `data.ms`. If a user mounted
its `data.ms` on a large partition, there is no reason he could not load a big
dump because his `/tmp` was too small. This solves the issue; now the dump is
extracted and indexed on the same partition the `data.ms` will lay.

fix #1833

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-01-10 17:57:16 +00:00
Tamo
c9c7da3626
fix(dump): Uncompress the dump IN the data.ms
When loading a dump with docker, we had two problems.
After creating a tempdirectory, uncompressing and re-indexing the dump:
1. We try to `move` the new “data.ms” onto the currently present
   one. The problem is that if the `data.ms` is a mount point because
   that's what peoples do with docker usually. We can't override
   a mount point, and thus we were throwing an error.
2. The tempdir is created in `/tmp`, which is usually quite small AND may not
   be on the same partition as the `data.ms`. This means when we tried to move
   the dump over the `data.ms`, it was also failing because we can't move data
   between two partitions.
==============
1 was fixed by deleting the *content* of the `data.ms` and moving the *content*
of the tempdir *inside* the `data.ms`. If someone tries to create volumes inside
the `data.ms` that's his problem, not ours.
2 was fixed by creating the tempdir *inside* of the `data.ms`. If a user mounted
its `data.ms` on a large partition, there is no reason he could not load a big
dump because his `/tmp` was too small. This solves the issue; now the dump is
extracted and indexed on the same partition the `data.ms` will lay.

fix #1833
2022-01-10 14:56:03 +01:00
Marin Postma
5fb4ed60e7 chore(all) set rust edition to 2021 2022-01-06 13:30:45 +01:00
Tamo
eea483c470
fix(dump): Fix the loading of dump with empty indexes 2022-01-05 15:08:21 +01:00
bors[bot]
2f3faadcbf
Merge #2034
2034: Fix typo r=curquiza a=curquiza

Fix `Meilisearch` typo into `MeiliSearch`

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2022-01-03 09:40:56 +00:00
Clémentine Urquizar
f529c46598
Fix typo in error messages and comments 2021-12-21 16:01:38 +01:00
Maxime Legendre
1ba49d2ddb Bug(FS): Consider empty pre-created directory as unexisting DB 2021-12-21 15:30:11 +01:00
Maxime Legendre
a845cd8880 Fix(auth): Forbid index creation on alternates routes
Forbid index creation on alternates routes when the action `index.create` is not given

fix #2024
2021-12-20 14:48:18 +01:00
Marin Postma
b28a465304 bug(lib): drop env on last use
fixes the `too many open files` error when running tests by closing the
environment on last drop
2021-12-16 10:57:55 +01:00
bors[bot]
845d3114ea
Merge #2008
2008: bug(lib): fix get dumps bad error code r=curquiza a=MarinPostma

fix bad error code being returned whet getting a dump status, and add a test
close #1994

Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-12-15 18:58:17 +00:00
Marin Postma
7ddab7ef31 bug(lib): fix get dumps bad error code 2021-12-15 16:58:05 +01:00
Marin Postma
d534a7f7c8 bug(lib): ignore primary if already set on document addition 2021-12-15 14:58:37 +01:00
many
ee7970f603 feat(auth): Extend API keys
- Add API keys in snapshots
- Add API keys in dumps
- Rename action indexes.add to indexes.create
- fix QA #1979

fix #1979
fix #1995
fix #2001
fix #2003
related to #1890
2021-12-14 17:33:39 +01:00
Clémentine Urquizar
ae73386723
Update version for the next release (v0.25.0) 2021-12-07 14:00:43 +01:00
Marin Postma
23e35fa526 feat(dumps): drop dump V1 support 2021-12-07 10:36:27 +01:00
many
ae2b0e7aa7
Use milli reexported tokenizer instead of importing meilisearch-tokenizer dependency 2021-12-06 17:18:28 +01:00
bors[bot]
948615537b
Merge #1965
1965: Reintroduce engine version file r=MarinPostma a=irevoire

Right now if you boot up MeiliSearch and point it to a DB directory created with a previous version of MeiliSearch the existing indexes will be deleted. This [used to be](51d7c84e73) prevented by a startup check which would compare the current engine version vs what was stored in the DB directory's version file, but this functionality seems to have been lost after a few refactorings of the code.

In order to go back to the old behavior we'll need to reintroduce the `VERSION` file that used to be present; I considered reusing the `metadata.json` file used in the dumps feature, but this seemed like the simpler and more approach. As the intent is just to restore functionality, the implementation is quite basic. I imagine that in the future we could build on this and do things like compatibility across major/minor versions and even migrating between formats.

This PR was made thanks to `@mbStavola` and is basically a port of his PR #1860 after a big refacto of the code #1796.

Closes #1840

Co-authored-by: Matt Stavola <m.freitas@offensive-security.com>
2021-12-06 13:39:37 +00:00
Matt Stavola
a0e129304c
feat(lib): Reintroduce engine version file
Right now if you boot up MeiliSearch and point it to a DB directory created with a previous version of MeiliSearch the existing indexes will be deleted. This used to be prevented by a startup check which would compare the current engine version vs what was stored in the DB directory's version file, but this functionality seems to have been lost after a few refactorings of the code.
In order to go back to the old behavior we'll need to reintroduce the VERSION file that used to be present; I considered reusing the metadata.json file used in the dumps feature, but this seemed like the simpler and more approach. As the intent is just to restore functionality, the implementation is quite basic. I imagine that in the future we could build on this and do things like compatibility across major/minor versions and even migrating between formats.

This PR was made thanks to @mbStavola

Closes #1840
2021-12-06 14:30:56 +01:00
many
ffefd0caf2
feat(auth): API keys
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0085-api-keys.md

- Add tests on API keys management route (meilisearch-http/tests/auth/api_keys.rs)
- Add tests checking authorizations on each meilisearch routes (meilisearch-http/tests/auth/authorization.rs)
- Implement API keys management routes (meilisearch-http/src/routes/api_key.rs)
- Create module to manage API keys and authorizations (meilisearch-auth)
- Reimplement GuardedData to extend authorizations (meilisearch-http/src/extractors/authentication/mod.rs)
- Change X-MEILI-API-KEY by Authorization Bearer (meilisearch-http/src/extractors/authentication/mod.rs)
- Change meilisearch routes to fit to the new authorization feature (meilisearch-http/src/routes/)

- close #1867
2021-12-06 09:52:41 +01:00
Marin Postma
a30e02c18c feat(all): Task store
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0060-refashion-updates-apis.md

linked PR:

- #1889
- #1891
- #1892
- #1902
- #1906
- #1911
- #1914
- #1915
- #1916
- #1918
- #1924
- #1925
- #1926
- #1930
- #1936
- #1937
- #1942
- #1944
- #1945
- #1946
- #1947
- #1950
- #1951
- #1957
- #1959
- #1960
- #1961
- #1962
- #1964

- https://github.com/meilisearch/milli/pull/414
- https://github.com/meilisearch/milli/pull/409
- https://github.com/meilisearch/milli/pull/406
- https://github.com/meilisearch/milli/pull/418

- close #1687
- close #1786
- close #1940
- close #1948
- close #1949
- close #1932
- close #1956
2021-12-02 20:14:42 +01:00
bors[bot]
c9f3726447
Merge #1893
1893: Make matches work with numerical value r=MarinPostma a=Thearas

# Pull Request

## What does this PR do?

Implement #1883.

I have test this PR with unit test. It appears to be working properly:
![image](https://user-images.githubusercontent.com/44015907/141141082-dad8cd18-e803-408f-ad6a-c7a212b7ec88.png)

PTAL `@curquiza` 

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?


Co-authored-by: Thearas <thearas850@gmail.com>
2021-11-29 13:38:57 +00:00
Thearas
ac5535055f Make matches work with numerical value 2021-11-10 23:10:30 +08:00
many
cc6306c0e1
Update milli version 2021-11-04 14:57:45 +01:00
many
b664a46e91
Update milli version 2021-11-03 16:11:20 +01:00
many
06e6eaa7b4
Remove useless Facet variant 2021-11-03 16:11:09 +01:00
many
30a094cbb2
Change lacking errors 2021-11-03 14:33:33 +01:00
Tamo
76a4f86e0c
rename user-id to instance-uid 2021-10-29 17:25:52 +02:00
Tamo
c752c14c46
refactorize the dump and snapshot 2021-10-29 17:25:51 +02:00
Tamo
87a8bf5e96
write and load the user-id in the dumps 2021-10-29 17:25:51 +02:00
bors[bot]
cf67964133
Merge #1848
1848: Error format and Definition r=MarinPostma a=ManyTheFish



Co-authored-by: many <maxime@meilisearch.com>
2021-10-28 14:15:35 +00:00
many
3a29cbf0ae
Use milli v0.20.0 2021-10-28 15:59:06 +02:00
many
cbaca2b579
Fix PR comments 2021-10-28 15:42:42 +02:00
Clémentine Urquizar
a76d9b15c9
Update version for the next release (v0.24.0) 2021-10-28 12:24:49 +02:00
many
59636fa688
Pimp error where no document is provided 2021-10-28 12:13:51 +02:00
many
7464720426
Fix some errors 2021-10-28 10:47:59 +02:00
bors[bot]
d905bbf961
Merge #1787
1787: Handle empty dump r=MarinPostma a=irevoire

Fixes #1701

Co-authored-by: Tamo <tamo@meilisearch.com>
2021-10-27 12:47:45 +00:00
many
61c15b69fb
Change malformed_payload error 2021-10-27 11:13:12 +02:00
marin postma
4ac005b094
optimize document transform
fix error types

bump milli
2021-10-26 13:51:15 +02:00
Tamo
5e3a53b576
fix a bug in the generation of empty dumps 2021-10-25 14:17:57 +02:00
many
36ab7b3ebd
Fix small typo 2021-10-18 14:17:32 +02:00
many
b4038597ba
Keep persisting tmp files in database directory and put non-persisting tmp files in default tmp dir 2021-10-18 14:16:35 +02:00
bors[bot]
79817bd465
Merge #1813
1813: Apply highlight tags on numbers in the formatted search result output r=irevoire a=Jhnbrn90

This is my first ever Rust related PR. 

As described in #1758, I've attempted to highlighting numbers correctly under the `_formatted` key.

Additionally, I added a test which should assert appropriate highlighting. 

I'm open to suggestions and improvements. 


Co-authored-by: John Braun <john@brn.email>
2021-10-18 09:05:01 +00:00
John Braun
be75426e64 Apply formatting according code style guidelines 2021-10-15 21:32:29 +02:00
Clémentine Urquizar
0f342ac46e
Update MeiliSearch version 2021-10-12 16:43:31 +02:00
Clémentine Urquizar
29ac324e90
Update milli version to v0.17.3 2021-10-12 16:12:16 +02:00
Clémentine Urquizar
3edbc74430
Merge branch 'main' into stable 2021-10-11 18:30:10 +02:00
Clémentine Urquizar
60473637fe
Update milli version 2021-10-11 16:21:19 +02:00
Tom Parker-Shemilt
6c46fbbc57 Remove memmap dependency 2021-10-10 22:33:40 +01:00
John Braun
3756f5a0ca Add test for highlighting numbers 2021-10-08 15:07:45 +02:00
John Braun
5b4e4bb858 Highlight numbers (int) as string in formatted JSON 2021-10-08 15:07:15 +02:00
many
a92a0c3ed3
Log the error instead of returning it when deletion fails 2021-10-07 17:38:22 +02:00
many
0774b1efa5
Close index's heed environment when index is deleted 2021-10-07 17:09:41 +02:00
many
7fc7eb7457
Make sure to remove newly created index if uid is already taken 2021-10-07 16:49:21 +02:00
Tamo
66dbd3cd34
makes clippy happy 2021-10-06 17:39:04 +02:00
bors[bot]
ddbcf449da
Merge #1763
1763: Index tests r=MarinPostma a=MarinPostma

This pr aims to test more thorougly the usage on index in the meilisearch database, by writing unit tests.

work included:
- [x] Create index mock and stub methods
- [x] Test snapshot creation
- [x] Test Dumps
- [x] Test search

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-10-06 14:39:53 +00:00
mpostma
9fa61439b1 fix clippy warning & unsafety 2021-10-06 14:51:46 +02:00
mpostma
a38215de98 edit documentation 2021-10-06 14:35:18 +02:00
mpostma
85b5260d9d simple search unit test 2021-10-06 14:20:05 +02:00
mpostma
4b4ebad9a9 test dumps 2021-10-06 14:10:26 +02:00
mpostma
ece4c739f4 update store tests 2021-10-06 14:10:26 +02:00
mpostma
85ae34cf9f test snapshots 2021-10-06 14:10:23 +02:00
mpostma
0448f0ce56 handle panic in stubs 2021-10-06 14:09:04 +02:00
mpostma
4835d82a0b implement index mock 2021-10-06 14:09:01 +02:00
Tamo
fca686e7f8
bump meilisearch 2021-10-04 13:52:37 +02:00
mpostma
4eb3817b03
missing payload error 2021-09-30 16:58:13 +02:00
mpostma
ddd40d87a7
malformed payload error 2021-09-30 16:58:13 +02:00
Kerollmops
6a691db7f8
Do not commit transaction on failed updates 2021-09-30 15:46:03 +02:00
mpostma
ee372a7b30
implement new dump v2 2021-09-30 14:49:13 +02:00
mpostma
66f39aaa92
fix dump v3 2021-09-30 14:49:13 +02:00
mpostma
03af99650d
fix dumpv1 2021-09-30 14:49:13 +02:00
Clémentine Urquizar
b17dae9ac0
Update version for the next release (v0.23.0) 2021-09-29 18:40:35 +02:00
mpostma
311933614e bump milli to v0.17.0 2021-09-29 15:44:54 +02:00
mpostma
8fa6502b16 review changes 2021-09-29 14:17:41 +02:00
mpostma
1f537e1b60 jsonl support 2021-09-29 11:28:02 +02:00
mpostma
911630000f split csv and json document routes 2021-09-29 00:12:25 +02:00
mpostma
6e8a3fe8de move csv parsing to document_formats 2021-09-28 22:58:48 +02:00
many
2a14948123 Use an existing revision of milli 2021-09-28 22:30:34 +02:00
many
61e5eed493 Call csv specialized function 2021-09-28 22:29:26 +02:00
many
d30830a55c Add csv deserializer for documents 2021-09-28 22:28:13 +02:00
mpostma
102c46f88b clippy + fmt 2021-09-28 22:22:59 +02:00
mpostma
5fa9bc67d7 remove unused dependencies 2021-09-28 22:16:18 +02:00
mpostma
3503fbf7fe re-export milli from meilisearch_lib 2021-09-28 22:08:03 +02:00
mpostma
1cc733f801 fix get_info 2021-09-28 22:02:04 +02:00
mpostma
7a27cbcc78 rename RegisterUpdate to store::Update 2021-09-28 20:20:13 +02:00
mpostma
6f8e670dee move json reader to document_formats module 2021-09-28 20:13:26 +02:00
mpostma
df4e9f4e1e restore dump v1 2021-09-28 19:49:25 +02:00
mpostma
3747f5bdd8 replace unwraps with correct error 2021-09-28 19:29:14 +02:00
mpostma
692c676625 fix tests 2021-09-28 18:57:36 +02:00
mpostma
bcaee4d179 fix uuid store size 2021-09-28 18:17:56 +02:00
Tamo
539a57026d
fix the sort error messages 2021-09-28 14:50:26 +02:00
Tamo
654f49ccec
[WIP] put milli on branch main 2021-09-28 14:50:26 +02:00
Tamo
c1376a9f2a
add the geosearch to Meilisearch 2021-09-28 14:50:26 +02:00
mpostma
9ac999ca59 remove uuid resolver and index actor 2021-09-28 12:00:35 +02:00
mpostma
6a1964f146 restore dumps 2021-09-28 11:59:55 +02:00
mpostma
90018755c5 restore snapshots 2021-09-27 16:48:03 +02:00
mpostma
b9d189bf12 restore document deletion routes 2021-09-24 15:21:07 +02:00
mpostma
c32012c44a restore settings updates 2021-09-24 14:55:57 +02:00
mpostma
42a6260b65 introduce index resolver 2021-09-24 11:53:11 +02:00
mpostma
5353be74c3 refactor index actor 2021-09-22 15:07:04 +02:00
mpostma
12542bf922 refactor update actor 2021-09-22 11:52:50 +02:00
mpostma
def737edee refactor uuid resolver 2021-09-22 10:49:59 +02:00
mpostma
60518449fc split meilisearch-http and meilisearch-lib 2021-09-21 13:23:22 +02:00