marin postma
8e4928c7ea
fix tests
2021-06-24 16:36:22 +02:00
marin postma
d078cbf39b
remove authentication middleware
2021-06-24 16:36:21 +02:00
marin postma
561596d8bc
update stats routes
2021-06-24 16:36:18 +02:00
marin postma
549b489c8a
update settings routes
2021-06-24 16:35:48 +02:00
marin postma
1e9f374ff8
update running route
2021-06-24 16:35:12 +02:00
marin postma
817fcfdd88
update keys route
2021-06-24 16:35:12 +02:00
marin postma
fab50256bc
update index routes
2021-06-24 16:35:04 +02:00
marin postma
b044608b25
update health route
2021-06-24 16:32:45 +02:00
marin postma
ce4fb8ce20
update dump route
2021-06-24 16:32:43 +02:00
marin postma
adf91d286b
update documents and search routes
2021-06-24 16:32:15 +02:00
marin postma
0c1c7a3dd9
implement authentication policies
2021-06-24 16:31:30 +02:00
marin postma
5b71751391
policies macros
2021-06-24 16:31:30 +02:00
marin postma
12f6709e1c
move authencation to extractor mod
2021-06-24 16:31:28 +02:00
marin postma
5229f1e220
experimental auth extractor
2021-06-24 16:30:15 +02:00
bors[bot]
b6ca7929eb
Merge #240
...
240: Rework error messages r=irevoire a=MarinPostma
Simplify the error messages, and make them more compliant with legacy Meilisearch.
Basically, stop composing the messages, and simply forward the message of inner errors.
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-24 11:36:11 +00:00
Tamo
ad8d9a97d6
debug the body of every http request
2021-06-24 11:22:11 +02:00
Tamo
36f32f58d4
add the log_level variable to the cli and reduce the log level of milli and grenad
2021-06-24 11:20:52 +02:00
Tamo
b4fd4212ad
reduce the log level of some info!
2021-06-24 11:20:52 +02:00
marin postma
a1d34faaad
decompose error messages
2021-06-24 10:57:28 +02:00
bors[bot]
a2368db154
Merge #239
...
239: Bump milli to 0.6.0 r=MarinPostma a=MarinPostma
fix #231
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-24 08:08:41 +00:00
marin postma
74bb748a4e
bump milli to 0.6.0
2021-06-23 18:40:19 +02:00
bors[bot]
8638c9ab77
Merge #232
...
232: Fix payload size limit r=MarinPostma a=MarinPostma
Fix #223
This was due to the fact that Payload ignores the limit payload size limit. I fixed it by implementing my own `Payload` extractor that checks that the size of the payload is not too large.
I also refactored the `create_app` a bit.
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-23 16:06:08 +00:00
bors[bot]
b676b10cfe
Merge #238
...
238: Fix settings subroutes get r=MarinPostma a=MarinPostma
Fix #225
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-23 15:45:50 +00:00
marin postma
f68c257452
move flush in write_to_file function
2021-06-23 16:49:25 +02:00
marin postma
880fc069bd
remove dbg
2021-06-23 16:49:25 +02:00
marin postma
a838238a63
move payload to own module
2021-06-23 16:49:25 +02:00
marin postma
834995b130
clippy + fmt
2021-06-23 16:49:23 +02:00
marin postma
b000ae7614
remove file if write to update file fails
2021-06-23 16:48:33 +02:00
marin postma
f62779671b
change error message for payload size limit
2021-06-23 16:48:33 +02:00
marin postma
4b292c6e9b
add payload limit to app config
2021-06-23 16:48:33 +02:00
marin postma
1c13100948
implement custom payload
2021-06-23 16:48:31 +02:00
marin postma
71226feb74
refactor create_app macro
2021-06-23 16:47:15 +02:00
bors[bot]
3175f09989
Merge #235
...
235: Fix dump not found error r=MarinPostma a=MarinPostma
fix #233
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-23 14:21:07 +00:00
marin postma
322d6b8cfe
fix serialization bug in settings
2021-06-23 15:25:56 +02:00
marin postma
da36a6b5cd
fix not found error
2021-06-23 15:06:36 +02:00
bors[bot]
f2b2ca6d55
Merge #227
...
227: improve mini dashboard routing r=MarinPostma a=MarinPostma
The dependency we use to statically serve the mini-dashboard used globing to serve the mini-dashboard files. This caused all unfound routes to be caught by the "/" serving the dashboard assets. This fix makes it so that the assets have a dedicated route, and any unfound route is caught by the default service and return a 404.
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-23 13:01:40 +00:00
Clémentine Urquizar
ec3140a29e
Fix clippy
2021-06-23 11:23:57 +02:00
Clémentine Urquizar
00b0a00fc5
Add exhaustiveFacetsCount
2021-06-23 11:05:30 +02:00
Clémentine Urquizar
6d24a4744f
Roll back facetsDistribution
2021-06-23 10:04:01 +02:00
marin postma
b1a5ef0aab
improve mini dashboard routing
2021-06-22 21:49:05 +02:00
marin postma
3456a78552
refactor formatter
...
share the analyzer instance between the formatter and the
compute_matches function
2021-06-22 18:28:20 +02:00
marin postma
eb3d63691a
add tests
2021-06-22 18:12:53 +02:00
marin postma
c4ee937635
optimize fromat string
2021-06-22 18:12:53 +02:00
marin postma
f6d1fb7ac2
fmt
2021-06-22 18:12:53 +02:00
marin postma
97ef4a6c22
implement matches
2021-06-22 18:12:52 +02:00
bors[bot]
db7215eaa9
Merge #213
...
213: Implement all the CLI options r=MarinPostma a=irevoire
closes #206
And I looked into #204 , I fixed some default values and tried to test as many options as possible, and I think the cli is already mostly working.
If someone knows any issues about it, I would like to hear more 🙂
Co-authored-by: Tamo <tamo@meilisearch.com>
2021-06-22 15:04:05 +00:00
bors[bot]
4b37a4a415
Merge #211 #218
...
211: fix index deletion race condition r=MarinPostma a=MarinPostma
Make update store block if the currently processing update is from an index we are trying to delete. This ensure that no write to the index can occur after it has been deleted.
218: Update milli version to v0.5.0 r=MarinPostma a=curquiza
Co-authored-by: marin postma <postma.marin@protonmail.com>
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-06-22 14:36:34 +00:00
marin postma
caa231aebe
fix race condition
2021-06-22 16:09:07 +02:00
marin postma
9cc31c2258
fix get search crop len
2021-06-22 16:01:40 +02:00
Clémentine Urquizar
2e3d85c31a
Update milli version to v0.5.0
2021-06-22 15:57:46 +02:00
bors[bot]
25af262e79
Merge #210
...
210: Error handling r=MarinPostma a=MarinPostma
This pr implements the error handling for meilisearch.
Rather than grouping errors by types, this implementation groups them by scope, each scope enclosing errors from a scope further down, or new errors within this scope. This makes the tracking of the origins of errors easier , and error handling easier at the module level.
All errors that are eventually returned to the user implement the `Into<ResponseError>` trait. `ReponseError` in turn implements the `ErrorCode` trait from `meilisearch-error`.
Some new errors have been introduced with the new engine for which we haven't defined error codes yet. It has been decided with @gmourier that those would return the `internal-error` code until the correct error code is specified.
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-22 13:21:33 +00:00
marin postma
d0ef1ef174
change errors codes
2021-06-22 11:58:01 +02:00
marin postma
9092d35a3c
fix payload error handler
2021-06-21 21:51:38 +02:00
marin postma
2bdaa70f31
invalid update payload returns bad_request
2021-06-21 18:56:22 +02:00
marin postma
f91a3bc6ab
set error content type to json
2021-06-21 18:48:05 +02:00
marin postma
1e4592dd7e
enable errors in updates
2021-06-21 18:42:47 +02:00
Tamo
76727455ca
ignore all the options related to the indexer
2021-06-21 18:13:00 +02:00
Tamo
2097554c09
fix the cli
2021-06-21 16:34:49 +02:00
marin postma
56686dee40
review changes
2021-06-21 13:57:32 +02:00
marin postma
763ee521be
fix rebase errors
2021-06-21 12:11:09 +02:00
marin postma
fa573dabf0
fmt
2021-06-21 12:11:09 +02:00
marin postma
abdf642d68
integrate milli errors
2021-06-21 12:11:08 +02:00
marin postma
0dfd1b74c8
fix tests
2021-06-21 12:11:08 +02:00
marin postma
0d3fb5ee0d
factorize internal error macro
2021-06-21 12:11:08 +02:00
marin postma
02277ec2cf
reintroduce anyhow
2021-06-21 12:11:06 +02:00
marin
8fc12b1526
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: Clément Renault <clement@meilisearch.com>
2021-06-21 11:06:06 +02:00
marin postma
439db1aae0
enable response error for search routes
2021-06-21 11:00:14 +02:00
marin postma
8afbb9c462
enable response error for documents routes
2021-06-21 10:59:41 +02:00
marin postma
5c52a1393f
enable response error for settings routes
2021-06-21 10:59:41 +02:00
marin postma
112cd1787c
change error message for uuid resolver
2021-06-21 10:59:40 +02:00
marin postma
d1550670a8
enable response error for index routes
2021-06-21 10:59:40 +02:00
marin postma
58f9974be4
remove anyhow refs & implement missing errors
2021-06-21 10:59:38 +02:00
marin postma
3a2e7d3c3b
optimize cropping
2021-06-20 16:59:31 +02:00
bors[bot]
c1b6f0e833
Merge #183
...
183: Add cropping and update `_formatted` behavior r=curquiza a=MarinPostma
TODO:
- [x] Solves #5
- [x] Solves #203
- [x] integrate the new milli highlight (according to the query words)
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-06-18 11:18:37 +00:00
Clémentine Urquizar
5d8a21b0de
Fix clippy errors
2021-06-17 18:51:07 +02:00
Clémentine Urquizar
623b71e81e
Fix clippy errors
2021-06-17 18:02:25 +02:00
Clémentine Urquizar
c5c7e76805
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: marin <postma.marin@protonmail.com>
2021-06-17 18:00:02 +02:00
Clémentine Urquizar
e4b3d35ed8
Fix clippy errors
2021-06-17 17:03:43 +02:00
Clémentine Urquizar
33e55bd82e
Refactor the crop
2021-06-17 16:59:01 +02:00
Clémentine Urquizar
9543ab4db6
Use mut instead of returning the hashmap
2021-06-17 13:51:27 +02:00
Clémentine Urquizar
97909ce56e
Use BTreeMap and remove ids_in_formatted
2021-06-16 19:30:06 +02:00
bors[bot]
2f2484e186
Merge #212
...
212: bump milli to 0.4.0 r=MarinPostma a=MarinPostma
Co-authored-by: marin postma <postma.marin@protonmail.com>
2021-06-16 15:42:34 +00:00
bors[bot]
2062b10b79
Merge #209
...
209: Integrate amplitude r=MarinPostma a=irevoire
And merge the sentry and amplitude usage under one “Enable analytics” flag
closes #180
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-06-16 15:25:31 +00:00
Clémentine Urquizar
a0b022afee
Add Cow
2021-06-16 17:25:02 +02:00
marin postma
5a47cef9a8
bump milli to 0.4.0
2021-06-16 17:15:56 +02:00
Clémentine Urquizar
9538790b33
Decompose into two functions
2021-06-16 17:13:21 +02:00
Tamo
4e2568fd6e
disable amplitude on debug build
2021-06-16 17:12:49 +02:00
Clémentine Urquizar
dc5a3d4a62
Use BTreeSet instead of HashSet
2021-06-16 16:20:10 +02:00
Clémentine Urquizar
7b02fdaddc
Rename functions
2021-06-16 14:23:08 +02:00
Irevoire
c0d169e79e
Apply suggestions from code review
...
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-06-16 11:12:46 +02:00
Clémentine Urquizar
9840b5c7fb
Refacto
2021-06-15 18:44:56 +02:00
Clémentine Urquizar
1ef061d92b
Fix clippy errors
2021-06-15 17:40:45 +02:00
Clémentine Urquizar
79a1212ebe
Do intersection with displayed ids instead of checking in loop
2021-06-15 17:40:45 +02:00
Clémentine Urquizar
8d0269fcc4
Create function to create fomatted_options
2021-06-15 17:40:45 +02:00
Clémentine Urquizar
5e656bb58a
Rename parse_facets into parse_filter
2021-06-15 17:40:45 +02:00
Clémentine Urquizar
d9c0190497
Redo to_retrieve_ids
2021-06-15 17:40:45 +02:00
Clémentine Urquizar
5dffe566fd
Remove useless comments
2021-06-15 17:40:45 +02:00
Clémentine Urquizar
b769877183
Make it compatible with the new milli highlighting
2021-06-15 17:40:44 +02:00
Clémentine Urquizar
446b66b0fe
Fix cargo clippy error
2021-06-15 17:40:44 +02:00
Clémentine Urquizar
d0ec081e49
Refacto
2021-06-15 17:40:44 +02:00
Clémentine Urquizar
65130d9ee7
Change crop_length type from Option(usize) to usize
2021-06-15 17:40:44 +02:00
Clémentine Urquizar
638009fb2b
Rename highlighter variable into formatter
2021-06-15 17:40:44 +02:00
Clémentine Urquizar
7f84f59472
Reorganize imports
2021-06-15 17:40:44 +02:00
Clémentine Urquizar
4f8c771bb5
Add new line
2021-06-15 17:40:43 +02:00
Clémentine Urquizar
9e69f33f3c
Fix clippy errors
2021-06-15 17:40:43 +02:00
Clémentine Urquizar
0da8fa115e
Add custom croplength for attributes to crop
2021-06-15 17:40:43 +02:00
Clémentine Urquizar
811bc2f421
Around to previous word
2021-06-15 17:40:43 +02:00
Clémentine Urquizar
caaf8d3f40
Fix tests
2021-06-15 17:40:43 +02:00
Marin Postma
7473cc6e27
implement crop around
2021-06-15 17:40:43 +02:00
Marin Postma
56c9633c53
simple crop before
2021-06-15 17:40:43 +02:00
Clémentine Urquizar
93002e734c
Fix tests
2021-06-15 17:40:42 +02:00
Clémentine Urquizar
60f6d1c373
First version of highlight after refacto
2021-06-15 17:40:42 +02:00
Clémentine Urquizar
a03d9d496e
Fix compilation errors
2021-06-15 17:40:42 +02:00
Marin Postma
7904637893
crop skeleton
2021-06-15 17:40:42 +02:00
Tamo
def1596eaf
Integrate amplitude
...
And merge the sentry and amplitude usage under one “Enable analytics”
flag
2021-06-15 15:36:30 +02:00
marin postma
a780cff8fd
fix clippy warning
2021-06-14 14:53:47 +02:00
Tamo
7cb2dcbdf8
add a comment
2021-06-14 14:47:53 +02:00
Tamo
f068d7f978
makes clippy happy
2021-06-14 14:47:53 +02:00
Tamo
18d4d6097a
implements the synonyms in transplant
2021-06-14 14:47:51 +02:00
bors[bot]
b119bb4ab0
Merge #197
...
197: Update milli (v0.3.1) with filterable attributes r=MarinPostma a=curquiza
Fixes #187 and #70
Also fixes #195
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-06-14 12:19:42 +00:00
bors[bot]
d65b5db97f
Merge #144 #173
...
144: Concurrent update run loop (refactor) r=MarinPostma a=MarinPostma
This PR allows multiple request to the update store to be performed concurently (i.e, one can list updates while an updates in being written to the update store).
173: Convert UpdateStatus to legacy meilisearch format r=MarinPostma a=MarinPostma
Returns the update statuses with the same format as legacy meilisearch.
The number of documents in a document addition/deletion is not known before processing, so it is only returned when the update is `processed`.
close #78
associated milli PR: https://github.com/meilisearch/milli/pull/178
Co-authored-by: marin postma <postma.marin@protonmail.com>
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-06-14 11:30:44 +00:00
Clémentine Urquizar
d4be4d80db
Fix after rebase
2021-06-14 13:27:18 +02:00
Clémentine Urquizar
88bf867a3e
Rename attributes for faceting into filterable attributes
2021-06-14 13:20:43 +02:00
Clémentine Urquizar
aa04124bfc
Add changes according to milli update
2021-06-14 13:20:37 +02:00
Marin Postma
11c81ab4cb
fix tests
2021-06-14 11:17:49 +02:00
marin postma
0f767e3743
conccurrent update run loop
2021-06-14 10:57:14 +02:00
bors[bot]
1e659bb17b
Merge #194
...
194: Bump sentry version r=MarinPostma a=irevoire
closes #102
Co-authored-by: tamo <tamo@meilisearch.com>
2021-06-14 08:34:04 +00:00
Marin Postma
e8bd5ea4e0
convert UpdateStatus to legacy meilisearch format
2021-06-14 10:21:57 +02:00
bors[bot]
d765397c82
Merge #179
...
179: Enable filter paramater during search r=MarinPostma a=MarinPostma
This pr makes the necessary changes to transplant in accordance with the specification on filters.
More precisely, it:
- Removes the `filters` parameter
- Renames `facetFilters` to `filter`
- Allows either a string or an array to be passed to the filter param.
It doesn't allow the mixed syntax, that needs to be handled by milli.
close #81
close #140
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-06-14 08:11:30 +00:00
mpostma
3ef0830c5d
review changes
2021-06-10 16:11:52 +02:00
mpostma
eb7616ca0f
remove dbg
2021-06-10 16:03:48 +02:00
Irevoire
20e1caef47
makes clippy happy
2021-06-10 16:03:48 +02:00
Irevoire
99551fc21b
fix encoding bug
2021-06-10 16:03:48 +02:00
mpostma
2716c1aebb
fix update store lock
2021-06-09 16:19:45 +02:00
mpostma
1a65eed724
fix index creation bug
2021-06-09 11:52:36 +02:00
marin postma
75c0718691
fix update loop infinite loop
2021-06-02 17:29:50 +02:00
Tamo
2d7785ae0c
remove the dump_batch_size option from the CLI
2021-06-01 20:42:06 +02:00
Tamo
d0552e765e
forbid deserialization of Setting<Checked>
2021-06-01 20:41:45 +02:00
marin
df6ba0e824
Apply suggestions from code review
...
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-06-01 11:18:37 +02:00
Marin Postma
6609f9e3be
review edits
2021-05-31 18:41:37 +02:00
Marin Postma
1c4f0b2ccf
clippy, fmt & tests
2021-05-31 16:03:39 +02:00
Marin Postma
10fc870684
improve dump info reports
2021-05-31 15:49:04 +02:00
tamo
dffbaca63b
bump sentry version
2021-05-31 13:59:31 +02:00
Marin Postma
b3c8f0e1f6
fix empty index error
2021-05-31 10:58:51 +02:00
Marin Postma
bc5a5e37ea
fix dump v1
2021-05-31 10:42:31 +02:00
Marin Postma
33c6c4f0ee
add timestamos to dump info
2021-05-30 15:55:17 +02:00
Marin Postma
39c16c0fe4
fix dump import
2021-05-30 12:35:17 +02:00
Marin Postma
1cb64caae4
dump content is now only uuid
2021-05-29 00:08:17 +02:00
Marin Postma
b258f4f394
fix dump import
2021-05-27 14:30:20 +02:00
Marin Postma
c47369839b
dump meta
2021-05-27 10:51:19 +02:00
Marin Postma
b924e897f1
load index dump
2021-05-27 10:27:47 +02:00
Marin Postma
e818c33fec
implement load uuid_resolver
2021-05-26 20:42:09 +02:00
Marin Postma
9278a6fe59
integrate in dump actor
2021-05-25 18:14:11 +02:00
Marin Postma
3593ebb8aa
dump updates
2021-05-25 16:44:58 +02:00
Marin Postma
464639aa0f
udpate actor error improvements
2021-05-25 16:44:58 +02:00
Marin Postma
4acbe8e473
implement index dump
2021-05-25 16:44:58 +02:00
Marin Postma
7ad553670f
index error handling
2021-05-25 16:44:58 +02:00
Marin Postma
2185fb8367
dump uuid resolver
2021-05-25 16:44:54 +02:00
tamo
89846d1656
improve panic message
2021-05-25 15:47:57 +02:00
tamo
1a6dcec83a
crash when the actor have no inbox
2021-05-25 15:23:13 +02:00
tamo
991d8e1ec6
fix the error printing
2021-05-25 10:48:57 +02:00
tamo
49a0e8aa19
use a RwLock instead of a Mutex
2021-05-24 18:19:34 +02:00
tamo
912f0286b3
remove the dump_inner trickery
2021-05-24 18:06:20 +02:00
tamo
dcf29e1081
fix the error handling in case there is a panic while creating a dump
2021-05-24 17:33:42 +02:00
tamo
529f7962f4
handle parallel requests for the dump actor
2021-05-24 15:42:12 +02:00
tamo
8a11c6c429
Implements the legacy behaviour of the dump
...
When asked if a dump exists we check if it's the current dump, and if
it's not then we check on the filesystem for any file matching our
`uid.dump`
2021-05-24 12:35:46 +02:00
tamo
4cbf866821
merge with main
2021-05-12 18:12:37 +02:00
tamo
e0e23636c6
fix the serializer + reformat the file
2021-05-12 17:04:24 +02:00
Marin Postma
295f496e8a
atomic index dump load
2021-05-12 16:21:37 +02:00
tamo
6d837e3e07
the route to create a dump must return a 202
2021-05-11 17:34:34 +02:00
Marin Postma
1b671d4302
fix-snapshot
2021-05-11 13:57:18 +02:00
tamo
c30b32e173
add the criterion attribute when importing dumps from the v1
2021-05-11 13:21:36 +02:00
tamo
9e798fea75
fix the import of dump without unprocessing updates
2021-05-11 13:03:47 +02:00
tamo
384afb3455
fix the way we return the settings
2021-05-11 11:47:04 +02:00
tamo
92a7c8cd17
make clippy happy
2021-05-11 00:27:22 +02:00
tamo
8b7735c20a
move the import of the updates in the v2 and ignore the v1 for now
2021-05-11 00:20:55 +02:00
tamo
7d748fa384
integrate the new Settings in the dumps
2021-05-10 20:48:06 +02:00
tamo
d767990424
fix the import of the updates in the dump
2021-05-10 20:25:12 +02:00
tamo
ef438852cd
fix the v1
2021-05-10 20:25:12 +02:00
tamo
40ced3ff8d
first working version
2021-05-10 20:25:12 +02:00
tamo
5f5402a3ab
provide a way to access the internal content path of all processing State
2021-05-10 20:25:12 +02:00
tamo
26dcb9e66d
bump milli version and fix a performance issue for large dumps
2021-05-10 20:25:12 +02:00
Marin Postma
956012da95
fix dump lock
2021-05-10 20:25:12 +02:00
tamo
24192fc550
fix tests
2021-05-10 20:25:12 +02:00
tamo
efca63f9ce
[WIP] rebase on main
2021-05-10 20:25:09 +02:00
tamo
c3552cecdf
WIP rebase on main
2021-05-10 20:24:18 +02:00
tamo
0f94ef8abc
WIP: dump
2021-05-10 20:24:18 +02:00
tamo
0275b36fb0
[WIP] rebase on main
2021-05-10 20:24:14 +02:00
tamo
1b5fc61eb6
[WIP] rebase on main
2021-05-10 20:23:12 +02:00
tamo
0fee81678e
[WIP] rebase on main
2021-05-10 20:22:18 +02:00
tamo
c4d898a265
split the dumps between v1 and v2
2021-05-10 20:20:57 +02:00
tamo
e389c088eb
WIP: rebasing on master
2021-05-10 20:20:57 +02:00
Marin Postma
0cc79d414f
add test
2021-05-10 18:34:25 +02:00
Marin Postma
8d11b368d1
implement check
2021-05-10 18:22:41 +02:00
Marin Postma
706643dfed
type setting struct
2021-05-10 17:30:09 +02:00
Marin Postma
b192cb9c1f
enable string syntax for the filters
2021-05-06 12:48:31 +02:00
Marin Postma
ec7eb7798f
remove facet setting
2021-05-04 22:36:31 +02:00
Marin Postma
a717925caa
remove filters, rename facet_filters to filter
2021-05-04 18:20:56 +02:00
Marin Postma
78217bcf18
Fix cors authentication issue
2021-04-29 16:28:12 +02:00
bors[bot]
8bc7dd8b03
Merge #143
...
143: Shared update store r=irevoire a=MarinPostma
This PR changes the updates process so that only one instance of an update store is shared among indexes.
This allows updates to always be processed sequentially without additional synchronization, and fixes the bug where all the first pending update for each index were reported as processing whereas only one was.
EDIT:
I ended having to rewrite the whole `UpdateStore` to allow updates being really queued and processed sequentially in the ordered they were added. For that purpose I created a `pending_queue` that orders the updates by a global update id.
To find the next `update_id` to use, both globally and for each index, I have created another database that contains the next id to use.
Finally, all updates that have been processed (with success or otherwise) are all stores in an `updates` database.
The layout for the keys of these databases are such that it is easy to iterate over the elements for a particular index, and greatly reduces the amount of code to do so, compared to the former implementation.
I have also simplified the locking mechanism for the update store, thanks to the StateLock data structure, that allow both an arbitrary number of readers and a single writer to concurrently access the state. The current state can be either Idle, Processing, or Snapshotting. When an update or snapshotting is ongoing, the process holds the state lock until it is done processing its task. When it is done, it sets bask the state to Idle.
I have made other small improvements here and there, and have let some other for work, such as:
- When creating an update file to hold a request's content, it would be preferable to first create a temporary file, and then atomically persist it when we have written to it. This would simplify the case when there is no data to be written to the file, since we wouldn't have to take care about cleaning after ourselves.
- The logic for content validation must be factored.
- Some more tests related to error handling in the process_pending_update function.
- The issue #159
close #114
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-04-27 18:41:55 +00:00
Marin Postma
a961f0ce75
fix clippy warnings
2021-04-27 18:28:46 +02:00
Marin Postma
bb79a15c04
reenable ranking rules route
2021-04-27 15:29:00 +02:00
Marin Postma
4fe2a13c71
rewrite update store
2021-04-27 15:20:52 +02:00
Marin Postma
51829ad85e
review fixes
2021-04-27 15:10:57 +02:00
Marin Postma
c78f351300
fix tests
2021-04-27 15:10:57 +02:00
Marin Postma
ee675eadf1
fix stats
2021-04-27 15:10:55 +02:00
Marin Postma
33830d5ecf
fix snapshots
2021-04-27 15:09:55 +02:00
Marin Postma
2b154524bb
fix filtered out pending update
2021-04-27 15:09:23 +02:00
Marin Postma
b626d02ffe
simplify index actor run loop
2021-04-27 15:09:22 +02:00
Marin Postma
9ce68d11a7
single update store instance
2021-04-27 15:09:21 +02:00
Marin Postma
5a38f13cae
multi_index udpate store
2021-04-27 15:07:13 +02:00
Clémentine Urquizar
1ba46f8f77
Disable clippy rule
2021-04-27 12:43:00 +02:00
Clémentine Urquizar
f80ea24d2b
Add tests on every platform and fix clippy errors
2021-04-27 12:42:59 +02:00
Marin Postma
c2461e5066
review fixes
2021-04-26 10:20:46 +02:00
Marin Postma
6af769af20
bump mini-dashboard
2021-04-22 10:45:05 +02:00
Marin Postma
6bcf20c70e
serve static site
2021-04-22 10:26:54 +02:00
Clémentine Urquizar
a72d2f66cd
use <em> tags instead of <mark> tags for highlighting
2021-04-21 19:14:55 +02:00
Marin Postma
dd2914873b
fix document fields order
2021-04-20 21:30:30 +02:00
Marin Postma
d9a29cae60
fix ignored displayed attributes
2021-04-20 21:23:35 +02:00
Marin Postma
7a737d2bd3
support wildcard
2021-04-20 21:23:35 +02:00
Marin Postma
881b099c8e
add tests
2021-04-20 21:23:34 +02:00
Marin Postma
c6bb36efa5
implement _formated
2021-04-20 21:23:28 +02:00
Marin Postma
526a05565e
add SearchHit structure
2021-04-20 21:22:48 +02:00
Marin Postma
6c470cf687
enable distinct-attribute setting route
2021-04-20 11:34:18 +02:00
mpostma
ec230c2835
enable distinct
2021-04-20 11:29:06 +02:00
Clémentine Urquizar
8eceba98d3
Handle star in attributes_to_retrieve
2021-04-19 18:20:19 +02:00
tamo
ec3a08ea0c
remove another unused legacy file
2021-04-15 14:44:43 +02:00
Alexey Shekhirin
f87afbc558
fix(http): commit date & SHA in startup message
2021-04-13 20:16:18 +03:00
Alexey Shekhirin
9eaf048a06
fix(http): use BTreeMap instead of HashMap to preserve stats order
2021-04-13 11:59:07 +03:00
Alexey Shekhirin
adfdb99abc
feat(http): calculate updates' and uuids' dbs size
2021-04-09 15:59:12 +03:00
Alexey Shekhirin
ae1655586c
fixes after review
2021-04-09 14:40:48 +03:00
Alexey Shekhirin
698a1ea582
feat(http): store processing as RwLock<Option<Uuid>> in index_actor
2021-04-09 14:34:43 +03:00
Alexey Shekhirin
87412f63ef
feat(http): implement is_indexing for stats
2021-04-09 14:34:42 +03:00
Alexey Shekhirin
dd9eae8c26
feat(http): stats route
2021-04-09 14:34:42 +03:00
bors[bot]
a1d04fbff5
Merge #136
...
136: Rename update status "pending" into "enqueued" r=curquiza a=curquiza
Closes #107
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-08 16:46:12 +00:00
Alexey Shekhirin
51ba1bd7d3
fix(http, index): init analyzer with optional stop words
...
Next release
update tokenizer
2021-04-08 17:16:13 +03:00
bors[bot]
f881e8691e
Merge #135
...
135: Add stop words r=curquiza a=irevoire
closes #21
Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-08 11:29:00 +00:00
Clémentine Urquizar
cb23775d18
Rename pending into enqueued
2021-04-07 19:46:36 +02:00
bors[bot]
0344cf5874
Merge #122
...
122: Update display r=MarinPostma a=curquiza
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-04-07 12:33:25 +00:00
tamo
b1962c8e02
remove legacy files from meilisearch that have been replaced by a macro in routes/settings/mod.rs
2021-04-06 16:29:04 +02:00
tamo
40ef9a3c6a
push a first implementation of the stop_words
2021-04-06 16:29:04 +02:00
bors[bot]
6cb8052d3d
Merge #104
...
104: Update all the response format (issue #64 ) r=MarinPostma a=irevoire
closes #64
Co-authored-by: Irevoire <tamo@meilisearch.com>
Co-authored-by: tamo <tamo@meilisearch.com>
2021-04-01 14:22:57 +00:00
tamo
73973e2b9e
fix more settings routes
2021-04-01 15:50:45 +02:00
Marin Postma
248e9b3808
Merge remote-tracking branch 'origin/main' into snapshots
2021-04-01 15:10:33 +02:00
tamo
79c63049d7
update the settings routes
2021-04-01 11:52:26 +02:00
Irevoire
96cffeab1e
update all the response format to be ISO with meilisearch, see #64
2021-04-01 11:43:03 +02:00
Alexey Shekhirin
3d51db5929
fix(ci, http): commit_sha and commit_date in docker builds
...
chore(ci): cache dependencies in Docker build
2021-03-31 13:56:28 +03:00
tamo
0f2143e7fd
remove the now useless dead_code flags
2021-03-26 14:15:12 +01:00
Clémentine Urquizar
b9f79c8df0
Update display
2021-03-26 12:12:55 +01:00
mpostma
d7c077cffb
atomic snapshot import
2021-03-25 14:48:51 +01:00
mpostma
7d6ec7f3d3
resolve merge
2021-03-25 14:21:05 +01:00
mpostma
f3dc853be3
Merge remote-tracking branch 'origin/main' into snapshots
2021-03-25 13:45:07 +01:00
mpostma
48507460b2
add snapshot tests
2021-03-25 12:02:10 +01:00
mpostma
d029464de8
fix snapshot path
2021-03-25 10:23:31 +01:00
mpostma
79d09705d8
perform snapshot on startup
2021-03-25 09:35:15 +01:00
bors[bot]
868658f3d8
Merge #109
...
109: Make updates atomic r=curquiza a=MarinPostma
Until now, the index_uid->uuid mapping was done before the update was written to disk in the case of automatic index creation. This was an issue when the update failed, and the index would still exists in the uuid resolver.
This is fixed by this pr, by first creating the update with an uuid if the index does not exist, and then register this uuid to the uuid resolver.
This is preliminary work to the implementation of snapshots (#19 ).
This pr also changes the `resolve` method on the `UuidResolver` to `get` to make it clearer.
The `create_uuid` method may be bound to disappear when the index name resolution is handled by a remote machine.
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-24 12:24:32 +00:00
bors[bot]
fe87477238
Merge #115
...
115: Add the exhaustiveNbHits in search response body (returns always false) r=curquiza a=irevoire
closes #103
Co-authored-by: tamo <irevoire@protonmail.ch>
Co-authored-by: Irevoire <irevoire@protonmail.ch>
2021-03-24 12:16:53 +00:00
mpostma
d892a2643e
fix clippy
2021-03-24 12:38:59 +01:00
mpostma
83ffdc888a
remove bad file name test
2021-03-24 12:38:59 +01:00
mpostma
4041d9dc48
format code
2021-03-24 12:38:59 +01:00
mpostma
1f16c8d224
integration test snapshot
2021-03-24 12:38:59 +01:00
mpostma
06f9dae0f3
remove prints
2021-03-24 12:38:59 +01:00
mpostma
48d5f88c1a
fix snapshot dir already exists
2021-03-24 12:38:59 +01:00
mpostma
eb53ed4cc1
load snapshot
2021-03-24 12:38:59 +01:00
mpostma
46293546f3
add tests and mocks
2021-03-24 12:38:59 +01:00
mpostma
3cc3637e2d
refactor for tests
2021-03-24 12:38:56 +01:00
mpostma
1f51fc8baf
create indexes snapshots concurrently
2021-03-24 12:38:12 +01:00
mpostma
e9da191b7d
fix snapshot bugs
2021-03-24 12:38:12 +01:00
mpostma
d73fbdef2e
remove from snapshot
2021-03-24 12:38:12 +01:00
mpostma
44dcfe29aa
clean snapshot creation
2021-03-24 12:38:12 +01:00
mpostma
a85e7abb0c
fix snapshot creation
2021-03-24 12:38:12 +01:00
mpostma
4847884165
restore snapshots
2021-03-24 12:38:12 +01:00
mpostma
7f6a54cb12
add lock to prevent snapshot during update
2021-03-24 12:38:12 +01:00
mpostma
520f7c09ba
sequential index snapshot
2021-03-24 12:38:12 +01:00
mpostma
35a7b800eb
snapshot indexes
2021-03-24 12:38:12 +01:00
mpostma
c966b1dd94
use options to schedule snapshot
2021-03-24 12:38:11 +01:00
mpostma
ee838be41b
implement snapshot scheduler
2021-03-24 12:38:11 +01:00
Irevoire
127e944866
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 19:13:22 +01:00
Irevoire
cc81aca6a4
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-03-23 10:47:19 +01:00
Irevoire
46d7cedb18
Update meilisearch-http/src/index/search.rs
...
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 10:46:59 +01:00
mpostma
5f33672f0e
change payload send to use stream methods
2021-03-22 19:49:21 +01:00
mpostma
b690f1103a
fix typos
2021-03-22 19:25:56 +01:00
tamo
91089db444
add the exhaustive nb hits to be ISO, currently it's always set to false
2021-03-22 18:41:33 +01:00
bors[bot]
2f418ee767
Merge #108
...
108: use write senders for updates r=MarinPostma a=MarinPostma
Use write senders to send updates to the `IndexActor`, so updates are performed sequentially on all indexes.
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-22 14:18:43 +00:00
Irevoire
7ecefe37da
fix root route
2021-03-19 11:34:54 +01:00
mpostma
147756750b
create uuid on successful update addition
...
also change resolve to get in uuid resolver
2021-03-18 09:09:26 +01:00
mpostma
8b99860e85
use write sender for updates
2021-03-18 08:32:05 +01:00
Irevoire
c8b05712fa
return 202 on settings update / reset
2021-03-17 14:44:32 +01:00
bors[bot]
ca3b343b1f
Merge #96
...
96: Check json payload on document addition r=curquiza a=MarinPostma
Check if the json payload in updates is valid. It uses a json validator to avoid allocation, and only serializes the json in case of error, to return a pretty message.
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 17:20:44 +00:00
bors[bot]
f8ea081df5
Merge #98
...
98: replace body with json r=curquiza a=MarinPostma
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 17:12:30 +00:00
tamo
233c1e304d
use json instead of body when crafting the request
2021-03-16 17:45:59 +01:00
tamo
a268d0e283
return a 200 on health check
2021-03-16 17:42:01 +01:00
bors[bot]
764ced8b5c
Merge #88
...
88: restore name field in index meta r=MarinPostma a=MarinPostma
Makes the IndexMetadata payload iso with legacy meilisearch and closes #67
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 15:50:08 +00:00
mpostma
3c25ab0d50
replace body with json
2021-03-16 16:46:07 +01:00
bors[bot]
c6dbd81823
Merge #90
...
90: restore version route r=MarinPostma a=MarinPostma
close #74
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 13:53:23 +00:00
mpostma
204c743bcc
add json payload check on document addition
2021-03-16 14:28:13 +01:00
bors[bot]
2138f54954
Merge #89
...
89: delete index returns 204 instead of 200 r=curquiza a=MarinPostma
close #63
Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-16 13:01:32 +00:00
mpostma
f4cf96915a
remove guard on add documetn route
2021-03-16 12:04:32 +01:00
mpostma
6a742ee62c
restore version route
2021-03-15 19:11:27 +01:00
mpostma
58fab035bb
delete index returns 204 instead of 200
2021-03-15 18:44:33 +01:00
mpostma
94bd14ede3
add name to index_metadata
2021-03-15 18:35:16 +01:00
mpostma
dd324807f9
last review edits + fmt
2021-03-15 18:11:10 +01:00
mpostma
abbea59732
fix clippy warnings
2021-03-15 16:52:05 +01:00
mpostma
01479dcf99
rename name to uid in code
2021-03-15 14:43:47 +01:00
mpostma
55fadd7f87
change facetedAttributes to attributesForFaceting
2021-03-15 13:53:50 +01:00
mpostma
fcf1d4e922
fix displayed attributes in search
2021-03-15 12:20:33 +01:00
mpostma
c079f60346
fixup! fix displayed attributes in document retrieval
2021-03-15 11:01:14 +01:00
mpostma
77c0a0fba5
add test get document displayed attributes
2021-03-15 10:36:12 +01:00
mpostma
adc71a70ce
fix displayed attributes in document retrieval
2021-03-15 10:17:41 +01:00
mpostma
99c89cf2ba
use options max db sizes
2021-03-13 10:09:10 +01:00
mpostma
49b74b587a
enable jemalloc only on linux
2021-03-12 17:47:40 +01:00
mpostma
2ee2e6a9b2
clean project
2021-03-12 14:57:24 +01:00
mpostma
c4846dafca
implement update index
2021-03-12 14:48:43 +01:00
mpostma
77d5dd452f
remove open_or_create
2021-03-12 14:16:54 +01:00
mpostma
e4d45b0500
fix various bugs
2021-03-12 00:37:43 +01:00
mpostma
7d9637861f
fix add primary key on index creation
2021-03-11 22:55:29 +01:00
mpostma
271c8ba991
change index name to uid
2021-03-11 22:47:29 +01:00
mpostma
8617bcf8bd
add ranking rules
2021-03-11 22:39:16 +01:00
mpostma
66b64c1f80
correct error on settings delete unexisting index
2021-03-11 22:33:31 +01:00
mpostma
30dd790884
handle badly formatted index uid
2021-03-11 22:23:48 +01:00
mpostma
40b3451a4e
fix unexisting update store + race conditions
2021-03-11 22:11:58 +01:00
mpostma
3f68460d6c
fix update dedup
2021-03-11 20:58:51 +01:00
mpostma
79a4bc8129
use meta from milli
2021-03-11 19:40:18 +01:00
mpostma
1fad72e019
fix test bug with tempdir
2021-03-11 17:59:47 +01:00
mpostma
2ae90f9c5d
lazy load update store
2021-03-11 14:23:11 +01:00
mpostma
53cf500e36
uuid resolver hard state
2021-03-10 18:04:20 +01:00
mpostma
5ecf514d28
restructure project
2021-03-10 13:46:49 +01:00
