Commit Graph

165 Commits

Author SHA1 Message Date
Kerollmops
68856e5e2f
Disable the default snappy compression for the http-ui crate 2021-09-08 14:17:32 +02:00
Clémentine Urquizar
eb7b9d9dbf
Update version for the next release (v0.13.0) 2021-09-08 10:59:30 +02:00
bors[bot]
5cbe879325
Merge #308
308: Implement a better parallel indexer r=Kerollmops a=ManyTheFish

Rewrite the indexer:
- enhance memory consumption control
- optimize parallelism using rayon and crossbeam channel
- factorize the different parts and make new DB implementation easier
- optimize and fix prefix databases


Co-authored-by: many <maxime@meilisearch.com>
2021-09-02 15:03:52 +00:00
Clémentine Urquizar
285849e3a6
Update version for the next release (v0.12.0) 2021-09-02 10:08:41 +02:00
many
1d314328f0
Plug new indexer 2021-09-01 16:48:36 +02:00
Tamo
d106eb5b90
add the sortable attributes to http-ui and fix the tests 2021-08-30 16:25:10 +02:00
Kerollmops
af65485ba7
Reexport the grenad CompressionType from milli 2021-08-24 18:15:31 +02:00
Kerollmops
2f20257070
Update milli to the v0.11.0 2021-08-24 18:10:11 +02:00
Clément Renault
89d0758713
Revert "Revert "Sort at query time"" 2021-08-24 11:55:16 +02:00
Clémentine Urquizar
88f6c18665
Update version for the next release (v0.10.2) 2021-08-23 11:33:30 +02:00
Clémentine Urquizar
922f9fd4d5
Revert "Sort at query time" 2021-08-20 18:09:17 +02:00
bors[bot]
41fc0dcb62
Merge #309
309: Sort at query time r=Kerollmops a=Kerollmops

This PR:
 - Makes the `Asc/Desc` criteria work with strings too, it first returns documents ordered by numbers then by strings, and finally the documents that can't be ordered. Note that it is lexicographically ordered and not ordered by character, which means that it doesn't know about wide and short characters i.e. `a`, `丹`, `▲`.
 - Changes the syntax for the `Asc/Desc` criterion by now using a colon to separate the name and the order i.e. `title:asc`, `price:desc`.
 - Add the `Sort` criterion at the third position in the ranking rules by default.
 - Add the `sort_criteria` method to the `Search` builder struct to let the users define the `Asc/Desc` sortable attributes they want to use at query time. Note that we need to check that the fields are registered in the sortable attributes before performing the search.
 - Introduce a new `InvalidSortableAttribute` user error that is raised when the sort criteria declared at query time are not part of the sortable attributes.
 - `@ManyTheFish` introduced integration tests for the dynamic Sort criterion.

Fixes #305.

Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: many <maxime@meilisearch.com>
2021-08-18 16:55:32 +00:00
bors[bot]
198c416bd8
Merge #312
312: Update milli version to v0.10.1 r=Kerollmops a=curquiza



Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-08-18 12:08:04 +00:00
Clémentine Urquizar
6cb9c3b81f
Update milli version to v0.10.1 2021-08-18 13:46:27 +02:00
Clémentine Urquizar
42cf847a63
Update tokenizer version to v0.2.5 2021-08-18 13:37:41 +02:00
Kerollmops
5b88df508e
Use the new Asc/Desc syntax everywhere 2021-08-17 14:15:22 +02:00
Clémentine Urquizar
fcc520e49a
Update version for the next release (v0.10.0) 2021-08-16 12:00:28 +02:00
Clémentine Urquizar
7f26c75610
Update milli to v0.9.0 2021-08-04 16:04:55 +02:00
Kerollmops
341c244965
Bump milli to v0.8.1 2021-07-29 15:56:36 +02:00
Clémentine Urquizar
6a141694da
Update version for the next release (v0.8.0) 2021-07-27 16:38:42 +02:00
bors[bot]
cc54c41e30
Merge #283
283: Use the AlwaysFreePages flag when opening an index r=irevoire a=Kerollmops

We introduced a new flag in our fork of LMDB, this `AlwaysFreePages` flag forces LMDB to always free the single pages it uses before writing to the disk instead of keeping them in a linked list.

Declaring this flag reduces the memory print (leak) we have on memory after indexing a lot of documents.

Fixes #279.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2021-07-05 16:59:16 +00:00
Irevoire
4562b278a8
remove a warning and add a log
Co-authored-by: Clément Renault <clement@meilisearch.com>
2021-07-05 17:46:02 +02:00
Tamo
a57e522a67
introduce a die route let the program exit itself alone 2021-07-05 17:38:10 +02:00
Kerollmops
91c5d0c042
Use the AlwaysFreePages flag when opening an index 2021-07-05 16:36:13 +02:00
Kerollmops
a6b4069172
Bump to v0.7.2 2021-07-05 10:54:53 +02:00
Clémentine Urquizar
b489515f4d
Update milli version to v0.7.1 2021-06-30 13:52:46 +02:00
Clément Renault
80c6aaf1fd
Bump milli to 0.7.0 2021-06-28 18:31:56 +02:00
Clément Renault
bdc5599b73
Bump heed to use the git repo with v0.12.0 2021-06-28 18:26:20 +02:00
Kerollmops
98285b4b18
Bump milli to 0.6.0 2021-06-23 17:30:26 +02:00
Clémentine Urquizar
9885fb4159
Update version for the next release (v0.5.1) 2021-06-23 14:05:20 +02:00
bors[bot]
634201244c
Merge #250 #251
250: Add the limit field to http-ui r=Kerollmops a=irevoire



251: Fix the limit r=Kerollmops a=irevoire

There was no check on the limit and thus if a user specified a very large number this line could cause a panic.

Co-authored-by: Tamo <tamo@meilisearch.com>
2021-06-22 13:00:52 +00:00
Tamo
81643e6d70
add the limit field to http-ui 2021-06-22 14:47:23 +02:00
Tamo
77eb37934f
add jemalloc to http-ui and the benchmarks 2021-06-22 14:17:56 +02:00
Clémentine Urquizar
320670f8fe
Update version for the next release (v0.5.0) 2021-06-21 15:59:17 +02:00
Clémentine Urquizar
35fcc351a0
Update version for the next release (v0.4.2) 2021-06-20 17:37:24 +02:00
Kerollmops
ccd6f13793
Update version to the next release (0.4.1) 2021-06-17 15:01:20 +02:00
Tamo
9716fb3b36
format the whole project 2021-06-16 18:33:33 +02:00
Clémentine Urquizar
f5ff3e8e19
Update version for the next release (v0.4.0) 2021-06-16 14:01:05 +02:00
Kerollmops
78fe4259a9
Fix the http-ui crate 2021-06-14 18:06:23 +02:00
Clémentine Urquizar
7d5395c12b
Update Tokenizer version to v0.2.3 2021-06-10 17:00:04 +02:00
Clémentine Urquizar
dc64e139b9
Update version for the next release (v0.3.1) 2021-06-09 14:39:21 +02:00
Kerollmops
103dddba2f
Move the UpdateStore into the http-ui crate 2021-06-08 17:59:51 +02:00
Clémentine Urquizar
3b2b3aeea9
Update Cargo.toml for next release v0.3.0 2021-06-03 12:24:27 +02:00
Kerollmops
3b1cd4c4b4
Rename the FacetCondition into FilterCondition 2021-06-02 16:24:58 +02:00
Kerollmops
c10469ddb6
Patch the http-ui crate to support filterable fields 2021-06-02 16:24:58 +02:00
Tamo
608c5bad24
fix http-ui 2021-06-01 16:24:46 +02:00
Kerollmops
5012cc3a32
Fix the http-ui crate to support split facet databases 2021-05-25 11:31:06 +02:00
Clémentine Urquizar
1e11578ef0
Update version for the next release (v0.2.1) 2021-05-05 14:57:34 +02:00
Clémentine Urquizar
a8680887d8
Upgrade Milli version (v0.2.0) 2021-05-03 14:50:47 +02:00
Clémentine Urquizar
34e02aba42
Upgrade Tokenizer version (v0.2.2) 2021-05-03 10:55:55 +02:00
Kerollmops
e65bad16cc
Compute the words prefixes at the end of an update 2021-04-27 14:39:52 +02:00
Clément Renault
7aa5753ed2
Make the attribute positions range bounds to be fixed 2021-04-27 14:35:43 +02:00
Kerollmops
e8cc7f9cee
Expose a route in the http-ui to update the WordsLevelPositions 2021-04-27 14:25:34 +02:00
tamo
7fa3a1d23e
makes clippy happy http-ui 2021-04-21 12:36:45 +02:00
Clémentine Urquizar
127d3d028e
Update version for the next release (v0.1.1) 2021-04-19 14:48:13 +02:00
Alexey Shekhirin
33860bc3b7
test(update, settings): set & reset synonyms
fixes after review

more fixes after review
2021-04-18 11:24:17 +03:00
Alexey Shekhirin
e39aabbfe6
feat(search, update): synonyms 2021-04-18 11:24:17 +03:00
Clémentine Urquizar
2c5c79d68e
Update Tokenizer version to v0.2.1 2021-04-14 18:54:04 +02:00
Alexey Shekhirin
3af8fa194c
test(http): combine settings assert_(ser|de)_tokens into 1 test 2021-04-10 12:13:59 +03:00
Alexey Shekhirin
84c1dda39d
test(http): setting enum serialize/deserialize 2021-04-08 17:03:40 +03:00
Alexey Shekhirin
dc636d190d
refactor(http, update): introduce setting enum 2021-04-08 17:03:40 +03:00
tamo
a2f46029c7
implement a first version of the stop_words
The front must provide a BTreeSet containing the stop words
The stop_words are set at None if an empty Set is provided
add the stop-words in the http-ui interface

Use maplit in the test
and remove all the useless drop(rtxn) at the end of all tests
2021-04-01 13:57:55 +02:00
tamo
62a8f1d707
bump the version of the tokenizer 2021-04-01 13:49:22 +02:00
tamo
13ce0ebb87
stop requestings the facets if the user has hidden them 2021-03-31 16:27:32 +02:00
tamo
bcc131e866
add a button to display or hide the facets 2021-03-31 16:18:53 +02:00
tamo
73dcdb27f6
select a specific release of the tokenizer instead of using the latests git commit 2021-03-25 15:00:18 +01:00
Kerollmops
5af63c74e0
Speed-up the MatchingWords highlighting struct 2021-03-03 15:45:03 +01:00
Clément Renault
c62d2f56d8
Expose an http route for the WordsPrefixes update 2021-02-21 12:16:53 +01:00
Kerollmops
519b1cb5c9
Update dependencies 2021-02-21 10:26:04 +01:00
Clément Renault
fecf3d6fc1
Move the command lines helpers into different crates 2021-02-14 18:55:15 +01:00
Clément Renault
e8639517da
Change the project to become a workspace with milli as a default-member 2021-02-12 16:15:09 +01:00
mpostma
8f43698a60
fix httpui 2021-02-01 19:49:51 +01:00
mpostma
3b60432687
Use update_id in UpdateBuilder
Add `the update_id` to the to the updates. The rationale is the
following:
- It allows for better tracability of the update events, thus improved
  debugging and logging.
- The enigne is now aware of what he's already processed, and can return
  it if asked. It may not make sense now, but in the future, the update
  store may not work the same way, and this information about the state
  of the engine will be desirable (distributed environement).
2021-02-01 19:46:34 +01:00
Clément Renault
65b821b192
Rename the Index facets method into facets_distribution 2021-01-27 14:15:33 +01:00
Kerollmops
61dbcfa44a
Bump the roaring to 0.6.4 2021-01-26 14:38:43 +01:00
Clément Renault
b0c31500fc
Simplify the front page 2021-01-26 14:14:38 +01:00
Kerollmops
7be275b692
Add the count to the facet distribution 2021-01-26 14:14:37 +01:00
Kerollmops
33945a3115
Introduce a new facet filters query field 2021-01-26 14:09:08 +01:00
Clément Renault
a8e3269ad6
Introduce a basic front to display facets 2021-01-26 14:06:28 +01:00
Clément Renault
2cd8675734
Show facet values even for empty queries 2021-01-26 14:06:28 +01:00
Kerollmops
aa129dd7e8
Display the number of candidates instead of the returned document count 2021-01-26 14:06:28 +01:00
Kerollmops
510df4729c
Append the facet value to the facet query on click 2021-01-26 14:06:28 +01:00
Kerollmops
d25a859985
Display the facet values on the HTML debug page 2021-01-26 14:06:28 +01:00
Kerollmops
3b64735058
Introduce a struct to compute facets values 2021-01-26 14:06:27 +01:00
mpostma
87a56d2bc9
Fix settings bug
replace ids with str in settings

This allows for better maintainability of the settings code, since
updating the searchable attributes is now straightforward.

criterion use string

fix reindexing fieldid remaping

add tests for primary_key compute

fix tests

fix http-ui

fixup! add tests for primary_key compute

code improvements settings

update deps

fixup! code improvements settings

fixup! refactor settings updates and fix bug

fixup! Fix settings bug

fixup! Fix settings bug

fixup! Fix settings bug

Update src/update/index_documents/transform.rs

Co-authored-by: Clément Renault <clement@meilisearch.com>

fixup! Fix settings bug
2021-01-26 13:53:08 +01:00
Clément Renault
44c0dd0762
Fix an fst Set related warning 2021-01-13 11:03:03 +01:00
mpostma
4f7f7538f7
highlight with new tokenizer 2021-01-11 21:59:37 +01:00
mpostma
1ae761311e
integrate with meilisearch tokenizer 2021-01-07 16:14:27 +01:00
Kerollmops
a576c7ae4b
Display the update meta result content on the update page 2020-12-22 13:42:43 +01:00
Kerollmops
6c7db3d956
Display the time it took to process an update 2020-12-22 13:42:43 +01:00
Kerollmops
9fcbc83ebc
Accept and mirror compression of documents additions 2020-12-22 13:42:42 +01:00
mpostma
49a016b53d create update handler trait
fix type inference error
2020-12-22 12:59:15 +01:00
Kerollmops
77e951e933
Use the byte-unit crate to ease library usage 2020-12-20 12:00:37 +01:00
Clément Renault
5a23417499
Compress updates content using gzip 2020-12-17 10:59:58 +01:00
Clément Renault
e7f2ab9138
Bump grenad to fix an indexing bug 2020-12-05 16:39:15 +01:00
Clément Renault
0959e1501f
Introduce the FacetRevRange Iterator struct 2020-12-04 12:02:23 +01:00
Clément Renault
61b383f422
Introduce the criteria update setting 2020-12-04 12:02:22 +01:00
Clément Renault
58a1f9081c
Allow users to abort pending updates, one by one or all at once 2020-12-01 14:51:05 +01:00
Clément Renault
ba4ba685f9
Make the facet levels maps to previous level groups and don't split them 2020-11-28 12:43:43 +01:00
Clément Renault
fc686aaca7
Use the De Morgan law to simplify the NOT operation 2020-11-23 16:43:56 +01:00