Commit Graph

10373 Commits

Author SHA1 Message Date
Clément Renault
656a851830
Introduce the Transform struct transforming CSVs
This allows us to:
  - transform a CSV, a JSON or a JSON lines data type into the same
    Grenad x Obkv streamable data type and creates the new FieldsIdsMap.
  - Extract all the documents user ids in advance to be able to delete
    the existing documents before re-indexing them.
  - Keep the last documents with the same user id avoiding duplicates
    in the same request.
2020-10-24 13:37:38 +02:00
Clément Renault
8d82e37ec0
Introduce the AvailableDocumentsIds iterator 2020-10-23 12:07:01 +02:00
Charlotte Vermandel
9dbf43d3e7 Update readme accordingly 2020-10-22 20:33:20 +02:00
Charlotte Vermandel
772f4d6671 Remove not maintained cvs movies dataset 2020-10-22 20:33:20 +02:00
bors[bot]
1b57218739
Merge #1040
1040: Update movie posters r=Kerollmops a=bidoubiwa

This PR resolves 3 issues: 

1. update posters URLs that changed
2. All posters point to a smaller image ( +- 20kb instead of 500kb+-) this was done by changing the width size from 1280 px to 500 px. 
3. Remove films that are not in the tmdb database

Co-authored-by: Charlotte Vermandel <charlottevermandel@gmail.com>
2020-10-22 16:38:41 +00:00
Charlotte Vermandel
8767269b47 Update movie posters 2020-10-22 18:07:57 +02:00
bors[bot]
baceaed582
Merge #1038
1038: Add Sandbox section to README.md r=LegendreM a=eskombro

This PR adds a link to [MeiliSearch Sandbox](https://sandbox.meilisearch.com/) in the README.md

Co-authored-by: Samuel Jimenez <sjimenezre@gmail.com>
2020-10-22 15:25:23 +00:00
Samuel Jimenez
62a28bc2a1 Add Sandbox section to README.md 2020-10-22 17:04:45 +02:00
bors[bot]
f83caa6c40
Merge #1008
1008: Dump info r=Kerollmops a=LegendreM

fix #998 
fix #988 
fix #1009
fix #1010
fix #1033


Co-authored-by: many <maxime@meilisearch.com>
2020-10-22 14:23:50 +00:00
many
53b1483e71
fix pr comments 2020-10-22 16:12:55 +02:00
many
a0eafea200
fix tests 2020-10-22 15:46:20 +02:00
many
10dace305d
snapshot at start 2020-10-22 15:46:20 +02:00
many
1eace79f77
change error message to be absolute 2020-10-22 15:46:20 +02:00
many
e6033e174d
fix #1010 2020-10-22 15:46:20 +02:00
many
f1925b8f71
fix #1009 2020-10-22 15:46:20 +02:00
many
834f3cc192
rename folder to dir 2020-10-22 15:46:20 +02:00
many
e049aead16
improve dump status 2020-10-22 15:46:20 +02:00
Clément Renault
2a4cd81c86
Add documentation to the Index methods 2020-10-22 15:44:12 +02:00
Clément Renault
566a7c3039
Make the FieldsIdsMap serialization more stable by using a BTreeMap 2020-10-22 14:53:20 +02:00
Clément Renault
9133f38138
Introduce the FieldsIdsMap type 2020-10-22 12:56:35 +02:00
bors[bot]
0a9c9670e7
Merge #1028
1028: Clean external contributions r=Kerollmops a=LegendreM

We accepted some unperfect external PRs, this one is here to clean this:
-  clean PR #946 (remove changelog line and add forgotten newline)
- remove useless function after health route refacto #1026

Co-authored-by: many <maxime@meilisearch.com>
Co-authored-by: Many <legendre.maxime.isn@gmail.com>
2020-10-22 10:46:19 +00:00
Many
1744dcebfe
Merge branch 'master' into clean_external_contributions 2020-10-22 12:23:51 +02:00
Clément Renault
802e925fd7
Switch to a JSON protocol for the front page 2020-10-21 18:26:29 +02:00
bors[bot]
29712916e6
Merge #1034
1034: Remove outdated settings file r=Kerollmops a=bidoubiwa

Unnecessary settings files in the dataset folder should be removed. 

Co-authored-by: Charlotte Vermandel <charlottevermandel@gmail.com>
2020-10-21 15:42:48 +00:00
Charlotte Vermandel
4d2783bb04 Remove outdated settings file 2020-10-21 17:12:10 +02:00
Clément Renault
5caf523fd9
Move the Index to its own module 2020-10-21 15:55:48 +02:00
Clément Renault
2210818114
Introduce the obkv heed codec 2020-10-21 15:51:48 +02:00
Clément Renault
f6eecb855e
Send a basic progressing status to the updates front page 2020-10-21 15:38:28 +02:00
Clément Renault
4eeeccb9cd
Change the UpdateStore to have different processed and pending meta types 2020-10-21 13:52:15 +02:00
Clément Renault
16ab3e02a9
Change the UpdateStore internal meta serializer 2020-10-21 13:42:49 +02:00
Clément Renault
f948a03be2
Optimise the merge functions to avoid allocations 2020-10-20 16:40:50 +02:00
many
50f0fbb05c
remove useless function after health route refacto #1026 2020-10-20 16:21:46 +02:00
Clément Renault
cde8478388
Replace the panic in the merge function by actual errors 2020-10-20 16:19:07 +02:00
Clément Renault
8ed8abb9df
Introduce an append-only indexing system 2020-10-20 15:00:58 +02:00
Clément Renault
a122d3d466
Export the indexing part into a module 2020-10-20 14:22:09 +02:00
Clément Renault
eb92e72e6c
Updates can send progress update status 2020-10-20 12:28:10 +02:00
Clément Renault
341046c96c
Remove the js map file from the filesize.js script 2020-10-20 12:20:42 +02:00
Clément Renault
3a934b7020
Split the update attributes on the updates front page 2020-10-20 12:19:48 +02:00
Clément Renault
03ca1ff634
Make the updates page interactive 2020-10-20 12:09:38 +02:00
Clément Renault
35c9a3c558
Brodacast the updates infos to every ws clients 2020-10-20 11:19:34 +02:00
Clément Renault
56c3a61d83
Introduce a new updates page 2020-10-19 19:57:15 +02:00
many
5a842ec94a
clean PR #946 2020-10-19 17:16:25 +02:00
Clément Renault
871222aebd
Introduce some new routes to handle live indexing 2020-10-19 16:06:43 +02:00
bors[bot]
372680e2ab
Merge #1026
1026: refactor /health  r=LegendreM a=frbimo

Fixes: #940 

Testing:
`cargo test` and `cargo build --release` passed

Co-authored-by: frbimo <fr.bimo@gmail.com>
2020-10-19 13:57:15 +00:00
Clément Renault
d3145be744
Rename the meta UpdateStore method 2020-10-19 14:00:00 +02:00
Clément Renault
8bfa43f9a7
Update the iter_metas UpdateStore method 2020-10-19 13:58:08 +02:00
Clément Renault
65e32fecb1
Move the binaries into one with subcommands 2020-10-19 13:44:17 +02:00
Clément Renault
ff389f1270
Update heed-types to 0.7.1 2020-10-19 11:52:59 +02:00
frbimo
6465a3f549 refactor /health on meilisearch-http that complies:
1. NEEDS to ensure that service is completely up if it returns 204
2. DOES NOT block service process (write transaction)
3. NEEDS to use the less network bandwidth as possible when it's triggered
4. NEEDS to use the less service resources as possible when it's triggered
5. DOES NOT NEED any authentication
6. MAY be named /health
2020-10-19 14:30:43 +08:00
Clément Renault
5b4eda670b
Add two tests for the UpdateStore 2020-10-18 18:55:09 +02:00