Commit Graph

7600 Commits

Author SHA1 Message Date
Kerollmops
bdc4263883
Introduce the validate_documents_batch function 2022-07-12 14:55:51 +02:00
Kerollmops
a97d4d63b9
Fix the benchmarks 2022-07-12 14:55:50 +02:00
Kerollmops
f29114f94a
Fix http-ui to fit with the new DocumentsBatchBuilder/Reader structs 2022-07-12 14:52:56 +02:00
Kerollmops
a4ceef9624
Fix the cli for the new DocumentsBatchBuilder/Reader structs 2022-07-12 14:52:56 +02:00
Kerollmops
6d0498df24
Fix the fuzz tests 2022-07-12 14:52:56 +02:00
Kerollmops
e8297ad27e
Fix the tests for the new DocumentsBatchBuilder/Reader 2022-07-12 14:52:56 +02:00
Kerollmops
419ce3966c
Rework the DocumentsBatchBuilder/Reader to use grenad 2022-07-12 14:52:55 +02:00
Kerollmops
eb63af1f10
Update grenad to 0.4.2 2022-07-12 14:52:55 +02:00
Kerollmops
048e174efb
Do not allocate when parsing CSV headers 2022-07-12 14:52:55 +02:00
bors[bot]
01a47e2db5
Merge #2598
2598: Bring `stable` into `main` (v0.28.0) r=curquiza a=curquiza



Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Janith Petangoda <22471198+janithpet@users.noreply.github.com>
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-07-11 14:06:02 +00:00
Clémentine Urquizar
cfacc79ad7
Remove duplicate step in CI 2022-07-11 15:18:15 +02:00
Clémentine Urquizar - curqui
8e370ed9ab
Merge branch 'main' into stable 2022-07-11 14:41:15 +02:00
Clémentine Urquizar
32d6af6527
Merge remote-tracking branch 'origin/release-v0.28.0' into stable 2022-07-11 14:37:21 +02:00
ManyTheFish
5d79617a56 Chores: Enhance smart-crop code comments 2022-07-07 16:28:09 +02:00
bors[bot]
be3240d2dd
Merge #2592
2592: Chores: Add a dedicated section for Language Support in the issue template r=curquiza a=ManyTheFish



This new section in put upper than the feature proposal because language support is kind of a sub-category of it,
and so, in the reading order, we choose to create a feature proposal only if it is not related to Language.


Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-07-07 12:37:36 +00:00
ManyTheFish
2de6868858 Chores: Add a dedicated section for Language Support in the issue template
This new section in put apper than feature proposal because language-support is kind of a sub-category of it,
and so, in the reading order, we choose to create a feature proposal only if it is not related to Language.
2022-07-07 13:48:47 +02:00
bors[bot]
d419a91207
Merge #2591
2591: Introduce the Tasks Seen event when filtering r=Kerollmops a=Kerollmops

This PR fixes #2377 by introducing the Tasks Seen analytics events.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-07-07 11:41:01 +00:00
Kerollmops
a9fb5a4d50
Introduce the Tasks Seen event when filtering 2022-07-07 11:39:23 +02:00
bors[bot]
0353537fef
Merge #2579
2579: API keys: adds action * for actions r=irevoire a=phdavis1027

# Pull Request
This is PR builds on `@janithpet's` addition to DocumentsAll; it's basically a copy-and-paste job, except I used ```iter.filter()``` to avoid the possibility for duplication that they mentioned. I'm not sure how much that matters.

Also, hi! This is my first open-source contribution and my first attempt to write Rust for anyone other than myself, so any feedback whatsoever is appreciated. 

## What does this PR do?
Fixes #2560

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Phillip Davis <phdavis1027@gmail.com>
2022-07-07 08:54:23 +00:00
bors[bot]
da7729e4a8
Merge #2589
2589: Update create-issue-dependencies.yml r=ManyTheFish a=curquiza

Minor change to update the format in the description of the issue created: remove the useless newlines

See the format without this change: https://github.com/meilisearch/meilisearch/issues/2588

Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
2022-07-07 08:37:33 +00:00
bors[bot]
ce90fc628a
Merge #583
583: Use BufReader to read datasets in benchmarks r=ManyTheFish a=loiclec

## What does this PR do?
Ensure that the datasets used by the benchmarks are read efficiently by using a `BufReader`.

## Why?
Using a `BufReader` is more representative of how `meilisearch` works. It will also make performance comparisons between different branches of `milli` more  accurate.




Co-authored-by: Loïc Lecrenier <loic@meilisearch.com>
2022-07-07 08:13:07 +00:00
Phillip Davis
074a6a0cce
Fix typos in HeadAuthStore::put_api_key 2022-07-06 22:24:46 -04:00
Loïc Lecrenier
aae03356cb Use BufReader to read datasets in benchmarks 2022-07-06 18:20:15 +02:00
bors[bot]
755b1a59a2
Merge #2584
2584: Format API keys in hexa instead of base64 r=curquiza a=ManyTheFish

This PR:
- Changes API key generation and formatting to ease the generation of the key made by our users
- updates the `uuid` crate version

The API key can now be generated in bash as below:
```sh
echo -n $HYPHENATED_UUID | openssl dgst -sha256 -hmac $MASTER_KEY
```

fixes the issue raised in [product/discussion#421](https://github.com/meilisearch/product/discussions/421#discussioncomment-3079410), this should not impact anything in documentation nor integration but ease the key generation on the user sides.

poke `@gmourier` 

Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-07-06 12:49:04 +00:00
Clémentine Urquizar - curqui
bb5b18b82c
Update create-issue-dependencies.yml 2022-07-06 11:01:25 +02:00
bors[bot]
01d9560318
Merge #2587
2587: Update mini-dashboard to v0.2.0 r=curquiza a=mdubus

# Pull Request

## What does this PR do?
Fixes #2469

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Morgane Dubus <30866152+mdubus@users.noreply.github.com>
2022-07-06 08:54:16 +00:00
Morgane Dubus
719879d4d2
Update mini-dashboard to v0.2.0 2022-07-06 08:12:17 +02:00
Phillip Davis
fb9b298645
Leave actions as HashSet 2022-07-05 21:52:50 -04:00
Phillip Davis
23f02f241e
Run the code formatter 2022-07-05 21:08:56 -04:00
Phillip Davis
5d80ff41a2
Clean up put_key impl 2022-07-05 21:06:58 -04:00
bors[bot]
f4989590db
Merge #2585
2585: Add CI creates issue updating dependencies r=curquiza a=VasiliySoldatkin

Adds CI, which uses cron GHA to create Issue "Upgrade dependencies" every 3 months 
Context: [#2569](https://github.com/meilisearch/meilisearch/issues/2569)


Co-authored-by: Vasiliy Soldatkin <vasiliy.soldatkin@gmail.com>
Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
2022-07-05 19:05:43 +00:00
Clémentine Urquizar - curqui
bba5fab5e5
Update .github/workflows/create-issue-dependencies.yml 2022-07-05 21:03:08 +02:00
Clémentine Urquizar - curqui
05ffe24d64
Update .github/workflows/create-issue-dependencies.yml 2022-07-05 21:02:40 +02:00
Clémentine Urquizar - curqui
6f95ae9879
Update .github/workflows/create-issue-dependencies.yml 2022-07-05 21:02:32 +02:00
Vasiliy Soldatkin
480b881e15 Remove template and add GHA from review 2022-07-05 21:08:59 +03:00
Clémentine Urquizar - curqui
43fecbf382
Update .github/workflows/create-issue-dependencies.yml 2022-07-05 19:35:09 +02:00
Clémentine Urquizar - curqui
5588a6415a
Update .github/workflows/create-issue-dependencies.yml 2022-07-05 19:34:57 +02:00
Clémentine Urquizar - curqui
b0757e75c4
Update .github/workflows/create-issue-dependencies.yml 2022-07-05 19:34:49 +02:00
bors[bot]
106be03ba8
Merge #2539
2539: Update Docker credentials r=curquiza a=curquiza

This is to avoid using the tpayet credentials and use `@meili-bot` credentials instead.

The `DOCKER_USERNAME` and `DOCKER_PASSWORD` are still present as secret, I will remove them once v0.28.0 is fully merged (they are still used on `release-v0.28.0`)

I tested by created a tag on the branch, it worked: the tag was pushed to docker hub by meili-bot

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2022-07-05 17:22:26 +00:00
bors[bot]
70c55208f1
Merge #2586
2586: Fix Clippy r=irevoire a=Kerollmops

This PR fixes clippy on the `release-v0.28.0` branch.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-07-05 15:55:00 +00:00
Kerollmops
d56bf66022
Make clippy happy 2022-07-05 17:48:44 +02:00
Vasiliy Soldatkin
2c300c72c9 Add CI creates issue updating dependencies 2022-07-05 18:28:29 +03:00
bors[bot]
ebddfdb9a3
Merge #578
578: Bump uuid to 1.1.2 r=ManyTheFish a=Kerollmops

Just to [align the version with Meilisearch](https://github.com/meilisearch/meilisearch/pull/2584).

Co-authored-by: Kerollmops <clement@meilisearch.com>
2022-07-05 14:56:08 +00:00
bors[bot]
eeba196053
Merge #572
572: Add reindexing benchmarks r=Kerollmops a=irevoire

With #557 coming, we should add benchmarks that measure our impact on the reindexing process.

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-07-05 14:43:01 +00:00
Kerollmops
1bfdcfc84f
Bump uuid to 1.1.2 2022-07-05 16:23:36 +02:00
bors[bot]
dd1e606f13
Merge #557
557: Fasten documents deletion and update r=Kerollmops a=irevoire

When a document deletion occurs, instead of deleting the document we mark it as deleted in the new “soft deleted” bitmap. It is then removed from the search and all the other endpoints.

I ran the benchmarks against main;
```
% ./compare.sh indexing_main_83ad1aaf.json indexing_fasten-document-deletion_abab51fb.json
group                                                                     indexing_fasten-document-deletion_abab51fb    indexing_main_83ad1aaf
-----                                                                     ------------------------------------------    ----------------------
indexing/-geo-delete-facetedNumber-facetedGeo-searchable-                 1.05      2.0±0.40ms        ? ?/sec           1.00  1904.9±190.00µs        ? ?/sec
indexing/-movies-delete-facetedString-facetedNumber-searchable-           1.00     10.3±2.64ms        ? ?/sec           961.61      9.9±0.12s        ? ?/sec
indexing/-movies-delete-facetedString-facetedNumber-searchable-nested-    1.00     15.1±3.90ms        ? ?/sec           554.63      8.4±0.12s        ? ?/sec
indexing/-songs-delete-facetedString-facetedNumber-searchable-            1.00     45.1±7.53ms        ? ?/sec           710.15     32.0±0.10s        ? ?/sec
indexing/-wiki-delete-searchable-                                         1.00    277.8±7.97ms        ? ?/sec           1946.57    540.8±3.15s        ? ?/sec
indexing/Indexing geo_point                                               1.00      12.0±0.20s        ? ?/sec           1.03      12.4±0.19s        ? ?/sec
indexing/Indexing movies in three batches                                 1.00      19.3±0.30s        ? ?/sec           1.01      19.4±0.16s        ? ?/sec
indexing/Indexing movies with default settings                            1.00      18.8±0.09s        ? ?/sec           1.00      18.9±0.10s        ? ?/sec
indexing/Indexing nested movies with default settings                     1.00      25.9±0.19s        ? ?/sec           1.00      25.9±0.12s        ? ?/sec
indexing/Indexing nested movies without any facets                        1.00      24.8±0.17s        ? ?/sec           1.00      24.8±0.18s        ? ?/sec
indexing/Indexing songs in three batches with default settings            1.00      65.9±0.96s        ? ?/sec           1.03      67.8±0.82s        ? ?/sec
indexing/Indexing songs with default settings                             1.00      58.8±1.11s        ? ?/sec           1.02      59.9±2.09s        ? ?/sec
indexing/Indexing songs without any facets                                1.00      53.4±0.72s        ? ?/sec           1.01      54.2±0.88s        ? ?/sec
indexing/Indexing songs without faceted numbers                           1.00      57.9±1.17s        ? ?/sec           1.01      58.3±1.20s        ? ?/sec
indexing/Indexing wiki                                                    1.00   1065.2±13.26s        ? ?/sec           1.00   1065.8±12.66s        ? ?/sec
indexing/Indexing wiki in three batches                                   1.00    1182.4±6.20s        ? ?/sec           1.01    1190.8±8.48s        ? ?/sec
```

Most things do not change, we lost 0.1ms on the indexing of geo point (I don’t get why), and then we are between 500 and 1900 times faster when we delete documents.


Co-authored-by: Tamo <tamo@meilisearch.com>
2022-07-05 14:14:38 +00:00
ManyTheFish
a146fd45b9 Format API keys in hexa instead of base64 2022-07-05 16:14:18 +02:00
Tamo
250be9fe6c
put the threshold back to 10k 2022-07-05 15:57:44 +02:00
bors[bot]
62692c171d
Merge #577
577: Fix deserialisation of NDJson documents in benchmarks r=irevoire a=loiclec

Previously, the first document in the NDJson file was read over and over again. So the `geo_point` benchmark was not working properly: it only indexed one document.

Co-authored-by: Loïc Lecrenier <loic@meilisearch.com>
2022-07-05 13:54:47 +00:00
Loïc Lecrenier
9bc7627e27 Fix deserialisation of NDJson documents in benchmarks 2022-07-05 15:51:06 +02:00