Commit Graph

3344 Commits

Author SHA1 Message Date
bors[bot]
09ee8e34a5
Merge #2192
2192: Fix max dbs error r=Kerollmops a=MarinPostma

Factor the way we open environments to make sure they are always opened with the same options.


The issue was that indexes were first opened in snapshots with incorrect options, and heed cache returned an environment with incorrect open options on subsequent index open.

fix #2190


Co-authored-by: ad hoc <postma.marin@protonmail.com>
2022-02-23 16:18:23 +00:00
ad hoc
7e832105d7
bug(snapshot): Correctly open environments in snapshots 2022-02-23 17:12:08 +01:00
bors[bot]
ce6ff294cf
Merge #2171
2171: Update LICENSE with Meili SAS r=curquiza a=curquiza

Check with thomas, we must put the real name of the company

Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
2022-02-15 18:29:03 +00:00
Clémentine Urquizar - curqui
b318ab46cb
Update LICENSE 2022-02-15 15:54:45 +01:00
bors[bot]
5890600101
Merge #2170
2170: Update version (v0.26.0) r=curquiza a=curquiza



Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2022-02-14 15:22:09 +00:00
Clémentine Urquizar
e2a9414c7a
Update version (v0.26.0) 2022-02-14 16:11:07 +01:00
bors[bot]
216965e9d9
Merge #2122
2122: fix: docker image failed to boot on arm64 node r=curquiza a=Thearas

# Pull Request

## What does this PR do?
Fixes #2115.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?


Co-authored-by: Thearas <thearas850@gmail.com>
Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
2022-02-14 11:41:18 +00:00
Thearas
d0ddbcc2b2 update 2022-02-14 18:08:57 +08:00
Clémentine Urquizar - curqui
41db2601a6
Update Dockerfile 2022-02-14 10:44:03 +01:00
Thearas
42cb94e1f4 fix: docker image failed to boot on arm64 node 2022-02-14 13:23:20 +08:00
bors[bot]
6e7a0cc65d
Merge #2157
2157: fix(auth): fix env being closed when dumping r=Kerollmops a=MarinPostma

When creating a dump, the auth store environment would be closed on drop, so subsequent dumps couldn't reopen the environment. I have added a flag in the environment to prevent the closing of the environment on drop when dumping.


Co-authored-by: ad hoc <postma.marin@protonmail.com>
2022-02-09 15:04:19 +00:00
ad hoc
23eba82038
fix(auth): fix env being closed when dumping 2022-02-09 13:56:26 +01:00
bors[bot]
001b9acb63
Merge #2155
2155: Fix curl command in download-latest script r=Kerollmops a=curquiza

Fixes #2151 

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2022-02-08 15:29:51 +00:00
Clémentine Urquizar
af65ccfd6a
Fix curl command in download-latest script 2022-02-08 16:28:26 +01:00
bors[bot]
0c7251475d
Merge #2150
2150: Bump milli to v0.22.1 r=curquiza a=curquiza

Fixes https://github.com/meilisearch/meilisearch/issues/2138 and https://github.com/meilisearch/meilisearch/issues/2123 by bumping milli to [v0.22.1](https://github.com/meilisearch/milli/releases/tag/v0.22.1)

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2022-02-08 15:05:06 +00:00
Clémentine Urquizar
1a87b2f37d
Bump milli to v0.22.1 2022-02-08 11:21:44 +01:00
bors[bot]
752a0e13ad
Merge #2136
2136: Refactoring CI regarding ARM binary publish r=curquiza a=curquiza

Fixes https://github.com/meilisearch/meilisearch/issues/1909

- Remove CI file to publish aarch64 binary and put the logic into `publish-binary.yml`
- Remove the job to publish armv8 binary
- Fix download-latest script accordingly
- Adapt dowload-latest with the specific case of the MacOS m1

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: meili-bot <74670311+meili-bot@users.noreply.github.com>
2022-02-07 16:07:46 +00:00
Clémentine Urquizar
ccaca33446
Add --fail-with-body flag to curl in script 2022-02-07 16:16:49 +01:00
Clémentine Urquizar
2a90e805a2
Fix script 2022-02-07 16:05:48 +01:00
Clémentine Urquizar
c4a2d70d19
Fix error handler for curl command in script 2022-02-07 16:01:50 +01:00
meili-bot
f7e4a0177d
Update download-latest.sh
Co-authored-by: Clément Renault <clement@meilisearch.com>
2022-02-07 13:53:30 +01:00
bors[bot]
cca65499de
Merge #2145
2145: Update LICENSE r=meili-bot a=curquiza



Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
2022-02-07 12:51:50 +00:00
Clémentine Urquizar - curqui
80fa7dbbfa
Update LICENSE 2022-02-05 18:29:47 +01:00
bors[bot]
c24b1e5250
Merge #2135
2135: bug(auth): Make API keys accept Null descriptions r=curquiza a=ManyTheFish

Fix #2116


Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-02-03 15:26:11 +00:00
Clémentine Urquizar
78cf8f1f9f
Fix typo 2022-02-02 19:32:20 +01:00
Clémentine Urquizar
1da7277817
Fix dowload-latest.sh according to the new name of the binary 2022-02-02 19:25:52 +01:00
Clémentine Urquizar
c71c95feb0
Refactor CIs to publish aaarch64 binary 2022-02-02 19:25:28 +01:00
ManyTheFish
3bee31e6c7 bug(auth): Make API keys accept Null descriptions 2022-02-02 18:18:17 +01:00
bors[bot]
9448ca58aa
Merge #2005
2005: auto batching r=MarinPostma a=MarinPostma

This pr implements auto batching. The basic functioning of this is that all updates that can be batched together are batched together while the previous batch is being processed.

For now, the only updates that can be batched together are the document addition updates (both update and replace), for a single index.

The batching is disabled by default for multiple reasons:
- We need more experimentation with the scheduling techniques
- Right now, if one task fails in a batch, the whole batch fails. We need more permissive error handling when processing document indexation.

There are four CLI options, for now, to interact with how the batch is scheduled:
- `enable-autobatching`: enable the autobatching feature.
- `debounce-duration-sec`: When an update is received, wait that number of seconds before batching and performing the updates. Defaults to 0s.
- `max-batch-size`: the maximum number of tasks per batch, defaults to unlimited.
- `max-documents-per-batch`: the maximum number of documents in a batch, defaults to unlimited. The batch will always contain a least 1 task, no matter the number of documents in that task.

# Implementation

The current implementation is made of 3 major components:

## TaskStore
The `TaskStore` contains all the tasks. When a task is pushed, it is directly registered to the task store.

## Scheduler
The scheduler is in charge of making the batches. At its core, there is a `TaskQueue` and a job queue. `Job`s are always processed first. They are *volatile* tasks, that is, they don't have a TaskId and are not persisted to disk. Snapshots and dumps are examples of Jobs.

If no `Job` is available for processing, then the scheduler attempts to make a `Task` batch from the `TaskQueue`. The first step is to gather new tasks from the `TaskStore` to populate the `TaskQueue`. When this is done, we can prepare our batch. The `TaskQueue` is itself a `BinaryHeap` of `Tasklist`. Each `index_uid` is associated with a `TaskList` that contains all the updates associated with that index uid. Each `TaskList` in the `TaskQueue` is ordered by the id of its first task.

When preparing a batch, the `TaskList` at the top of the `TaskQueue` is popped, and the tasks are popped from the list to make the next batch. If there are remaining tasks in the list, the list is inserted back in the `TaskQueue`.

## UpdateLoop
The `UpdateLoop` role is to perform batch sequentially. Each time updates are pushed to the update store, the scheduler is notified, and will in turn notify the update loop that work can be performed. When notified, the update loop waits some time to wait for more incoming update and then asks the scheduler for the next batch to perform and perform it. When it is done, the status of the task is put back into the store, and the next batch is processed.


Co-authored-by: mpostma <postma.marin@protonmail.com>
2022-02-02 11:04:30 +00:00
mpostma
c9a236b0af
feat(lib): auto-batching 2022-02-01 18:06:20 +01:00
bors[bot]
622c15e825
Merge #2096
2096: feat(auth): Tenant token r=Kerollmops a=ManyTheFish

Make meilisearch support JWT authentication signed with meilisearch API keys
using HS256, HS384 or HS512 algorithms.

Related spec: [specifications#89](https://github.com/meilisearch/specifications/pull/89) [rendered](https://github.com/meilisearch/specifications/blob/scoped-api-keys/text/0089-tenant-tokens.md)
Fix #1991 


Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-01-27 10:38:41 +00:00
bors[bot]
054598734a
Merge #2120
2120: Bring `stable` into `main` r=curquiza a=curquiza

I forgot to do it, tell me `@Kerollmops` or `@irevoire` if it's useful or not. I would say yes, otherwise I will have conflict when I will try to bring `main` into `stable` for the next release. Maybe I'm wrong

Co-authored-by: Irevoire <tamo@meilisearch.com>
Co-authored-by: mpostma <postma.marin@protonmail.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: Clémentine Urquizar - curqui <clementine@meilisearch.com>
2022-01-27 09:35:21 +00:00
ManyTheFish
7ca647f0d0 feat(auth): Implement Tenant token
Make meilisearch support JWT authentication signed with meilisearch API keys
using HS256, HS384 or HS512 algorithms.

Related spec: https://github.com/meilisearch/specifications/pull/89
Fix #1991
2022-01-27 08:25:39 +01:00
Clémentine Urquizar - curqui
aa50fcb1f0
Merge branch 'main' into stable 2022-01-26 20:17:41 +01:00
Clémentine Urquizar - curqui
b408de0761
Merge pull request #2117 from meilisearch/rebranding
Changes related to the rebranding
2022-01-26 19:58:54 +01:00
Tamo
72d9c5ee5c
fix(rebranding): Update the ascii art (#2118) 2022-01-26 18:53:07 +01:00
Clémentine Urquizar
2b7440d4b5
Fix some typo 2022-01-26 17:56:18 +01:00
Clémentine Urquizar - curqui
3bc6a18bcd
Update README.md
Co-authored-by: Clément Renault <clement@meilisearch.com>
2022-01-26 17:54:51 +01:00
Clémentine Urquizar - curqui
db56d6cb11
Update download-latest.sh 2022-01-26 17:54:22 +01:00
Clémentine Urquizar - curqui
a5759139bf
Update CONTRIBUTING.md
Co-authored-by: Clément Renault <clement@meilisearch.com>
2022-01-26 17:51:38 +01:00
Clémentine Urquizar
8a959da120
Update MeiliSearch into Meilisearch everywhere 2022-01-26 17:43:16 +01:00
Clémentine Urquizar
0a78750465
Replace meilisearch by Meilisearch 2022-01-26 17:35:56 +01:00
Clémentine Urquizar
372f4fc924
Replace logo 2022-01-26 17:34:31 +01:00
meili-bot
ae5b401e74 Update README.md 2022-01-26 16:31:04 +01:00
meili-bot
c562655be7 Update CONTRIBUTING.md 2022-01-26 16:31:03 +01:00
bors[bot]
c8bb54cd94
Merge #2098
2098: feat(dump): Provide the same cli options as the snapshots r=MarinPostma a=irevoire

Add two cli options for the dump:
- `--ignore-missing-dump`
- `--ignore-dump-if-db-exists`

Fix #2087

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-01-26 14:32:23 +00:00
bors[bot]
8c80326dd5
Merge #2086
2086: feat(analytics): send the whole set of cli options instead of only the snapshot r=MarinPostma a=irevoire

Fixes #2088 

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-01-26 13:53:20 +00:00
Tamo
bad4bed439
feat(dump): Provide the same cli options as the snapshots
Add two cli options for the dump:
- `--ignore-missing-dump`
- `--ignore-dump-if-db-exists`

Fix #2087
2022-01-26 14:34:06 +01:00
Tamo
7828da15c3
feat(analytics): send the whole set of cli options instead of only the snapshot 2022-01-26 13:52:41 +01:00
bors[bot]
7e2f6063ae
Merge #2099 #2108
2099: feat(analytics): Set the timestamp of the aggregated event as the first aggregate r=MarinPostma a=irevoire



2108: meta(auth): Enhance tests on authorization r=MarinPostma a=ManyTheFish

Enhance auth tests in order to be able to add new actions without changing tests.

Helping #2080 

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-01-24 15:13:01 +00:00