Commit Graph

44 Commits

Author SHA1 Message Date
Tamo
953d3a44bd make the new_shared function synchronous and stop indexing documents when it's not required 2024-07-30 14:24:57 +02:00
Tamo
e5345fb0eb shave off 15s by providing a shared instance to the integration tests 2024-07-30 14:24:55 +02:00
Clément Renault
f07256971a
Fix tests 2024-07-10 16:28:14 +02:00
Tamo
3d6b61d8d2 fix flakyness for real 2024-06-26 09:24:09 +02:00
Tamo
1374b661d1 fix a possibly flaky test 2024-06-26 09:14:59 +02:00
Tamo
b130917933 add the content type in the webhook + improve the test 2024-03-05 11:22:29 +01:00
Tamo
77254765e8
get rids of env loggegr and fix the tests 2024-02-08 15:03:33 +01:00
Tamo
c83a33017e
stream and chunk the data 2023-12-19 10:35:51 +01:00
Tamo
be72326c0a
gzip the tasks 2023-12-19 10:35:51 +01:00
Tamo
0b2fff27f2
update and fix the test 2023-12-19 10:35:51 +01:00
Tamo
3adbc2b942
return a task view instead of a task 2023-12-19 10:35:51 +01:00
Tamo
fbea721378
add a first working test with actixweb 2023-12-19 10:35:51 +01:00
Tamo
391eb72137
start writing a test with actix but it doesn't works 2023-12-19 10:35:50 +01:00
Tamo
056b2c387d refactor the tests suite slightly 2023-09-11 16:56:26 +02:00
Tamo
2acc3ec5ee
fix the type of the document deletion by filter tasks 2023-05-30 15:18:52 +02:00
Tamo
b9dd092a62 make the details return null in the originalFilter field if no filter was provided + add a big test on the details 2023-05-24 11:48:22 +02:00
meili-bors[bot]
a95128df6b
Merge #3550
3550: Delete documents by filter r=irevoire a=dureuill

# Prototype `prototype-delete-by-filter-0`

Usage:
A new route is available under `POST /indexes/{index_uid}/documents/delete` that allows you to delete your documents by filter.
The expected payload looks like that:
```json
{
  "filter": "doggo = bernese",
}
```

It'll then enqueue a task in your task queue that'll delete all the documents matching this filter once it's processed.
Here is an example of the associated details;
```json
  "details": {
    "deletedDocuments": 53,
    "originalFilter": "\"doggo = bernese\""
  }
```

----------


# Pull Request

## Related issue
Related to https://github.com/meilisearch/meilisearch/issues/3477

## What does this PR do?

### User standpoint

- Modifies the `/indexes/{:indexUid}/documents/delete-batch` route to accept either the existing array of documents ids, or a JSON object with a `filter` field representing a filter to apply. If that latter variant is used, any document matching the filter will be deleted.

### Implementation standpoint

- (processing time version) Adds a new BatchKind that is not autobatchable and that performs the delete by filter
- Reuse the `documentDeletion` task with a new `originalFilter` detail that replaces the `providedIds` detail.

## Example

<details>
<summary>Sample request, response and task result</summary>

Request:

```
curl \
  -X POST 'http://localhost:7700/indexes/index-10/documents/delete-batch' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "filter" : "mass = 600"}'
```

Response:

```
{
  "taskUid": 3902,
  "indexUid": "index-10",
  "status": "enqueued",
  "type": "documentDeletion",
  "enqueuedAt": "2023-02-28T20:50:31.667502Z"
}
```

Task log:

```json
    {
      "uid": 3906,
      "indexUid": "index-12",
      "status": "succeeded",
      "type": "documentDeletion",
      "canceledBy": null,
      "details": {
        "deletedDocuments": 3,
        "originalFilter": "\"mass = 600\""
      },
      "error": null,
      "duration": "PT0.001819S",
      "enqueuedAt": "2023-03-07T08:57:20.11387Z",
      "startedAt": "2023-03-07T08:57:20.115895Z",
      "finishedAt": "2023-03-07T08:57:20.117714Z"
    }
```

</details>

## Draft status

- [ ] Error handling
- [ ] Analytics
- [ ] Do we want to reuse the `delete-batch` route in this way, or create a new route instead?
- [ ] Should the filter be applied at request time or when the deletion task is processed? 
  - The first commit in this PR applies the filter at request time, meaning that even if a document is modified in a way that no longer matches the filter in a later update, it will be deleted as long as the deletion task is processed after that update. 
  - The other commits in this PR apply the filter only when the asynchronous deletion task is processed, meaning that documents that match the filter at processing time are deleted even if they didn't match the filter at request time.
- [ ] If keeping the filter at request time, find a more elegant way to recover the user document ids from the internal document ids. The current way implemented in the first commit of this PR involves getting all the documents matching the filter, looking for the value of their primary key, and turning it into a string by copy-pasting routines found in milli...
- [ ] Security consideration, if any
- [ ] Fix the tests (but waiting until product questions are resolved)
- [ ] Add delete by filter specific tests



Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
2023-05-04 10:44:41 +00:00
Tamo
f9ddd32545
implement the auto-deletion of tasks 2023-05-04 00:06:49 +02:00
Tamo
143acb9cdc
update the tests 2023-05-03 17:41:49 +02:00
Louis Dureuil
c2f4b6ced0
Test: await for the deletion task to complete before trying to add another task 2023-04-13 18:22:42 +02:00
Louis Dureuil
1e6cbcaf12
Update test comment
Co-authored-by: Tamo <tamo@meilisearch.com>
2023-04-13 17:27:12 +02:00
Louis Dureuil
066c6bd875
test task db full now checks that a task can be successfully added after deleting tasks 2023-04-13 17:20:06 +02:00
Tamo
b3f60ee805 try to fix the ci 2023-04-13 10:18:58 +02:00
Tamo
b4fabce36d
update the error message + update the task db size to 20GiB with a limit at 50% 2023-04-12 18:54:11 +02:00
Tamo
9350a7b017
improve the test and try to understand the issue happening on windows 2023-04-12 18:54:11 +02:00
Tamo
be69ab320d
stops receiving tasks once the task queue is full 2023-04-12 18:54:11 +02:00
Louis Dureuil
d2420f5c8f
Fix non insta tests 2023-01-19 16:10:05 +01:00
Louis Dureuil
72e2b220ed
Fix tests 2023-01-19 15:48:20 +01:00
Tamo
e8e7070cc6
improve the error message when no task filter are specified for the cancelation or deletion of tasks 2023-01-19 12:42:08 +01:00
bors[bot]
3e5b3df487
Merge #3370 #3373 #3375
3370: make the swap indexes not found errors return an IndexNotFound error-code r=irevoire a=irevoire

Fix https://github.com/meilisearch/meilisearch/issues/3368

3373: fix a wrong error code and add tests on the document resource r=irevoire a=irevoire

Fix https://github.com/meilisearch/meilisearch/issues/3371

3375: Avoid deleting all task invalid canceled by r=irevoire a=Kerollmops

Fixes #3369 by making sure that at least one `canceledBy` task filter parameter matches something.

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
2023-01-18 15:21:11 +00:00
Kerollmops
d3c796af38
Add a new test to check that invalid canceledBy works correctly 2023-01-18 15:50:46 +01:00
Kerollmops
182eea1f17
Introduce a canceledBy filter for the tests 2023-01-18 15:50:42 +01:00
Tamo
57da80900d
make the swap indexes not found errors return an IndexNotFound error code 2023-01-18 14:16:00 +01:00
Loïc Lecrenier
9194508a0f Refactor query parameter deserialisation logic 2023-01-17 11:07:07 +01:00
Loïc Lecrenier
436ae4e466 Improve error messages generated by deserr
Split Json and Query Parameter error types
2023-01-17 09:43:07 +01:00
Loïc Lecrenier
a09b6a341d Move tasks route to deserr 2023-01-12 13:57:29 +01:00
Tamo
387874ea26
test the error codes on the task routes 2023-01-12 13:46:19 +01:00
Loïc Lecrenier
b0b7ad7caf
Apply review suggestions 2023-01-11 19:08:39 +01:00
Loïc Lecrenier
1fc11264e8
Refactor deserr integration 2023-01-11 19:08:39 +01:00
Tamo
d308684395 remove two ununsed error codes + fix the sort error_code 2023-01-10 11:32:11 +01:00
Tamo
e706628bb1 fix the error code of the swap index route 2023-01-06 14:48:25 +01:00
Tamo
50ce0409bc
Integrate deserr on the most important routes 2023-01-05 20:48:29 +01:00
Loïc Lecrenier
2d74678b51 Replace underscores with hyphens in doc link to error code 2023-01-05 10:09:02 +01:00
Colby Allen
ad2b1467da Renames meilisearch-http to meilisearch 2022-12-08 08:22:53 -07:00