searchengines-web/MeiliSearch
searchengines-web/MeiliSearch
1
0
Fork 0
mirror of https://github.com/meilisearch/MeiliSearch synced 2025-02-05 18:13:28 +01:00
Code Issues Releases Wiki Activity
9,810 Commits 192 Branches 415 Tags
Commit Graph

925 Commits

Author SHA1 Message Date
Tamo
c5f22be6e1 add boolean support for csv documents 2023-03-09 11:12:49 +01:00
ManyTheFish
b4b859ec8c Fix typos 2023-03-09 10:58:35 +01:00
ManyTheFish
7e2fd82e41 Use Language allow list in the highlighter 2023-03-08 12:44:16 +01:00
ManyTheFish
3092cf0448 Fix clippy errors 2023-03-08 10:53:42 +01:00
Louis Dureuil
2f5b9fbbd8
Restore contribution of the index sizes to the db size 
- the index size now contributes to the db size even if the index is not authorized
 2023-03-07 14:05:27 +01:00
Tamo
d34faa8f9c put back the sleep as it was and fix the from 2023-03-06 18:09:09 +01:00
Tamo
e5d0bef6d8 update a comment 2023-03-06 17:04:24 +01:00
Louis Dureuil
076a3d371c Eagerly compute stats as fallback to the cache. 
- Refactor all around to avoid spawning indexes more times than necessary
 2023-03-06 16:57:31 +01:00
Tamo
fd5c48941a Add cache on the indexes stats 2023-03-06 16:57:31 +01:00
Tamo
e704728ee7 fix the snapshots permissions on unix system 2023-03-06 16:28:40 +01:00
bors[bot]
3d1046369c
Merge #3529 
3529: Add an analytics on the geo bounding box feature r=ManyTheFish a=irevoire

Fixes #3527

[The specification of the geoBoundingBox](https://github.com/meilisearch/specifications/pull/223) feature has been updated and now introduces a new analytics to follow the usage of the geoBoundingBox feature in the search requests.

Co-authored-by: Tamo <tamo@meilisearch.com>
 2023-03-02 11:58:39 +00:00
bors[bot]
4f1ccbc495
Merge #3525 
3525: Fix phrase search containing stop words r=ManyTheFish a=ManyTheFish

# Summary
A search with a phrase containing only stop words was returning an HTTP error 500,
this PR filters the phrase containing only stop words dropping them before the search starts, a query with a phrase containing only stop words now behaves like a placeholder search.

fixes https://github.com/meilisearch/meilisearch/issues/3521

related v1.0.2 PR on milli: https://github.com/meilisearch/milli/pull/779



Co-authored-by: ManyTheFish <many@meilisearch.com>
 2023-03-02 10:55:37 +00:00
Louis Dureuil
d4d4702f1b
Rephrase hint message 2023-02-27 13:46:16 +01:00
Tamo
7ae10abb6b fix the auth tests 2023-02-23 17:27:42 +01:00
Tamo
dc533584c6 Forbid the usage of the metrics route if your API key have a limitation on the indexes 2023-02-23 17:13:22 +01:00
bors[bot]
89ac1015f3
Merge #3524 
3524: Update the metrics route r=irevoire a=irevoire

Fixes #3523

Make the metrics available by default without a feature flag.
+ Rename the cli-flag to `experimental-enable-metrics`.

Co-authored-by: Tamo <tamo@meilisearch.com>
 2023-02-23 15:11:10 +00:00
bors[bot]
ca25904c26
Merge #3331 
3331: Limit the number of concurrently opened indexes r=dureuill a=dureuill

# Pull Request

## Related issue
Relevant to #1841, fixes #3382

## What does this PR do?

### User standpoint

- Limit the number of concurrently opened indexes (currently, the number of indexes that can be concurrently opened is computed at startup)
- When too many an index is opened, the least recently used one is closed and its virtual memory released.
- This allows a user to have an arbitrary number of indexes of an arbitrary size

### Implementation standpoint

- Added a LRU cache map in `index-scheduler::lru`. A more complete implementation  (eg with helper functions not used here) is available but would better fit a dedicated crate.
- Use the LRU cache map in the `IndexScheduler`. To simplify the lifecycle of indexes, they are never removed from the cache when they are in the middle of a resize or delete operation. To achieve this, an intermediate `Vec` stores the UUIDs of the indexes that are in the middle of such an operation.
- Upon creating the index scheduler object, compute the total virtual memory that is adressable by using a dichotomic search on the max size of an index. Use this as a base to compute the number of indexes that can be open with 2TiB per index. If the virtual memory address space is lower than 2TiB, then only allow for 1 index of a fraction of that size.

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2023-02-23 14:20:52 +00:00
Tamo
8a1b1a95f3 comment the right of the metrics 2023-02-23 13:59:01 +01:00
Tamo
8d47d2d018 update the auth api after the rebase 2023-02-23 13:15:51 +01:00
Tamo
750a2b6842 Update meilisearch/src/option.rs 
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2023-02-23 12:26:22 +01:00
Tamo
bc7d4112d9 send the cli experimental feature in the analytics 2023-02-23 12:26:22 +01:00
Tamo
88a18677d0 rename the metrics cli flag 2023-02-23 12:26:22 +01:00
Tamo
68e30214ca remove the feature flag and reorganize the module slightly 2023-02-23 12:26:21 +01:00
bors[bot]
b985b96e4e
Merge #3530 
3530: Fix highlighter bug r=Kerollmops a=ManyTheFish

# Pull Request

There was a highlighting issue on CJK's character, we were highlighting too many characters and these additional characters were duplicated after the highlight tag.

## Related issue
Fixes #3517 
Fixes #3526 

## What does this PR do?
- add a test showcasing the bug
- fix the bug by activating the char_map creation of the tokenizer during the highlighting process


Co-authored-by: ManyTheFish <many@meilisearch.com>
 2023-02-23 10:59:43 +00:00
Louis Dureuil
3db613ff77
Don't iterate all indexes manually 2023-02-23 11:29:09 +01:00
Louis Dureuil
5822764be9
Skip computing index budget in tests 2023-02-23 11:23:39 +01:00
Louis Dureuil
c63294f331
Switch to 2TiB default index size, updates documentation 2023-02-23 11:23:39 +01:00
Louis Dureuil
80b060f920
Use LRU cache 2023-02-23 11:23:39 +01:00
bors[bot]
f62703cd67
Merge #3534 
3534: Update the csv error code from InvalidIndexCsvDelimiter to InvalidDocumentCsvDelimiter r=Kerollmops a=irevoire

Fixes #3533

Co-authored-by: Tamo <tamo@meilisearch.com>
 2023-02-23 07:05:12 +00:00
Tamo
76f82c880d update the csv error code from InvalidIndexCsvDelimiter to InvalidDocumentCsvDelimiter 2023-02-22 19:26:48 +01:00
ManyTheFish
28d6a4466d Make the tokenizer creating a char map during highlighting 2023-02-22 17:43:10 +01:00
Louis Dureuil
1ba2fae3ae
multi-search/authentication: Add authentication tests 2023-02-22 17:04:12 +01:00
Louis Dureuil
28d6ab78de
multi-search: Add multi search tests 2023-02-22 17:04:12 +01:00
Louis Dureuil
3ba5dfb6ec
multi-search: Add test server search method for multi search 2023-02-22 17:04:12 +01:00
Louis Dureuil
a23fbf6c7b
multi-search: Add search with an array of indexes 2023-02-22 17:04:12 +01:00
Louis Dureuil
596a98f7c6
multi-search: Add basic analytics 2023-02-22 16:37:18 +01:00
Louis Dureuil
14c4a222da
Authentication: AuthFilter::allow_index_creation both check that the index is authorized and the IndexCreate action 2023-02-22 16:37:13 +01:00
Louis Dureuil
690bb2e5cc
Authentication: Make allow_index_creation a private field 2023-02-22 16:35:52 +01:00
Louis Dureuil
42577403d8
Authentication: Directly pass the authfilter to the index scheduler 2023-02-22 16:35:52 +01:00
Louis Dureuil
c8c5944094
Authentication: is_index_authorized takes into account API key indexes even with a tenant token 2023-02-22 16:35:52 +01:00
Louis Dureuil
4b65851793
Authentication: Refactor authentication check to work for tenant token even without an index in URL 
Callers need to manually check `is_index_authorized` when using the route without an index in URL
 2023-02-22 16:35:51 +01:00
ManyTheFish
ad35edfa32 Add test 2023-02-22 15:47:15 +01:00
Tamo
033417e9cc add an analytics on the geo bounding box feature 2023-02-22 15:35:26 +01:00
bors[bot]
ac5a1e4c4b
Merge #3423 
3423: Add min and max facet stats r=dureuill a=dureuill

# Pull Request

## Related issue
Fixes #3426

## What does this PR do?

### User standpoint

- When using a `facets` parameter in search, the facets that have numeric values are displayed in a new section of the response called `facetStats` that contains, per facet, the numeric min and max value of the hits returned by the search.

<details>
<summary>
Sample request/response
</summary>

```json
❯ curl \
  -X POST 'http://localhost:7700/indexes/meteorites/search?facets=mass' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "LL6", "facets":["mass", "recclass"], "limit": 5 }' | jsonxf
{
  "hits": [
    {
      "name": "Niger (LL6)",
      "id": "16975",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 3.3,
      "fall": "Fell"
    },
    {
      "name": "Appley Bridge",
      "id": "2318",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 15000,
      "fall": "Fell",
      "_geo": {
        "lat": 53.58333,
        "lng": -2.71667
      }
    },
    {
      "name": "Athens",
      "id": "4885",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 265,
      "fall": "Fell",
      "_geo": {
        "lat": 34.75,
        "lng": -87.0
      }
    },
    {
      "name": "Bandong",
      "id": "4935",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 11500,
      "fall": "Fell",
      "_geo": {
        "lat": -6.91667,
        "lng": 107.6
      }
    },
    {
      "name": "Benguerir",
      "id": "30443",
      "nametype": "Valid",
      "recclass": "LL6",
      "mass": 25000,
      "fall": "Fell",
      "_geo": {
        "lat": 32.25,
        "lng": -8.15
      }
    }
  ],
  "query": "LL6",
  "processingTimeMs": 15,
  "limit": 5,
  "offset": 0,
  "estimatedTotalHits": 42,
  "facetDistribution": {
    "mass": {
      "110000": 1,
      "11500": 1,
      "1161": 1,
      "12000": 1,
      "1215.5": 1,
      "127000": 1,
      "15000": 1,
      "1676": 1,
      "1700": 1,
      "1710.5": 1,
      "18000": 1,
      "19000": 1,
      "220000": 1,
      "2220": 1,
      "22300": 1,
      "25000": 2,
      "265": 1,
      "271000": 1,
      "2840": 1,
      "3.3": 1,
      "3000": 1,
      "303": 1,
      "32000": 1,
      "34000": 1,
      "36.1": 1,
      "45000": 1,
      "460": 1,
      "478": 1,
      "483": 1,
      "5500": 2,
      "600": 1,
      "6000": 1,
      "67.8": 1,
      "678": 1,
      "680.5": 1,
      "6930": 1,
      "8": 1,
      "8300": 1,
      "840": 1,
      "8400": 1
    },
    "recclass": {
      "L/LL6": 3,
      "LL6": 39
    }
  },
  "facetStats": {
    "mass": {
      "min": 3.3,
      "max": 271000.0
    }
  }
}
```

</details>

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ ] Have you read the contributing guidelines?
- [ ] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2023-02-22 13:06:43 +00:00
ManyTheFish
6841f167b4 Add test 2023-02-21 18:02:52 +01:00
bors[bot]
39407885c2
Merge #3347 
3347: Enhance language detection r=irevoire a=ManyTheFish

## Summary

Some completely unrelated Languages can share the same characters, in Meilisearch we detect the Languages using `whatlang`, which works well on large texts but fails on small search queries leading to a bad segmentation and normalization of the query.

This PR now stores the Languages detected during the indexing in order to reduce the Languages list that can be detected during the search.

## Detail

- Create a 19th database mapping the scripts and the Languages detected with the documents where the Language is detected
- Fill the newly created database during indexing
- Create an allow-list with this database and pass it to Charabia
- Add a test ensuring that a Japanese request containing kanjis only is detected as Japanese and not Chinese

## Related issues
Fixes #2403
Fixes #3513

Co-authored-by: f3r10 <frledesma@outlook.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
 2023-02-21 10:52:13 +00:00
bors[bot]
a3e41ba33e
Merge #3496 
3496: Fix metrics feature r=irevoire a=james-2001

# Pull Request

## Related issue

Resolves: #3469
See also: #2763

## What does this PR do?
As reported the metrics feature was broken by still using and old reference to `meilisearch_auth::actions`. This commit switches to the new location, `meilisearch_types::keys::actions`.

The original issue was not *that* clear as to exactly what was broken, and the build logs have disappeared, but it seemed to just be this one line fix. If this is not the case and I've missed the mark let me know, and i'll head back to the drawing board.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Co-authored-by: James <james.a.may.2001@gmail.com>
 2023-02-21 10:13:11 +00:00
James
ce807d760b Fix formatting issue on Opt struct 
tab in enable_metrics_route to fix cargo fmt issues

Resolves: #3469
See also: #2763
 2023-02-21 09:45:18 +00:00
James
5cff435bf6 Add feature flags to Opt structure 
Resolves: #3469
See also: #2763
 2023-02-21 07:41:41 +00:00
ManyTheFish
8aa808d51b Merge branch 'main' into enhance-language-detection 2023-02-20 18:14:34 +01:00