MeiliSearch/README.md

199 lines
9.9 KiB
Markdown
Raw Normal View History

2020-03-26 12:23:09 +01:00
<h1 align="center">
<img src="assets/logo.png" alt="MeiliSearch" />
</h1>
<h1 align="center">MeiliSearch</h1>
2019-11-26 11:06:55 +01:00
[![Build Status](https://github.com/meilisearch/MeiliSearch/workflows/Cargo%20test/badge.svg)](https://github.com/meilisearch/MeiliSearch/actions)
[![dependency status](https://deps.rs/repo/github/meilisearch/MeiliSearch/status.svg)](https://deps.rs/repo/github/meilisearch/MeiliSearch)
2019-11-28 14:11:35 +01:00
[![License](https://img.shields.io/badge/license-MIT-informational)](https://github.com/meilisearch/MeiliSearch/blob/master/LICENSE)
2020-03-25 19:57:22 +01:00
[![Slack](https://img.shields.io/badge/slack-MeiliSearch-blue.svg?logo=slack)](https://slack.meilisearch.com)
2020-03-26 12:23:09 +01:00
⚡ Ultra relevant and instant full-text **search API** 🔍
2019-11-12 16:51:08 +01:00
2020-03-26 12:23:09 +01:00
**MeiliSearch** is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box.
2020-03-26 11:59:40 +01:00
For more information about features go to [our documentation](https://docs.meilisearch.com/).
2019-12-02 17:17:16 +01:00
[![crates.io demo gif](misc/crates-io-demo.gif)](https://crates.meilisearch.com)
> Meili helps the Rust community find crates on [crates.meilisearch.com](https://crates.meilisearch.com)
## Features
2020-03-26 12:23:09 +01:00
* Search as-you-type experience (answers < 50 milliseconds)
2019-11-26 14:23:56 +01:00
* Full-text search
2020-03-26 12:23:09 +01:00
* Typo tolerant (understands typos and miss-spelling)
2019-11-26 14:23:56 +01:00
* Supports Kanji
* Supports Synonym
* Easy to install, deploy, and maintain
2020-03-26 11:59:40 +01:00
* Whole documents are returned
2019-11-26 14:23:56 +01:00
* Highly customizable
2020-03-26 11:59:40 +01:00
* RESTful API
2020-03-26 11:59:40 +01:00
## Get started
### Deploy the Server
2019-12-04 15:36:15 +01:00
#### Run it using Docker
2019-12-04 15:36:15 +01:00
```bash
docker run -it -p 7700:7700 --rm getmeili/meilisearch
2019-12-04 15:36:15 +01:00
```
2019-12-02 17:17:16 +01:00
2020-03-26 11:59:40 +01:00
#### Installing with Homebrew
2020-01-08 11:30:33 +01:00
```bash
brew update && brew install meilisearch
meilisearch
```
2020-03-26 11:59:40 +01:00
#### Installing with APT
```bash
echo "deb [trusted=yes] https://apt.fury.io/meilisearch/ /" > /etc/apt/sources.list.d/fury.list
apt update && apt install meilisearch-http
meilisearch
```
2019-12-04 15:36:15 +01:00
#### Download the binary
```bash
2019-12-02 17:17:16 +01:00
curl -L https://install.meilisearch.com | sh
./meilisearch
```
2019-12-11 17:24:22 +01:00
#### Run it on heroku
[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/meilisearch/MeiliSearch)
2019-12-04 15:36:15 +01:00
#### Compile and run it from sources
2020-03-26 11:59:40 +01:00
If you have the Rust toolchain already installed on your local system, clone the repository and change it to your working directory. In the cloned repository, compile MeiliSearch.
2019-12-04 15:36:15 +01:00
```bash
git clone https://github.com/meilisearch/MeiliSearch.git
cd MeiliSearch
cargo run --release
```
### Create an Index and Upload Some Documents
2020-03-26 11:59:40 +01:00
Let's create an index! If you need a sample dataset, use [this movie database](https://www.notion.so/meilisearch/A-movies-dataset-to-test-Meili-1cbf7c9cfa4247249c40edfa22d7ca87#b5ae399b81834705ba5420ac70358a65). You can also find it in the `datasets/` directory.
2019-12-02 17:17:16 +01:00
```bash
2019-12-10 16:49:22 +01:00
curl -L 'https://bit.ly/2PAcw9l' -o movies.json
2019-12-02 17:17:16 +01:00
```
2020-03-26 11:59:40 +01:00
MeiliSearch can serve multiple indexes, with different kinds of documents.
It is required to create an index before sending documents to it.
```bash
2019-11-29 18:02:32 +01:00
curl -i -X POST 'http://127.0.0.1:7700/indexes' --data '{ "name": "Movies", "uid": "movies" }'
```
2020-03-26 11:59:40 +01:00
Now that the server knows about your brand new index, you're ready to send it some data.
```bash
2019-11-29 18:02:32 +01:00
curl -i -X POST 'http://127.0.0.1:7700/indexes/movies/documents' \
--header 'content-type: application/json' \
2019-12-02 17:17:16 +01:00
--data-binary @movies.json
```
### Search for Documents
#### In command line
2020-03-26 12:23:09 +01:00
The search engine is now aware of your documents and can serve those via a HTTP server.
2020-03-26 11:59:40 +01:00
The [`jq` command-line tool](https://stedolan.github.io/jq/) can greatly help you read the server responses.
```bash
2019-12-02 17:17:16 +01:00
curl 'http://127.0.0.1:7700/indexes/movies/search?q=botman+robin&limit=2' | jq
```
```json
{
"hits": [
{
2019-12-02 17:17:16 +01:00
"id": "415",
"title": "Batman & Robin",
"poster": "https://image.tmdb.org/t/p/w1280/79AYCcxw3kSKbhGpx1LiqaCAbwo.jpg",
"overview": "Along with crime-fighting partner Robin and new recruit Batgirl...",
"release_date": "1997-06-20",
},
{
2019-12-02 17:17:16 +01:00
"id": "411736",
"title": "Batman: Return of the Caped Crusaders",
"poster": "https://image.tmdb.org/t/p/w1280/GW3IyMW5Xgl0cgCN8wu96IlNpD.jpg",
"overview": "Adam West and Burt Ward returns to their iconic roles of Batman and Robin...",
"release_date": "2016-10-08",
}
],
"offset": 0,
"limit": 2,
"processingTimeMs": 1,
2019-12-02 17:17:16 +01:00
"query": "botman robin"
}
```
2020-03-26 11:59:40 +01:00
#### Use the Web Interface
2020-03-26 12:23:09 +01:00
We also deliver an **out-of-the-box web interface** in which you can test MeiliSearch interactively.
2020-03-26 11:59:40 +01:00
You can access the web interface in your web browser at the root of the server. The default URL is [http://127.0.0.1:7700](http://127.0.0.1:7700). All you need to do is open your web browser and enter MeiliSearchs address to visit it. This will lead you to a web page with a search bar that allows you to search in a given set of documents.
2019-12-02 17:17:16 +01:00
### Documentation
2020-03-26 11:59:40 +01:00
Now that your MeiliSearch server is up and running, you can learn more about how to tune your search engine in [the documentation](https://docs.meilisearch.com).
2019-12-02 17:17:16 +01:00
## How it works
2020-03-26 11:59:40 +01:00
MeiliSearch uses [LMDB](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database) as the internal key-value store. The key-value store allows to handle updates and queries with small memory and CPU overheads. The whole ranking system is [data oriented](https://github.com/meilisearch/MeiliSearch/issues/82) and ensures great performances.
2019-12-02 17:17:16 +01:00
2020-03-26 11:59:40 +01:00
You can read [this document](deep-dive.md) if you want to dive deeper into the engine. The whole process of generating updates and handling queries is described in it. Besides, to learn the default rules used for sorting documents, you can take a look at this [typos and ranking rules explanation](typos-ranking-rules.md).
2019-12-02 17:17:16 +01:00
### Technical features
- Provides [6 default ranking criteria](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-core/src/criterion/mod.rs#L106-L111) used to [bucket sort](https://en.wikipedia.org/wiki/Bucket_sort) documents
- Accepts [custom criteria](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-core/src/criterion/mod.rs#L20-L29) and can apply them in any custom order
2020-03-26 11:59:40 +01:00
- Supports [ranged queries](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-core/src/query_builder.rs#L342), useful for paginating results
2019-12-02 17:17:16 +01:00
- Can [distinct](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-core/src/query_builder.rs#L324-L329) and [filter](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-core/src/query_builder.rs#L313-L318) returned documents based on context defined rules
- Searches for [concatenated](https://github.com/meilisearch/MeiliSearch/pull/164) and [splitted query words](https://github.com/meilisearch/MeiliSearch/pull/232) to improve the search quality.
- Can store complete documents or only [user schema specified fields](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/datasets/movies/schema.toml)
2020-03-26 12:23:09 +01:00
- The [default tokenizer](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-tokenizer/src/lib.rs) can index Latin and Kanji based languages
2019-12-02 17:17:16 +01:00
- Returns [the matching text areas](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-types/src/lib.rs#L49-L65), useful to highlight matched words in results
- Accepts query time search config like the [searchable attributes](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-core/src/query_builder.rs#L331-L336)
- Supports [runtime incremental indexing](https://github.com/meilisearch/MeiliSearch/blob/3ea5aa18a209b6973b921542d46a79e1c753c163/meilisearch-core/src/store/mod.rs#L143-L212)
## Performances
2020-03-26 11:59:40 +01:00
When processing a dataset composed of _100 353_ documents with _352_ attributes each and _3_ of them indexed, which means more than _300 000_ fields indexed for _35 million_ stored, MeiliSearch is able to carry out more than _2.8k req/sec_ with an average response time of _9 ms_ on an Intel i7-7700 (8) @ 4.2GHz.
2019-11-26 14:23:56 +01:00
Requests are made using [wrk](https://github.com/wg/wrk) and scripted to simulate real users' queries.
```
Running 10s test @ http://localhost:2230
2 threads and 25 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.52ms 7.61ms 99.25ms 84.58%
Req/Sec 1.41k 119.11 1.78k 64.50%
28080 requests in 10.01s, 7.42MB read
Requests/sec: 2806.46
Transfer/sec: 759.17KB
```
2020-03-26 11:59:40 +01:00
We also indexed a dataset containing about _12 millions_ cities names in _24 minutes_ on a _8 cores_, _64 GB of RAM_, and a _300 GB NMVe_ SSD machine.<br/>
The size of the resulting database reached _16 GB_ and search results were presented between _30 ms_ and _4 seconds_ for short prefix queries.
2019-11-14 19:09:04 +01:00
### Notes
2020-03-26 11:59:40 +01:00
In Rust 1.32, the allocator has been [changed to use the system allocator](https://blog.rust-lang.org/2019/01/17/Rust-1.32.0.html#jemalloc-is-removed-by-default).
We observed significant performance improvements when [using jemalloc as the global allocator](https://github.com/alexcrichton/jemallocator#documentation).
2019-11-26 14:23:56 +01:00
## Contributing
2020-03-26 11:59:40 +01:00
Hey! We're glad you're thinking about contributing to MeiliSearch! If you think something is missing or could be improved, please open issues and pull requests. If you'd like to help this project grow, we'd love to have you! To start contributing, checking [issues tagged as "good-first-issue"](https://github.com/meilisearch/MeiliSearch/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) is a good start!
2019-11-21 19:15:33 +01:00
### Analytic Events
2020-03-26 11:59:40 +01:00
Once a day, events are being sent to our Amplitude instance so we can know how many people are using MeiliSearch.<br/>
Only information concerning the platform on which the server runs is concerned. No other information is being sent.<br/>
If this doesn't suit you, you can disable these analytics by using the `MEILI_NO_ANALYTICS` env variable.