MeiliDB

A full-text search database using a key-value store internally.

It uses RocksDB as the internal key-value store. The key-value store allows us to handle updates and queries with small memory and CPU overheads.

You can read the deep dive if you want more information on the engine, it describes the whole process of generating updates and handling queries.

We will be proud if you submit issues and pull requests. You can help to grow this project and start contributing by checking issues tagged "good-first-issue". It is a good start!

The project is only a library yet. It means that there is no binary provided yet. To get started, you can check the examples wich are made to work with the data located in the misc/ folder.

MeiliDB will be a binary in a near future so you will be able to use it as a database out-of-the-box. We should be able to query it using a to-be-defined protocol. This is our current goal, see the milestones. In the end, the binary will be a bunch of network protocols and wrappers around the library - which will also be published on crates.io. Both the binary and the library will follow the same update cycle.

Performances

These information are outdated (October 2018) and will be updated soon

We made some tests on remote machines and found that MeiliDB easily handles a dataset of near 280k products, on a $5/month server with a single vCPU and 1GB of RAM, running the same index, with a simple query:

• near 190 concurrent users with an average response time of 90ms
• 150 concurrent users with an average response time of 70ms
• 100 concurrent users with an average response time of 45ms

Servers were located in Amsterdam and tests were made between two different locations.

Notes

The default Rust allocator has recently been changed to use the system allocator. We have seen much better performances when using jemalloc as the global allocator.

Usage and examples

MeiliDB runs with an index like most search engines. So to test the library you can create one by indexing a simple csv file.

cargo run --release --example create-database -- test.mdb misc/kaggle.csv

Once the command is executed, the index should be in the test.mdb folder.

You are now able to run the query-database example and play with MeiliDB.

cargo run --release --example query-database -- test.mdb