673 Commits

Author	SHA1	Message	Date
bors[bot]	ddbcf449da	Merge #1763 ... 1763: Index tests r=MarinPostma a=MarinPostma This pr aims to test more thorougly the usage on index in the meilisearch database, by writing unit tests. work included: - [x] Create index mock and stub methods - [x] Test snapshot creation - [x] Test Dumps - [x] Test search Co-authored-by: mpostma <postma.marin@protonmail.com>	2021-10-06 14:39:53 +00:00
mpostma	85ae34cf9f	test snapshots	2021-10-06 14:10:23 +02:00
Tamo	fca686e7f8	bump meilisearch	2021-10-04 13:52:37 +02:00
Clémentine Urquizar	b17dae9ac0	Update version for the next release (v0.23.0)	2021-09-29 18:40:35 +02:00
bors[bot]	5fad37aebd	Merge #1711 ... 1711: MeiliSearch refactor introducing OBKV format r=MarinPostma a=MarinPostma This PR refactor some multiple components of meilisearch, and introduce the obkv document format to meilisearch - [x] Split meilisearch-http and meilisearch-lib - [x] Replace `IndexActor` and `UuidResolver` with `IndexResolver` - [x] Remove mentions to Actor - [x] Remove Actor traits to simplify code - [x] Integrate obkv document format - [x] Remove `Data` - [x] Restore all route - [x] Replace `Box<dyn error>` with `anyhow::Error` - [x] Introduce update file store - [x] Update file store error handling - [x] Fix dumps - [x] Fix snapshots - [x] Fix tests - [x] Update module documentation - [x] add csv suppport (feat `@ManyTheFish` #1729 ) - [x] add jsonl support - [x] integrate geosearch (feat `@irevoire` #1725) partially implements #1691 and #1690. The error handling is very basic now, I will finish it in the next pr. Some unit tests have been disabled, I will re-enable them ASAP, but they need a bit more work. close #1531 P.S: sorry for this monstrous PR :'( Co-authored-by: mpostma <postma.marin@protonmail.com> Co-authored-by: Tamo <tamo@meilisearch.com> Co-authored-by: many <maxime@meilisearch.com>	2021-09-29 14:38:55 +00:00
mpostma	311933614e	bump milli to v0.17.0	2021-09-29 15:44:54 +02:00
mpostma	1f537e1b60	jsonl support	2021-09-29 11:28:02 +02:00
mpostma	6e8a3fe8de	move csv parsing to document_formats	2021-09-28 22:58:48 +02:00
many	2a14948123	Use an existing revision of milli	2021-09-28 22:30:34 +02:00
many	d30830a55c	Add csv deserializer for documents	2021-09-28 22:28:13 +02:00
mpostma	5fa9bc67d7	remove unused dependencies	2021-09-28 22:16:18 +02:00
mpostma	3503fbf7fe	re-export milli from meilisearch_lib	2021-09-28 22:08:03 +02:00
mpostma	692c676625	fix tests	2021-09-28 18:57:36 +02:00
Tamo	654f49ccec	[WIP] put milli on branch main	2021-09-28 14:50:26 +02:00
Tamo	c1376a9f2a	add the geosearch to Meilisearch	2021-09-28 14:50:26 +02:00
mpostma	5353be74c3	refactor index actor	2021-09-22 15:07:04 +02:00
Tamo	2e99fa8251	remove the cargo.lock again	2021-09-22 11:30:33 +02:00
mpostma	aa6c5df0bc	Implement documents format ... document reader transform remove update format support document sequences fix document transform clean transform improve error handling add documents! macro fix transform bug fix tests remove csv dependency Add comments on the transform process replace search cli fmt review edits fix http ui fix clippy warnings Revert "fix clippy warnings" This reverts commit a1ce3cd96e603633dbf43e9e0b12b2453c9c5620. fix review comments remove smallvec in transform loop review edits	2021-09-21 16:58:33 +02:00
mpostma	60518449fc	split meilisearch-http and meilisearch-lib	2021-09-21 13:23:22 +02:00
mpostma	e14640e530	refactor meilisearch	2021-09-20 14:54:20 +02:00
happysalada	770b6d25ae	deps: unify pest dependency	2021-09-15 12:15:44 +09:00
Felix Yan	a57943b77e	Use tikv-jemallocator instead of jemallocator ... `jemallocator` has been abandoned for nearly two years, and `rustc` itself moved to use `tikv-jemallocator` instead: 3965773ae7 Let's switch to a better maintained version.	2021-09-14 18:30:24 +03:00
mpostma	0f7625e29a	bump dependencies	2021-09-13 15:17:08 +02:00
Clémentine Urquizar	6d2f7af642	Bump milli: fix some crashes	2021-09-13 12:14:54 +02:00
Tamo	49c918defa	bump milli	2021-09-08 17:41:47 +02:00
Clémentine Urquizar	33514b28be	Merge pull request #1588 from meilisearch/test-new-indexer ... Integrate the new indexer	2021-09-06 10:21:42 +02:00
Clément Renault	7e80337e5b	Bump milli to v0.12.0	2021-09-02 18:19:12 +02:00
many	8d4723d91b	Update lock file	2021-09-02 18:19:12 +02:00
Clémentine Urquizar	63e67f72e3	Update tokenizer and new milli version	2021-09-02 18:19:00 +02:00
Kerollmops	0cd66c3a89	Bump the milli version	2021-09-02 18:19:00 +02:00
Clément Renault	b092a624ed	Introduce the MaxMemory struct that defaults to 2/3 of the available memory	2021-09-02 18:18:59 +02:00
Clémentine Urquizar	24e84d7ca1	Test new indexer	2021-09-02 18:11:20 +02:00
Clémentine Urquizar	90116155b4	Update version for the next release (v0.22.0)	2021-09-01 12:33:30 +02:00
bors[bot]	19c5c74291	Merge #1652 #1654 #1657 ... 1652: Remove dependabot r=MarinPostma a=curquiza Fixes #1649 Dependabot for vulnerability and security updates is still activated. 1654: Add Script for Windows r=MarinPostma a=singh08prashant fixes #1570 changes: 1. added script for detecting windows os running git bash 2. appended `.exe` to `$release_file` for windows as listed [here](https://github.com/meilisearch/MeiliSearch/releases/) 3. removed global `$BINARY_NAME='meilisearch'` as windows require `.exe` file 1657: Bring vergen hotfix from `stable` to `main` r=MarinPostma a=curquiza Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com> Co-authored-by: singh08prashant <singh08prashant@gmail.com> Co-authored-by: Kerollmops <clement@meilisearch.com> Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>	2021-08-31 14:31:42 +00:00
Kerollmops	b8c954eb3f	Bump the MeiliSearch version to v0.21.1	2021-08-30 17:41:25 +02:00
Kerollmops	1782753387	Bump vergen and remove unused build feature	2021-08-30 15:03:45 +02:00
Kerollmops	51387b2c80	Introduce the new invalid sortable error codes	2021-08-25 17:29:30 +02:00
Irevoire	672fcee8aa	remove sentry	2021-08-24 12:38:31 +02:00
Clémentine Urquizar	149f46c184	Fix pacic when decoding	2021-08-23 12:37:51 +02:00
Clémentine Urquizar	38fc876704	Update tokenizer and new milli version with new tags	2021-08-18 14:55:10 +02:00
Clémentine Urquizar	39d5a99095	Update milli and tokenizer version	2021-08-18 12:09:34 +02:00
bors[bot]	2beb306834	Merge #1577 ... 1577: Update milli dependency: fix facet values bugs r=Kerollmops a=curquiza Fixes #1576 Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>	2021-08-16 16:13:42 +00:00
Clémentine Urquizar	f3e595e2f0	Update milli dependency	2021-08-16 13:36:42 +02:00
Clémentine Urquizar	7531280764	Update tokenizer version to v0.2.4	2021-08-12 13:55:47 +02:00
bors[bot]	7e3b2ddff2	Merge #1554 ... 1554: Fix dump v1 (attributesForFaceting, and criteria) r=curquiza a=MarinPostma close #1553 Co-authored-by: mpostma <postma.marin@protonmail.com>	2021-08-05 19:45:52 +00:00
mpostma	8f05d8d546	fix clippy warnings	2021-08-05 16:00:47 +02:00
Clémentine Urquizar	31fe263356	Update milli to v0.9.0	2021-08-05 13:08:27 +02:00
bors[bot]	9810f6b695	Merge #1540 ... 1540: Update milli to version 0.8.1 r=curquiza a=curquiza Integrates this fix into MeiliSearch https://github.com/meilisearch/milli/pull/296 Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>	2021-07-29 17:15:52 +00:00
mpostma	53b4b2fcbc	Use serdeval for validating json format.	2021-07-29 18:02:54 +02:00
Clémentine Urquizar	9a8629a6a9	Update milli	2021-07-29 17:45:31 +02:00
Clémentine Urquizar	e5dabf265a	Update milli version to v0.8.0	2021-07-28 10:52:47 +02:00
Tamo	0038b3848a	add a simple github cache	2021-07-26 15:31:26 +02:00
Tamo	f79b8287f5	update vergen	2021-07-26 15:25:30 +02:00
Kerollmops	0353fbb5df	Bump the tokenizer version to v0.2.4	2021-07-22 17:14:45 +02:00
Kerollmops	838ed1cd32	Use an u16 field id instead of one byte	2021-07-06 11:58:03 +02:00
Kerollmops	91c5d0c042	Use the AlwaysFreePages flag when opening an index	2021-07-05 16:36:13 +02:00
Tamo	9949a2a930	bump milli to 0.7.2	2021-07-05 12:19:27 +02:00
Kerollmops	a6b4069172	Bump to v0.7.2	2021-07-05 10:54:53 +02:00
Tamo	fc09d77e89	fix the benchmarks dependcies	2021-07-01 11:38:30 +02:00
Clémentine Urquizar	3c149d8a43	Update tokenizer version to v0.2.3	2021-06-30 18:41:35 +02:00
Clémentine Urquizar	4f0d3b065f	Update milli	2021-06-30 18:39:06 +02:00
Clémentine Urquizar	01de7f9e36	Update version	2021-06-30 15:59:59 +02:00
Clémentine Urquizar	b489515f4d	Update milli version to v0.7.1	2021-06-30 13:52:46 +02:00
Tamo	ec809ca487	use rustls instead of openssl and remove all default-features of reqwest	2021-06-29 13:07:40 +02:00
Clémentine Urquizar	c09e610bb5	Update heed and milli	2021-06-29 10:25:47 +02:00
Clément Renault	80c6aaf1fd	Bump milli to 0.7.0	2021-06-28 18:31:56 +02:00
Clément Renault	bdc5599b73	Bump heed to use the git repo with v0.12.0	2021-06-28 18:26:20 +02:00
Tamo	a59f437ee3	use only half of the computer threads for the indexation by default	2021-06-28 14:35:50 +02:00
marin postma	74bb748a4e	bump milli to 0.6.0	2021-06-23 18:40:19 +02:00
Kerollmops	98285b4b18	Bump milli to 0.6.0	2021-06-23 17:30:26 +02:00
Clémentine Urquizar	9885fb4159	Update version for the next release (v0.5.1)	2021-06-23 14:05:20 +02:00
Clémentine Urquizar	0de696feaf	Update version for alpha 6	2021-06-22 18:40:51 +02:00
Clémentine Urquizar	e2844f3a92	Update tokenizer version to v0.2.3	2021-06-22 15:57:47 +02:00
Clémentine Urquizar	2e3d85c31a	Update milli version to v0.5.0	2021-06-22 15:57:46 +02:00
Tamo	77eb37934f	add jemalloc to http-ui and the benchmarks	2021-06-22 14:17:56 +02:00
Clémentine Urquizar	320670f8fe	Update version for the next release (v0.5.0)	2021-06-21 15:59:17 +02:00
marin postma	0bfdf9a785	bump milli	2021-06-21 12:11:09 +02:00
marin postma	abdf642d68	integrate milli errors	2021-06-21 12:11:08 +02:00
Clémentine Urquizar	35fcc351a0	Update version for the next release (v0.4.2)	2021-06-20 17:37:24 +02:00
Kerollmops	ccd6f13793	Update version to the next release (0.4.1)	2021-06-17 15:01:20 +02:00
bors[bot]	2f2484e186	Merge #212 ... 212: bump milli to 0.4.0 r=MarinPostma a=MarinPostma Co-authored-by: marin postma <postma.marin@protonmail.com>	2021-06-16 15:42:34 +00:00
bors[bot]	2062b10b79	Merge #209 ... 209: Integrate amplitude r=MarinPostma a=irevoire And merge the sentry and amplitude usage under one “Enable analytics” flag closes #180 Co-authored-by: Tamo <tamo@meilisearch.com> Co-authored-by: Irevoire <tamo@meilisearch.com>	2021-06-16 15:25:31 +00:00
marin postma	5a47cef9a8	bump milli to 0.4.0	2021-06-16 17:15:56 +02:00
Clémentine Urquizar	f5ff3e8e19	Update version for the next release (v0.4.0)	2021-06-16 14:01:05 +02:00
Tamo	def1596eaf	Integrate amplitude ... And merge the sentry and amplitude usage under one “Enable analytics” flag	2021-06-15 15:36:30 +02:00
Clémentine Urquizar	fe5a494035	Update alpha for the next release	2021-06-14 17:55:04 +02:00
Kerollmops	312c2d1d8e	Use the Error enum everywhere in the project	2021-06-14 16:58:38 +02:00
bors[bot]	b119bb4ab0	Merge #197 ... 197: Update milli (v0.3.1) with filterable attributes r=MarinPostma a=curquiza Fixes #187 and #70 Also fixes #195 Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>	2021-06-14 12:19:42 +00:00
Clémentine Urquizar	9996c59183	Update with milli 0.3.1	2021-06-14 13:20:43 +02:00
Clémentine Urquizar	aa04124bfc	Add changes according to milli update	2021-06-14 13:20:37 +02:00
marin postma	0f767e3743	conccurrent update run loop	2021-06-14 10:57:14 +02:00
bors[bot]	1e659bb17b	Merge #194 ... 194: Bump sentry version r=MarinPostma a=irevoire closes #102 Co-authored-by: tamo <tamo@meilisearch.com>	2021-06-14 08:34:04 +00:00
Clémentine Urquizar	7d5395c12b	Update Tokenizer version to v0.2.3	2021-06-10 17:00:04 +02:00
Clémentine Urquizar	dc64e139b9	Update version for the next release (v0.3.1)	2021-06-09 14:39:21 +02:00
Kerollmops	103dddba2f	Move the UpdateStore into the http-ui crate	2021-06-08 17:59:51 +02:00
Kerollmops	29824d05ab	Reduce the length of the benchmarks names	2021-06-03 15:59:43 +02:00
tamo	4536dfccd0	add a way to provide primary_key or autogenerate documents ids	2021-06-02 11:13:20 +02:00
tamo	dffbaca63b	bump sentry version	2021-05-31 13:59:31 +02:00
tamo	26dcb9e66d	bump milli version and fix a performance issue for large dumps	2021-05-10 20:25:12 +02:00
Clémentine Urquizar	1e11578ef0	Update version for the next release (v0.2.1)	2021-05-05 14:57:34 +02:00
Clémentine Urquizar	eb03a3ccb1	Upgrade Milli and Tokenizer	2021-05-04 17:56:19 +02:00
Marin Postma	928fb34eff	bump milli and fix tests	2021-05-04 15:10:22 +02:00
Clémentine Urquizar	a8680887d8	Upgrade Milli version (v0.2.0)	2021-05-03 14:50:47 +02:00
Clémentine Urquizar	34e02aba42	Upgrade Tokenizer version (v0.2.2)	2021-05-03 10:55:55 +02:00
Marin Postma	78217bcf18	Fix cors authentication issue	2021-04-29 16:28:12 +02:00
Marin Postma	a961f0ce75	fix clippy warnings	2021-04-27 18:28:46 +02:00
Marin Postma	4fe2a13c71	rewrite update store	2021-04-27 15:20:52 +02:00
Marin Postma	5a38f13cae	multi_index udpate store	2021-04-27 15:07:13 +02:00
many	0d7d3ce802	Update roaring package	2021-04-27 14:39:53 +02:00
many	71740805a7	Fix forgotten typo tests	2021-04-27 14:39:53 +02:00
Kerollmops	0f4c0beffd	Introduce the Attribute criterion	2021-04-27 14:25:34 +02:00
Clémentine Urquizar	f3b6bf55a6	Update version for the next release (alpha4)	2021-04-26 19:05:16 +02:00
Marin Postma	e4bd1bc5ce	update actix-web-static-file rev	2021-04-22 11:42:41 +02:00
Marin Postma	6bcf20c70e	serve static site	2021-04-22 10:26:54 +02:00
Marin Postma	bb79695e44	load mini-dashboard assets	2021-04-22 10:26:54 +02:00
Marin Postma	ea5517bc8c	add mini-dashboard feature	2021-04-22 10:26:54 +02:00
Marin Postma	c6bb36efa5	implement _formated	2021-04-20 21:23:28 +02:00
Clémentine Urquizar	b8e535579f	Update version for the next release (alpha3)	2021-04-20 16:11:07 +02:00
Marin Postma	f8c338e3a7	add test for dedicated distinct route	2021-04-20 15:49:17 +02:00
mpostma	ec230c2835	enable distinct	2021-04-20 11:29:06 +02:00
Marin Postma	45665245dc	bump actix	2021-04-20 11:07:23 +02:00
Kerollmops	c9b2d3ae1a	Warn instead of returning an error when a conversion fails	2021-04-20 10:23:31 +02:00
Kerollmops	51767725b2	Simplify integer and float functions trait bounds	2021-04-20 10:23:31 +02:00
Kerollmops	efbfa81fa7	Merge the Float and Integer enum variant into the Number one	2021-04-20 10:23:30 +02:00
Clémentine Urquizar	2c380731b9	Update milli version to v0.1.1	2021-04-19 16:03:39 +02:00
Clémentine Urquizar	127d3d028e	Update version for the next release (v0.1.1)	2021-04-19 14:48:13 +02:00
Clémentine Urquizar	b0717b75d9	Update tokenizer to v0.2.1	2021-04-14 19:06:18 +02:00
Clémentine Urquizar	2c5c79d68e	Update Tokenizer version to v0.2.1	2021-04-14 18:54:04 +02:00
Alexey Shekhirin	3af8fa194c	test(http): combine settings assert_(ser|de)_tokens into 1 test	2021-04-10 12:13:59 +03:00
Alexey Shekhirin	ae1655586c	fixes after review	2021-04-09 14:40:48 +03:00
Alexey Shekhirin	dd9eae8c26	feat(http): stats route	2021-04-09 14:34:42 +03:00
Alexey Shekhirin	51ba1bd7d3	fix(http, index): init analyzer with optional stop words ... Next release update tokenizer	2021-04-08 17:16:13 +03:00
Alexey Shekhirin	84c1dda39d	test(http): setting enum serialize/deserialize	2021-04-08 17:03:40 +03:00
Alexey Shekhirin	dc636d190d	refactor(http, update): introduce setting enum	2021-04-08 17:03:40 +03:00
tamo	40ef9a3c6a	push a first implementation of the stop_words	2021-04-06 16:29:04 +02:00
Clémentine Urquizar	4ee6ce7871	Next release	2021-04-01 17:16:16 +02:00
tamo	62a8f1d707	bump the version of the tokenizer	2021-04-01 13:49:22 +02:00
tamo	73dcdb27f6	select a specific release of the tokenizer instead of using the latests git commit	2021-03-25 15:00:18 +01:00
mpostma	46293546f3	add tests and mocks	2021-03-24 12:38:59 +01:00
mpostma	8b99860e85	use write sender for updates	2021-03-18 08:32:05 +01:00
bors[bot]	ca3b343b1f	Merge #96 ... 96: Check json payload on document addition r=curquiza a=MarinPostma Check if the json payload in updates is valid. It uses a json validator to avoid allocation, and only serializes the json in case of error, to return a pretty message. Co-authored-by: mpostma <postma.marin@protonmail.com>	2021-03-16 17:20:44 +00:00
mpostma	204c743bcc	add json payload check on document addition	2021-03-16 14:28:13 +01:00
mpostma	6a742ee62c	restore version route	2021-03-15 19:11:27 +01:00
mpostma	c29b86849b	use actix cors git dependency	2021-03-15 17:40:20 +01:00
mpostma	f727dcc8c6	update milli	2021-03-15 14:26:59 +01:00
mpostma	80d0f9c49d	methods to update index time metadata	2021-03-15 14:05:47 +01:00
mpostma	adc71a70ce	fix displayed attributes in document retrieval	2021-03-15 10:17:41 +01:00
mpostma	3f68460d6c	fix update dedup	2021-03-11 20:58:51 +01:00
mpostma	79a4bc8129	use meta from milli	2021-03-11 19:40:18 +01:00
mpostma	a56e8c1a0c	fix tests	2021-03-10 14:47:04 +01:00
mpostma	5ecf514d28	restructure project	2021-03-10 13:46:49 +01:00
mpostma	562da9dd3f	fix test compilation	2021-03-10 11:56:51 +01:00
Clément Renault	b18ec00a7a	Add a logging_timer macro to te criterion next methods	2021-03-08 16:12:06 +01:00
Kerollmops	636a9df177	Temporarily fix the tinytemplate doc hidden issue	2021-03-08 15:57:45 +01:00
mpostma	7d28f8cff0	implement get single udpate	2021-03-06 10:51:52 +01:00
mpostma	f090f42e7a	multi index store ... create two channels for Index handler, one for writes and one for reads, so write are processed one at a time, while reads are processed in parallel.	2021-03-04 19:18:01 +01:00
Kerollmops	07784c8990	Tune the words prefixes threshold to compute for 1/1000 instead	2021-03-03 15:51:28 +01:00
Kerollmops	79a143b32f	Introduce the query tree data structure	2021-03-03 13:40:18 +01:00
mpostma	62532b8f79	WIP concurent index store	2021-03-02 14:05:03 +01:00
Clément Renault	9423310816	Introduce an helpers crate that export the database to stdout	2021-03-01 19:55:04 +01:00
mpostma	61ce749122	update tokio and disable all routes	2021-02-26 09:10:04 +01:00
Kerollmops	519b1cb5c9	Update dependencies	2021-02-21 10:26:04 +01:00
mpostma	91d6e90d5d	enable faceted searches	2021-02-16 19:20:39 +01:00
Clément Renault	fecf3d6fc1	Move the command lines helpers into different crates	2021-02-14 18:55:15 +01:00
Clément Renault	d8f3421608	Update the dependencies and remove the unused ones	2021-02-14 18:32:46 +01:00
Clément Renault	e8639517da	Change the project to become a workspace with milli as a default-member	2021-02-12 16:15:09 +01:00
mpostma	f8f02af23e	incorporate review changes	2021-02-04 13:21:15 +01:00
mpostma	9af0a08122	post review fixes	2021-02-02 17:34:06 +01:00
mpostma	17c463ca61	remove unused deps	2021-02-01 13:32:21 +01:00
mpostma	e9c95f6623	remove useless files	2021-01-28 19:43:54 +01:00
mpostma	6c63ee6798	implement list all indexes	2021-01-28 18:32:24 +01:00
mpostma	74410d8c6b	architecture rework	2021-01-28 14:12:34 +01:00
Clément Renault	433ac8c38a	Remove the ordered-float serde feature	2021-01-27 14:11:10 +01:00
Kerollmops	61dbcfa44a	Bump the roaring to 0.6.4	2021-01-26 14:38:43 +01:00
Clément Renault	51a37de885	Introduce the FacetValue enum type	2021-01-26 14:09:09 +01:00
mpostma	87a56d2bc9	Fix settings bug ... replace ids with str in settings This allows for better maintainability of the settings code, since updating the searchable attributes is now straightforward. criterion use string fix reindexing fieldid remaping add tests for primary_key compute fix tests fix http-ui fixup! add tests for primary_key compute code improvements settings update deps fixup! code improvements settings fixup! refactor settings updates and fix bug fixup! Fix settings bug fixup! Fix settings bug fixup! Fix settings bug Update src/update/index_documents/transform.rs Co-authored-by: Clément Renault <clement@meilisearch.com> fixup! Fix settings bug	2021-01-26 13:53:08 +01:00
mpostma	6a3f625e11	WIP: refactor IndexController ... change the architecture of the index controller to allow it to own an index store.	2021-01-16 15:09:48 +01:00
mpostma	686f987180	fix compile errors	2021-01-14 11:27:07 +01:00
mpostma	d22fab5bae	implement open index	2021-01-13 18:20:14 +01:00
mpostma	ddd7789713	WIP: IndexController	2021-01-13 17:50:36 +01:00
mpostma	4f7f7538f7	highlight with new tokenizer	2021-01-11 21:59:37 +01:00
mpostma	1ae761311e	integrate with meilisearch tokenizer	2021-01-07 16:14:27 +01:00
mpostma	b4d447b5cb	temp	2021-01-01 16:59:49 +01:00
mpostma	d1e9ded76f	setting builder takes ownership	2020-12-31 00:50:30 +01:00
mpostma	d9dc2036a7	support error & return document count on addition	2020-12-30 18:44:33 +01:00
mpostma	54861335a0	retrieve update status	2020-12-30 18:16:07 +01:00
mpostma	0cd9e62fc6	search first iteration	2020-12-24 12:58:34 +01:00
mpostma	1a38bfd31f	data add documents	2020-12-23 13:52:28 +01:00
mpostma	55e1552957	update queue refactor, first iteration	2020-12-22 17:13:50 +01:00
mpostma	7c9eaaeadb	clean code, and fix errors	2020-12-22 14:02:41 +01:00
Kerollmops	77e951e933	Use the byte-unit crate to ease library usage	2020-12-20 12:00:37 +01:00
Clément Renault	e7f2ab9138	Bump grenad to fix an indexing bug	2020-12-05 16:39:15 +01:00
Clément Renault	0959e1501f	Introduce the FacetRevRange Iterator struct	2020-12-04 12:02:23 +01:00
Clément Renault	61b383f422	Introduce the criteria update setting	2020-12-04 12:02:22 +01:00
Clément Renault	a0adfb5e8e	Introduce a real pest parser and support every facet filter conditions	2020-11-23 16:43:55 +01:00
Clément Renault	07a0c82790	Bump heed to 0.10.4 to use be able to lazily decode roaring bitmaps	2020-11-23 16:43:53 +01:00
Clément Renault	38c76754ef	Make the facet level search system generic on f64 and i64	2020-11-23 16:43:52 +01:00
Clément Renault	b255be93fa	Bump heed to 0.10.3	2020-11-23 16:43:49 +01:00
Clément Renault	a18d9a1f87	Parse and store the faceted fields	2020-11-13 16:13:51 +01:00
Clément Renault	640c7d748a	Modify the highlight function to support any JSON type	2020-11-05 13:59:32 +01:00
Clément Renault	0408c9d66a	Move the http server into its own sub-module	2020-11-05 11:16:39 +01:00
Clément Renault	4fded5bd0e	Bump heed to be able to reference a RoTxn from multiple threads	2020-11-02 12:49:23 +01:00
Clément Renault	f0d028d3a4	Update the Transform struct to support JSON updates	2020-10-31 20:52:49 +01:00
Clément Renault	9d47ee52b4	Generate a uuid v4 based document id when missing	2020-10-31 15:11:06 +01:00
Clément Renault	085d3b9d94	Update heed to 0.10.0	2020-10-30 11:42:00 +01:00
Clément Renault	b5d52b6b45	Prefer using a smallstr instead of a real String to reduce allocations	2020-10-29 14:32:32 +01:00
Clément Renault	98fc24cbdf	Bump heed to fix a prefix iter bug	2020-10-28 10:55:21 +01:00
Clément Renault	b44b04d25b	Serialize the CSV record values as JSON strings	2020-10-24 14:43:46 +02:00
Clément Renault	802e925fd7	Switch to a JSON protocol for the front page	2020-10-21 18:26:29 +02:00
Clément Renault	2210818114	Introduce the obkv heed codec	2020-10-21 15:51:48 +02:00
Clément Renault	f948a03be2	Optimise the merge functions to avoid allocations	2020-10-20 16:40:50 +02:00
Clément Renault	cde8478388	Replace the panic in the merge function by actual errors	2020-10-20 16:19:07 +02:00
Clément Renault	35c9a3c558	Brodacast the updates infos to every ws clients	2020-10-20 11:19:34 +02:00
Clément Renault	871222aebd	Introduce some new routes to handle live indexing	2020-10-19 16:06:43 +02:00
Clément Renault	65e32fecb1	Move the binaries into one with subcommands	2020-10-19 13:44:17 +02:00
Clément Renault	ff389f1270	Update heed-types to 0.7.1	2020-10-19 11:52:59 +02:00
Clément Renault	eca49e3a03	Introduce a notification channel for the UpdateStore	2020-10-18 16:37:37 +02:00
Clément Renault	83c1db8763	Introduce the UpdateStore	2020-10-18 15:26:57 +02:00
Clément Renault	9021b2dba6	Introduce the enable-chunk-fusing flag	2020-10-14 18:44:59 +02:00
Kerollmops	f980422c57	Move from oxidized-mtbl to grenad	2020-10-14 12:47:32 +02:00
Kerollmops	4e9bd1fef5	Bump oxidized-mtbl	2020-10-07 14:23:22 +02:00
Kerollmops	433d9bbc6e	Use CompressionType::from_str rather than a custom function	2020-10-06 13:50:34 +02:00
Kerollmops	4b819457c9	Enable the strucopt/clap warp help feature	2020-10-06 13:06:22 +02:00
Clément Renault	770f29fd05	Bump the oxidized-mtbl dependency	2020-10-04 17:04:33 +02:00
Kerollmops	68f4af7d2e	Improve the display of the number of processed documents	2020-09-29 16:08:58 +02:00
Clément Renault	ed05999f63	Replace the arc cache by a simple linked hash map	2020-09-23 14:50:52 +02:00
Clément Renault	d6fa9c0414	Index the intra documents word pair proximities	2020-09-22 14:04:33 +02:00
Kerollmops	3ded98e5fa	Bump the roaring version that fix a deserialization bug	2020-09-10 22:37:51 +02:00
Kerollmops	d5e5baa20f	Bump the oxidized-mtbl dependency	2020-09-10 13:29:12 +02:00
Kerollmops	0fb086f241	Use the crates.io raoring library	2020-09-08 15:16:04 +02:00
Clément Renault	bb1ab428db	Use another function to define the proximity	2020-09-06 17:55:07 +02:00
Clément Renault	f928b91e9d	Specify the exact rev for the near-proximity dep	2020-09-06 17:21:38 +02:00
Clément Renault	1c504471d3	Introduce the plane-sweep algorithm	2020-09-05 18:25:27 +02:00
Clément Renault	dc88a86259	Store the word positions under the documents	2020-09-05 18:03:06 +02:00
Kerollmops	580ed1119a	Make the engine to return csv string records as documents and headers	2020-08-31 19:02:00 +02:00
Clément Renault	bad0663138	Come back to the old tokenizer	2020-08-31 13:34:38 +02:00
Clément Renault	3fe497e129	Improve the Mtbl heed codec to only encode MTBL databases	2020-08-29 11:20:39 +02:00
Clément Renault	d19f394630	Make the indexer support gzipped CSV as input	2020-08-21 18:10:24 +02:00
Clément Renault	ff479c865d	Replace pipe by ringtail to improve stdin read performances	2020-08-21 17:45:52 +02:00
Clément Renault	8806fcd545	Introduce a better query and document lexer	2020-08-16 14:36:54 +02:00
Clément Renault	1e358e3ae8	Introduce the AstarBagIter that iterates through best paths	2020-08-15 16:24:06 +02:00
Clément Renault	fae694a102	Put the documents into an MTBL database	2020-08-07 12:14:40 +02:00
Clément Renault	405a71d3a4	Accept csv from stdin	2020-08-06 13:38:21 +02:00
Clément Renault	6508d497ce	Replace the regex highlighting by a simple algorithm	2020-08-05 13:52:27 +02:00
Clément Renault	bd4b18541c	Introduce a new indexer which uses an MTBL sorter	2020-08-04 15:44:37 +02:00
Kerollmops	085c376655	Use the regex crate to highlight "hello"	2020-07-14 11:28:40 +02:00
Kerollmops	12358476da	Use the log crate instead of stderr	2020-07-12 10:55:09 +02:00
Kerollmops	2c62eeea3c	Rename the project milli	2020-07-12 00:16:41 +02:00
Kerollmops	f6eae91c7d	Pretty print the new dashboard numbers	2020-07-11 14:17:37 +02:00
Kerollmops	11c7fef80a	Implement a memory dumper ... It moves the in memory HashMaps used when indexing to a disk based MTBL file	2020-07-07 16:48:49 +02:00
Kerollmops	7178b6c2c4	First basic version using MTBL again	2020-07-07 11:32:33 +02:00
Kerollmops	2a3b03138b	Use heed 0.8.1 with the RwIter append method	2020-07-05 19:50:28 +02:00
Kerollmops	46ced5c828	Introduce the RwIter append heed API	2020-07-04 12:34:10 +02:00
Kerollmops	2ae3f40971	Make the indexer ignore certain words ... This is a preparation for making the indexing fully parallel by making the indexer only be aware of certain words for each threads to avoid postings lists conflicts for each words	2020-07-01 17:49:46 +02:00
Kerollmops	f98b615bf3	Replace the LRU by an Arc cache	2020-06-29 20:48:57 +02:00
Kerollmops	07abebfc46	Introduce a (too big) LRU cache	2020-06-29 18:15:03 +02:00
Kerollmops	5f0088594b	Index by writing directly into LMDB	2020-06-29 13:54:47 +02:00
Kerollmops	d6705d5529	Introduce the criterion dependency to bench the engine	2020-06-19 18:32:25 +02:00
Kerollmops	55a8941922	Optimize things	2020-06-19 17:48:17 +02:00
Kerollmops	a8cda248b4	Introduce a customized A* algorithm. ... This custom algo lazily compute the intersections between words, to avoid too much set operations and database reads	2020-06-14 12:51:57 +02:00
Kerollmops	0a83a86e65	Fix multiple bugs	2020-06-11 11:55:03 +02:00
Kerollmops	13977d9338	squash-me	2020-06-09 23:06:59 +02:00
Kerollmops	dfdaceb410	Introduce a first basic working positions-based engine	2020-06-05 20:13:19 +02:00
Kerollmops	3a23dc242e	More efficiently merge MTBLs, more than two at a time	2020-06-04 16:17:24 +02:00
Kerollmops	dff68a339a	Use OnceCell to cache levenshtein builders	2020-05-31 19:27:11 +02:00
Kerollmops	a26553c90a	Reintroduce a simple HTTP server	2020-05-31 17:48:13 +02:00
Kerollmops	ba9527abc0	Support typos with a levenshtein automata	2020-05-31 17:01:11 +02:00
Kerollmops	6c726df9b9	Support multiple space seperated words	2020-05-31 16:09:34 +02:00
Kerollmops	24587148fd	Introduce MTBL parallel merging before LMDB writing	2020-05-31 14:22:57 +02:00
Kerollmops	3a998cf39c	Far better usage of rayon to fold indexed data	2020-05-31 14:22:57 +02:00
Kerollmops	1237306ca8	Introduce a thread that write to heed	2020-05-31 14:22:57 +02:00
Kerollmops	a81f201fad	Inroduce the use of RocksDB instead of sled (RAM)	2020-05-31 14:22:06 +02:00
Kerollmops	91ba938953	Initial commit	2020-05-31 14:22:06 +02:00