684 Commits

Author	SHA1	Message	Date
mpostma	8b99860e85	use write sender for updates	2021-03-18 08:32:05 +01:00
bors[bot]	ca3b343b1f	Merge #96 ... 96: Check json payload on document addition r=curquiza a=MarinPostma Check if the json payload in updates is valid. It uses a json validator to avoid allocation, and only serializes the json in case of error, to return a pretty message. Co-authored-by: mpostma <postma.marin@protonmail.com>	2021-03-16 17:20:44 +00:00
mpostma	204c743bcc	add json payload check on document addition	2021-03-16 14:28:13 +01:00
mpostma	6a742ee62c	restore version route	2021-03-15 19:11:27 +01:00
mpostma	c29b86849b	use actix cors git dependency	2021-03-15 17:40:20 +01:00
mpostma	f727dcc8c6	update milli	2021-03-15 14:26:59 +01:00
mpostma	80d0f9c49d	methods to update index time metadata	2021-03-15 14:05:47 +01:00
mpostma	adc71a70ce	fix displayed attributes in document retrieval	2021-03-15 10:17:41 +01:00
mpostma	3f68460d6c	fix update dedup	2021-03-11 20:58:51 +01:00
mpostma	79a4bc8129	use meta from milli	2021-03-11 19:40:18 +01:00
mpostma	a56e8c1a0c	fix tests	2021-03-10 14:47:04 +01:00
mpostma	5ecf514d28	restructure project	2021-03-10 13:46:49 +01:00
mpostma	562da9dd3f	fix test compilation	2021-03-10 11:56:51 +01:00
Clément Renault	b18ec00a7a	Add a logging_timer macro to te criterion next methods	2021-03-08 16:12:06 +01:00
Kerollmops	636a9df177	Temporarily fix the tinytemplate doc hidden issue	2021-03-08 15:57:45 +01:00
mpostma	7d28f8cff0	implement get single udpate	2021-03-06 10:51:52 +01:00
mpostma	f090f42e7a	multi index store ... create two channels for Index handler, one for writes and one for reads, so write are processed one at a time, while reads are processed in parallel.	2021-03-04 19:18:01 +01:00
Kerollmops	07784c8990	Tune the words prefixes threshold to compute for 1/1000 instead	2021-03-03 15:51:28 +01:00
Kerollmops	79a143b32f	Introduce the query tree data structure	2021-03-03 13:40:18 +01:00
mpostma	62532b8f79	WIP concurent index store	2021-03-02 14:05:03 +01:00
Clément Renault	9423310816	Introduce an helpers crate that export the database to stdout	2021-03-01 19:55:04 +01:00
mpostma	61ce749122	update tokio and disable all routes	2021-02-26 09:10:04 +01:00
Kerollmops	519b1cb5c9	Update dependencies	2021-02-21 10:26:04 +01:00
mpostma	91d6e90d5d	enable faceted searches	2021-02-16 19:20:39 +01:00
Clément Renault	fecf3d6fc1	Move the command lines helpers into different crates	2021-02-14 18:55:15 +01:00
Clément Renault	d8f3421608	Update the dependencies and remove the unused ones	2021-02-14 18:32:46 +01:00
Clément Renault	e8639517da	Change the project to become a workspace with milli as a default-member	2021-02-12 16:15:09 +01:00
mpostma	f8f02af23e	incorporate review changes	2021-02-04 13:21:15 +01:00
mpostma	9af0a08122	post review fixes	2021-02-02 17:34:06 +01:00
mpostma	17c463ca61	remove unused deps	2021-02-01 13:32:21 +01:00
mpostma	e9c95f6623	remove useless files	2021-01-28 19:43:54 +01:00
mpostma	6c63ee6798	implement list all indexes	2021-01-28 18:32:24 +01:00
mpostma	74410d8c6b	architecture rework	2021-01-28 14:12:34 +01:00
Clément Renault	433ac8c38a	Remove the ordered-float serde feature	2021-01-27 14:11:10 +01:00
Kerollmops	61dbcfa44a	Bump the roaring to 0.6.4	2021-01-26 14:38:43 +01:00
Clément Renault	51a37de885	Introduce the FacetValue enum type	2021-01-26 14:09:09 +01:00
mpostma	87a56d2bc9	Fix settings bug ... replace ids with str in settings This allows for better maintainability of the settings code, since updating the searchable attributes is now straightforward. criterion use string fix reindexing fieldid remaping add tests for primary_key compute fix tests fix http-ui fixup! add tests for primary_key compute code improvements settings update deps fixup! code improvements settings fixup! refactor settings updates and fix bug fixup! Fix settings bug fixup! Fix settings bug fixup! Fix settings bug Update src/update/index_documents/transform.rs Co-authored-by: Clément Renault <clement@meilisearch.com> fixup! Fix settings bug	2021-01-26 13:53:08 +01:00
mpostma	6a3f625e11	WIP: refactor IndexController ... change the architecture of the index controller to allow it to own an index store.	2021-01-16 15:09:48 +01:00
mpostma	686f987180	fix compile errors	2021-01-14 11:27:07 +01:00
mpostma	d22fab5bae	implement open index	2021-01-13 18:20:14 +01:00
mpostma	ddd7789713	WIP: IndexController	2021-01-13 17:50:36 +01:00
mpostma	4f7f7538f7	highlight with new tokenizer	2021-01-11 21:59:37 +01:00
mpostma	1ae761311e	integrate with meilisearch tokenizer	2021-01-07 16:14:27 +01:00
mpostma	b4d447b5cb	temp	2021-01-01 16:59:49 +01:00
mpostma	d1e9ded76f	setting builder takes ownership	2020-12-31 00:50:30 +01:00
mpostma	d9dc2036a7	support error & return document count on addition	2020-12-30 18:44:33 +01:00
mpostma	54861335a0	retrieve update status	2020-12-30 18:16:07 +01:00
mpostma	0cd9e62fc6	search first iteration	2020-12-24 12:58:34 +01:00
mpostma	1a38bfd31f	data add documents	2020-12-23 13:52:28 +01:00
mpostma	55e1552957	update queue refactor, first iteration	2020-12-22 17:13:50 +01:00
mpostma	7c9eaaeadb	clean code, and fix errors	2020-12-22 14:02:41 +01:00
Kerollmops	77e951e933	Use the byte-unit crate to ease library usage	2020-12-20 12:00:37 +01:00
Clément Renault	e7f2ab9138	Bump grenad to fix an indexing bug	2020-12-05 16:39:15 +01:00
Clément Renault	0959e1501f	Introduce the FacetRevRange Iterator struct	2020-12-04 12:02:23 +01:00
Clément Renault	61b383f422	Introduce the criteria update setting	2020-12-04 12:02:22 +01:00
Clément Renault	a0adfb5e8e	Introduce a real pest parser and support every facet filter conditions	2020-11-23 16:43:55 +01:00
Clément Renault	07a0c82790	Bump heed to 0.10.4 to use be able to lazily decode roaring bitmaps	2020-11-23 16:43:53 +01:00
Clément Renault	38c76754ef	Make the facet level search system generic on f64 and i64	2020-11-23 16:43:52 +01:00
Clément Renault	b255be93fa	Bump heed to 0.10.3	2020-11-23 16:43:49 +01:00
Clément Renault	a18d9a1f87	Parse and store the faceted fields	2020-11-13 16:13:51 +01:00
Clément Renault	640c7d748a	Modify the highlight function to support any JSON type	2020-11-05 13:59:32 +01:00
Clément Renault	0408c9d66a	Move the http server into its own sub-module	2020-11-05 11:16:39 +01:00
Clément Renault	4fded5bd0e	Bump heed to be able to reference a RoTxn from multiple threads	2020-11-02 12:49:23 +01:00
Clément Renault	f0d028d3a4	Update the Transform struct to support JSON updates	2020-10-31 20:52:49 +01:00
Clément Renault	9d47ee52b4	Generate a uuid v4 based document id when missing	2020-10-31 15:11:06 +01:00
Clément Renault	085d3b9d94	Update heed to 0.10.0	2020-10-30 11:42:00 +01:00
Clément Renault	b5d52b6b45	Prefer using a smallstr instead of a real String to reduce allocations	2020-10-29 14:32:32 +01:00
Clément Renault	98fc24cbdf	Bump heed to fix a prefix iter bug	2020-10-28 10:55:21 +01:00
Clément Renault	b44b04d25b	Serialize the CSV record values as JSON strings	2020-10-24 14:43:46 +02:00
Clément Renault	802e925fd7	Switch to a JSON protocol for the front page	2020-10-21 18:26:29 +02:00
Clément Renault	2210818114	Introduce the obkv heed codec	2020-10-21 15:51:48 +02:00
Clément Renault	f948a03be2	Optimise the merge functions to avoid allocations	2020-10-20 16:40:50 +02:00
Clément Renault	cde8478388	Replace the panic in the merge function by actual errors	2020-10-20 16:19:07 +02:00
Clément Renault	35c9a3c558	Brodacast the updates infos to every ws clients	2020-10-20 11:19:34 +02:00
Clément Renault	871222aebd	Introduce some new routes to handle live indexing	2020-10-19 16:06:43 +02:00
Clément Renault	65e32fecb1	Move the binaries into one with subcommands	2020-10-19 13:44:17 +02:00
Clément Renault	ff389f1270	Update heed-types to 0.7.1	2020-10-19 11:52:59 +02:00
Clément Renault	eca49e3a03	Introduce a notification channel for the UpdateStore	2020-10-18 16:37:37 +02:00
Clément Renault	83c1db8763	Introduce the UpdateStore	2020-10-18 15:26:57 +02:00
Clément Renault	9021b2dba6	Introduce the enable-chunk-fusing flag	2020-10-14 18:44:59 +02:00
Kerollmops	f980422c57	Move from oxidized-mtbl to grenad	2020-10-14 12:47:32 +02:00
Kerollmops	4e9bd1fef5	Bump oxidized-mtbl	2020-10-07 14:23:22 +02:00
Kerollmops	433d9bbc6e	Use CompressionType::from_str rather than a custom function	2020-10-06 13:50:34 +02:00
Kerollmops	4b819457c9	Enable the strucopt/clap warp help feature	2020-10-06 13:06:22 +02:00
Clément Renault	770f29fd05	Bump the oxidized-mtbl dependency	2020-10-04 17:04:33 +02:00
Kerollmops	68f4af7d2e	Improve the display of the number of processed documents	2020-09-29 16:08:58 +02:00
Clément Renault	ed05999f63	Replace the arc cache by a simple linked hash map	2020-09-23 14:50:52 +02:00
Clément Renault	d6fa9c0414	Index the intra documents word pair proximities	2020-09-22 14:04:33 +02:00
Kerollmops	3ded98e5fa	Bump the roaring version that fix a deserialization bug	2020-09-10 22:37:51 +02:00
Kerollmops	d5e5baa20f	Bump the oxidized-mtbl dependency	2020-09-10 13:29:12 +02:00
Kerollmops	0fb086f241	Use the crates.io raoring library	2020-09-08 15:16:04 +02:00
Clément Renault	bb1ab428db	Use another function to define the proximity	2020-09-06 17:55:07 +02:00
Clément Renault	f928b91e9d	Specify the exact rev for the near-proximity dep	2020-09-06 17:21:38 +02:00
Clément Renault	1c504471d3	Introduce the plane-sweep algorithm	2020-09-05 18:25:27 +02:00
Clément Renault	dc88a86259	Store the word positions under the documents	2020-09-05 18:03:06 +02:00
Kerollmops	580ed1119a	Make the engine to return csv string records as documents and headers	2020-08-31 19:02:00 +02:00
Clément Renault	bad0663138	Come back to the old tokenizer	2020-08-31 13:34:38 +02:00
Clément Renault	3fe497e129	Improve the Mtbl heed codec to only encode MTBL databases	2020-08-29 11:20:39 +02:00
Clément Renault	d19f394630	Make the indexer support gzipped CSV as input	2020-08-21 18:10:24 +02:00
Clément Renault	ff479c865d	Replace pipe by ringtail to improve stdin read performances	2020-08-21 17:45:52 +02:00
Clément Renault	8806fcd545	Introduce a better query and document lexer	2020-08-16 14:36:54 +02:00
Clément Renault	1e358e3ae8	Introduce the AstarBagIter that iterates through best paths	2020-08-15 16:24:06 +02:00
Clément Renault	fae694a102	Put the documents into an MTBL database	2020-08-07 12:14:40 +02:00
Clément Renault	405a71d3a4	Accept csv from stdin	2020-08-06 13:38:21 +02:00
Clément Renault	6508d497ce	Replace the regex highlighting by a simple algorithm	2020-08-05 13:52:27 +02:00
Clément Renault	bd4b18541c	Introduce a new indexer which uses an MTBL sorter	2020-08-04 15:44:37 +02:00
Kerollmops	085c376655	Use the regex crate to highlight "hello"	2020-07-14 11:28:40 +02:00
Kerollmops	12358476da	Use the log crate instead of stderr	2020-07-12 10:55:09 +02:00
Kerollmops	2c62eeea3c	Rename the project milli	2020-07-12 00:16:41 +02:00
Kerollmops	f6eae91c7d	Pretty print the new dashboard numbers	2020-07-11 14:17:37 +02:00
Kerollmops	11c7fef80a	Implement a memory dumper ... It moves the in memory HashMaps used when indexing to a disk based MTBL file	2020-07-07 16:48:49 +02:00
Kerollmops	7178b6c2c4	First basic version using MTBL again	2020-07-07 11:32:33 +02:00
Kerollmops	2a3b03138b	Use heed 0.8.1 with the RwIter append method	2020-07-05 19:50:28 +02:00
Kerollmops	46ced5c828	Introduce the RwIter append heed API	2020-07-04 12:34:10 +02:00
Kerollmops	2ae3f40971	Make the indexer ignore certain words ... This is a preparation for making the indexing fully parallel by making the indexer only be aware of certain words for each threads to avoid postings lists conflicts for each words	2020-07-01 17:49:46 +02:00
Kerollmops	f98b615bf3	Replace the LRU by an Arc cache	2020-06-29 20:48:57 +02:00
Kerollmops	07abebfc46	Introduce a (too big) LRU cache	2020-06-29 18:15:03 +02:00
Kerollmops	5f0088594b	Index by writing directly into LMDB	2020-06-29 13:54:47 +02:00
Kerollmops	d6705d5529	Introduce the criterion dependency to bench the engine	2020-06-19 18:32:25 +02:00
Kerollmops	55a8941922	Optimize things	2020-06-19 17:48:17 +02:00
Kerollmops	a8cda248b4	Introduce a customized A* algorithm. ... This custom algo lazily compute the intersections between words, to avoid too much set operations and database reads	2020-06-14 12:51:57 +02:00
Kerollmops	0a83a86e65	Fix multiple bugs	2020-06-11 11:55:03 +02:00
Kerollmops	13977d9338	squash-me	2020-06-09 23:06:59 +02:00
Kerollmops	dfdaceb410	Introduce a first basic working positions-based engine	2020-06-05 20:13:19 +02:00
Kerollmops	3a23dc242e	More efficiently merge MTBLs, more than two at a time	2020-06-04 16:17:24 +02:00
Kerollmops	dff68a339a	Use OnceCell to cache levenshtein builders	2020-05-31 19:27:11 +02:00
Kerollmops	a26553c90a	Reintroduce a simple HTTP server	2020-05-31 17:48:13 +02:00
Kerollmops	ba9527abc0	Support typos with a levenshtein automata	2020-05-31 17:01:11 +02:00
Kerollmops	6c726df9b9	Support multiple space seperated words	2020-05-31 16:09:34 +02:00
Kerollmops	24587148fd	Introduce MTBL parallel merging before LMDB writing	2020-05-31 14:22:57 +02:00
Kerollmops	3a998cf39c	Far better usage of rayon to fold indexed data	2020-05-31 14:22:57 +02:00
Kerollmops	1237306ca8	Introduce a thread that write to heed	2020-05-31 14:22:57 +02:00
Kerollmops	a81f201fad	Inroduce the use of RocksDB instead of sled (RAM)	2020-05-31 14:22:06 +02:00
Kerollmops	91ba938953	Initial commit	2020-05-31 14:22:06 +02:00