Commit Graph

35 Commits

Author SHA1 Message Date
Kerollmops
46ced5c828
Introduce the RwIter append heed API 2020-07-04 12:34:10 +02:00
Kerollmops
7e7440c431
Finalize the LMDB indexing design 2020-07-01 22:45:43 +02:00
Kerollmops
2ae3f40971
Make the indexer ignore certain words
This is a preparation for making the indexing fully parallel by making the
indexer only be aware of certain words for each threads to avoid postings lists
conflicts for each words
2020-07-01 17:49:46 +02:00
Kerollmops
a3ac2623d5
Introduce multiple functions to clean up the code 2020-07-01 17:24:55 +02:00
Kerollmops
014a25697d
Use only one ARC cache based on the words 2020-07-01 12:03:18 +02:00
Kerollmops
fc4013a43f
Fix the ARC cache 2020-07-01 10:35:07 +02:00
Kerollmops
2fcae719ad
Use another LRU impl which uses hashbrown 2020-06-29 22:26:06 +02:00
Kerollmops
f98b615bf3
Replace the LRU by an Arc cache 2020-06-29 20:48:57 +02:00
Kerollmops
07abebfc46
Introduce a (too big) LRU cache 2020-06-29 18:15:03 +02:00
Kerollmops
5f0088594b
Index by writing directly into LMDB 2020-06-29 13:54:47 +02:00
Kerollmops
6a2834f2b0
Add a jobs parameter to set the number of threads the indexer uses 2020-06-28 12:13:10 +02:00
Kerollmops
374ec6773f
Introduce a database to store all docids for a word and attribute 2020-06-22 19:24:20 +02:00
Kerollmops
a044cb6cc8
Clean up the warnings for prefix postings 2020-06-22 18:10:31 +02:00
Kerollmops
ba3e805981
Document the Index types and the internal LMDB databases 2020-06-22 18:09:22 +02:00
Kerollmops
c84012d655
Accept queries from standard input when not given as argument 2020-06-20 12:01:15 +02:00
Kerollmops
0a83a86e65
Fix multiple bugs 2020-06-11 11:55:03 +02:00
Kerollmops
4e86ecf807
Retrieve the words before the intersect loops 2020-06-10 22:05:01 +02:00
Kerollmops
2a6d6a7f69
Introduce a first draft of the best_proximity algorithm 2020-06-09 10:11:43 +02:00
Kerollmops
dfdaceb410
Introduce a first basic working positions-based engine 2020-06-05 20:13:19 +02:00
Kerollmops
f51a63e4ef
Store documents ids under attribute ids 2020-06-05 16:32:14 +02:00
Kerollmops
eefc6d7c44
Add support for quoted query phrases 2020-06-04 20:25:51 +02:00
Kerollmops
5d1c625b74
Change the page index texts 2020-06-04 18:20:57 +02:00
Kerollmops
c42d3c19e2
Merge the whole list of generated MTBL in one go 2020-06-04 17:38:43 +02:00
Kerollmops
3a23dc242e
More efficiently merge MTBLs, more than two at a time 2020-06-04 16:17:24 +02:00
Kerollmops
1df1f88fe1
Directly write to LMDB without intermediate final MTBL 2020-06-01 21:30:39 +02:00
Kerollmops
2174042994
Merge only 3 MTBL at the same time 2020-06-01 19:49:58 +02:00
Kerollmops
5cc81a0179
Merge many MTBL into one a the same time 2020-06-01 18:39:58 +02:00
Kerollmops
6a047519f6
Do a merge two by two 2020-06-01 18:27:26 +02:00
Kerollmops
5404776f7a
Add a little bit more debug 2020-06-01 17:52:43 +02:00
Kerollmops
dff68a339a
Use OnceCell to cache levenshtein builders 2020-05-31 19:27:11 +02:00
Kerollmops
dde3e01a59
Introduce prefix postings ids for better perfs 2020-05-31 18:20:49 +02:00
Kerollmops
a26553c90a
Reintroduce a simple HTTP server 2020-05-31 17:48:13 +02:00
Kerollmops
2a10b2275e
Support prefix typo tolerant search 2020-05-31 17:18:13 +02:00
Kerollmops
ba9527abc0
Support typos with a levenshtein automata 2020-05-31 17:01:11 +02:00
Kerollmops
6c726df9b9
Support multiple space seperated words 2020-05-31 16:09:34 +02:00