4 Commits

Author	SHA1	Message	Date
Clément Renault	abf7191eec	feat: Make the Tokenizer able to support tokenizing sequences	2019-04-29 14:32:00 +02:00
Clément Renault	a745819ddf	feat: Simplify the Tokenizer to use the LinearStrGroupBy type	2019-04-29 14:31:37 +02:00
Clément Renault	19e67dcf0b	feat: Move query splitting into the tokenizer workspace	2019-04-29 14:31:37 +02:00
Clément Renault	1897da5348	feat: Move tokenizer things into the meilidb-tokenizer workspace	2019-04-29 14:31:37 +02:00