3 Commits

Author	SHA1	Message	Date
Clément Renault	acd2a63879	Introduce a simple FST based chinese word segmenter	2020-10-04 17:04:33 +02:00
Clément Renault	e5adfaade0	Replace the token filter by a filter mapper	2020-09-22 10:24:31 +02:00
Clément Renault	bad0663138	Come back to the old tokenizer	2020-08-31 13:34:38 +02:00