MeiliSearch/milli/src
tamo da036dcc3e
Revert "Integrate the stop_words in the querytree"
This reverts commit 12fb509d84.
We revert this commit because it's causing the bug #150.
The initial algorithm we implemented for the stop_words was:

1. remove the stop_words from the dataset
2. keep the stop_words in the query to see if we can generate new words by
   integrating typos or if the word was a prefix
=> This was causing the bug since, in the case of “The hobbit”, we were
   **always** looking for something starting with “t he” or “th e”
   instead of ignoring the word completely.

For now we are going to fix the bug by completely ignoring the
stop_words in the query.
This could cause another problem were someone mistyped a normal word and
ended up typing a stop_word.

For example imagine someone searching for the music “Won't he do it”.
If that person misplace one space and write “Won' the do it” then we
will loose a part of the request.

One fix would be to update our query tree to something like that:

---------------------
OR
  OR
    TOLERANT hobbit # the first option is to ignore the stop_word
    AND
      CONSECUTIVE   # the second option is to do as we are doing
        EXACT t	    # currently
        EXACT he
      TOLERANT hobbit
---------------------

This would increase drastically the size of our query tree on request
with a lot of stop_words. For example think of “The Lord Of The Rings”.

For now whatsoever we decided we were going to ignore this problem and consider
that it doesn't reduce too much the relevancy of the search to do that
while it improves the performances.
2021-04-12 18:35:33 +02:00
..
facet Change the project to become a workspace with milli as a default-member 2021-02-12 16:15:09 +01:00
heed_codec Introduce heed codecs to retrieve the length of roaring bitmaps 2021-02-18 14:30:47 +01:00
search Revert "Integrate the stop_words in the querytree" 2021-04-12 18:35:33 +02:00
update test(http): setting enum serialize/deserialize 2021-04-08 17:03:40 +03:00
criterion.rs update the default ordering of the criterion 2021-04-01 19:45:31 +02:00
external_documents_ids.rs feat(index): update fields distribution in clear & delete operations 2021-04-01 19:12:35 +03:00
fields_ids_map.rs Change the project to become a workspace with milli as a default-member 2021-02-12 16:15:09 +01:00
index.rs feat(index): update fields distribution in clear & delete operations 2021-04-01 19:12:35 +03:00
lib.rs feat(index): update fields distribution in clear & delete operations 2021-04-01 19:12:35 +03:00
proximity.rs Change the project to become a workspace with milli as a default-member 2021-02-12 16:15:09 +01:00
update_store.rs Change the project to become a workspace with milli as a default-member 2021-02-12 16:15:09 +01:00