1791 Commits

Author	SHA1	Message	Date
Louis Dureuil	4e91707a06	Rename test	2023-10-31 09:41:17 +01:00
Louis Dureuil	de10f20732	Fix field distribution again	2023-10-30 17:47:22 +01:00
Louis Dureuil	be395c7944	Change order of arguments to tokenizer_builder	2023-10-30 16:26:29 +01:00
Louis Dureuil	9fedd8101a	Fix tests	2023-10-30 15:11:07 +01:00
Louis Dureuil	54d07a8da3	Update field distribution taking into account both deletions and additions	2023-10-30 14:47:51 +01:00
Louis Dureuil	58690dfb19	Fix tests compilation after changes to ExternalDocumentsIds API	2023-10-30 13:34:07 +01:00
Louis Dureuil	abf424ebfc	Remove unused FromIterator	2023-10-30 11:41:56 +01:00
Clément Renault	dfab6293c9	Use an LMDB database to store the external documents ids	2023-10-30 11:41:23 +01:00
Louis Dureuil	fdf3f7f627	Fix facet distribution test	2023-10-30 11:41:23 +01:00
Louis Dureuil	6260cff65f	Actually delete documents from DB when the merge function says so	2023-10-30 11:41:22 +01:00
Louis Dureuil	8e0d9c9a5e	Recover delete_documents tests that were too eagerly deleted	2023-10-30 11:41:22 +01:00
Louis Dureuil	ae4ec8ea55	Add delete_document_using_wtxn to TempIndex	2023-10-30 11:41:22 +01:00
Louis Dureuil	9a2dccc3bc	Add iterator to find external ids of a bitmap of internal ids	2023-10-30 11:41:22 +01:00
Louis Dureuil	a35988550c	Fix some snapshots	2023-10-30 11:41:22 +01:00
Louis Dureuil	e78281785c	Actually execute the transform even if there are only documents to delete	2023-10-30 11:41:22 +01:00
Louis Dureuil	3c15881818	Add simple delete test	2023-10-30 11:41:22 +01:00
Louis Dureuil	73c06d31d9	snapshot always display stuff in consistent order	2023-10-30 11:41:22 +01:00
Louis Dureuil	290e773d23	remove more warnings and fix some tests	2023-10-30 11:41:22 +01:00
Louis Dureuil	fa6c7f65ca	Add TmpIndex::delete_documents	2023-10-30 11:41:22 +01:00
Louis Dureuil	113527f466	Remove soft-deleted related methods from Index	2023-10-30 11:41:22 +01:00
Louis Dureuil	c534a1b687	Stop using delete documents pipeline in batch runner	2023-10-30 11:41:22 +01:00
Louis Dureuil	2263dff02b	Stop using removed delete pipelines almost everywhere	2023-10-30 11:41:22 +01:00
Louis Dureuil	d651b3ef01	Remove delete documents files	2023-10-30 11:41:20 +01:00
ManyTheFish	762b0b47e6	Use deladd merging function in chunks mergers	2023-10-30 11:40:20 +01:00
Louis Dureuil	01d5eedf2f	Remove some warnings	2023-10-30 11:40:20 +01:00
Louis Dureuil	073f89db79	Fix facet tests	2023-10-30 11:40:20 +01:00
Louis Dureuil	8370fbc92b	Fix snaps	2023-10-30 11:40:20 +01:00
Louis Dureuil	85f42fbc03	Handle external to internal id mapping from TypedChunk::Documents	2023-10-30 11:40:20 +01:00
Louis Dureuil	c6b3c18c85	WIP: Comment out document deletion in other pipelines than update ... TODO: fix calls to DELETE route	2023-10-30 11:40:20 +01:00
Louis Dureuil	bafeb892a7	Modify Index after changes to ExternalDocumentsIds	2023-10-30 11:40:20 +01:00
Louis Dureuil	8fb221dae3	Refactor ExternalDocumentsIds ... - Remove soft deleted - Add apply method that takes a list of operations to encapsulate modifications to the external -> internal mapping	2023-10-30 11:40:20 +01:00
Louis Dureuil	946c762d28	WIP: reset documents in TypedChunk::Documents	2023-10-30 11:40:20 +01:00
Louis Dureuil	cda6ca1ee6	Remove TypedChunk::NewDocumentIds	2023-10-30 11:40:18 +01:00
Louis Dureuil	696fcf4d18	Fix document insertion into LMDB	2023-10-30 11:39:31 +01:00
ManyTheFish	476e4d3dbe	Use value buffer instead of the initial value when writting the final result in the sorter	2023-10-30 11:39:31 +01:00
Clément Renault	576fa9c6da	Remove useless comment	2023-10-30 11:39:31 +01:00
Kerollmops	77dcbff6b2	Remove and Insert the DelAdd geo points	2023-10-30 11:39:31 +01:00
Kerollmops	544440c363	Ignore geo fields when the Del and Add content is the same	2023-10-30 11:39:31 +01:00
Clément Renault	a3dae4db9b	Extract the geo fields DelAdd and generate a new DelAdd obkv with it	2023-10-30 11:39:31 +01:00
ManyTheFish	ba90a5ec0e	update extract fid word count docids	2023-10-30 11:39:31 +01:00
Louis Dureuil	b26dc9aabe	Explanatory code comment	2023-10-30 11:39:31 +01:00
Louis Dureuil	66abac9364	Use specialized KvReaderDelAdd type ... Co-authored-by: Clément Renault <clement@meilisearch.com>	2023-10-30 11:39:31 +01:00
Louis Dureuil	59f88c14b3	Simplify facet update after removing Index::faceted_documents_ids	2023-10-30 11:39:29 +01:00
Louis Dureuil	14832cb324	Remove Index::faceted_documents_ids	2023-10-30 11:37:32 +01:00
Louis Dureuil	04ec293024	Facet Incremental update	2023-10-30 11:37:30 +01:00
Louis Dureuil	f67ff3a738	Facets Bulk update	2023-10-30 11:36:40 +01:00
Clément Renault	560e8f5613	Introduce the CboRoaringBitmapCodec merge_deladd_into and use it	2023-10-30 11:34:55 +01:00
Clément Renault	2d3f15f82c	Introduce a function to only serialize the Add side of a DelAdd obkv	2023-10-30 11:34:55 +01:00
Clément Renault	40186bf403	Rename FieldIdWordCountDocids correctly	2023-10-30 11:34:50 +01:00
ManyTheFish	87e3d27878	update extract word pair proximity to support deladd obkvs	2023-10-30 11:34:02 +01:00
ManyTheFish	6bcf8b4f8c	update extract word position docids	2023-10-30 11:34:02 +01:00
ManyTheFish	46aa75abdb	update extract word docids	2023-10-30 11:34:02 +01:00
ManyTheFish	2597bbd107	Make script language docids map taking a tuple of roaring bitmaps expressing the deletions and the additions	2023-10-30 11:34:00 +01:00
Clément Renault	e2bc054604	Update extract_facet_string_docids to support deladd obkvs	2023-10-30 11:32:36 +01:00
Clément Renault	fcd3a1434d	Update extract_facet_number_docids to support deladd obkvs	2023-10-30 11:31:04 +01:00
Clément Renault	a82dee21e0	Rename docid_fid into fid_docid	2023-10-30 11:31:02 +01:00
Clément Renault	bc45c1206d	Implement all the facet extraction paths and simplify them	2023-10-30 11:29:08 +01:00
Clément Renault	6ae4100f07	Generate the DelAdd for is_null, is_empty, and exists	2023-10-30 11:29:08 +01:00
Clément Renault	0c47defeee	Work on fid docid facet values rewrite	2023-10-30 11:29:06 +01:00
ManyTheFish	313b16bec2	Support diff indexing on extract_docid_word_positions	2023-10-30 11:24:19 +01:00
ManyTheFish	1dd97578a8	Make the transform struct return diff-based documents obkvs	2023-10-30 11:22:07 +01:00
ManyTheFish	f5ef69293b	deactivate prefix dbs	2023-10-30 11:22:07 +01:00
ManyTheFish	1c5705c164	clean PR warnings	2023-10-30 11:22:05 +01:00
ManyTheFish	66c2c82a18	Split wpp in several sorters	2023-10-30 11:15:02 +01:00
ManyTheFish	28a8d0ccda	Fix word pair proximity	2023-10-30 11:15:02 +01:00
ManyTheFish	96be85396d	Use a vecDeque in wpp database	2023-10-30 11:15:02 +01:00
ManyTheFish	df9e5c8651	Generalize usage of CboRoaringBitmap codec to ease the use	2023-10-30 11:15:02 +01:00
ManyTheFish	b541d48847	Add buffer to the obkv writter	2023-10-30 11:15:02 +01:00
ManyTheFish	8ccf32d1a0	Compute word_fid_docids before word_docids and exact_word_docids	2023-10-30 11:15:02 +01:00
ManyTheFish	db1ca21231	add puffin in sorter into reeder function	2023-10-30 11:15:00 +01:00
ManyTheFish	11ea5acff9	Fix	2023-10-30 11:13:10 +01:00
ManyTheFish	8d77736a67	Fix fid_word_docids	2023-10-30 11:13:10 +01:00
ManyTheFish	748b333161	Add usefull debug assert before key insertion in database	2023-10-30 11:13:10 +01:00
ManyTheFish	17b647dfe5	Wip	2023-10-30 11:13:08 +01:00
Tamo	e7244aa485	fix warnings	2023-10-30 11:00:46 +01:00
Louis Dureuil	2bae9550c8	Add explanatory comment	2023-10-23 12:06:28 +02:00
Vivek Kumar	5fe7c4545a	compute all candidates correctly when skipping	2023-10-23 12:02:45 +02:00
meili-bors[bot]	5e0485d8dd	Merge #4131 ... 4131: Reduce proximity range from 7 to 3 r=Kerollmops a=ManyTheFish ## Summary This PR aims to reduce the impact of the proximity databases on the indexing time and on the database size by reducing the maximum distance between two words to be indexed in the proximity database. ## Stats ### Impact on database size and indexing time  ### Impact on search relevancy <details> | dataset_name | host_name | Relevancy rate (Precision) | completion_rate 25.00% | completion_rate 50.00% | completion_rate 75.00% | completion_rate 100.00% | |--------------|------------------|------------------------------------|-----------------|-----------------|-----------------|-----------------| | FBIS | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | FBIS | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | FBIS | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 5.56% | | FBIS | 1_4_0 | percentile-75 | 0.00% | 12.50% | 35.00% | 45.00% | | FBIS | 1_4_0 | percentile-90 | 20.00% | 40.00% | | 100.00% | | FBIS | 1_4_0 | average | 5.78% | 11.16% | 21.90% | 26.29% | | FBIS | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | FBIS | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | FBIS | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 5.56% | | FBIS | reduce_proximity | percentile-75 | 0.00% | 15.00% | 35.00% | 40.00% | | FBIS | reduce_proximity | percentile-90 | 20.00% | 40.00% | 85.00% | 100.00% | | FBIS | reduce_proximity | average | 5.55% | 11.34% | 21.75% | 26.14% | | FR94 | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | FR94 | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | FR94 | 1_4_0 | percentile-50 | 0.00% | 0.00% | 0.00% | 0.00% | | FR94 | 1_4_0 | percentile-75 | 0.00% | 5.00% | 15.00% | 42.11% | | FR94 | 1_4_0 | percentile-90 | 15.00% | 54.55% | 100.00% | 100.00% | | FR94 | 1_4_0 | average | 5.95% | 12.07% | 18.70% | 25.57% | | FR94 | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | FR94 | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | FR94 | reduce_proximity | percentile-50 | 0.00% | 0.00% | 0.00% | 0.00% | | FR94 | reduce_proximity | percentile-75 | 0.00% | 5.00% | 15.00% | 42.11% | | FR94 | reduce_proximity | percentile-90 | 15.00% | 54.55% | 100.00% | 100.00% | | FR94 | reduce_proximity | average | 5.79% | 12.00% | 18.70% | 25.53% | | FT | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | FT | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | FT | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 10.00% | | FT | 1_4_0 | percentile-75 | 0.00% | 15.00% | 30.00% | 40.00% | | FT | 1_4_0 | percentile-90 | 20.00% | 50.00% | 65.00% | 100.00% | | FT | 1_4_0 | average | 5.08% | 12.58% | 20.00% | 25.49% | | FT | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | FT | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | FT | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 10.00% | | FT | reduce_proximity | percentile-75 | 0.00% | 15.00% | 30.00% | 40.00% | | FT | reduce_proximity | percentile-90 | 10.00% | 45.00% | 60.00% | 100.00% | | FT | reduce_proximity | average | 5.01% | 12.64% | 20.10% | 25.53% | | LAT | 1_4_0 | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | LAT | 1_4_0 | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | LAT | 1_4_0 | percentile-50 | 0.00% | 0.00% | 5.00% | 5.00% | | LAT | 1_4_0 | percentile-75 | 5.00% | 15.00% | 30.00% | 30.00% | | LAT | 1_4_0 | percentile-90 | 15.00% | 45.00% | 60.00% | 80.00% | | LAT | 1_4_0 | average | 4.80% | 11.80% | 17.88% | 21.62% | | LAT | reduce_proximity | percentile-10 | 0.00% | 0.00% | 0.00% | 0.00% | | LAT | reduce_proximity | percentile-25 | 0.00% | 0.00% | 0.00% | 0.00% | | LAT | reduce_proximity | percentile-50 | 0.00% | 0.00% | 5.00% | 5.00% | | LAT | reduce_proximity | percentile-75 | 0.00% | 11.11% | 25.00% | 35.00% | | LAT | reduce_proximity | percentile-90 | 15.00% | 45.00% | 55.00% | 80.00% | | LAT | reduce_proximity | average | 4.43% | 11.23% | 17.32% | 21.45% | </details> ### Impact on Search time | dataset_name | host_name | 25.00% | 50.00% | 75.00% | 100.00% | Average | |--------------|------------------|------------:|------------:|------------:|------------:|-------------| | FBIS | 1_4_0 | 3.45 | 7.446666667 | 9.773489933 | 9.620300752 | 7.572614338 | | FBIS | reduce_proximity | 2.983333333 | 5.316666667 | 6.911073826 | 7.637218045 | 5.712072968 | | FR94 | 1_4_0 | 2.236666667 | 4.45 | 5.523489933 | 4.560150376 | 4.192576744 | | FR94 | reduce_proximity | 2.09 | 3.991666667 | 4.981543624 | 4.266917293 | 3.832531896 | | FT | 1_4_0 | 5.956666667 | 9.656666667 | 13.86912752 | 10.83270677 | 10.0787919 | | FT | reduce_proximity | 4.51 | 5.981666667 | 7.701342282 | 6.766917293 | 6.23998156 | | LAT | 1_4_0 | 5.856666667 | 9.233333333 | 12.98322148 | 10.78759398 | 9.715203865 | | LAT | reduce_proximity | 6.91 | 6.706666667 | 8.463087248 | 8.265037594 | 7.586197877 | ## Technical approach - Ensure the MAX_DISTANCE constant is used everywhere needed - Reduce the MAX_DISTANCE from 8 to 4 ## Related TBD Co-authored-by: ManyTheFish <many@meilisearch.com>	2023-10-18 14:56:08 +00:00
ManyTheFish	27eec21415	Fix tests	2023-10-18 16:03:22 +02:00
Clément Renault	62dfd09dc6	Add more puffin logs to the deletion functions	2023-10-13 13:11:09 +02:00
meili-bors[bot]	f343ef5f2f	Merge #4108 ... 4108: Fix bug where search with distinct attribute and no ranking, returns offset+limit hits r=curquiza a=vivek-26 # Pull Request ## Related issue Fixes #4078 ## What does this PR do? This PR - - Fixes bug where search with distinct attribute and no ranking, returns offset+limit hits. - Adds unit and integration tests. ## PR checklist Please check if your PR fulfills the following requirements: - [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)? - [x] Have you read the contributing guidelines? - [x] Have you made sure that the title is accurate and descriptive of the changes? Thank you so much for contributing to Meilisearch! Co-authored-by: Vivek Kumar <vivek.26@outlook.com>	2023-10-12 07:51:29 +00:00
Vivek Kumar	d4da06ff47	fix bug where distinct search with no ranking returns offset+limit hits	2023-10-11 19:02:16 +05:30
Tamo	c0f2724c2d	get rids of the new introduced error code in favor of an io::Error	2023-10-10 15:12:23 +02:00
Tamo	d772073dfa	use a bufreader everytime there is a grenad<file>	2023-10-10 15:00:30 +02:00
ManyTheFish	43989fe2e4	Reduce porximity range from 7 to 3	2023-10-03 12:16:48 +02:00
meili-bors[bot]	487d493f49	Merge #4043 ... 4043: Bring back hotfixes from v1.3.3 into v1.4.0 r=Kerollmops a=curquiza Co-authored-by: curquiza <curquiza@users.noreply.github.com> Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com> Co-authored-by: Kerollmops <clement@meilisearch.com> Co-authored-by: curquiza <clementine@meilisearch.com>	2023-09-11 12:27:34 +00:00
Vivek Kumar	abfa7ded25	use a new temp index in the test	2023-09-08 12:32:47 +05:30
Vivek Kumar	f2837aaec2	add another test case	2023-09-08 11:39:54 +05:30
Vivek Kumar	11df155598	fix highlighting bug when searching for a phrase with cropping	2023-09-08 11:39:52 +05:30
meili-bors[bot]	256cf33bca	Merge #4039 ... 4039: Fix multiple vectors dimensions r=ManyTheFish a=Kerollmops This PR fixes #4035, making providing multiple vectors in documents possible. This is fixed by extracting the vectors from the non-flattened version of the documents. Co-authored-by: Kerollmops <clement@meilisearch.com>	2023-09-07 09:25:58 +00:00
Kerollmops	679c0b0f97	Extract the vectors from the non-flattened version of the documents	2023-09-06 12:26:00 +02:00
Kerollmops	e02d0064bd	Add a test case scenario	2023-09-06 12:26:00 +02:00
meili-bors[bot]	dc3d9c90d9	Merge #3994 ... 3994: Fix synonyms with separators r=Kerollmops a=ManyTheFish # Pull Request ## Related issue Fixes #3977 ## Available prototype ``` $ docker pull getmeili/meilisearch:prototype-fix-synonyms-with-separators-0 ``` ## What does this PR do? - add a new test - filter the empty synonyms after normalization Co-authored-by: ManyTheFish <many@meilisearch.com>	2023-09-05 14:42:46 +00:00
ManyTheFish	66aa6d5871	Ignore tokens with empty normalized value during indexing process	2023-09-05 15:44:14 +02:00
Kerollmops	8ac5b765bc	Fix synonyms normalization	2023-09-04 16:12:48 +02:00
Kerollmops	085aad0a94	Add a test	2023-09-04 14:39:33 +02:00
meili-bors[bot]	ccf3ba3f32	Merge #4019 ... 4019: Bringing back changes from `v1.3.2` onto `main` r=irevoire a=Kerollmops Co-authored-by: Kerollmops <clement@meilisearch.com> Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com> Co-authored-by: irevoire <irevoire@users.noreply.github.com> Co-authored-by: Clément Renault <clement@meilisearch.com>	2023-08-28 12:14:11 +00:00
Clément Renault	8c0ebd1331	Update milli/src/search/new/bucket_sort.rs ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2023-08-23 16:40:39 +02:00
Kerollmops	5130e06b41	Temporarily disable an assert in the ranking rules	2023-08-23 16:11:54 +02:00
meili-bors[bot]	914b125c5f	Merge #3945 ... 3945: Do not leak field information on error r=Kerollmops a=vivek-26 # Pull Request ## Related issue Fixes #3865 ## What does this PR do? This PR ensures that `InvalidSortableAttribute`and `InvalidFacetSearchFacetName` errors do not leak field information i.e. fields which are not part of `displayedAttributes` in the settings are hidden from the error message. ## PR checklist Please check if your PR fulfills the following requirements: - [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)? - [x] Have you read the contributing guidelines? - [x] Have you made sure that the title is accurate and descriptive of the changes? Thank you so much for contributing to Meilisearch! Co-authored-by: Vivek Kumar <vivek.26@outlook.com>	2023-08-22 18:55:27 +00:00
Kerollmops	c53841e166	Accept the null JSON value as the value of _vectors	2023-08-14 16:03:55 +02:00
meili-bors[bot]	e4e49e63d0	Merge #3993 ... 3993: Bringing back changes from v1.3.1 to `main` r=irevoire a=curquiza Co-authored-by: irevoire <irevoire@users.noreply.github.com> Co-authored-by: meili-bors[bot] <89034592+meili-bors[bot]@users.noreply.github.com> Co-authored-by: Tamo <tamo@meilisearch.com> Co-authored-by: ManyTheFish <many@meilisearch.com>	2023-08-10 14:30:02 +00:00
ManyTheFish	5a7c1bde84	Fix clippy	2023-08-10 11:27:56 +02:00
ManyTheFish	6b2d671be7	Fix PR comments	2023-08-10 10:44:07 +02:00
Many the fish	43c13faeda	Update milli/src/update/index_documents/extract/extract_docid_word_positions.rs ... Co-authored-by: Tamo <tamo@meilisearch.com>	2023-08-10 10:05:03 +02:00
meili-bors[bot]	44c1900f36	Merge #3986 ... 3986: Fix geo bounding box with strings r=ManyTheFish a=irevoire # Pull Request When sending a document with one geofield of type string (i.e.: `{ "_geo": { "lat": 12, "lng": "13" }}`), the geobounding box would exclude this document. This PR fixes this issue by automatically parsing the string value in case we're working on a geofield. ## Related issue Fixes https://github.com/meilisearch/meilisearch/issues/3973 ## What does this PR do? - Automatically parse the facet value iif we're working on a geofield. - Make insta works with snapshots in loops or closure executed multiple times. (you may need to update your cli if it panics after this PR: `cargo install cargo-insta`). - Add one integration test in milli and in meilisearch to ensure it works forever. - Add three snapshots for the dump that mysteriously disappeared I don't know how Co-authored-by: Tamo <tamo@meilisearch.com>	2023-08-09 07:58:15 +00:00
ManyTheFish	8dc5acf998	Try fix	2023-08-08 16:52:36 +02:00
ManyTheFish	35758db9ec	Truncate the the normalized long facets used in search for facet value	2023-08-08 16:38:30 +02:00
Tamo	4988199bb9	ensure the geoboundingbox works with strings and int geofields in milli and meilisearch	2023-08-08 16:29:25 +02:00
Tamo	9d061cec26	automatically parse the filterable attribute to float if it's a geo field	2023-08-08 16:28:07 +02:00
ManyTheFish	4a21fecf67	Merge branch 'main' into settings-customizing-tokenization	2023-08-08 16:08:16 +02:00
Vivek Kumar	dd57873f8e	hide fields not in the displayedAttributes list from errors	2023-08-05 16:03:10 +05:30
ManyTheFish	b45c36cd71	Merge branch 'main' into tmp-release-v1.3.0	2023-08-01 15:05:17 +02:00
ManyTheFish	9d5e3457e5	Fix clippy	2023-07-27 14:21:19 +02:00
meili-bors[bot]	939b2fc6fd	Merge #3949 ... 3949: Fix score details casing r=Kerollmops a=ManyTheFish # Pull Request Fixes #3941 Co-authored-by: ManyTheFish <many@meilisearch.com>	2023-07-26 14:14:59 +00:00
ManyTheFish	b0c1a9504a	ensure the synonyms are updated when the tokenizer settings are changed	2023-07-26 09:33:42 +02:00
meili-bors[bot]	be72be7c0d	Merge #3942 ... 3942: Normalize for the search the facets values r=ManyTheFish a=Kerollmops This PR improves and fixes the search for facet values feature. Searching for _bre_ wasn't returning facet values like _brévent_ or _brô_. The issue was related to the fact that facets are normalized but not in the same way as the `searchableAttributes` are. We decided to normalize them further and add another intermediate database where the key is the normalized facet value, and the value is a set of the non-normalized facets. We then use these non-normalized ones to get the correct counts by fetching the associated databases. ### What's missing in this PR? - [x] Apply the change to the whole set of `SearchForFacetValue::execute` conditions. - [x] Factorize the code that does an intermediate normalized value fetch in a function. - [x] Add or modify the search for facet value test. Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Kerollmops <clement@meilisearch.com>	2023-07-25 14:37:17 +00:00
ManyTheFish	88559a2d54	Fix score details casing	2023-07-25 15:49:33 +02:00
ManyTheFish	d57026cd96	Support synonyms sinergies	2023-07-25 15:01:42 +02:00
Kerollmops	29ab54b259	Replace the hnsw crate by the instant-distance one	2023-07-25 12:37:35 +02:00
ManyTheFish	d4ff59fcf5	Fix clippy	2023-07-24 18:42:26 +02:00
ManyTheFish	9c485f8563	Make the search and the indexing work	2023-07-24 18:35:20 +02:00
Kerollmops	691a536893	Implement the facet search with the normalized index	2023-07-24 17:56:17 +02:00
ManyTheFish	d8d12d5979	Be able to set and reset settings	2023-07-24 17:00:18 +02:00
Clément Renault	df528b41d8	Normalize for the search the facets values	2023-07-20 17:57:07 +02:00
Kerollmops	eef95de30e	First iteration on exposing puffin profiling	2023-07-18 17:38:13 +02:00
Kerollmops	d383afc82b	Fix the geo sort when lat and lng are strings	2023-07-17 18:28:04 +02:00
Louis Dureuil	4310928803	Fixes #3912	2023-07-12 10:08:56 +02:00
Louis Dureuil	74315b4ea8	Fixes #3911	2023-07-12 10:08:29 +02:00
Louis Dureuil	40fa59d64c	Sort by lexicographic order after normalization	2023-07-10 09:26:59 +02:00
Louis Dureuil	55cd7738b9	Update snapshots	2023-07-04 16:31:01 +02:00
Louis Dureuil	48409c9183	Add missing exactness.matchingWords, exactness.maxMatchingWords	2023-07-04 16:31:01 +02:00
Louis Dureuil	324d448236	Format let-else ❤️ 🎉	2023-07-03 10:20:28 +02:00
meili-bors[bot]	661d1f90dc	Merge #3866 ... 3866: Update charabia v0.8.0 r=dureuill a=ManyTheFish # Pull Request Update Charabia: - enhance Japanese segmentation - enhance Latin Tokenization - words containing `_` are now properly segmented into several words - brackets `{([])}` are no more considered as context separators so word separated by brackets are now considered near together for the proximity ranking rule - fixes #3815 - fixes #3778 - fixes [product#151](https://github.com/meilisearch/product/discussions/151) > Important note: now the float numbers are segmented around the `.` so `3.22` is segmented as [`3`, `.`, `22`] but the middle dot isn't considered as a hard separator, which means that if we search `3.22` we find documents containing `3.22` Co-authored-by: ManyTheFish <many@meilisearch.com>	2023-06-29 15:24:36 +00:00
ManyTheFish	6ec7541026	Update inta snapshots	2023-06-29 17:18:39 +02:00
ManyTheFish	a82c49ab08	Update test	2023-06-29 15:56:36 +02:00
ManyTheFish	84845de9ef	Update Charabia	2023-06-29 15:56:32 +02:00
Clément Renault	7c157fc442	Document that the LevelEntry fields order is important	2023-06-29 14:33:32 +02:00
Clément Renault	0b97596c93	Replace unwraps with ?	2023-06-29 14:33:32 +02:00
Clément Renault	a0e0fce677	Simplify a Rust lifetime trick	2023-06-29 14:33:32 +02:00
Clément Renault	b951830461	Add more tests	2023-06-29 14:33:32 +02:00
Kerollmops	b132e859f7	Make clippy happy	2023-06-29 14:33:31 +02:00
Kerollmops	9917bf046a	Move the sortFacetValuesBy in the faceting settings	2023-06-29 14:33:31 +02:00
Kerollmops	d9fea0143f	Make Clippy happy	2023-06-29 14:33:31 +02:00
Kerollmops	a385642ec3	Replace the BTreeMap by an IndexMap to return values in order	2023-06-29 14:33:31 +02:00
Kerollmops	34b2e98fe9	Expose a sortFacetValuesBy parameter to the user	2023-06-29 14:33:00 +02:00
Kerollmops	80bbd4b6f3	Clean and make the facet order configurable internally	2023-06-29 14:31:17 +02:00
Kerollmops	f42bef2f66	Make the search to always return the facets ordered by count	2023-06-29 14:31:17 +02:00
Kerollmops	bd3c026406	First to-test version of the algorithm	2023-06-29 14:31:17 +02:00
Kerollmops	84f8938f33	Rename facet distribution to be explicit on the order to find them	2023-06-29 14:31:15 +02:00