mirror of
https://github.com/meilisearch/MeiliSearch
synced 2024-12-25 06:00:08 +01:00
457a473b72
* Fix error code of the "duplicate index found" error * Use the content of the ProcessingTasks in the tasks cancelation system * Change the missing_filters error code into missing_task_filters * WIP Introduce the invalid_task_uid error code * Use more precise error codes/message for the task routes + Allow star operator in delete/cancel tasks + rename originalQuery to originalFilters + Display error/canceled_by in task view even when they are = null + Rename task filter fields by using their plural forms + Prepare an error code for canceledBy filter + Only return global tasks if the API key action `index.*` is there * Add canceledBy task filter * Update tests following task API changes * Rename original_query to original_filters everywhere * Update more insta-snap tests * Make clippy happy They're a happy clip now. * Make rustfmt happy >:-( * Fix Index name parsing error message to fit the specification * Bump milli version to 0.35.1 * Fix the new error messages * fix the error messages and add tests * rename the error codes for the sake of consistency * refactor the way we send the cli informations + add the analytics for the config file and ssl usage * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> * add a comment over the new infos structure * reformat, sorry @kero * Store analytics for the documents deletions * Add analytics on all the settings * Spawn threads with names * Spawn rayon threads with names * update the distinct attributes to the spec update * update the analytics on the search route * implements the analytics on the health and version routes * Fix task details serialization * Add the question mark to the task deletion query filter * Add the question mark to the task cancelation query filter * Fix tests * add analytics on the task route * Add all the missing fields of the new task query type * Create a new analytics for the task deletion * Create a new analytics for the task creation * batch the tasks seen events * Update the finite pagination analytics * add the analytics of the swap-indexes route * Stop removing the DB when failing to read it * Rename originalFilters into originalFilters * Rename matchedDocuments into providedIds * Add `workflow_dispatch` to flaky.yml * Bump grenad to 0.4.4 * Bump milli to version v0.37.0 * Don't multiply total memory returned by sysinfo anymore sysinfo now returns bytes rather than KB * Add a dispatch to the publish binaries workflow * Fix publish release CI * Don't use gold but the default linker * Always display details for the indexDeletion task * Fix the insta tests * refactorize the whole test suite 1. Make a call to assert_internally_consistent automatically when snapshoting the scheduler. There is no point in snapshoting something broken and expect the dumb humans to notice. 2. Replace every possible call to assert_internally_consistent by a snapshot of the scheduler. It takes as many lines and ensure we never change something without noticing in any tests ever. 3. Name every snapshots: it's easier to debug when something goes wrong and easier to review in general. 4. Stop skipping breakpoints, it's too easy to miss something. Now you must explicitely show which path is the scheduler supposed to use. 5. Add a timeout on the channel.recv, it eases the process of writing tests, now when something file you get a failure instead of a deadlock. * rebase on release-v0.30 * makes clippy happy * update the snapshots after a rebase * try to remove the flakyness of the failing test * Add more analytics on the ranking rules positions * Update the dump test to check for the dumpUid dumpCreation task details * send the ranking rules as a string because amplitude is too dumb to process an array as a single value * Display a null dumpUid until we computed the dump itself on disk * Update tests * Check if the master key is missing before returning an error Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com> Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com> Co-authored-by: Kerollmops <clement@meilisearch.com> Co-authored-by: ManyTheFish <many@meilisearch.com> Co-authored-by: Tamo <tamo@meilisearch.com> Co-authored-by: Louis Dureuil <louis@meilisearch.com>
391 lines
14 KiB
Rust
391 lines
14 KiB
Rust
use std::hash::Hash;
|
|
use std::str::FromStr;
|
|
|
|
use enum_iterator::Sequence;
|
|
use serde::{Deserialize, Serialize};
|
|
use serde_json::{from_value, Value};
|
|
use time::format_description::well_known::Rfc3339;
|
|
use time::macros::{format_description, time};
|
|
use time::{Date, OffsetDateTime, PrimitiveDateTime};
|
|
use uuid::Uuid;
|
|
|
|
use crate::error::{Code, ErrorCode};
|
|
use crate::index_uid::{IndexUid, IndexUidFormatError};
|
|
use crate::star_or::StarOr;
|
|
|
|
type Result<T> = std::result::Result<T, Error>;
|
|
|
|
pub type KeyId = Uuid;
|
|
|
|
#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Serialize)]
|
|
pub struct Key {
|
|
#[serde(skip_serializing_if = "Option::is_none")]
|
|
pub description: Option<String>,
|
|
#[serde(skip_serializing_if = "Option::is_none")]
|
|
pub name: Option<String>,
|
|
pub uid: KeyId,
|
|
pub actions: Vec<Action>,
|
|
pub indexes: Vec<StarOr<IndexUid>>,
|
|
#[serde(with = "time::serde::rfc3339::option")]
|
|
pub expires_at: Option<OffsetDateTime>,
|
|
#[serde(with = "time::serde::rfc3339")]
|
|
pub created_at: OffsetDateTime,
|
|
#[serde(with = "time::serde::rfc3339")]
|
|
pub updated_at: OffsetDateTime,
|
|
}
|
|
|
|
impl Key {
|
|
pub fn create_from_value(value: Value) -> Result<Self> {
|
|
let name = match value.get("name") {
|
|
None | Some(Value::Null) => None,
|
|
Some(des) => from_value(des.clone())
|
|
.map(Some)
|
|
.map_err(|_| Error::InvalidApiKeyName(des.clone()))?,
|
|
};
|
|
|
|
let description = match value.get("description") {
|
|
None | Some(Value::Null) => None,
|
|
Some(des) => from_value(des.clone())
|
|
.map(Some)
|
|
.map_err(|_| Error::InvalidApiKeyDescription(des.clone()))?,
|
|
};
|
|
|
|
let uid = value.get("uid").map_or_else(
|
|
|| Ok(Uuid::new_v4()),
|
|
|uid| from_value(uid.clone()).map_err(|_| Error::InvalidApiKeyUid(uid.clone())),
|
|
)?;
|
|
|
|
let actions = value
|
|
.get("actions")
|
|
.map(|act| {
|
|
from_value(act.clone()).map_err(|_| Error::InvalidApiKeyActions(act.clone()))
|
|
})
|
|
.ok_or(Error::MissingParameter("actions"))??;
|
|
|
|
let indexes = value
|
|
.get("indexes")
|
|
.map(|ind| {
|
|
from_value::<Vec<String>>(ind.clone())
|
|
// If it's not a vec of string, return an API key parsing error.
|
|
.map_err(|_| Error::InvalidApiKeyIndexes(ind.clone()))
|
|
.and_then(|ind| {
|
|
ind.into_iter()
|
|
// If it's not a valid Index uid, return an Index Uid parsing error.
|
|
.map(|i| StarOr::<IndexUid>::from_str(&i).map_err(Error::from))
|
|
.collect()
|
|
})
|
|
})
|
|
.ok_or(Error::MissingParameter("indexes"))??;
|
|
|
|
let expires_at = value
|
|
.get("expiresAt")
|
|
.map(parse_expiration_date)
|
|
.ok_or(Error::MissingParameter("expiresAt"))??;
|
|
|
|
let created_at = OffsetDateTime::now_utc();
|
|
let updated_at = created_at;
|
|
|
|
Ok(Self { name, description, uid, actions, indexes, expires_at, created_at, updated_at })
|
|
}
|
|
|
|
pub fn update_from_value(&mut self, value: Value) -> Result<()> {
|
|
if let Some(des) = value.get("description") {
|
|
let des =
|
|
from_value(des.clone()).map_err(|_| Error::InvalidApiKeyDescription(des.clone()));
|
|
self.description = des?;
|
|
}
|
|
|
|
if let Some(des) = value.get("name") {
|
|
let des = from_value(des.clone()).map_err(|_| Error::InvalidApiKeyName(des.clone()));
|
|
self.name = des?;
|
|
}
|
|
|
|
if value.get("uid").is_some() {
|
|
return Err(Error::ImmutableField("uid".to_string()));
|
|
}
|
|
|
|
if value.get("actions").is_some() {
|
|
return Err(Error::ImmutableField("actions".to_string()));
|
|
}
|
|
|
|
if value.get("indexes").is_some() {
|
|
return Err(Error::ImmutableField("indexes".to_string()));
|
|
}
|
|
|
|
if value.get("expiresAt").is_some() {
|
|
return Err(Error::ImmutableField("expiresAt".to_string()));
|
|
}
|
|
|
|
if value.get("createdAt").is_some() {
|
|
return Err(Error::ImmutableField("createdAt".to_string()));
|
|
}
|
|
|
|
if value.get("updatedAt").is_some() {
|
|
return Err(Error::ImmutableField("updatedAt".to_string()));
|
|
}
|
|
|
|
self.updated_at = OffsetDateTime::now_utc();
|
|
|
|
Ok(())
|
|
}
|
|
|
|
pub fn default_admin() -> Self {
|
|
let now = OffsetDateTime::now_utc();
|
|
let uid = Uuid::new_v4();
|
|
Self {
|
|
name: Some("Default Admin API Key".to_string()),
|
|
description: Some("Use it for anything that is not a search operation. Caution! Do not expose it on a public frontend".to_string()),
|
|
uid,
|
|
actions: vec![Action::All],
|
|
indexes: vec![StarOr::Star],
|
|
expires_at: None,
|
|
created_at: now,
|
|
updated_at: now,
|
|
}
|
|
}
|
|
|
|
pub fn default_search() -> Self {
|
|
let now = OffsetDateTime::now_utc();
|
|
let uid = Uuid::new_v4();
|
|
Self {
|
|
name: Some("Default Search API Key".to_string()),
|
|
description: Some("Use it to search from the frontend".to_string()),
|
|
uid,
|
|
actions: vec![Action::Search],
|
|
indexes: vec![StarOr::Star],
|
|
expires_at: None,
|
|
created_at: now,
|
|
updated_at: now,
|
|
}
|
|
}
|
|
}
|
|
|
|
fn parse_expiration_date(value: &Value) -> Result<Option<OffsetDateTime>> {
|
|
match value {
|
|
Value::String(string) => OffsetDateTime::parse(string, &Rfc3339)
|
|
.or_else(|_| {
|
|
PrimitiveDateTime::parse(
|
|
string,
|
|
format_description!(
|
|
"[year repr:full base:calendar]-[month repr:numerical]-[day]T[hour]:[minute]:[second]"
|
|
),
|
|
).map(|datetime| datetime.assume_utc())
|
|
})
|
|
.or_else(|_| {
|
|
PrimitiveDateTime::parse(
|
|
string,
|
|
format_description!(
|
|
"[year repr:full base:calendar]-[month repr:numerical]-[day] [hour]:[minute]:[second]"
|
|
),
|
|
).map(|datetime| datetime.assume_utc())
|
|
})
|
|
.or_else(|_| {
|
|
Date::parse(string, format_description!(
|
|
"[year repr:full base:calendar]-[month repr:numerical]-[day]"
|
|
)).map(|date| PrimitiveDateTime::new(date, time!(00:00)).assume_utc())
|
|
})
|
|
.map_err(|_| Error::InvalidApiKeyExpiresAt(value.clone()))
|
|
// check if the key is already expired.
|
|
.and_then(|d| {
|
|
if d > OffsetDateTime::now_utc() {
|
|
Ok(d)
|
|
} else {
|
|
Err(Error::InvalidApiKeyExpiresAt(value.clone()))
|
|
}
|
|
})
|
|
.map(Option::Some),
|
|
Value::Null => Ok(None),
|
|
_otherwise => Err(Error::InvalidApiKeyExpiresAt(value.clone())),
|
|
}
|
|
}
|
|
|
|
#[derive(Copy, Clone, Serialize, Deserialize, Debug, Eq, PartialEq, Hash, Sequence)]
|
|
#[repr(u8)]
|
|
pub enum Action {
|
|
#[serde(rename = "*")]
|
|
All = 0,
|
|
#[serde(rename = "search")]
|
|
Search,
|
|
#[serde(rename = "documents.*")]
|
|
DocumentsAll,
|
|
#[serde(rename = "documents.add")]
|
|
DocumentsAdd,
|
|
#[serde(rename = "documents.get")]
|
|
DocumentsGet,
|
|
#[serde(rename = "documents.delete")]
|
|
DocumentsDelete,
|
|
#[serde(rename = "indexes.*")]
|
|
IndexesAll,
|
|
#[serde(rename = "indexes.create")]
|
|
IndexesAdd,
|
|
#[serde(rename = "indexes.get")]
|
|
IndexesGet,
|
|
#[serde(rename = "indexes.update")]
|
|
IndexesUpdate,
|
|
#[serde(rename = "indexes.delete")]
|
|
IndexesDelete,
|
|
#[serde(rename = "indexes.swap")]
|
|
IndexesSwap,
|
|
#[serde(rename = "tasks.*")]
|
|
TasksAll,
|
|
#[serde(rename = "tasks.cancel")]
|
|
TasksCancel,
|
|
#[serde(rename = "tasks.delete")]
|
|
TasksDelete,
|
|
#[serde(rename = "tasks.get")]
|
|
TasksGet,
|
|
#[serde(rename = "settings.*")]
|
|
SettingsAll,
|
|
#[serde(rename = "settings.get")]
|
|
SettingsGet,
|
|
#[serde(rename = "settings.update")]
|
|
SettingsUpdate,
|
|
#[serde(rename = "stats.*")]
|
|
StatsAll,
|
|
#[serde(rename = "stats.get")]
|
|
StatsGet,
|
|
#[serde(rename = "metrics.*")]
|
|
MetricsAll,
|
|
#[serde(rename = "metrics.get")]
|
|
MetricsGet,
|
|
#[serde(rename = "dumps.*")]
|
|
DumpsAll,
|
|
#[serde(rename = "dumps.create")]
|
|
DumpsCreate,
|
|
#[serde(rename = "version")]
|
|
Version,
|
|
#[serde(rename = "keys.create")]
|
|
KeysAdd,
|
|
#[serde(rename = "keys.get")]
|
|
KeysGet,
|
|
#[serde(rename = "keys.update")]
|
|
KeysUpdate,
|
|
#[serde(rename = "keys.delete")]
|
|
KeysDelete,
|
|
}
|
|
|
|
impl Action {
|
|
pub const fn from_repr(repr: u8) -> Option<Self> {
|
|
use actions::*;
|
|
match repr {
|
|
ALL => Some(Self::All),
|
|
SEARCH => Some(Self::Search),
|
|
DOCUMENTS_ALL => Some(Self::DocumentsAll),
|
|
DOCUMENTS_ADD => Some(Self::DocumentsAdd),
|
|
DOCUMENTS_GET => Some(Self::DocumentsGet),
|
|
DOCUMENTS_DELETE => Some(Self::DocumentsDelete),
|
|
INDEXES_ALL => Some(Self::IndexesAll),
|
|
INDEXES_CREATE => Some(Self::IndexesAdd),
|
|
INDEXES_GET => Some(Self::IndexesGet),
|
|
INDEXES_UPDATE => Some(Self::IndexesUpdate),
|
|
INDEXES_DELETE => Some(Self::IndexesDelete),
|
|
INDEXES_SWAP => Some(Self::IndexesSwap),
|
|
TASKS_ALL => Some(Self::TasksAll),
|
|
TASKS_CANCEL => Some(Self::TasksCancel),
|
|
TASKS_DELETE => Some(Self::TasksDelete),
|
|
TASKS_GET => Some(Self::TasksGet),
|
|
SETTINGS_ALL => Some(Self::SettingsAll),
|
|
SETTINGS_GET => Some(Self::SettingsGet),
|
|
SETTINGS_UPDATE => Some(Self::SettingsUpdate),
|
|
STATS_ALL => Some(Self::StatsAll),
|
|
STATS_GET => Some(Self::StatsGet),
|
|
METRICS_ALL => Some(Self::MetricsAll),
|
|
METRICS_GET => Some(Self::MetricsGet),
|
|
DUMPS_ALL => Some(Self::DumpsAll),
|
|
DUMPS_CREATE => Some(Self::DumpsCreate),
|
|
VERSION => Some(Self::Version),
|
|
KEYS_CREATE => Some(Self::KeysAdd),
|
|
KEYS_GET => Some(Self::KeysGet),
|
|
KEYS_UPDATE => Some(Self::KeysUpdate),
|
|
KEYS_DELETE => Some(Self::KeysDelete),
|
|
_otherwise => None,
|
|
}
|
|
}
|
|
|
|
pub const fn repr(&self) -> u8 {
|
|
*self as u8
|
|
}
|
|
}
|
|
|
|
pub mod actions {
|
|
use super::Action::*;
|
|
|
|
pub(crate) const ALL: u8 = All.repr();
|
|
pub const SEARCH: u8 = Search.repr();
|
|
pub const DOCUMENTS_ALL: u8 = DocumentsAll.repr();
|
|
pub const DOCUMENTS_ADD: u8 = DocumentsAdd.repr();
|
|
pub const DOCUMENTS_GET: u8 = DocumentsGet.repr();
|
|
pub const DOCUMENTS_DELETE: u8 = DocumentsDelete.repr();
|
|
pub const INDEXES_ALL: u8 = IndexesAll.repr();
|
|
pub const INDEXES_CREATE: u8 = IndexesAdd.repr();
|
|
pub const INDEXES_GET: u8 = IndexesGet.repr();
|
|
pub const INDEXES_UPDATE: u8 = IndexesUpdate.repr();
|
|
pub const INDEXES_DELETE: u8 = IndexesDelete.repr();
|
|
pub const INDEXES_SWAP: u8 = IndexesSwap.repr();
|
|
pub const TASKS_ALL: u8 = TasksAll.repr();
|
|
pub const TASKS_CANCEL: u8 = TasksCancel.repr();
|
|
pub const TASKS_DELETE: u8 = TasksDelete.repr();
|
|
pub const TASKS_GET: u8 = TasksGet.repr();
|
|
pub const SETTINGS_ALL: u8 = SettingsAll.repr();
|
|
pub const SETTINGS_GET: u8 = SettingsGet.repr();
|
|
pub const SETTINGS_UPDATE: u8 = SettingsUpdate.repr();
|
|
pub const STATS_ALL: u8 = StatsAll.repr();
|
|
pub const STATS_GET: u8 = StatsGet.repr();
|
|
pub const METRICS_ALL: u8 = MetricsAll.repr();
|
|
pub const METRICS_GET: u8 = MetricsGet.repr();
|
|
pub const DUMPS_ALL: u8 = DumpsAll.repr();
|
|
pub const DUMPS_CREATE: u8 = DumpsCreate.repr();
|
|
pub const VERSION: u8 = Version.repr();
|
|
pub const KEYS_CREATE: u8 = KeysAdd.repr();
|
|
pub const KEYS_GET: u8 = KeysGet.repr();
|
|
pub const KEYS_UPDATE: u8 = KeysUpdate.repr();
|
|
pub const KEYS_DELETE: u8 = KeysDelete.repr();
|
|
}
|
|
|
|
#[derive(Debug, thiserror::Error)]
|
|
pub enum Error {
|
|
#[error("`{0}` field is mandatory.")]
|
|
MissingParameter(&'static str),
|
|
#[error("`actions` field value `{0}` is invalid. It should be an array of string representing action names.")]
|
|
InvalidApiKeyActions(Value),
|
|
#[error("`indexes` field value `{0}` is invalid. It should be an array of string representing index names.")]
|
|
InvalidApiKeyIndexes(Value),
|
|
#[error("{0}")]
|
|
InvalidApiKeyIndexUid(IndexUidFormatError),
|
|
#[error("`expiresAt` field value `{0}` is invalid. It should follow the RFC 3339 format to represents a date or datetime in the future or specified as a null value. e.g. 'YYYY-MM-DD' or 'YYYY-MM-DD HH:MM:SS'.")]
|
|
InvalidApiKeyExpiresAt(Value),
|
|
#[error("`description` field value `{0}` is invalid. It should be a string or specified as a null value.")]
|
|
InvalidApiKeyDescription(Value),
|
|
#[error(
|
|
"`name` field value `{0}` is invalid. It should be a string or specified as a null value."
|
|
)]
|
|
InvalidApiKeyName(Value),
|
|
#[error("`uid` field value `{0}` is invalid. It should be a valid UUID v4 string or omitted.")]
|
|
InvalidApiKeyUid(Value),
|
|
#[error("The `{0}` field cannot be modified for the given resource.")]
|
|
ImmutableField(String),
|
|
}
|
|
|
|
impl From<IndexUidFormatError> for Error {
|
|
fn from(e: IndexUidFormatError) -> Self {
|
|
Self::InvalidApiKeyIndexUid(e)
|
|
}
|
|
}
|
|
|
|
impl ErrorCode for Error {
|
|
fn error_code(&self) -> Code {
|
|
match self {
|
|
Self::MissingParameter(_) => Code::MissingParameter,
|
|
Self::InvalidApiKeyActions(_) => Code::InvalidApiKeyActions,
|
|
Self::InvalidApiKeyIndexes(_) | Self::InvalidApiKeyIndexUid(_) => {
|
|
Code::InvalidApiKeyIndexes
|
|
}
|
|
Self::InvalidApiKeyExpiresAt(_) => Code::InvalidApiKeyExpiresAt,
|
|
Self::InvalidApiKeyDescription(_) => Code::InvalidApiKeyDescription,
|
|
Self::InvalidApiKeyName(_) => Code::InvalidApiKeyName,
|
|
Self::InvalidApiKeyUid(_) => Code::InvalidApiKeyUid,
|
|
Self::ImmutableField(_) => Code::ImmutableField,
|
|
}
|
|
}
|
|
}
|