Monitoring
Every model can be monitored with a set of predefined and custom metrics. Conceptually metrics can be grouped into the following categories:
 Perrequest Metrics — every request is evaluated against all assigned to the model metrics.
 Batch Metrics — requests are collected into a batch and the batch is proceeded to the assigned metrics for the the calculation.
 Overall Metrics — metrics are calculated against all data that was collected during production inference.
Perrequest Metrics
Perrequest metrics allow you to make immediate judgments against an incoming request. Metrics described below are heavily focused on the anomaly detection task. We divide algorithms by the type of data that your models works with.
Numerical

KNN — this algorithm uses distance to the nearest neighbors from the training dataset as a way to measure incoming sample outlier score.

IsolationForest — this algorithm is an autoregressive stateful model. We fit IsolationForest on 5 consequent data samples and then decide whether an incoming sample is an outlier or not.
Image
 IsolationForest with EfficientNet — this algorithm uses IsolationForests trained on the features, extracted from the last layer of the EfficientNet.
Text

Unknown Words Counter — this algorithm simply counts how many unknown words are present in the observed sample.

KMeans — this algorithm uses KMeans to score each incoming sample against predefined clusters and decide if the observing sample is an outlier or not.
All
 Autoencoder — this algorithm trains an autoencoder on the training/production data and shows a reconstruction error of each incoming sample.
Batch Metrics
Batch metrics let you compute metrics in a fixedsize windows, which gives you an opportunity to compare distributions, validate hypothesis, etc.
Overall Metrics
Overall metrics give you an idea how all of your data looks like through profile calculations. Profiles, like algorithms, are differentiated by the data type.
Numerical

Common statistics
 Total request amount
 Unique request amount
 Unique request percentage
 Missing values amount
 Missing values percentage

Quantile statistics
 Min
 5th percentile
 Q1
 Median
 Q3
 95th percentile
 Range
 Interquartile rage
 Max

Descriptive statistics
 Standard deviation
 Coefficient of variation
 Kurtosis
 Variance
 Mean
 Skewness
Text
 Common statistics
 Mean token length
 Mean character length
 Mean tree depth
 Mean unique lemma ratio
 Mean sentiment score
 Mean language proba
 Mean POS proba