Every model can be monitored with a set of pre-defined and custom metrics. Conceptually metrics can be grouped into the following categories:
- Per-request Metrics — every request is evaluated against all assigned to the model metrics.
- Batch Metrics — requests are collected into a batch and the batch is proceeded to the assigned metrics for the the calculation.
- Overall Metrics — metrics are calculated against all data that was collected during production inference.
Per-request metrics allow you to make immediate judgments against an incoming request. Metrics described below are heavily focused on the anomaly detection task. We divide algorithms by the type of data that your models works with.
KNN — this algorithm uses distance to the nearest neighbors from the training dataset as a way to measure incoming sample outlier score.
IsolationForest — this algorithm is an autoregressive stateful model. We fit IsolationForest on 5 consequent data samples and then decide whether an incoming sample is an outlier or not.
- IsolationForest with EfficientNet — this algorithm uses IsolationForests trained on the features, extracted from the last layer of the EfficientNet.
Unknown Words Counter — this algorithm simply counts how many unknown words are present in the observed sample.
KMeans — this algorithm uses KMeans to score each incoming sample against predefined clusters and decide if the observing sample is an outlier or not.
- Autoencoder — this algorithm trains an autoencoder on the training/production data and shows a reconstruction error of each incoming sample.
Batch metrics let you compute metrics in a fixed-size windows, which gives you an opportunity to compare distributions, validate hypothesis, etc.
Overall metrics give you an idea how all of your data looks like through profile calculations. Profiles, like algorithms, are differentiated by the data type.
- Total request amount
- Unique request amount
- Unique request percentage
- Missing values amount
- Missing values percentage
- 5th percentile
- 95th percentile
- Interquartile rage
- Standard deviation
- Coefficient of variation
- Common statistics
- Mean token length
- Mean character length
- Mean tree depth
- Mean unique lemma ratio
- Mean sentiment score
- Mean language proba
- Mean POS proba