Predictive quality metrics for context based models

This function computes and returns predictive quality metrics for context based models such as VLMC and VLMC with covariates.

Usage

metrics(model, ...)

Arguments

model: The context based model on which to compute predictive metrics.
...: Additional parameters for predictive metrics computation.

Value

The returned value is guaranteed to have at least three components

accuracy: the accuracy of the predictions
conf_mat: the confusion matrix of the predictions, with predicted values in rows and true values in columns
auc: the AUC of the predictive model

Details

A context based model computes transition probabilities for its contexts. Using a maximum transition probability decision rule, this can be used to predict the new state that is the more likely to follow the current one, given the context (see predict.vlmc()). The quality of these predictions is evaluated using standard metrics including:

accuracy
the full confusion matrix
the area under the roc curve (AUC), considering the context based model as a (conditional) probability estimator. We use Hand and Till (2001) multiclass AUC in case of a state space with more than 2 states

References

David J. Hand and Robert J. Till (2001). "A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems." Machine Learning 45(2), p. 171--186. DOI: doi:10.1023/A:1010920819831 .

Examples

pc <- powerconsumption[powerconsumption$week == 5, ]
breaks <- c(
  0,
  median(powerconsumption$active_power, na.rm = TRUE),
  max(powerconsumption$active_power, na.rm = TRUE)
)
labels <- c(0, 1)
dts <- cut(pc$active_power, breaks = breaks, labels = labels)
model <- vlmc(dts)
metrics(model)
#> VLMC context tree on 0, 1 
#>  cutoff: 1.921 (quantile: 0.05)
#>  Number of contexts: 69 
#>  Maximum context length: 63 
#>  Confusion matrix: 
#>     0   1   
#>   0 318 30  
#>   1 32  628 
#>  Accuracy: 0.9385 
#>  AUC: 0.9518