mlr3 中的自定义 Precision-Recall AUC 测量答案

【问题标题】：Custom Precision-Recall AUC measure in mlr3mlr3 中的自定义 Precision-Recall AUC 测量
【发布时间】：2020-08-20 02:25:56
【问题描述】：

我想在 mlr3 中创建自定义 Precision-Recall AUC 度量。

我在关注mlr3 book chapter on creating custom measures.

我觉得我几乎在那里，但是 R 抛出了一个我不知道如何解释的恼人的错误。

让我们定义度量：

PRAUC = R6::R6Class("PRAUC",
  inherit = mlr3::MeasureClassif,
    public = list(
      initialize = function() {
        super$initialize(
          # custom id for the measure
          id = "classif.prauc",

          # additional packages required to calculate this measure
          packages = c('PRROC'),

          # properties, see below
          properties = character(),

          # required predict type of the learner
          predict_type = "prob",

          # feasible range of values
          range = c(0, 1),

          # minimize during tuning?
          minimize = FALSE
        )
      }
    ),

    private = list(
      # custom scoring function operating on the prediction object
      .score = function(prediction, ...) {

        truth1 <- ifelse(prediction$truth == levels(prediction$truth)[1], 1, 0) # Function PRROC::pr.curve assumes binary response is numeric, positive class is 1, negative class is 0
        PRROC::pr.curve(scores.class0 = prediction$prob, weights.class0 = truth1)

      }
    )
)

mlr3::mlr_measures$add("classif.prauc", PRAUC)

让我们看看它是否有效：

task_sonar <- tsk('sonar')
learner <- lrn('classif.rpart', predict_type = 'prob')
learner$train(task_sonar)
pred <- learner$predict(task_sonar)
pred$score(msr('classif.prauc'))

# Error in if (sum(weights < 0) != 0) { : 
#  missing value where TRUE/FALSE needed

这是回溯：

11.
check(length(sorted.scores.class0), weights.class0) 
10.
compute.pr(scores.class0, scores.class1, weights.class0, weights.class1, 
    curve, minStepSize, max.compute, min.compute, rand.compute, 
    dg.compute) 
9.
PRROC::pr.curve(scores.class0 = prediction$prob, weights.class0 = truth1) 
8.
measure$.__enclos_env__$private$.score(prediction = prediction, 
    task = task, learner = learner, train_set = train_set) 
7.
measure_score(self, prediction, task, learner, train_set) 
6.
m$score(prediction = self, task = task, learner = learner, train_set = train_set) 
5.
FUN(X[[i]], ...) 
4.
vapply(.x, .f, FUN.VALUE = .value, USE.NAMES = FALSE, ...) 
3.
map_mold(.x, .f, NA_real_, ...) 
2.
map_dbl(measures, function(m) m$score(prediction = self, task = task, 
    learner = learner, train_set = train_set)) 
1.
pred$score(msr("classif.prauc"))

似乎故障来自PRROC::pr.curve。但是，在实际预测对象pred 上尝试此功能时，它工作得很好：

PRROC::pr.curve(
  scores.class0 = pred$prob[, 1], 
  weights.class0 =  ifelse(pred$truth == levels(pred$truth)[1], 1, 0)
)

#  Precision-recall curve
#
#    Area under curve (Integral):
#     0.9081261
#
#    Area under curve (Davis & Goadrich):
#     0.9081837 
#
#    Curve not computed ( can be done by using curve=TRUE )

发生错误的一种可能情况是，在PRAUC 内部，PRROC::pr.curve 的参数weights.class0 是NA。我无法确认这一点，但我怀疑weights.class0 正在接收NA 而不是数字，导致PRROC::pr.curve 在PRAUC 内部发生故障。如果是这样的话，我不知道为什么会这样。

可能还有其他我没有想到的场景。任何帮助将不胜感激。

编辑

missuse 的回答帮助我意识到为什么我的措施不起作用。首先，

PRROC::pr.curve(scores.class0 = prediction$prob, weights.class0 = truth1)

应该是

PRROC::pr.curve(scores.class0 = prediction$prob[, 1], weights.class0 = truth1)。

其次，函数pr.curve 返回一个类PRROC 的对象，而我定义的mlr3 度量实际上期待numeric。所以应该是

PRROC::pr.curve(scores.class0 = prediction$prob[, 1], weights.class0 = truth1)[[2]]

或

PRROC::pr.curve(scores.class0 = prediction$prob[, 1], weights.class0 = truth1)[[3]],

取决于用于计算 AUC 的方法（请参阅?PRROC::pr.curve）。

请注意，虽然MLmetrics::PRAUC 远没有PRROC::pr.curve 令人困惑，但它看起来像the former is poorly implemented。

以下是实际有效的PRROC::pr.curve 措施的实现：

PRAUC = R6::R6Class("PRAUC",
  inherit = mlr3::MeasureClassif,
    public = list(
      initialize = function() {
        super$initialize(
          # custom id for the measure
          id = "classif.prauc",

          # additional packages required to calculate this measure
          packages = c('PRROC'),

          # properties, see below
          properties = character(),

          # required predict type of the learner
          predict_type = "prob",

          # feasible range of values
          range = c(0, 1),

          # minimize during tuning?
          minimize = FALSE
        )
      }
    ),

    private = list(
      # custom scoring function operating on the prediction object
      .score = function(prediction, ...) {

        truth1 <- ifelse(prediction$truth == levels(prediction$truth)[1], 1, 0) # Looks like in mlr3 the positive class in binary classification is always the first factor level
        PRROC::pr.curve(
          scores.class0 = prediction$prob[, 1], # Looks like in mlr3 the positive class in binary classification is always the first of two columns
          weights.class0 = truth1
        )[[2]]

      }
    )
)

mlr3::mlr_measures$add("classif.prauc", PRAUC)

例子：

task_sonar <- tsk('sonar')
learner <- lrn('classif.rpart', predict_type = 'prob')
learner$train(task_sonar)
pred <- learner$predict(task_sonar)
pred$score(msr('classif.prauc'))

#classif.prauc 
#     0.923816

但是，现在的问题是改变正类会导致不同的分数：

task_sonar <- tsk('sonar')
task_sonar$positive <- 'R' # Now R is the positive class
learner <- lrn('classif.rpart', predict_type = 'prob')
learner$train(task_sonar)
pred <- learner$predict(task_sonar)
pred$score(msr('classif.prauc'))

#classif.prauc 
#    0.9081261

【问题讨论】：

尽管MLmetrics::PRAUC 可能不如PRROC::pr.curve，但在mlr3 中实现度量的总体思路是相同的。有趣的是我忘记了this。应该使用 prediction$prob[,1] 是正确的，因为它对应于正类。至于编辑过的问题，我不确定问题出在哪里？这就像说当你改变正面类时灵敏度会改变。这是度量类型固有的。
我的错。我认为 PR AUC 和标准 ROC AUC 一样是对称的。例如。 MLmetrics::AUC(pred$data$prob[, 1], as.integer(pred$truth == "M")) == MLmetrics::AUC(pred$data$prob[, 2], as.integer(pred$truth == "R")) 是 TRUE，而 MLmetrics::PRAUC(pred$data$prob[, 1], as.integer(pred$truth == "M")) == MLmetrics::PRAUC(pred$data$prob[, 2], as.integer(pred$truth == "R")) 是 FALSE。
我用更精确的PRROC::pr.curve 编辑了我的答案以指向您的实现。一切顺利。

标签： r machine-learning precision-recall mlr3

【解决方案1】：

?PRROC::pr.curve 比较混乱，所以我将使用MLmetrics::PRAUC 来计算 PRAUC：

library(mlr3measures)
library(mlr3)

PRAUC = R6::R6Class("PRAUC",
                    inherit = mlr3::MeasureClassif,
                    public = list(
                      initialize = function() {
                        super$initialize(
                          # custom id for the measure
                          id = "classif.prauc",

                          # additional packages required to calculate this measure
                          packages = c('MLmetrics'),

                          # properties, see below
                          properties = character(),

                          # required predict type of the learner
                          predict_type = "prob",

                          # feasible range of values
                          range = c(0, 1),

                          # minimize during tuning?
                          minimize = FALSE
                        )
                      }
                    ),

                    private = list(
                      # custom scoring function operating on the prediction object
                      .score = function(prediction, ...) {

                        MLmetrics::PRAUC(prediction$prob[,1], #probs for 1st (positive class is in first column) class
                                         as.integer(prediction$truth == levels(prediction$truth)[1])) #truth for 1st class

                      }
                    )
)

验证它是否有效：

mlr3::mlr_measures$add("classif.prauc", PRAUC)
task_sonar <- tsk('sonar')
learner <- lrn('classif.rpart', predict_type = 'prob')
learner$train(task_sonar)
pred <- learner$predict(task_sonar)
pred$score(msr('classif.prauc'))
classif.prauc 
     0.8489383  

MLmetrics::PRAUC(pred$data$prob[,1],
                 as.integer(pred$truth == "M"))
0.8489383

编辑：使用PRROC::pr.curve 的措施实施作为对上述问题的编辑给出。建议使用该实现，因为与MLmetrics::PRAUC 相比，PRROC::pr.curve 更精确。

【讨论】：