如何告诉 h2o 深度学习网格具有 AUC 而不是残余偏差答案

【问题标题】：How can I tell h2o deep learning grid to have AUC instead of residual deviance如何告诉 h2o 深度学习网格具有 AUC 而不是残余偏差
【发布时间】：2017-05-17 07:07:54
【问题描述】：

我想通过寻找 AUC 或准确度来衡量模型的性能。在网格搜索中，我得到residual deviance 的结果，我如何告诉 h2o 深度学习网格具有 AUC 而不是残余偏差，并将结果呈现为如下所示的表格？

train <- read.table(text = "target birds    wolfs     snakes
                              0        9         7 a
                              0        8         4 b
                              1        2         8 c
                              1        2         3 a
                              1        8         3 a
                              0        1         2 a
                              0        7         1 b
                              0        1         5 c
                              1        9         7 c
                              1        8         7 c
                              0        2         7 b
                              1        2         3 b
                              1        6         3 c
                              0        1         1 a
                              0        3         9 a
                              1        1         1 b ",header = TRUE)
trainHex <- as.h2o(train)

g <- h2o.grid("deeplearning",
              hyper_params = list(
                  seed = c(123456789,12345678,1234567),
                  activation = c("Rectifier", "Tanh", "TanhWithDropout", "RectifierWithDropout", "Maxout", "MaxoutWithDropout")
              ),
              reproducible = TRUE,
              x = 2:4,
              y = 1,
              training_frame = trainHex,
              validation_frame = trainHex,
              epochs = 50,
              )
g
model_ids <- g@summary_table
model_ids<-as.data.frame(model_ids)

我得到的结果表：

     Hyper-Parameter Search Summary: ordered by increasing residual_deviance
             activation      seed                                                  model_ids   residual_deviance
1                Maxout  12345678 Grid_DeepLearning_train_model_R_1483217086840_112_model_10 0.07243775676256235
2                Maxout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_16 0.10060885040861599
3     MaxoutWithDropout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_5  0.1706496158406441
4                Maxout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_4 0.17243125875659948
5                  Tanh 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_1 0.18326527198894926
6                  Tanh  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_7 0.18763395264761593
7                  Tanh   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_13 0.18791531211136187
8       TanhWithDropout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_2 0.19808063817007837
9       TanhWithDropout  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_8 0.19815190962052193
10      TanhWithDropout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_14 0.19832946889767458
11            Rectifier 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_0 0.20679125165086842
12    MaxoutWithDropout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_17 0.21971759565380736
13 RectifierWithDropout 123456789  Grid_DeepLearning_train_model_R_1483217086840_112_model_3 0.22337599298253263
14    MaxoutWithDropout  12345678 Grid_DeepLearning_train_model_R_1483217086840_112_model_11 0.22440661112729862
15 RectifierWithDropout   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_15  0.2284671685474275
16 RectifierWithDropout  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_9 0.23163744415703522
17            Rectifier   1234567 Grid_DeepLearning_train_model_R_1483217086840_112_model_12  0.2516917276707789
18            Rectifier  12345678  Grid_DeepLearning_train_model_R_1483217086840_112_model_6  0.2642221616447725

【问题讨论】：

顺便说一句，将validation_frame 设置为与training_frame 相同是默认行为，因此无需指定。请注意，通过不使用验证和测试数据集，您正在针对过度拟合最佳的深度学习参数进行优化。我什至不确定您对随机种子对结果变化的影响的了解是否适用于看不见的数据。（当然它仍然可以是一个有趣的实验：例如，我以前做过这个，看看需要多少隐藏节点/层/时期才能完美地拟合数据。）

标签： r h2o auc

【解决方案1】：

您可以使用h2o.getGrid() 执行此操作。从您的示例代码开始：

g_rmse <- h2o.getGrid(g@grid_id, "rmse")
g_rmse  #Output it

我在那里选择了 root-MSE。 AUC 不适用于您的样本数据：它必须是二项式分类，并且您正在进行回归。

您进行回归的原因是您的 y 包含 0 和 1，因此 H2O 猜测它是数字。您需要在该列上使用as.factor()，将其上传到 H2O 后。

train <-  ...
trainHex <- as.h2o(train)
trainHex[,1] = as.factor(trainHex[,1])  #Add this

g <- ...

那么你可以这样做：

g_auc <- h2o.getGrid(g@grid_id, "auc", decreasing = TRUE)
g_auc

我已将其设置为 decreasing=TRUE，以便最佳 AUC 位于顶部。

【讨论】：

非常感谢@Darren Cook 的详细回答。