如何使用网格搜索和使用 tidymodels 的单个验证折叠来调整模型？

Question

我刚刚了解了 KNN 算法和机器学习。这对我来说很多，我们在 R 中使用tidymodels 来练习。

现在，我知道如何使用 k 折交叉验证来实现网格搜索，如下所示：

hist_data_split <- initial_split(hist_data, strata = fraud)
hist_data_train <- training(hist_data_split)
hist_data_test <- testing(hist_data_split)
folds <- vfold_cv(hist_data_train, strata = fraud)
nearest_neighbor_grid <- grid_regular(neighbors(range = c(1, 500)), levels = 25)
knn_rec_1 <- recipe(fraud ~ ., data = hist_data_train)
knn_spec_1 <- nearest_neighbor(mode = "classification", engine = "kknn", neighbors = tune(), weight_func = "rectangular")
knn_wf_1 <- workflow(preprocessor = knn_rec_1, spec = knn_spec_1)
knn_fit_1 <- tune_grid(knn_wf_1, resamples = folds, metrics = metric_set(accuracy, sens, spec, roc_auc), control = control_resamples(save_pred = T), grid = nearest_neighbor_grid)

在上述情况下，我实际上是在运行 10 倍交叉验证网格搜索来调整我的模型。然而，hist_data 的大小是 169173，它给出了大约 411 的最佳 K，并且使用 10 倍交叉验证，调整将永远需要，所以给出的提示是使用单个验证折叠而不是交叉验证。

因此，我想知道如何调整我的代码来实现这一点。当我在vfold_cv 中添加参数v = 1 时，R 向我抛出一个错误，上面写着“至少应该为分析集选择一行”。我是否应该将tune_grid 中的resamples = folds 改为resamples = 1？

任何直观的建议将不胜感激:)

附言在没有提供数据的意义上，我没有包括 MWE，因为我觉得这是一个非常微不足道的问题，可以按原样回答！

Answer 1

如果您无法进行交叉验证拆分，无论出于何种原因，您都可以进行概念上非常接近v = 1 交叉验证的验​​证拆分。

library(tidymodels)

hist_data_split <- initial_split(ames, strata = Street)
hist_data_train <- training(hist_data_split)
hist_data_test <- testing(hist_data_split)

folds <- validation_split(hist_data_train, strata = Street)

nearest_neighbor_grid <- grid_regular(
  neighbors(range = c(1, 500)), 
  levels = 25
)

knn_rec_1 <- recipe(Street ~ ., data = ames)
knn_spec_1 <- nearest_neighbor(neighbors = tune()) %>%
  set_mode("classification") %>%
  set_engine("kknn") %>%
  set_args(weight_func = "rectangular")

knn_wf_1 <- workflow(preprocessor = knn_rec_1, spec = knn_spec_1)

knn_fit_1 <- tune_grid(
  knn_wf_1,
  resamples = folds,
  metrics = metric_set(accuracy, sens, spec, roc_auc),
  control = control_resamples(save_pred = T),
  grid = nearest_neighbor_grid
)

knn_fit_1
#> # Tuning results
#> # Validation Set Split (0.75/0.25)  using stratification 
#> # A tibble: 1 × 5
#>   splits             id         .metrics           .notes           .predictions
#>   <list>             <chr>      <list>             <list>           <list>      
#> 1 <split [1647/550]> validation <tibble [100 × 5]> <tibble [0 × 3]> <tibble>

knn_fit_1 %>%
  collect_metrics()
#> # A tibble: 100 × 7
#>    neighbors .metric  .estimator  mean     n std_err .config              
#>        <int> <chr>    <chr>      <dbl> <int>   <dbl> <chr>                
#>  1         1 accuracy binary     0.996     1      NA Preprocessor1_Model01
#>  2         1 roc_auc  binary     0.5       1      NA Preprocessor1_Model01
#>  3         1 sens     binary     0         1      NA Preprocessor1_Model01
#>  4         1 spec     binary     1         1      NA Preprocessor1_Model01
#>  5        21 accuracy binary     0.996     1      NA Preprocessor1_Model02
#>  6        21 roc_auc  binary     0.495     1      NA Preprocessor1_Model02
#>  7        21 sens     binary     0         1      NA Preprocessor1_Model02
#>  8        21 spec     binary     1         1      NA Preprocessor1_Model02
#>  9        42 accuracy binary     0.996     1      NA Preprocessor1_Model03
#> 10        42 roc_auc  binary     0.486     1      NA Preprocessor1_Model03
#> # … with 90 more rows

^{由reprex package (v2.0.1) 于 2022-09-06 创建}