提高 R 中 For 循环的性能以计算所有 Shapley 值

Question

我目前正在 R 中处理 For 循环。如果我对自己的数据运行 For 循环，则需要很长时间，我相信是因为我在代码中做了一些低效的事情。你能帮我改进一下吗？

# Loop through the samples, explaining one instance at a time.
shap_values <- vector("list", nrow(X))  # initialize the results list.
system.time({
  for (i in seq_along(shap_values)) {
    set.seed(224)
    shap_values[[i]] <- iml::Shapley$new(predictor, x.interest = X[i, ],sample.size = 30)$results
    shap_values[[i]]$predicted_value <- iml::Shapley$new(predictor, x.interest = X[i, ],sample.size = 30)$y.hat.interest
    shap_values[[i]]$sample_num <- i  # identifier to track our instances.
  }
  data_shap_values <- dplyr::bind_rows(shap_values)  # collapse the list.
})

我相信我的问题出在

shap_values[[i]]$sample_num

变量，因为我在那里重做之前的计算

shap_values[[i]]$predicted_value

变量。我添加该变量的原因是因为我需要

$y.hat.interest

作为新数据框的一部分（称为“shap_values”，后来称为“data_shap_values”）。

可重复的示例：（从“这是重要的部分：）开始：）

#Example Shapley
#https://cran.r-project.org/web/packages/iml/vignettes/intro.html

data("Boston", package  = "MASS")
head(Boston)

set.seed(42)
#install.packages("iml")
library("iml")
library("randomForest")
data("Boston", package  = "MASS")
rf = randomForest(medv ~ ., data = Boston, ntree = 50)

# We create a Predictor object, that holds the model and the data. 
# The iml package uses R6 classes: New objects can be created by calling Predictor$new()
X = Boston[which(names(Boston) != "medv")]
predictor = Predictor$new(rf, data = X, y = Boston$medv)

# Feature Importance
## Shifting each future, and measring how much the performance drops ## 
imp = FeatureImp$new(predictor, loss = "mae")
plot(imp)


# Shapley value. Assume that for 1 data point, the feature values play a game together, in which 
# they get the prediction as payout. Tells us how fairly distibute the payout among the feature values. 
View(X)
shapley = Shapley$new(predictor, x.interest = X[1,])
shapley$plot()

# Reuse the object to explain other data points 
shapley$explain(x.interest = X[2,])
shapley$plot()

# Results in data.frame form can be extracted like this: 
results = shapley$results
head(results)


# THIS IS THE IMPORTANT PART: 

# It might make sense for testing, to reduce the data: 
X = X[1:10,]

# Loop through the samples, explaining one instance at a time.
shap_values <- vector("list", nrow(X))  # initialize the results list.
system.time({
  for (i in seq_along(shap_values)) {
    set.seed(224)
    shap_values[[i]] <- iml::Shapley$new(predictor, x.interest = X[i, ],sample.size = 30)$results
    shap_values[[i]]$predicted_value <- iml::Shapley$new(predictor, x.interest = X[i, ],sample.size = 30)$y.hat.interest
    shap_values[[i]]$sample_num <- i  # identifier to track our instances.
  }
  data_shap_values <- dplyr::bind_rows(shap_values)  # collapse the list.
})

更新

根据@Ralf Stubner 的要求，分析 for 循环：

Answer 1

通过使用相同的参数调用两次imp::Shapely$new，您将运行时间加倍。作为替代方案，您可以创建一次对象并提取两个值：

system.time({
    for (i in seq_along(shap_values)) {
        set.seed(224)
        shapley <- iml::Shapley$new(predictor, x.interest = X[i, ],sample.size = 30)
        shap_values[[i]] <- shapley$results
        shap_values[[i]]$predicted_value <- shapley$y.hat.interest
        shap_values[[i]]$sample_num <- i  # identifier to track our instances.
    }
    data_shap_values <- dplyr::bind_rows(shap_values)  # collapse the list.
})

如果您有足够的 RAM 来多次存储数据，您还可以尝试使用 parallel、foreach 或 future.apply 进行并行处理。