如何避免R中的for循环[三重循环又名三重威胁]

Question

目前，我遇到了计算时间问题，因为我在 R 中运行了一个三重 for 循环，以便为每个唯一 ID 创建星期几和小时级别的异常阈值。

我的原始数据框： 唯一 ID、事件日期小时、事件日期、事件星期几、事件小时、数值变量 1、数值变量 2 等。

df <- read.csv("mm.csv",header=TRUE,sep=",")

for (i in unique(df$customer_id)) {
  #I initialize the output data frame so I can rbind as I loop though the grains. This data frame is always emptied out once we move onto our next customer_id
  output.final.df <- data_frame(seller_name = factor(), is_anomaly_date = integer(), event_date_hr = double(), event_day_of_wk = integer(), event_day = double(), ...)

  for (k in unique(df$event_day_of_wk)) {
    for (z in unique(df$event_hr)) {
      merchant.df = df[df$merchant_customer_id==i & df$event_day_of_wk==k & df$event_hr==z,10:19] #columns 10:19 are the 9 different numeric variables I am creating anomaly thresholds

      #1st anomaly threshold - I have multiple different anomaly thresholds

      # TRANSFORM VARIABLES - sometime within the for loop I run another loop that transforms the subset of data within it.
      for(j in names(merchant.df)){
        merchant.df[[paste(j,"_log")]] <- log(merchant.df[[j]]+1)
        #merchant.df[[paste(j,"_scale")]] <- scale(merchant.df[[j]])
        #merchant.df[[paste(j,"_cube")]] <- merchant.df[[j]]**3
        #merchant.df[[paste(j,"_cos")]] <- cos(merchant.df[[j]])
      }

      mu_vector        = apply( merchant.df, 2, mean )
      sigma_matrix     = cov( merchant.df, use="complete.obs", method='pearson' )
      inv_sigma_matrix = ginv(sigma_matrix)
      det_sigma_matrix = det( sigma_matrix )

      z_probas = apply( merchant.df, 1, mv_gaussian, mu_vector, det_sigma_matrix, inv_sigma_matrix )
      eps = quantile(z_probas,0.01)
      mv_outliers = ifelse( z_probas<eps, TRUE, FALSE )

      #2nd anomaly threshold
      nov = ncol(merchant.df)
      pca_result <- PCA(merchant.df,graph = F, ncp = nov, scale.unit = T)
      pca.var <- pca_result$eig[['cumulative percentage of variance']]/100
      lambda <- pca_result$eig[, 'eigenvalue']
      anomaly_score = (as.matrix(pca_result$ind$coord) ^ 2) %*% (1 / as.matrix(lambda, ncol = 1))
      significance <- c (0.99)
      thresh = qchisq(significance, nov)
      pca_outliers = ifelse( anomaly_score > thresh , TRUE, FALSE )

      #This is where I bind the anomaly points with the original data frame and then I row bind to the final output data frame then the code goes back to the top and loops through the next hour and then day of the week. Temp.output.df is constantly remade and output.df is slowly growing bigger.
      temp.output.df <- cbind(merchant.df, mv_outliers, pca_outliers)
      output.df <- rbind(output.df, temp.output.df)
     }
    }
   #Again this is where I write the output for a particular unique_ID then output.df is recreated at the top for the next unique_ID
   write.csv(output.df,row.names=FALSE)
   }

以下代码显示了我在做什么的想法。如您所见，我运行了 3 个循环，在其中我以最低粒度（即一周中的每一天的小时级别）计算多个异常检测，然后一旦完成，我将每个唯一的 customer_id 级别输出到 csv 中。

总体而言，代码运行速度非常快；但是，执行三重 for 循环正在扼杀我的表现。鉴于我的原始数据框并且需要在每个 unique_id 级别输出 csv，有谁知道我可以通过其他方式进行这样的操作吗？

Answer 1

• 所以不要使用三重循环。使用 dplyr::group_by(customer_id, event_day_of_wk, event_hr) 或 data.table 等效项。两者都应该更快。
• 无需在每次迭代中显式附加 rbind 和 cbind，否则会影响性能。
• 另外，不需要cbind()你的整个输入df到你的输出df；你唯一的实际输出是mv_outliers, pca_outliers；你可以join() 稍后在customer_id, event_day_of_wk, event_hr 输入和输出dfs
• 编辑：由于您想整理每个customer_id 然后write.csv() 它们的所有结果，因此需要进入分组的外部级别，而group_by(event_day_of_wk, event_hr) 进入内部级别。

.

# Here is pseudocode, you can figure out the rest, do things incrementally
# It looks like seller_name, is_anomaly_date, event_date_hr, event_day_of_wk, event_day,... are variables from your input

require(dplyr)

output.df <- df %>%
  group_by(customer_id) %>%
    group_by(event_day_of_wk, event_hr) %>%

    # columns 10:19 ('foo','bar','baz'...) are the 9 different numeric variables I am creating anomaly thresholds
    # Either a) you can hardcode their names in mutate(), summarize() calls
    #  or b) you can reference the vars by string in mutate_(), summarize_() calls

    # TRANSFORM VARIABLES
    mutate(foo_log = log1p(foo), bar_log = log1p(bar), ...) %>%

    mutate(mu_vector = c(mean(foo_log), mean(bar_log)...) ) %>%
    # compute sigma_matrix, inv_sigma_matrix, det_sigma_matrix ...

    summarize(
       z_probas=mv_gaussian(mu_vector, det_sigma_matrix, inv_sigma_matrix),
       eps = quantile(z_probas,0.01),
       mv_outliers = (z_probas<eps)
    ) %>%

    # similarly, use mutate() and do.call() for your PCA invocation...

    # Your outputs are mv_outliers, pca_outliers
    # You don't necessarily need to `cbind(merchant.df, mv_outliers, pca_outliers)` i.e. cbind all your input data together with your output

    # Now remove all your temporary variables from your output:
    select(-foo_log, -bar_log, ...) %>%
    # or else just select(mv_outliers, pca_outliers) the variables you want to keep

  ungroup() %>%  # (this ends the group_by(event_day_of_wk, event_hr) and cbinds all the intermediate dataframes for you)

  write.csv( c(.$mv_outliers, .$pca_outliers), file='<this_customer_id>.csv')

ungroup()  # group_by(customer_id)

另见"write.csv() in dplyr chain"