R 在具有多个连接和过滤器的 dplyr 上表现不佳答案

【问题标题】：R underperforming on dplyr summarize with multiple joins and filtersR 在具有多个连接和过滤器的 dplyr 上表现不佳
【发布时间】：2021-09-23 16:10:49
【问题描述】：

我有以下包含 3 个数据框的示例数据集：

base_pop_ex <-
  structure(
    list(
      anon_id = c(
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7"
      ),
      session_number = c(1,
                         2),
      entrance_date = c("2021-06-28 11:43:21.633 Z", "2021-06-29 01:10:08.109 Z"),
      single_article_session = c(0, 0)
    ),
    .Names = c(
      "anon_id",
      "session_number",
      "entrance_date",
      "single_article_session"
    ),
    row.names = c(NA,-2L),
    class = c("tbl_df", "tbl", "data.frame")
  )

ad_views_ex <-
  structure(
    list(
      anon_id = c(
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7"
      ),
      ad_view_date = c(
        "2021-06-28 11:43:22.654 Z",
        "2021-06-28 11:44:15.360 Z",
        "2021-06-28 11:44:32.538 Z",
        "2021-06-28 12:07:19.557 Z",
        "2021-06-28 12:07:20.146 Z",
        "2021-06-29 01:10:08.706 Z",
        "2021-06-29 01:10:17.127 Z",
        "2021-06-29 01:40:30.726 Z",
        "2021-06-29 01:40:30.914 Z"
      ),
      ad_call_count = c(3, 1, 1, 1, 3,
                        3, 1, 1, 3)
    ),
    .Names = c("anon_id", "ad_view_date", "ad_call_count"),
    row.names = c(NA,-9L),
    class = c("tbl_df", "tbl", "data.frame")
  )

scroll_depth_ex <-
  structure(
    list(
      anon_id = c(
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7",
        "0003ff12-03b1-42b9-86cf-4b7c05e3e3a7"
      ),
      scroll_date = c(
        "2021-06-28 11:43:38.263 Z",
        "2021-06-28 11:43:41.593 Z",
        "2021-06-28 11:43:48.882 Z",
        "2021-06-28 11:43:49.339 Z",
        "2021-06-28 11:43:52.270 Z",
        "2021-06-28 11:43:57.995 Z",
        "2021-06-28 11:44:15.324 Z",
        "2021-06-28 11:44:16.955 Z",
        "2021-06-28 11:44:30.284 Z",
        "2021-06-28 11:44:44.197 Z",
        "2021-06-28 12:07:19.564 Z",
        "2021-06-28 12:07:19.581 Z",
        "2021-06-28 12:07:19.593 Z",
        "2021-06-28 12:07:19.600 Z",
        "2021-06-28 12:07:19.617 Z",
        "2021-06-28 12:07:19.639 Z",
        "2021-06-28 12:07:19.648 Z",
        "2021-06-28 12:07:19.664 Z",
        "2021-06-29 01:10:13.401 Z",
        "2021-06-29 01:10:25.065 Z",
        "2021-06-29 01:11:02.595 Z",
        "2021-06-29 01:11:45.444 Z",
        "2021-06-29 01:40:30.741 Z",
        "2021-06-29 01:40:30.747 Z",
        "2021-06-29 01:40:30.903 Z",
        "2021-06-29 01:40:30.909 Z"
      ),
      scroll_depth = c(
        10,
        20,
        30,
        40,
        50,
        60,
        70,
        80,
        90,
        100,
        10,
        20,
        30,
        40,
        50,
        60,
        70,
        80,
        10,
        20,
        30,
        40,
        10,
        20,
        30,
        40
      )
    ),
    .Names = c("anon_id", "scroll_date",
               "scroll_depth"),
    row.names = c(NA,-26L),
    class = c("tbl_df",
              "tbl", "data.frame")
  )

我想加入所有三个数据框，所以最后得到anon_id、entrance_date、session_number、ad_views 和scroll_depth：

ad_views 是数据框 ad_views_ex 的所有 ad_call_counts 的总和，其中 ad_view_date 大于表 base_pop_ex 中的 entrance_date，同时，两个日期之间的分钟差更小超过 60
scroll_depth 对连接使用与前一个指标相同的逻辑。但是，我在这里计算每组事件的最大值

下面的代码完成了它的工作：

library(tidyr)
library(lubridate)

combined_ex <- base_pop_ex %>%
      left_join(ad_views_ex, by = c("anon_id")) %>%
      filter(
        entrance_date <= ad_view_date &
          difftime(ad_view_date, entrance_date, units = "mins") <= 60
      ) %>%
      group_by(anon_id, entrance_date, session_number) %>%
      summarize(
        ad_views = sum(ad_call_count, na.rm = TRUE)
        )
      )
    
    combined_ex2 <- combined_ex %>%
      left_join(scroll_depth_ex, by = c("anon_id")) %>%
      filter(
        entrance_date <= scroll_date &
          difftime(scroll_date, entrance_date, units = "mins") <= 60
      ) %>%
      group_by(anon_id, entrance_date, session_number, ad_views) %>%
      summarize(
        scroll_depth = max(scroll_depth, na.rm = TRUE)
      )
    )

最终结果为combined_ex2this：

|               anon_id              |       entrance_date        | session_number | ad_views | scroll_depth |
|------------------------------------|----------------------------|----------------|----------|--------------|
|0003ff12-03b1-42b9-86cf-4b7c05e3e3a7| 2021-06-28 11:43:21.633 Z  |       1        |   162    |      100     |      
|0003ff12-03b1-42b9-86cf-4b7c05e3e3a7| 2021-06-29 01:10:08.109 Z  |       2        |    64    |      40      |

但是，当我将其扩展到我的真实数据时，Rstudio 需要大约 1 分钟来创建第一个组合数据框，而需要 8 分钟来创建第二个组合数据框。我的数据包含base_pop_ex 的 500K 行，ad_views_ex 的 140 万行和 scroll_depth_ex 的 370 万行，我不考虑太多。

谁能告诉我为什么我的代码在我的数据上表现不佳？
另外，有没有一种方法可以完成相同的工作，而不必将连接、分组和汇总分成两个步骤？

【问题讨论】：

您的日期时间存储为字符。转换为日期时间有帮助吗？否则，我认为主要瓶颈是您正在使用时间戳进行非 equi 连接，而 dplyr 本身无法处理。连接+过滤器的解决方法往往会随着数据的平方而增加内存和时间，因此使用处理非 equi 连接的 data.table 或 sqldf 解决方案可能会快得多。
加快速度的一种简单方法是在第二行添加dtplyr::lazy_dt() %>%，最后添加as_tibble()。这会将计算转移到data.table，这比 dplyr 更快地计算分组数据。在我的测试中，它的速度提高了大约 2 倍。但是为了更快，我认为 data.table 或 sqldf 中的非 equi 连接会更有帮助。
感谢 cmets。我将计算转移到data.table，它运行得更快......但是，我最终做的是直接使用sqldf生成组合数据帧。

标签： r dplyr summarize

【解决方案1】：

我认为这是filter() 步骤。尝试将这两个部分分开，将difftime() 放在第二位。

例如：

  ...
  filter(entrance_date <= ad_view_date) %>%
  filter(difftime(ad_view_date, entrance_date, units = "mins") <= 60) %>%
  ...

【讨论】：

不幸的是，完成所需的时间大致相同。结果还可以。