【问题标题】:How to concatenate rows based on group as quickly as possible如何尽快根据组连接行
【发布时间】:2020-08-11 06:15:24
【问题描述】:

我有一个如下的数据框

ClientVisitGUID LineNum TextCol
1                 1      This was a great
1                 2      report I did
2                 3      was performed today
2                 1      Another great report
2                 2      for this person
3                 2      good stuff
3                 1      I really write very
3                 3      when I put my
3                 4      mind to it

我想根据ClientVisitGUID 和行号连接行,以便获得以下输出

ClientVisitGUID TextCol
1               This was a great report I did
2               Another great report for this person was performed today
3               I really write very good stuff when I put my mind to it

我尝试了dplyr,但它需要很长时间并且无法处理我所拥有的数千行

  resultset2<-resultset %>%
    group_by(ClientVisitGUID) %>%
    arrange(LineNum) %>%
    summarize_all(paste, collapse=",")

有没有更快的方法?我对 data.table 不是很熟悉,但这很快吗?

【问题讨论】:

  • 我不是 data.table 用户。我知道它超级快,但它不如 dplyr 友好。您是否尝试过并行化您的工作? (blog.aicry.com/multidplyr-dplyr-meets-parallel-processing/…)。您还可以手动将 tibble 拆分为块并将 group-by 应用于每个块,然后合并。
  • @lmeninato 我确实想到了这一点,但想看看是否有一个简单的 dplyr 替代方案就足够了。我认为我最终不得不并行化。谢谢

标签: r performance dplyr data.table concatenation


【解决方案1】:

第二个data.table 选项,同样使用stringi 来提高性能

library(data.table)
library(stringi)
setDT(df)
setkey(df, ClientVisitGUID, LineNum)
df1 <- df[, .(new = stri_c(TextCol, collapse = " ")), by = ClientVisitGUID]

结果

df1
#   ClientVisitGUID                                                      new
#1:               1                            This was a great report I did
#2:               2 Another great report for this person was performed today
#3:               3  I really write very good stuff when I put my mind to it

数据(感谢@ThomasIsCoding)

df <- structure(list(ClientVisitGUID = c(1L, 1L, 2L, 2L, 2L, 3L, 3L, 
3L, 3L), LineNum = c(1L, 2L, 3L, 1L, 2L, 2L, 1L, 3L, 4L), TextCol = c("This was a great", 
"report I did", "was performed today", "Another great report", 
"for this person", "good stuff", "I really write very", "when I put my", 
"mind to it")), class = "data.frame", row.names = c(NA, -9L))

【讨论】:

  • 如果我有更多列,如何同时连接它们。例如,我还有另一列包含一些文本,名为 Textcol2 。如何扩展您的解决方案以连接此列中的文本?
  • 最后,对于我更大的数据集(此处未显示),我将此解决方案中的列重新合并回原来有效的数据集中选定的唯一行
  • @SebastianZeki 你能用一个显示Textcol2 和所需输出的例子来更新你的帖子吗?
【解决方案2】:

如果你想要速度,data.table 确实是个不错的选择:

library(data.table)

setDT(resultset)
data.table::setkeyv(resultset, "ClientVisitGUID")
resultset <- resultset[order(ClientVisitGUID, LineNum)]
resultset[, .(lapply(.SD, paste, collapse = ",")), by = "ClientVisitGUID"]

一开始设置密钥需要一些时间,但之后您将获得更快的操作。设置键在连续的内存槽中重新排序属于同一组的行

示例

data = data.table("a" = c("aaa","ffff","ttt"), "b" = c(1,1,2))
data[, .(lapply(.SD, paste, collapse = ",")), by = "b"]

【讨论】:

  • 如果您的数据已经订购,那么resultset[order(ClientVisitGUID, LineNum)] 行可能是不必要的。由于我没有看到您的数据,因此我将其包含在以防万一
  • 我收到错误 .(lapply(.SD, paste, collapse = ",")) 中的错误:找不到函数“。”我尝试更新 data.table (1.12.8) 但仍然得到它
  • resultsetdata.table 对象吗? (inherits(resultset, "data.table")TRUE ?)我刚刚在我的电脑上测试了一个愚蠢的例子,它可以工作(添加到答案中)
  • 这是一个data.table和一个data.frame。那可能吗?似乎不能让它成为一个 data.frame 而已。那会是我遇到的错误的根源吗?
  • 那时一切正常。还有data.frame 类不是问题,这很正常。 . 应该是列表的快捷方式。如果您尝试resultset[, list(lapply(.SD, paste, collapse = ",")), by = "ClientVisitGUID"]resultset[, lapply(.SD, paste, collapse = ","), by = "ClientVisitGUID"],您仍然有错误吗? (你的data.table 版本比我的更新)
【解决方案3】:

基本 R 选项使用 aggregate

result <- aggregate(TextCol~ClientVisitGUID,
                    df[order(df$ClientVisitGUID,df$LineNum),],
                    paste0, 
                    collapse = " ")

给了

> result
  ClientVisitGUID                                                  TextCol
1               1                            This was a great report I did
2               2 Another great report for this person was performed today
3               3  I really write very good stuff when I put my mind to it

数据

df <- structure(list(ClientVisitGUID = c(1L, 1L, 2L, 2L, 2L, 3L, 3L, 
3L, 3L), LineNum = c(1L, 2L, 3L, 1L, 2L, 2L, 1L, 3L, 4L), TextCol = c("This was a great", 
"report I did", "was performed today", "Another great report", 
"for this person", "good stuff", "I really write very", "when I put my", 
"mind to it")), class = "data.frame", row.names = c(NA, -9L))

【讨论】:

  • 感谢@ThomasIsCoding,它看起来很快。如果我想通过 ClientVisitID 聚合超过 TextCol - 如果我也想聚合 LineNum 怎么办?
  • @SebastianZeki 是的,您可以在aggregate 中使用类似cbind(TextCol,col1,col2,col3)~ClientVisitGUID 的内容,其中col1,col2,col3 是您要对@ 执行类似操作的列名987654330@
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2012-03-14
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2017-01-24
相关资源
最近更新 更多