【问题标题】:R Joining unequal lists into a single dataframeR将不相等的列表加入单个数据框
【发布时间】:2016-06-27 08:51:16
【问题描述】:

我已将以下代码(基于此post)应用于我的示例data,以生成三个不同的列表,我试图将它们合并到一个数据框中。

idNodes <- getNodeSet(plans, "//person[@id]") ids <- lapply(idNodes, function(x) xmlAttrs(x)['id']) attribact <- lapply(idNodes, xpathApply, path = "./plan[@selected='yes']//act", xmlAttrs) attribleg <- lapply(idNodes, xpathApply, path = "./plan[@selected='yes']//leg", xmlAttrs)

为了生成数据框,我尝试使用x &lt;- do.call(rbind.data.frame, mapply(cbind, ids, attribact, attribleg)),但它给了我以下错误:

(函数 (..., deparse.level = 1, make.row.names = TRUE) 中的错误: 参数的列数不匹配另外:有 50 个或更多警告(使用 warnings() 查看前 50 个)

我还想指出,上面的do.call 命令适用于小样本数据(带有警告),但不适用于大样本。

期望的输出

id        type   link   x              y              start_time end_time   mode  dep_time   trav_time arr_time
10000061  home   21258  334867.243653  3126570.70778  03:00:00   15:07:00   ride  15:07:00   00:03:28  15:10:28 
10000061  shop   13904  332634.86999   3127078.96383  15:12:00   16:21:00   car   16:21:00   00:09:02  16:30:02 
10000061  shop   14129  331666.364904  3129306.48785  16:25:00   17:37:00   ride  17:37:00   00:10:33  17:47:33 
10000061  home   21258  334867.243653  3126570.70778  17:45:00   26:59:00   NA    NA         NA        NA
10000302  home   21256  334598.361546  3126269.05167  03:00:00   07:56:00   car   07:56:00   00:03:31  07:59:31 
10000302  work   14057  335957.065395  3128105.16619  08:04:00   10:28:00   car   10:28:00   00:06:47  10:34:47 
10000302  social 21191  333032.807855  3128759.66141  10:33:00   11:52:00   car   11:52:00   00:07:50  11:59:50 
10000302  home   21256  334598.361546  3126269.05167  11:59:00   12:11:00   car   12:11:00   00:04:49  12:15:49 
10000302  social 13906  332302.159169  3127536.46778  12:17:00   13:30:00   car   13:30:00   00:05:30  13:35:30 
10000302  home   21256  334598.361546  3126269.05167  13:36:00   26:59:00   NA    NA         NA        NA

样本数据

> dput(head(ids,2))
list(structure("10000061", .Names = "id"), structure("10000302", .Names = "id"))

> dput(head(attribact,2))
list(list(structure(c("home", "21258", "334867.243653", "3126570.70778", "03:00:00", "15:07:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("shop", "13904", "332634.86999", "3127078.96383", "15:12:00", "16:21:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("shop", "14129", "331666.364904", "3129306.48785", "16:25:00", "17:37:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("home", "21258", "334867.243653", "3126570.70778", "17:45:00", "26:59:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time"))), list(structure(c("home", "21256", "334598.361546", "3126269.05167", "03:00:00", "07:56:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("work", "14057", "335957.065395", "3128105.16619", "08:04:00", "10:28:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("social", "21191", "333032.807855", "3128759.66141", "10:33:00", "11:52:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("home", "21256", "334598.361546", "3126269.05167", "11:59:00", "12:11:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("social", "13906", "332302.159169", "3127536.46778", "12:17:00", "13:30:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time")), structure(c("home", "21256", "334598.361546", "3126269.05167", "13:36:00", "26:59:00"), .Names = c("type", "link", "x", "y", "start_time", "end_time"))))

> dput(head(attribleg,2))
list(list(structure(c("ride", "15:07:00", "00:03:28", "15:10:28"), .Names = c("mode", "dep_time", "trav_time", "arr_time")), structure(c("car", "16:21:00", "00:09:02", "16:30:02"), .Names = c("mode", "dep_time", "trav_time", "arr_time")), structure(c("ride", "17:37:00", "00:10:33", "17:47:33"), .Names = c("mode", "dep_time", "trav_time", "arr_time"))), list(structure(c("car", "07:56:00", "00:03:31", "07:59:31"), .Names = c("mode", "dep_time", "trav_time", "arr_time")), structure(c("car", "10:28:00", "00:06:47", "10:34:47"), .Names = c("mode", "dep_time", "trav_time", "arr_time")), structure(c("car", "11:52:00", "00:07:50", "11:59:50"), .Names = c("mode", "dep_time", "trav_time", "arr_time")), structure(c("car", "12:11:00", "00:04:49", "12:15:49"), .Names = c("mode", "dep_time", "trav_time", "arr_time")), structure(c("car", "13:30:00", "00:05:30", "13:35:30"), .Names = c("mode", "dep_time", "trav_time", "arr_time"))))

更新:

我尝试了以下解决方案。但是,就我的目的而言,它非常慢(尽管有预分配)。非常感谢任何提高效率的建议。

library(data.table)
df <- data.table(id=rep(0,10*length(ids)), type=rep("c",10*length(ids)), link=rep(0,10*length(ids)), x=rep(0,10*length(ids)), y=rep(0,10*length(ids)), start_time=rep("c",10*length(ids)), end_time=rep("c",10*length(ids)), mode=rep("c",10*length(ids)), dep_time=rep("c",10*length(ids)), trav_time=rep("c",10*length(ids)), arr_time=rep("c",10*length(ids)))
m <- 1
for (i in 1:length(ids))
{
  for(k in 1: length(attribact[[i]]))
  {
    df[m,id := ids[[i]]]
    df[m,type := attribact[[i]][[k]][[1]]]
    df[m,link := attribact[[i]][[k]][[2]]]
    df[m,x := attribact[[i]][[k]][[3]]]
    df[m,y := attribact[[i]][[k]][[4]]]
    df[m,start_time := attribact[[i]][[k]][[5]]]
    df[m,end_time := attribact[[i]][[k]][[6]]]
    df[m,mode := ifelse(length(attribleg[[i]])>=k, attribleg[[i]][[k]][[1]], NA)]
    df[m,dep_time := ifelse(length(attribleg[[i]])>=k, attribleg[[i]][[k]][[2]], NA)]
    df[m,trav_time := ifelse(length(attribleg[[i]])>=k, attribleg[[i]][[k]][[3]], NA)]
    df[m,arr_time := ifelse(length(attribleg[[i]])>=k, attribleg[[i]][[k]][[4]], NA)]
    m <- m+1
  }
}

【问题讨论】:

  • 发布了 dput 输出。

标签: xml r list data.table


【解决方案1】:

考虑到三个列表为“a”、“b”和“c”,这可能是一个选项

首先将列表'a'中的id分配为列表'b'和'c'的名称,然后rbind列表'b'和'c'中的每个元素,如下所示

names(b) = unlist(a) 
names(c) = unlist(a)

list1 = lapply(b, function(x) do.call(rbind, x)) # rbind list elements
list2 = lapply(c, function(x) do.call(rbind, x))

接下来cbindlist1 和list2 元素考虑list1 中列表元素的长度,最后使用rbind 将新的列表元素放在一起

out = do.call(rbind, 
      lapply(names(list1), 
        function(x){ 
          cbind(id = x, 
             data.frame(list1[[x]]), 
             data.frame(list2[[x]])[1:nrow(list1[[x]]),])
      }))


#> out
#         id   type  link             x             y start_time end_time mode
#1   10000061   home 21258 334867.243653 3126570.70778   03:00:00 15:07:00 ride
#2   10000061   shop 13904  332634.86999 3127078.96383   15:12:00 16:21:00  car
#3   10000061   shop 14129 331666.364904 3129306.48785   16:25:00 17:37:00 ride
#NA  10000061   home 21258 334867.243653 3126570.70778   17:45:00 26:59:00 <NA>
#11  10000302   home 21256 334598.361546 3126269.05167   03:00:00 07:56:00  car
#21  10000302   work 14057 335957.065395 3128105.16619   08:04:00 10:28:00  car
#31  10000302 social 21191 333032.807855 3128759.66141   10:33:00 11:52:00  car
#4   10000302   home 21256 334598.361546 3126269.05167   11:59:00 12:11:00  car
#5   10000302 social 13906 332302.159169 3127536.46778   12:17:00 13:30:00  car
#NA1 10000302   home 21256 334598.361546 3126269.05167   13:36:00 26:59:00 <NA>
#    dep_time trav_time arr_time
#1   15:07:00  00:03:28 15:10:28
#2   16:21:00  00:09:02 16:30:02
#3   17:37:00  00:10:33 17:47:33
#NA      <NA>      <NA>     <NA>
#11  07:56:00  00:03:31 07:59:31
#21  10:28:00  00:06:47 10:34:47
#31  11:52:00  00:07:50 11:59:50
#4   12:11:00  00:04:49 12:15:49
#5   13:30:00  00:05:30 13:35:30
#NA1     <NA>      <NA>     <NA>

【讨论】:

    【解决方案2】:

    我将使用 /* 将 actleg 标签放在一起,而不是三个单独的列表,并添加 id 作为列表名称。

    a <- lapply(idNodes, xpathApply, path = "./plan[@selected='yes']/*", xmlAttrs)
    names(a) <- sapply(idNodes, xmlGetAttr, "id")
    # combine using ldply
    library(plyr)
    x1 <- lapply(a, ldply, "rbind")
    x <- ldply( x1, "rbind", .id="id")
    

    现在您只需格式化 data.frame 并将 leg 属性向上移动 1 行(如果 leg 始终是 act 的下一个兄弟?)。

    n <- which(is.na(x$type) )
    x[n-1, 8:11] <- x[n,8:11]
    x <- subset(x,!is.na(type))
    rownames(x) <- NULL
    x   
             id   type  link             x             y start_time end_time mode dep_time trav_time arr_time
    1  10000061   home 21258 334867.243653 3126570.70778   03:00:00 15:07:00 ride 15:07:00  00:03:27 15:10:27
    2  10000061   shop 13904  332634.86999 3127078.96383   15:12:00 16:21:00  car 16:21:00  00:09:44 16:30:44
    3  10000061   shop 14129 331666.364904 3129306.48785   16:25:00 17:37:00 ride 17:37:00  00:09:46 17:46:46
    4  10000061   home 21258 334867.243653 3126570.70778   17:45:00 26:59:00 <NA>     <NA>      <NA>     <NA>
    5  10000302   home 21256 334598.361546 3126269.05167   03:00:00 07:56:00  car 07:56:00  00:03:00 07:59:00
    6  10000302   work 14057 335957.065395 3128105.16619   08:04:00 10:28:00  car 10:28:00  00:08:20 10:36:20
    7  10000302 social 21191 333032.807855 3128759.66141   10:33:00 11:52:00  car 11:52:00  00:08:33 12:00:33
    8  10000302   home 21256 334598.361546 3126269.05167   11:59:00 12:11:00  car 12:11:00  00:06:35 12:17:35
    9  10000302 social 13906 332302.159169 3127536.46778   12:17:00 13:30:00  car 13:30:00  00:05:30 13:35:30
    10 10000302   home 21256 334598.361546 3126269.05167   13:36:00 26:59:00 <NA>     <NA>      <NA>     <NA>
    

    另一种选择是跳过 idNodes,也许只是格式化下面的 xmlAttrsToDataFrame 输出。

    x <- XML:::xmlAttrsToDataFrame(plans["//person[@id]|//plan[@selected='yes']/*"])
    

    【讨论】:

    • 谢谢克里斯。您的第二个选项更有帮助(对于问题中的特定示例)。事实上,它的性能好几个数量级(比第一个选项好大约 9.5 倍)。这是我根据您的第二个选项提出的代码。 x &lt;- XML:::xmlAttrsToDataFrame(plans["//person[@id]|//plan[@selected='yes']/*"]) z1 &lt;- data.frame(x$id, lapply(x[,2:7], function (z) shift(z, 1, type='lead')), lapply(x[,8:11], function (z) shift(z, 2, type='lead'))) z1 &lt;- z1[rowSums(is.na(z1)) != ncol(z1),] z1 &lt;- cbind(na.locf(as.numeric(as.character(z1$x.id))), z1[,2:ncol(z1)])
    • 但是,令人惊讶的是,随着数据大小的增加,第二个代码的执行速度非常慢。在大约 114272 人的样本中,x &lt;-XML:::xmlAttrsToDataFrame(plans["//person[@id]|//plan[@selected='yes']/*"]) 非常慢(现在已经运行了 2 个多小时)。第一个版本下的整个代码用时不到 2 小时。您对为什么会这样有什么想法吗?
    • xmlToDataFrame 和相关的是便利函数,因此它们没有针对速度进行优化,并且最适合较小的文件。对于更快的 XML 解析方法的一些建议,也许谷歌 xmlToDataFrame 很慢?
    • 感谢克里斯的回答。
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2015-07-20
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多