在两个时间戳之间左加入 R答案

【问题标题】：Left joining in R between two timestamps在两个时间戳之间左加入 R
【发布时间】：2019-06-20 14:49:27
【问题描述】：

我的目标是在intervals 上执行左连接，其中bike_id 匹配并且records 中的created_at 时间戳在start 和end 之间intervals 表中

> class(records)
[1] "data.table" "data.frame"
> class(intervals)
[1] "data.table" "data.frame"

> records
  bike_id          created_at         resolved_at
1   28780 2019-05-03 08:29:18 2019-05-03 08:35:37
2   28780 2019-05-03 21:05:28 2019-05-03 21:07:28
3   28780 2019-05-04 21:13:39 2019-05-04 21:15:40
4   28780 2019-05-07 17:24:20 2019-05-07 17:26:39
5   28780 2019-05-08 11:34:32 2019-05-08 12:16:44
6   28780 2019-05-08 23:38:39 2019-05-08 23:40:36


> intervals
   bike_id               start                 end id
1:   28780 2019-05-03 04:44:45 2019-05-03 16:58:56  1
2:   28780 2019-05-04 07:07:39 2019-05-04 14:48:29  2
3:   28780 2019-05-07 23:28:32 2019-05-08 12:56:24  3
4:   28780 2019-05-10 06:06:21 2019-05-10 13:12:08  4
5:   28780 2019-05-12 05:21:24 2019-05-12 11:35:52  5
6:   28780 2019-05-13 08:44:54 2019-05-13 12:28:31  6

在这种情况下，输出看起来像

> output
  bike_id          created_at         resolved_at   id
1   28780 2019-05-03 08:29:18 2019-05-03 08:35:37    1
2   28780 2019-05-03 21:05:28 2019-05-03 21:07:28  NULL   
3   28780 2019-05-04 21:13:39 2019-05-04 21:15:40  NULL
4   28780 2019-05-07 17:24:20 2019-05-07 17:26:39  NULL
5   28780 2019-05-08 11:34:32 2019-05-08 12:16:44  NULL
6   28780 2019-05-08 23:38:39 2019-05-08 23:40:36  NULL

我曾尝试使用tidyverse 的解决方案posted here，但这会导致R 内存不足（尽管两个表中的记录量只有100K 左右）

fuzzy_left_join(
 records, intervals,
  by = c(
    "bike_id" = "bike_id",
    "created_at" = "start",
    "created_at" = "end"
    ),
  match_fun = list(`==`, `>=`, `<=`)
  ) %>%
  select(id, bike_id = bike_id.x, created_at, start, end)

这会引发错误：Error: vector memory exhausted (limit reached?)

是否有在data.table 甚至在基础R 中使用merge() 滚动连接的替代方法？什么是通过 id 连接两个数据帧的好方法以及连接表中其他两个数据帧之间的时间戳？

这是数据

dput(intervals)
structure(list(bike_id = c(28780L, 28780L, 28780L, 28780L, 28780L, 
28780L), start = structure(c(1556858685, 1556953659, 1557271712, 
1557468381, 1557638484, 1557737094), class = c("POSIXct", "POSIXt"
), tzone = "UTC"), end = structure(c(1556902736, 1556981309, 
1557320184, 1557493928, 1557660952, 1557750511), class = c("POSIXct", 
"POSIXt"), tzone = "UTC"), id = c(1, 2, 3, 4, 5, 6)), row.names = c(NA, 
-6L), class = c("data.table", "data.frame"), .internal.selfref = <pointer: 0x1030056e0>)

dput(records)
structure(list(bike_id = c(28780L, 28780L, 28780L, 28780L, 28780L, 
28780L), created_at = structure(c(1556872158.796, 1556917528.845, 
1557004419.928, 1557249860.939, 1557315272.396, 1557358719.333
), class = c("POSIXct", "POSIXt"), tzone = "UTC"), resolved_at = structure(c(1556872537.867, 
1556917648.118, 1557004540.056, 1557249999.892, 1557317804.183, 
1557358836.202), class = c("POSIXct", "POSIXt"), tzone = "UTC")), row.names = c(NA, 
6L), class = "data.frame")

【问题讨论】：

你可以试试nonequi join in data.tablesetDT(records)[intervals, on = .(bike_id, created_at >= start, created_at <= end)]

标签： r dplyr data.table tidyverse

【解决方案1】：

我们可以使用data.tablenonequi join

library(data.table)
setDT(records)[intervals, on = .(bike_id, created_at >= start, created_at <= end)]

【讨论】：

【解决方案2】：

我知道 OP 要求提供 tidyverse 或 data.table 解决方案，但 SQL 似乎是解决此问题的完美工具：

library(sqldf)

sqldf("select a.*, b.id 
        from records as a
        left join intervals as b
          on a.bike_id = b.bike_id and
            a.created_at >= b.start and
            a.created_at <= b.end")

或使用between 替代语法：

sqldf("select a.*, b.id 
        from records as a
        left join intervals as b
          on a.bike_id = b.bike_id and
            a.created_at between b.start and b.end")

编辑：如@G 所述。 Grothendieck，我们可以在读取数据之前设置环境的时区（使用Sys.setenv）以匹配OP的时区。

输出：

  bike_id          created_at         resolved_at id
1   28780 2019-05-03 08:29:18 2019-05-03 08:35:37  1
2   28780 2019-05-03 21:05:28 2019-05-03 21:07:28 NA
3   28780 2019-05-04 21:13:39 2019-05-04 21:15:40 NA
4   28780 2019-05-07 17:24:20 2019-05-07 17:26:39 NA
5   28780 2019-05-08 11:34:32 2019-05-08 12:16:44  3
6   28780 2019-05-08 23:38:39 2019-05-08 23:40:36 NA

数据：（OP 的 dput 确实有效，因为指针是从 data.table 创建的）

Sys.setenv(TZ = "GMT")

records <- structure(list(bike_id = c(28780L, 28780L, 28780L, 28780L, 28780L, 
28780L), created_at = c("2019-05-03 08:29:18", "2019-05-03 21:05:28", 
"2019-05-04 21:13:39", "2019-05-07 17:24:20", "2019-05-08 11:34:32", 
"2019-05-08 23:38:39"), resolved_at = c("2019-05-03 08:35:37", 
"2019-05-03 21:07:28", "2019-05-04 21:15:40", "2019-05-07 17:26:39", 
"2019-05-08 12:16:44", "2019-05-08 23:40:36")), class = "data.frame", row.names = c(NA, 
-6L))

intervals <- structure(list(bike_id = c(28780L, 28780L, 28780L, 28780L, 28780L, 
28780L), start = c("2019-05-03 04:44:45", "2019-05-04 07:07:39", 
"2019-05-07 23:28:32", "2019-05-10 06:06:21", "2019-05-12 05:21:24", 
"2019-05-13 08:44:54"), end = c("2019-05-03 16:58:56", "2019-05-04 14:48:29", 
"2019-05-08 12:56:24", "2019-05-10 13:12:08", "2019-05-12 11:35:52", 
"2019-05-13 12:28:31"), id = c(1, 2, 3, 4, 5, 6)), class = "data.frame", row.names = c(NA, 
-6L))

【讨论】：

我收到以下错误：Error in postgresqlNewConnection(drv, ...) : RS-DBI driver: (could not connect postgres@localhost:5432 on dbname "test": could not connect to server: Connection refused Is the server running on host "localhost" (::1) and accepting TCP/IP connections on port 5432? could not connect to server: Connection refused Is the server running on host "localhost" (127.0.0.1) and accepting TCP/IP connections on port 5432? ) Error in !dbPreExists : invalid argument type
@the_darkside 在运行我的代码之前，您是否已加载 RPostgreSQL？
是的，我卸载了它，然后设置了options(sqldf.driver = "SQLite")，它起作用了！谢谢
如果您将会话设置为 GMT：Sys.setenv(TZ = "GMT")，则无需转换为字符。也建议a.created_at between b.start and b.end
@G.Grothendieck 好点。我已根据您的建议编辑了我的答案。

【解决方案3】：

另一种方法是加入bike_id 和created_at 的日期部分，然后删除created_at 不在区间start-end 中的ID。这可能会通过将事情分解为单独的步骤来解决内存问题：

library(dplyr)
library(lubridate)
library(purrr)

intervals %>% 
    mutate(date = date(start)) %>% 
    right_join(mutate(records,
                      date = date(created_at)),
                      by = c("bike_id", "date")
              ) %>% 
    mutate(within = created_at %within% interval(start, end),
           within = replace_na(within, F),
           id = map2_dbl(id, within, ~ ifelse(.y, .x, NA))
           ) %>% 
    select(bike_id, id, created_at, resolved_at)

# A tibble: 6 x 4
  bike_id    id created_at          resolved_at        
    <int> <dbl> <dttm>              <dttm>             
1   28780     1 2019-05-03 08:29:18 2019-05-03 08:35:37
2   28780    NA 2019-05-03 21:05:28 2019-05-03 21:07:28
3   28780    NA 2019-05-04 21:13:39 2019-05-04 21:15:40
4   28780    NA 2019-05-07 17:24:20 2019-05-07 17:26:39
5   28780    NA 2019-05-08 11:34:32 2019-05-08 12:16:44
6   28780    NA 2019-05-08 23:38:39 2019-05-08 23:40:36

【讨论】：