如何对包含 AND 和 OR 运算符的语句执行条件连接？答案

【问题标题】：How to perform a conditional join on statement containing both AND and OR operators?如何对包含 AND 和 OR 运算符的语句执行条件连接？
【发布时间】：2022-10-12 21:39:32
【问题描述】：

我有一个大数据集，想形成满足某些条件的所有行对，然后根据满足条件的哪些部分计算一些变量。以下 MWE 说明了我想要实现的目标：

library(data.table)
set.seed(1234)

IDs <- data.table(id = letters[1:10],
                  c1 = sample(1:5, 10, replace = T),
                  c2 = sample(1:5, 10, replace = T),
                  c3 = sample(1:5, 10, replace = T),
                  c = 1)

IDs.joined <- IDs[IDs, on = 'c', allow.cartesian = T
                  ][c1 != i.c1 & (c2 == i.c2 | c3 == i.c3)  # condition defining which pairs are joined
                  ][, c('Ic2', 'Ic3') := .(c2 == i.c2, c3 == i.c3)
                  ][, overlap_id := fifelse(Ic2 == 1, 2, 3)
                  ][, overlap := Ic2 + Ic3
                  ][, -c('i.c1', 'i.c2', 'i.c3', 'Ic2', 'Ic3')]

问题是完整的数据集太大（约 500 万行）无法在自身上形成笛卡尔连接。我的问题是，有没有办法使用data.table 的语法直接执行这样的条件连接，而不是先通过笛卡尔连接，然后再施加所需的条件？

我在 SO 上看到过类似的问题，但这些问题通常可以表示为滚动连接，我不知道在滚动连接语法或 X != Y 条件中包含 X | Y 语句的方法。

【问题讨论】：

标签： r data.table self-join

【解决方案1】：

到目前为止，对于像这些相对简单的条件，我发现的最佳选择是绑定多个连接。它并不漂亮，但它速度快且内存效率高。

library(data.table)
set.seed(1234)

IDs <- data.table(id = 1:1e4,
                  c1 = sample(5e3, 1e4, replace = T),
                  c2 = sample(5e3, 1e4, replace = T),
                  c3 = sample(5e3, 1e4, replace = T),
                  c = 0L)

f1 <- function(dt) {
  dt[
    dt, on = 'c', allow.cartesian = TRUE
  ][
    c1 != i.c1 & (c2 == i.c2 | c3 == i.c3)
  ]
}

f2 <- function(dt) {
  unique(
    rbindlist(
      list(
        dt[dt, on = .(c1 > c1, c2 == c2), .(id = x.id, c1 = x.c1, c2 = x.c2, c3 = x.c3, c = x.c, i.id = i.id, i.c1 = i.c1, i.c2 = i.c2, i.c3 = i.c3), nomatch = 0],
        dt[dt, on = .(c1 < c1, c2 == c2), .(id = x.id, c1 = x.c1, c2 = x.c2, c3 = x.c3, c = x.c, i.id = i.id, i.c1 = i.c1, i.c2 = i.c2, i.c3 = i.c3), nomatch = 0],
        dt[dt, on = .(c1 > c1, c3 == c3), .(id = x.id, c1 = x.c1, c2 = x.c2, c3 = x.c3, c = x.c, i.id = i.id, i.c1 = i.c1, i.c2 = i.c2, i.c3 = i.c3), nomatch = 0],
        dt[dt, on = .(c1 < c1, c3 == c3), .(id = x.id, c1 = x.c1, c2 = x.c2, c3 = x.c3, c = x.c, i.id = i.id, i.c1 = i.c1, i.c2 = i.c2, i.c3 = i.c3), nomatch = 0]
      )
    )
  )
}

microbenchmark::microbenchmark(f1(IDs),
                               f2(IDs),
                               times = 10)
#> Unit: milliseconds
#>     expr       min        lq      mean    median        uq       max neval
#>  f1(IDs) 2553.3594 3305.0062 3256.9072 3343.6174 3396.6990 3470.7870    10
#>  f2(IDs)  375.0594  400.9712  428.4382  440.4604  449.4586  490.7598    10

identical(setorder(f1(IDs), id, i.id), setorder(f2(IDs), id, i.id))
#> [1] TRUE

【讨论】：