由于生产数据集包含 14 M 行，因此 OP 已请求任何可以相对较快地工作的建议。
不幸的是，PKumar 接受的答案在速度和内存消耗方面相当低效：
- 它创建了许多辅助向量，这些向量作为新列添加到
df，因此被存储两次
-
df 的每次更新都会复制整个对象
-
data.table 解决方案不使用data.table 语法来避免复制操作
-
POSIXlt 需要 52 个字节来存储一个日期时间实例，而 POSIXct 只需要 8 个字节
这是我使用data.table的建议：
# create sample data, see function definition below
df <- create_sample_data(n_id = 4L, n_hr = 24L * 2L)
str(df)
'data.frame': 192 obs. of 3 variables:
$ ID : chr "000001" "000001" "000001" "000001" ...
$ datetime: POSIXct, format: "2013-01-08 00:00:00" "2013-01-08 01:00:00" "2013-01-08 02:00:00" ...
$ measurem: num 1.207 0.277 1.084 2.346 0.429 ...
library(data.table)
# daily aggregates
setDT(df)[, .(sum_measurem = sum(measurem)),
by = .(ID, date = as.IDate(datetime))]
ID date sum_measurem
1: 000001 2013-01-08 18.01187
2: 000001 2013-01-09 22.53423
3: 000002 2013-01-08 21.77239
4: 000002 2013-01-09 15.57561
5: 000003 2013-01-08 14.79938
6: 000003 2013-01-09 20.09797
7: 000004 2013-01-08 15.21066
8: 000004 2013-01-09 25.47120
# bi-daily aggregates
setDT(df)[, .(sum_measurem = sum(measurem)),
by = .(ID, date = as.IDate(datetime), AM = hour(datetime) <= 12L)]
ID date AM sum_measurem
1: 000001 2013-01-08 TRUE 10.677509
2: 000001 2013-01-08 FALSE 7.334362
3: 000001 2013-01-09 TRUE 12.456765
4: 000001 2013-01-09 FALSE 10.077470
5: 000002 2013-01-08 TRUE 12.099480
6: 000002 2013-01-08 FALSE 9.672908
7: 000002 2013-01-09 TRUE 8.672189
8: 000002 2013-01-09 FALSE 6.903426
9: 000003 2013-01-08 TRUE 8.976965
10: 000003 2013-01-08 FALSE 5.822411
11: 000003 2013-01-09 TRUE 11.131718
12: 000003 2013-01-09 FALSE 8.966252
13: 000004 2013-01-08 TRUE 8.413315
14: 000004 2013-01-08 FALSE 6.797342
15: 000004 2013-01-09 TRUE 15.111185
16: 000004 2013-01-09 FALSE 10.360017
数据
create_sample_data <- function(n_id, n_hr) {
set.seed(1234L)
data.frame(
ID = rep(sprintf("%06i", seq_len(n_id)), each = n_hr),
datetime = rep(seq(as.POSIXct("2013-01-08"), length.out = n_hr, by = "1 hour"), n_id),
measurem = abs(rnorm(n_id * n_hr)),
stringsAsFactors = FALSE
)
}
基准测试
为了进行基准测试，我们为 100 个唯一的IDs 和 365 天的每小时数据创建了样本数据，每个数据产生了一个包含 876 K 行的样本数据集。当一些解决方案修改数据集时，copy() 用于为每次运行提供不受干扰的数据集。 copy() 也是定时的。
df0 <- create_sample_data(n_id = 100L, n_hr = 24L * 365L)
microbenchmark::microbenchmark(
copy = df <- copy(df0),
uwe_dt = {
df <- copy(df0)
setDT(df)[, .(sum_measurem = sum(measurem)),
by = .(ID, date = as.IDate(datetime), AM = hour(datetime) < 12L)]
},
PKumar_dt = {
df <- copy(df0)
datetime <- as.POSIXlt(df$datetime)
date <- as.Date(datetime)
ind <- ifelse(datetime$hour >= 12,"PM","AM")
df$ind <- ind
df$date <- date
dt <- setDT(df)
dt[,list(sum_measure = sum(measurem)),by=list(ID,date,ind)]
},
PKumar_baseR = {
df <- copy(df0)
datetime <- as.POSIXlt(df$datetime)
date <- as.Date(datetime)
ind <- ifelse(datetime$hour >= 12,"PM","AM")
df$ind <- ind
df$date <- date
fin <- aggregate(measurem ~ ID + date + ind, data = df, sum)
fin[order(fin$ID),]
},
times = 11L
)
Unit: milliseconds
expr min lq mean median uq max neval
copy 3.94761 4.391457 5.169909 5.537982 5.864401 5.997876 11
uwe_dt 271.89460 301.001006 339.913084 312.151541 344.251971 540.018306 11
PKumar_dt 417.57141 464.778485 575.547756 475.562955 689.848696 851.180584 11
PKumar_baseR 6356.93567 6707.847607 6896.174857 6863.069477 6903.442520 8112.316770 11
即使有这个中等规模的问题，基本的 R 解决方案也比 data.table 版本慢一个数量级。 Pkumar 的data.table 解决方案中的低效数据操作增加了 50% 的性能损失。此外，不必要地分配了 56 MB 的额外内存，而 df onyl 需要 17 MB。