如何从因子变量中按水平从大型数据集中提取每小时平均值并估计其他统计数据答案

【问题标题】：How to extract from a large dataset hourly daily means by level from a factor variable and estimate other statistics如何从因子变量中按水平从大型数据集中提取每小时平均值并估计其他统计数据
【发布时间】：2019-11-20 04:20:02
【问题描述】：

数据框df1 总结了人们在特定时间段内上过公共厕所的不同日期时间（假设在“2017-06-01”和“2017-06-30”之间）。 Zone 列指定了放置厕所的区域，这是一个具有两个级别的因素：A（聚会区域）或B（居住区域）。

我在下面展示了我所拥有的一个可重现的示例。此示例仅包含两天以减小示例数据集的大小。为了创建df1，我必须首先创建4个单独的数据框，然后将它们绑定以创建数据框df1（尝试一次创建df1时出现错误）。 df1 有 193 行。

options(digits.secs=3)
day_1_A<- data.frame(Datetime= ymd_hms(c("2017-06-01 00:04:17.986","2017-06-01 00:17:43.456","2017-06-01 00:22:43.456","2017-06-01 00:34:43.456","2017-06-01 00:45:43.456","2017-06-01 01:15:23.275","2017-06-01 01:41:32.609","2017-06-01 02:04:17.986","2017-06-01 02:17:43.456","2017-06-01 03:15:23.275","2017-06-01 03:41:32.609","2017-06-01 04:04:17.986","2017-06-01 04:17:43.456","2017-06-01 05:15:23.275","2017-06-01 05:41:32.609","2017-06-01 06:04:17.986","2017-06-01 06:17:43.456","2017-06-01 07:15:23.275","2017-06-01 07:41:32.609","2017-06-01 08:04:17.986","2017-06-01 08:17:43.456","2017-06-01 09:15:23.275","2017-06-01 09:41:32.609","2017-06-01 10:04:17.986","2017-06-01 10:17:43.456","2017-06-01 11:15:23.275","2017-06-01 11:41:32.609","2017-06-01 12:04:17.986","2017-06-01 12:17:43.456","2017-06-01 13:15:23.275","2017-06-01 13:41:32.609","2017-06-01 14:04:17.986","2017-06-01 14:17:43.456","2017-06-01 15:17:23.275","2017-06-01 15:41:32.609","2017-06-01 16:04:17.986","2017-06-01 16:17:43.456","2017-06-01 17:15:23.275","2017-06-01 17:41:32.609","2017-06-01 18:04:17.986","2017-06-01 18:17:43.456","2017-06-01 19:15:23.275","2017-06-01 19:41:32.609","2017-06-01 20:04:17.986","2017-06-01 20:17:43.456","2017-06-01 21:15:23.275","2017-06-01 21:41:32.609","2017-06-01 22:04:17.986","2017-06-01 22:17:43.456","2017-06-01 23:15:23.275","2017-06-01 23:41:32.609")),
                 ToiletZone = c("A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A"))

day_1_B<- data.frame(Datetime= ymd_hms(c("2017-06-01 00:04:17.986","2017-06-01 00:17:43.456","2017-06-01 01:15:23.275","2017-06-01 01:41:32.609","2017-06-01 02:04:17.986","2017-06-01 02:17:43.456","2017-06-01 03:15:23.275","2017-06-01 03:41:32.609","2017-06-01 04:04:17.986","2017-06-01 04:17:43.456","2017-06-01 05:15:23.275","2017-06-01 05:41:32.609","2017-06-01 06:04:17.986","2017-06-01 06:17:43.456","2017-06-01 07:15:23.275","2017-06-01 07:41:32.609","2017-06-01 08:04:17.986","2017-06-01 08:17:43.456","2017-06-01 09:15:23.275","2017-06-01 09:41:32.609","2017-06-01 10:04:17.986","2017-06-01 10:17:43.456","2017-06-01 11:15:23.275","2017-06-01 11:41:32.609","2017-06-01 12:04:17.986","2017-06-01 12:17:43.456","2017-06-01 13:15:23.275","2017-06-01 13:41:32.609","2017-06-01 14:04:17.986","2017-06-01 14:17:43.456","2017-06-01 15:15:23.275","2017-06-01 15:41:32.609","2017-06-01 16:04:17.986","2017-06-01 16:17:43.456","2017-06-01 17:15:23.275","2017-06-01 17:41:32.609","2017-06-01 18:04:17.986","2017-06-01 18:17:43.456","2017-06-01 19:15:23.275","2017-06-01 19:41:32.609","2017-06-01 20:04:17.986","2017-06-01 20:17:43.456","2017-06-01 21:15:23.275","2017-06-01 21:41:32.609","2017-06-01 22:04:17.986","2017-06-01 22:17:43.456","2017-06-01 23:15:23.275","2017-06-01 23:41:32.609")),
                 ToiletZone = c("B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B"))

day_2_A<- data.frame(Datetime= ymd_hms(c("2017-06-02 00:17:43.456","2017-06-02 00:48:43.456","2017-06-02 01:15:23.275","2017-06-02 01:52:23.275","2017-06-02 02:04:17.986","2017-06-02 02:17:43.456","2017-06-02 03:15:23.275","2017-06-02 03:41:32.609","2017-06-02 04:04:17.986","2017-06-02 04:17:43.456","2017-06-02 05:15:23.275","2017-06-02 05:41:32.609","2017-06-02 06:04:17.986","2017-06-02 06:17:43.456","2017-06-02 07:15:23.275","2017-06-02 07:41:32.609","2017-06-02 08:04:17.986","2017-06-02 08:17:43.456","2017-06-02 09:15:23.275","2017-06-02 09:41:32.609","2017-06-02 10:04:17.986","2017-06-02 10:17:43.456","2017-06-02 11:15:23.275","2017-06-02 11:41:32.609","2017-06-02 12:04:17.986","2017-06-02 12:17:43.456","2017-06-02 13:15:23.275","2017-06-02 13:41:32.609","2017-06-02 14:04:17.986","2017-06-02 14:17:43.456","2017-06-02 15:15:23.275","2017-06-02 15:41:32.609","2017-06-02 16:04:17.986","2017-06-02 16:17:43.456","2017-06-02 17:15:23.275","2017-06-02 17:41:32.609","2017-06-02 18:04:17.986","2017-06-02 18:17:43.456","2017-06-02 19:15:23.275","2017-06-02 19:41:32.609","2017-06-02 20:04:17.986","2017-06-02 20:17:43.456","2017-06-02 21:15:23.275","2017-06-02 21:41:32.609","2017-06-02 22:04:17.986","2017-06-02 22:17:43.456","2017-06-02 23:15:23.275","2017-06-02 23:41:32.609")),
                 ToiletZone = c("A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A","A"))

day_2_B<- data.frame(Datetime= ymd_hms(c("2017-06-02 00:04:17.986","2017-06-02 01:15:23.275","2017-06-02 02:04:17.986","2017-06-02 02:17:43.456","2017-06-02 03:15:23.275","2017-06-02 03:41:32.609","2017-06-02 04:04:17.986","2017-06-02 04:17:43.456","2017-06-02 05:15:23.275","2017-06-02 05:41:32.609","2017-06-02 06:04:17.986","2017-06-02 06:17:43.456","2017-06-02 07:15:23.275","2017-06-02 07:41:32.609","2017-06-02 08:04:17.986","2017-06-02 08:17:43.456","2017-06-02 09:15:23.275","2017-06-02 09:41:32.609","2017-06-02 10:04:17.986","2017-06-02 10:17:43.456","2017-06-02 11:15:23.275","2017-06-02 11:41:32.609","2017-06-02 12:04:17.986","2017-06-02 12:17:43.456","2017-06-02 13:15:23.275","2017-06-02 13:41:32.609","2017-06-02 14:04:17.986","2017-06-02 14:17:43.456","2017-06-02 15:15:23.275","2017-06-02 15:41:32.609","2017-06-02 16:04:17.986","2017-06-02 16:17:43.456","2017-06-02 17:15:23.275","2017-06-02 17:41:32.609","2017-06-02 18:04:17.986","2017-06-02 18:17:43.456","2017-06-02 19:15:23.275","2017-06-02 19:41:32.609","2017-06-02 20:04:17.986","2017-06-02 20:17:43.456","2017-06-02 21:15:23.275","2017-06-02 21:41:32.609","2017-06-02 22:04:17.986","2017-06-02 22:17:43.456","2017-06-02 23:15:23.275","2017-06-02 23:41:32.609")),
                 ToiletZone = c("B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B","B"))


df1<- rbind(day_1_A,day_1_B,day_2_A,day_2_B)
df1

> df1
                   Datetime ToiletZone
1   2017-06-01 00:04:17.986          A
2   2017-06-01 00:17:43.455          A
3   2017-06-01 00:22:43.455          A
4   2017-06-01 00:34:43.455          A
5   2017-06-01 00:45:43.455          A
6   2017-06-01 01:15:23.275          A
.               .                    .
.               .                    .
.               .                    .
193 2017-06-02 23:41:32.608          B

由于某些原因，我不会在这里解释，我需要为每个日期和每个区域计算一个名为 θ 的统计数据，它可以定义为“平均每小时访问次数”的除法系数白天的厕所”（Hourly_daily_μ）按“整个感兴趣期间的平均每小时访问次数”（Overall_hourly_μ）。

我在图片中展示了我对上一个示例的期望（列Hourly_daily_μ_A、Hourly_daily_μ_B、Overall_hourly_μ_A 和Overall_hourly_μ_A 被合并以澄清计算。我真正需要的列是@987654339 @ 和 θ_B):

为什么Hourly_daily_μ_A 是 2017-06-01 的 51/24？因为这一天有51个人上厕所。因此，如果我们除以 24，我们就得到了这一天上厕所的人的小时平均值。

为什么Overall_hourly_μ_A 对于不同日期的每个区域都相同？因为它是每个区域的总体平均值。在这里，我们想知道每小时上厕所的人的一般平均值是多少。在这个例子中，我们知道 99 人在 6 月 1 日和 6 月 2 日期间在 A 区上过厕所。所以我们将其除以总小时数（示例中为 48 小时），我们得到每小时的总平均值在 A 区上厕所的人数。这是每个区域的唯一值。

为什么θ_A 在 2017-06-01 上是 (51*48)/(24*99)？因为是Hourly_daily_μ_A (51/24) 除以Overall_hourly_μ_A (99/48) 的结果。

有人知道怎么做吗？我的数据框很大，所以我猜data.table 包可能是一个不错的选择。

【问题讨论】：

抱歉，我得到了一些不同的数字 -
如果我没记错的话，在示例中，6 月 1 日，A 区有 51 人上厕所，B 区有 48 人上厕所。 6 月 2 日，A 区有 48 人上厕所，B 区有 46 人上厕所。加起来是 193 人（示例数据集的行数）。
我尝试了几种方法，df1 %>% group_by(ToiletZone) %>% summarise(n = n()) 给出了 99 和 94 作为你的。接下来，我使用ceiling 将“日期时间”转换为每小时并检查频率
计算很“简单”，但我不知道该怎么做。我想为每一天和区域获得一个统计数据，该统计数据是今天上厕所的每小时平均人数（Hourly_daily_μ）除以每小时上厕所的平均人数数据集中的天数 (Overall_hourly_μ)。
可能是我错过了年份部分df1 %>% group_by(ToiletZone, date = as.Date(Datetime)) %>% summarise(v1 = n_distinct(ceiling_date(Datetime, 'hour')))

标签： r data.table tidyverse lubridate

【解决方案1】：

我认为您只需将日期设置为天单位，然后您就可以将其用于分组。与data.table：

setDT(df1)

df1[, Date := floor_date(Datetime, "day")]

daily <- df1[, .(DailyCount = .N, DailyAvg = .N / 24), by = .(ToiletZone, Date)]
overall <- daily[, .(Total = sum(DailyCount) / (.N * 24)), by = .(ToiletZone)]

overall[daily, .(ToiletZone, Date, Theta = DailyAvg / Total), on = "ToiletZone"]
   ToiletZone       Date     Theta
1:          A 2017-06-01 1.0303030
2:          B 2017-06-01 1.0212766
3:          A 2017-06-02 0.9696970
4:          B 2017-06-02 0.9787234

每小时也会类似，只需更改floor_date 并调整一些分母：

df1[, Date := floor_date(Datetime, "hour")]

hourly <- df1[, .(HourlyCount = .N), by = .(ToiletZone, Date)]
overall <- hourly[, .(Total = sum(HourlyCount) / .N), by = "ToiletZone"]

ans <- overall[hourly, .(ToiletZone, Date, Theta = HourlyCount / Total), on = "ToiletZone"]

顺便说一句，最后几行是一个连接，您可以将它们分别视为左连接， daily 和 hourly 作为左侧表格。

【讨论】：

谢谢亚历克西斯。一个疑问，如果我用数据增加周期，我可以使用相同的脚本吗？我也希望得到同样的结果：每个区域和每天都有一个“zheta”，它将这一天在这个区域上厕所的人的小时平均值（分子）与上厕所的人的“整体小时平均值”分开整个时期都在这个区域。我想我可以更改行数，它仍然可以工作，但以防万一我问你......
@Dekike 是的，我认为它应该可以工作，它不对您数据中的天数做出任何假设。
嗨@Alexis，很抱歉再次打扰您。如果我想按小时而不是按天计算“Theta”，我该怎么办？也就是说，我将每小时上厕所的次数除以整个期间上厕所的每小时平均值。我想我可能会在白天使用它而不是“Theta”......
@Dekike 我已经更新了答案，不适合发表评论。顺便说一句，您问题中的代码有错字，您使用了两次day_2_A。

【解决方案2】：

一个选项是按频率计数分组，做一些计算以获得预期的输出

library(dplyr)
library(tidyr)
library(lubridate)
df1 %>% 
  mutate(Date = floor_date(Datetime, "hour")) %>% 
  group_by(ToiletZone, Date) %>% 
  mutate(hourlyCount = n(), HourlyAvg = hourlyCount/24) %>% 
  group_by(ToiletZone) %>% 
  mutate(Total = sum(hourlyCount)/ n() * 24) %>% 
  group_by(Date = as.Date(Date), add = TRUE) %>% 
  summarise(Theta = hourlyCount[1]/Total[1]) %>%
  spread(ToiletZone, Theta)

【讨论】：

嗨，akrun，我添加了一个示例来尝试阐明我需要什么并了解您的代码是否满足我的需要......
我需要计算每个区域和每天上厕所的平均每小时人数。另一方面，我计算了在整个期间（在本例中为 2017 年 6 月 1 日和 2017 年 6 月 2 日）平均每小时上厕所的人数。最后，我得到了我需要的统计数据，即第一个参数和第二个参数之间的划分。
@Dekike 我得到了 99 部分，但 48 是什么
小时数。对于每日平均值，您除以 24（1 天 = 24 小时），但对于“总体平均值”，您必须除以该期间的总小时数，在这种情况下，因为该期间是两天，48 小时. 99 是两天内上厕所的人数，除以 48 小时，就是整个期间的每小时平均值。
@Dekike 我之前用过ceiling_date，但由于当时的混乱，不得不改变它。