【问题标题】:outlier replacement with their mean for multi-layered data in R using dplyr使用 dplyr 将异常值替换为 R 中多层数据的平均值
【发布时间】:2019-05-01 16:22:51
【问题描述】:

我的 df 与不同客户的销售数据一起,但有一些异常值,我想替换异常值(低于平均值 2 SD)(μ ± 2σ)并用他们的每个 customer_id 平均值替换它们。

structure(list(Date = c("6/29/2014", "7/6/2014", "7/13/2014", 
"7/20/2014", "7/27/2014", "8/3/2014", "8/10/2014", "8/17/2014", 
"8/24/2014", "6/29/2014", "7/6/2014", "7/13/2014", "7/20/2014", 
"7/27/2014", "8/3/2014", "8/10/2014", "8/17/2014", "8/24/2014", 
"7/6/2014", "7/13/2014", "7/20/2014", "7/27/2014", "8/3/2014", 
"8/10/2014", "8/17/2014", "8/24/2014"), customer_id = c("9000A", 
"9000A", "9000A", "9000A", "9000A", "9000A", "9000A", "9000A", 
"9000A", "80A09", "80A09", "80A09", "80A09", "80A09", "80A09", 
"80A09", "80A09", "80A09", "Y90BC", "Y90BC", "Y90BC", "Y90BC", 
"Y90BC", "Y90BC", "Y90BC", "Y90BC"), sales = c(20L, 40L, 0L, 
42L, 56L, 90L, 500L, 23L, 60L, 200L, 234L, 500L, 450L, 0L, 900L, 
459L, 347L, 895L, 380L, 390L, 432L, 320L, 400L, 10L, 0L, 1000L
)), class = "data.frame", row.names = c(NA, -26L))

有人可以帮助我使用 dplyr。 注意:所有不在 (μ ± 2σ) 中的“0”值和销售额都需要替换为尊重其 customer_id 的平均值

【问题讨论】:

  • “用他们的 customer_id 替换它们”是什么意思?您要替换哪一列?
  • 到目前为止你尝试了什么?
  • 每个客户都有不同的销售平均值。如果客户 1 的销售值在 (μ ± 2σ) 之外,则需要将该销售值替换为该特定客户 1@otwtm 的平均值跨度>

标签: r dplyr time-series outliers


【解决方案1】:

还有另一种使用 dplyr 的方法 :)

不完全确定您是否希望截止值基于全局平均值或按客户分组,因此有 2 个版本。

编辑:要检查

sales > mean(sales) + 2*sd(sales) | sales < mean(sales) - 2*sd(sales) | sales == 0

代码

# version to check for > global mean + 2 * global sd
# if sales-value > global cutoff sales-value gets replaced by customer mean
test_data2 = 
  test_data %>% group_by(customer_id) %>% 
  mutate(sales = ifelse(sales > mean(test_data$sales) + 2*sd(test_data$sales), mean(sales), sales))

# version to check for mean per customer + 2 * sd per customer
# if sales-value > customer cutoff sales-value gets replaced by customer mean
test_data2 = 
  test_data %>% group_by(customer_id) %>% 
  mutate(sales = ifelse(sales > mean(sales) + 2*sd(sales), mean(sales), sales))



### check if this is what we want

# calc global mean + global sd + cutoff global
mean(test_data$sales)
sd(test_data$sales)
mean(test_data$sales) + 2*sd(test_data$sales)

# calc mean, sd, cutoff for each customer
test_data %>% group_by(customer_id) %>% summarise(mean = mean(sales), sd = sd(sales), cutoff = mean + 2*sd(sales))



test_data$sales2 = test_data2$sales

test_data %>% filter(customer_id == "80A09")
test_data %>% filter(customer_id == "9000A")
test_data %>% filter(customer_id == "Y90BC")

单独的控制代码在两个版本之间不进行推断:

df = structure(list(Date = c("6/29/2014", "7/6/2014", "7/13/2014", 
                                    "7/20/2014", "7/27/2014", "8/3/2014", "8/10/2014", "8/17/2014", 
                                    "8/24/2014", "6/29/2014", "7/6/2014", "7/13/2014", "7/20/2014", 
                                    "7/27/2014", "8/3/2014", "8/10/2014", "8/17/2014", "8/24/2014", 
                                    "7/6/2014", "7/13/2014", "7/20/2014", "7/27/2014", "8/3/2014", 
                                    "8/10/2014", "8/17/2014", "8/24/2014"), customer_id = c("9000A", 
                                                                                            "9000A", "9000A", "9000A", "9000A", "9000A", "9000A", "9000A", 
                                                                                            "9000A", "80A09", "80A09", "80A09", "80A09", "80A09", "80A09", 
                                                                                            "80A09", "80A09", "80A09", "Y90BC", "Y90BC", "Y90BC", "Y90BC", 
                                                                                            "Y90BC", "Y90BC", "Y90BC", "Y90BC"), sales = c(20L, 40L, 0L, 
                                                                                                                                           42L, 56L, 90L, 500L, 23L, 60L, 200L, 234L, 500L, 450L, 0L, 900L, 
                                                                                                                                           459L, 347L, 895L, 380L, 390L, 432L, 320L, 400L, 10L, 0L, 1000L
                                                                                            )), class = "data.frame", row.names = c(NA, -26L))



test_data = df %>% group_by(customer_id) %>% mutate(sales =ifelse( sales > mean(sales) + 2*sd(sales) | sales < mean(sales) - 2*sd(sales) | sales == 0,mean(sales),sales))
test_data$sales_old = df$sales

df %>% group_by(customer_id) %>% summarise(mean = mean(sales), sd = sd(sales), cutoff = mean + 2*sd(sales))


test_data %>% filter(customer_id == "80A09" & sales != sales_old)
test_data %>% filter(customer_id == "9000A" & sales != sales_old)
test_data %>% filter(customer_id == "Y90BC" & sales != sales_old)

输出:

> df %>% group_by(customer_id) %>% summarise(mean = mean(sales), sd = sd(sales), cutoff = mean + 2*sd(sales))
# A tibble: 3 x 4
  customer_id  mean    sd cutoff
  <chr>       <dbl> <dbl>  <dbl>
1 80A09       443.   301.  1045.
2 9000A        92.3  155.   402.
3 Y90BC       366.   310.   986.
> test_data %>% filter(customer_id == "80A09" & sales != sales_old)
# A tibble: 1 x 4
# Groups:   customer_id [1]
  Date      customer_id sales sales_old
  <chr>     <chr>       <dbl>     <int>
1 7/27/2014 80A09        443.         0
> test_data %>% filter(customer_id == "9000A" & sales != sales_old)
# A tibble: 2 x 4
# Groups:   customer_id [1]
  Date      customer_id sales sales_old
  <chr>     <chr>       <dbl>     <int>
1 7/13/2014 9000A        92.3         0
2 8/10/2014 9000A        92.3       500
> test_data %>% filter(customer_id == "Y90BC" & sales != sales_old)
# A tibble: 2 x 4
# Groups:   customer_id [1]
  Date      customer_id sales sales_old
  <chr>     <chr>       <dbl>     <int>
1 8/17/2014 Y90BC        366.         0
2 8/24/2014 Y90BC        366.      1000

【讨论】:

  • test_data = df %&gt;% group_by(customer_id) %&gt;% mutate(sales =ifelse( sales &gt; mean(sales) + 2*sd(sales) | sales &lt; mean(sales) - 2*sd(sales) | sales == 0,mean(sales),sales)) 我试过了,但它被替换为全局平均值而不是客户平均值。
  • 嗯,奇怪。你能再试一次吗,因为对我来说它正在按预期工作。也许与其他版本的一些推论?
猜你喜欢
  • 1970-01-01
  • 2015-01-02
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多