规范化和控制极端数据集中的缺失数据

Question

我有以下数据集

structure(list(q1 = c(5, 40, 200, 100, 100, 3, 200, 10, 10, 50, 
50, 20, 600, 20, 15, 20, 80, 50, 0, 0, 45, 40, 20, 100, 20, 100, 
3, 30, 10, 3, 20, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0, 100, 0, 5, 5, 
5, 2, 0, 0, 0, 0, 0, 100, 0, 0, 0, 10, 5, 0, 50), q2 = c(5, 40, 
200, 80, 100, 2, 100, 11, 10, 5, 50, 60, 600, 10, 10, 30, 50, 
0, 0, 0, 45, 30, 10, 20, 20, 20, 5, 30, 30, 3, 20, 0, 20, 0, 
0, 0, 20, 0, 5, 2, 60, 0, 40, 10, 5, 0, 0, 0, 0, 5, 0, 0, 0, 
0, 0, 0, 10, 20, 0, 0), q3 = c(2, 70, 400, 160, 350, 100, 500, 
20, 100, 500, 300, 20, 1000, 20, 20, 200, 80, 100, 70, 50, 0, 
20, 40, 0, 0, 200, 5, 0, 100, 3, 50, 60, 0, 0, 0, 20, 100, 30, 
40, 50, 50, 1000, 60, 0, 10, 160, 20, 40, 40, 200, 20, 20, 15, 
150, 10, 15, 10, 100, 0, 10), q4 = c(50, 30, 300, 160, 300, 100, 
500, 20, 100, 25, 200, 30, 600, 20, 0, 0, 50, 20, 200, 50, 50, 
20, 30, 0, 0, 50, 3, 20, 60, 3, 0, 60, 0, 0, 0, 15, 100, 30, 
30, 20, 100, 1000, 30, 10, 10, 50, 3, 20, 0, 100, 15, 20, 1510, 
0, 10, 20, 0, 50, 0, 0), q5 = c(20, 50, 200, 40, 100, 100, 100, 
15, 20, 50, 50, 50, 1000, 20, 15, 30, 50, 30, 15, 15, 25, 20, 
20, 20, 20, 150, 3, 50, 30, 10, 30, 30, 50, 20, 20, 15, 20, 30, 
8, 20, 100, 500, 30, 10, 30, 20, 3, 20, 20, 15, 30, 0, 45, 20, 
0, 15, 30, 40, 20, 15), q6 = c(0, 70, 100, 160, 100, 100, 50, 
15, 10, 25, 1000, 50, 1000, 20, 0, 0, 80, 0, 0, 0, 35, 30, 10, 
20, 20, 100, 3, 10, 60, 10, 0, 100, 30, 50, 100, 15, 30, 30, 
17, 5, 30, 1000, 80, 20, 30, 80, 40, 80, 20, 20, 40, 30, 30, 
0, 0, 20, 10, 40, 20, 50), q7 = c(5, 50, 200, 100, 100, 5, 20, 
10, 0, 300, 50, 20, 300, 20, 0, 200, 80, 10, 15, 0, 30, 20, 40, 
20, 20, 100, 3, 15, 50, 15, 80, 20, 0, 30, 0, 15, 20, 30, 10, 
20, 30, 100, 70, 20, 3, 20, 30, 40, 30, 10, 15, 0, 30, 30, 0, 
5, 50, 30, 0, 30), q8 = c(0, 30, 50, 100, 20, 5, 5, 8, 10, 5, 
30, 20, 100, 20, 0, 0, 50, 20, 0, 0, 35, 20, 20, 0, 30, 20, 5, 
6, 30, 15, 10, 10, 30, 0, 0, 0, 20, 30, 6, 5, 50, 100, 10, 10, 
5, 35, 20, 80, 20, 20, 15, 0, 15, 0, 0, 5, 10, 40, 0, 15), q9 = c(20, 
40, 0, 180, 0, 0, 0, 1, 20, 500, 100, 20, 1000, 0, 20, 0, 80, 
50, 0, 15, 45, 20, 20, 0, 20, 200, 3, 80, 50, 15, 30, 30, 30, 
0, 20, 0, 50, 0, 45, 200, 0, 0, 5, 20, 10, 180, 50, 90, 20, 50, 
20, 0, 15, 0, 0, 30, 50, 40, 0, 30), q10 = c(10, 70, 0, 200, 
0, 0, 10, 1, 15, 15, 100, 20, 1000, 0, 0, 0, 80, 30, 0, 10, 30, 
30, 10, 0, 15, 20, 5, 30, 40, 15, 10, 30, 100, 0, 0, 5, 50, 30, 
20, 15, 30, 0, 5, 10, 10, 90, 25, 90, 15, 25, 20, 0, 15, 0, 0, 
35, 10, 20, 0, 15), q11 = c(20, 60, 200, 120, 100, 9, 100, 15, 
25, 150, 100, 30, 100, 20, 15, 50, 80, 50, 20, 15, 30, 20, 30, 
20, 15, 150, 10, 20, 50, 10, 35, 20, 50, 20, 0, 20, 0, 30, 35, 
20, 80, 100, 60, 20, 50, 20, 60, 20, 50, 25, 35, 0, 30, 0, 0, 
30, 30, 40, 20, 20), q12 = c(20, 50, 200, 120, 100, 3, 50, 12, 
10, 15, 50, 30, 100, 20, 0, 30, 60, 0, 0, 5, 25, 30, 10, 20, 
10, 1000, 5, 0, 60, 10, 20, 0, 5, 25, 0, 15, 0, 30, 31, 2, 35, 
1000, 10, 10, 15, 20, 25, 80, 50, 20, 35, 0, 20, 0, 0, 10, 20, 
30, 0, 15), q13 = c(200, 80, 0, 200, 25, 200, 10, 20, 50, 15, 
1000, 70, 1000, 50, 0, 0, 80, 40, 30, 0, 100, 30, 20, 20, 40, 
100, 5, 50, 100, 20, 0, 30, 30, 0, 50, 10, 30, 30, 45, 10, 120, 
1000, 50, 202, 100, 200, 15, 120, 25, 20, 35, 0, 45, 0, 50, 50, 
50, 30, 0, 30), q14 = c(0, 50, 200, 200, 0, 5, 100, 5, 20, 300, 
300, 40, 1000, 10020, 20, 0, 80, 30, 0, 15, 50, 50, 20, 0, 40, 
300, 3, 20, 100, 5, 0, 50, 100, 0, 0, 0, 30, 100, 20, 100, 40, 
100, 5, 10, 10, 10, 50, 120, 0, 50, 15, 50, 50, 0, 50, 15, 100, 
40, 0, 50), q15 = c(50, 40, 50, 150, 100, 30, 0, 8, 25, 100, 
100, 100, 0, 100, 0, 0, 50, 10, 0, 50, 150, 1000, 10, 0, 120, 
0, 5, 100, 20, 10, 10, 0, 100, 0, 0, 5, 100, 30, 45, 200, 100, 
200, 20, 5, 0, 0, 50, 100, 50, 100, 10, 0, 0, 0, 50, 30, 100, 
50, 0, 50), q16 = c(50, 50, 200, 100, 200, 15, 200, 15, 50, 500, 
150, 50, 1000, 20, 0, 100, 100, 30, 0, 50, 60, 30, 50, 100, 100, 
100, 10, 100, 100, 15, 200, 50, 30, 0, 0, 15, 30, 30, 5, 50, 
15, 1000, 5, 20, 100, 0, 80, 20, 0, 300, 20, 0, 100, 0, 0, 20, 
100, 100, 0, 200), q17 = c(0, 30, 100, 140, 100, 5, 100, 15, 
15, 15, 100, 60, 1000, 50, 0, 0, 50, 0, 0, 0, 60, 20, 10, 0, 
40, 100, 5, 30, 60, 15, 10, 30, 0, 0, 20, 15, 20, 30, 10, 10, 
50, 1000, 30, 10, 20, 30, 0, 80, 0, 50, 15, 0, 30, 0, 0, 15, 
10, 60, 0, 50), q18 = c(0, 60, 0, 80, 20, 5, 0, 5, 25, 500, 250, 
70, 800, 0, 20, 100, 100, 100, 50, 50, 70, 30, 50, 0, 50, 300, 
5, 100, 50, 15, 20, 50, 30, 0, 0, 0, 50, 0, 90, 100, 50, 100, 
0, 10, 1000, 0, 20, 80, 5, 100, 20, 0, 0, 0, 0, 30, 0, 100, 0, 
0), q19 = c(0, 30, 0, 80, 0, 5, 0, 15, 25, 15, 100, 60, 800, 
50, 0, 0, 80, 0, 0, 0, 45, 20, 10, 0, 20, 500, 5, 30, 60, 15, 
50, 50, 0, 0, 50, 0, 20, 0, 20, 15, 0, 0, 0, 10, 75, 100, 10, 
80, 5, 30, 20, 0, 15, 0, 0, 20, 0, 50, 10, 0), q20 = c(100, 60, 
200, 150, 200, 30, 200, 100, 50, 1500, 100, 40, 400, 5020, 35, 
150, 80, 100, 100, 50, 70, 30, 40, 100, 50, 200, 20, 0, 50, 10, 
100, 30, 0, 60, 30, 50, 20, 30, 63, 40, 100, 100, 0, 20, 50, 
200, 50, 50, 30, 50, 30, 0, 45, 35, 30, 45, 50, 50, 30, 40), 
    q21 = c(100, 30, 200, 150, 100, 40, 100, 10, 20, 15, 100, 
    30, 400, 20, 10, 0, 60, 0, 0, 0, 10, 20, 10, 20, 15, 20, 
    5, 30, 50, 10, 10, 20, 0, 0, 0, 15, 20, 30, 15, 10, 30, 100, 
    0, 10, 15, 0, 30, 120, 10, 10, 35, 0, 2525, 35, 50, 40, 10, 
    30, 20, 15), q22 = c(100, 70, 100, 150, 100, 5, 100, 5, 25, 
    250, 100, 50, 1000, 20, 15, 70, 80, 100, 10, 20, 30, 30, 
    20, 50, 50, 200, 10, 40, 40, 15, 100, 20, 50, 60, 20, 15, 
    30, 30, 10, 30, 100, 100, 25, 20, 10, 100, 80, 50, 25, 20, 
    35, 0, 30, 20, 0, 20, 50, 50, 0, 50), q23 = c(10, 40, 100, 
    150, 100, 3, 10, 10, 20, 4, 100, 60, 700, 20, 0, 0, 60, 0, 
    0, 0, 20, 20, 10, 20, 40, 20, 5, 2, 60, 15, 10, 20, 5, 0, 
    20, 0, 30, 30, 10, 2, 1010, 0, 10, 1010, 10, 10, 5, 80, 3, 
    20, 20, 0, 25, 0, 0, 20, 10, 30, 0, 15)), row.names = c(NA, 
-60L), class = "data.frame")

edit*: 0 没有丢失数据，因为它是 $ 中的值

当以图形方式查看时，它看起来远非理想

boxplot(as.matrix(example))
plot(density(as.matrix(example)))

我想通过异常值的转换和控制来规范化这些数据，所以我有两个问题：

问题 1

您将如何处理此数据集中的异常值。我不想丢失数据，所以我想替换它们，但是我不清楚使用哪种方法。在这个问题上，是否有任何包可以帮助我自动化这个？我也想看看使用方法的基本原理

问题 2

控制异常值后，我想将变量转换为正态。为此，我有两个我倾向于使用的包：

library(rcompanion)
a<- transformTukey(as.matrix(example))

和

library(LambertW)
b<-Gaussianize(example, type = "h")

但是我不太确定它们在数学上是如何工作的，以及如何评估它们是否做得很好，哪个更好，或者是否有其他更实用的解决方案。

Answer 1

目前尚不完全清楚您要如何处理数据，但我几乎会从简单的事情开始，然后从那里开始。例如， hist() 是什么样的以及所有正态分布（确保您可以在网上找到更好的地方）检查。我认为我在异常值中总是选择的是简单的lm()，它将包含异常值的图表以及如果您浏览这些图表，“截止”将在哪里。通常，数据类型也会让您对规范化方法有一些了解，但一般来说，log norm 通常是一个不错的默认选择