根据 2 列中的值 + 模糊匹配对表进行重复数据删除

Question

我有一个从 Zotero 导出的 CSV 文件，其中包含我的图书馆条目的元数据。我知道它包含很多重复项，但要摆脱它们并不容易：

• 并非所有标题相似的项目实际上都是重复的，例如

  | Year |            Author             |    Title     |
  +------+-------------------------------+--------------+
  | 2016 | Jones, Erik                   | Book Reviews |
  | 2016 | Hassner, Pierre; Jones, Erik  | Book Reviews |
  | 2010 | Adams, Laura L.; Gagnon, Chip | Book Reviews |
  

• 并非所有实际上相似的项目都具有 100% 相同的元数据字符串，例如

  |    Author     |                     Title                     |
  +---------------+-----------------------------------------------+
  | Tichý, Lukáš; | Can Iran Reduce EU Dependence on Russian Gas? |
  | Tichy, L.;    | "can iran reduce eu dependence onrussian gas" |
  

这是一个极端的例子（差异通常不会那么大），但正如您所见，预清洗并不能完全解决这个问题；所以我们的想法是消除在两列以上包含 相似 值的行 - 例如“作者”和“标题”。

到目前为止我尝试过/浏览过的内容：

• OpenRefine - 对它几乎不熟悉，因此无法想出或找到任何可行的方法。
• Excel fuzzy lookup extension - 并没有真正按照我需要的方式工作。
• Python - 再说一次，我不擅长这门语言；我找不到任何相关的解决方案/指南。
• R：尝试了一些想法：

首先在“作者”列的for循环中使用agrep来获取重复行的索引；然后对“标题”列执行相同的操作；然后比较向量并对值重合的行进行重复数据删除。不用说，我无法超越第 1 步：

titles <- unlist(corpus$"Title")
for (i in 1:length(titles)){
  Title_dupe_temp <- agrep(titles[i], titles[i+1:length(titles)], 
                           max.distance = 1, ignore.case = TRUE, fixed = FALSE)
  Title_dupes[i] <- paste(i, Title_dupe_temp, sep = " ")
}

结果（几乎）完全是胡言乱语；另外我收到警告消息：

In Title_dupes[i] <- paste(i, Title_dupe_temp, sep = " ") :
  number of items to replace is not a multiple of replacement length

我还阅读了fuzzywuzzyR 文档，但没有找到任何有用的功能。

最后，我尝试了RecordLinkage 包。尽管如此，我还是无法超越基础知识。 The documentation 相当沉重，对所有事情都不明确；指南很少，我找到的指南（例如this）使用已准备好身份向量的示例数据集 - 所以我无法弄清楚如何在我的数据上复制它。

所以在这一点上，我不在乎是否在 OpenRefine/R/Py/SQL/whatever 中执行，只是以任何方式执行。

Answer 1

解决方案一： 使用循环和库stringdist:

library(stringdist)
    zotero<-data.frame(
      Year=c(2016,2016,2010,2010,2010,2010),
      Author=c("Jones, Erik","Hassner, Pierre;","Adams, Laura L.;","Tichý, Lukáš;","Tichý, Lukáš;","Tichy, L.;"),
      Title=c("Book Reviews","Book Reviews","Book Reviews","Can Iran Reduce EU Dependence on Russian Gas?","Can Iran Reduce EU Dependence on Russian Gas?","can iran reduce eu dependence onrussian gas")
    )

    zotero$onestring<-paste0(zotero$Year,zotero$Author,zotero$Title)
    zotero<-zotero[order(zotero[,1],zotero[,2]),]

    atot<-NULL
    for (i in 2:dim(zotero)[1]){
      a<-stringdist(zotero$onestring[i-1],zotero$onestring[i])/(nchar(zotero$onestring[i-1])+nchar(zotero$onestring[i]))
      atot<-rbind(atot,a)
    }

    zotero<-cbind(zotero,threshold=c(1,atot))
    zotero[zotero$threshold>0.15,]

解决方案 II：使用矩阵计算它可能比使用循环更快：首先我根据您的数据样本创建一个数据框，其次我删除非 UTF 字符，第三我使用库 stringdist 来计算一个距离矩阵。您可以轻松地将这些转换为相似度百分比。

zotero<-data.frame(
  Year=c(2016,2016,2010,2010,2010,2010),
  Author=c("Jones, Erik","Hassner, Pierre;","Adams, Laura L.;","Tichý, Lukáš;","Tichý, Lukáš;","Tichy, L.;"),
  Title=c("Book Reviews","Book Reviews","Book Reviews","Can Iran Reduce EU Dependence on Russian Gas?","Can Iran Reduce EU Dependence on Russian Gas?","can iran reduce eu dependence onrussian gas")
)

zotero$onestring<-paste0(zotero$Year,zotero$Author,zotero$Title)

Encoding(zotero$onestring) <- "UTF-8"
zotero$onestring<-iconv(zotero$onestring, "UTF-8", "UTF-8",sub='')

library(stringdist)
stringdistmatrix(zotero$onestring)

结果：

> stringdistmatrix(zotero$onestring)
   1  2  3  4  5
2 11            
3 13 14         
4 47 45 44      
5 47 45 44  0   
6 47 45 42 13 13

Answer 2

我对@Nakx 有类似的方法，我喜欢矩阵解决方案。但是，您也可以尝试使用gsub 和iconv 进行更多清理，并使用 sapply 进行匹配（索引不是自身的最佳匹配值..0）。像这样的：

    > library(RecordLinkage)
> 
> zotero<-data.frame(
+   Year=c(2016,2016,2010,2010,2010,2010),
+   Author=c("Jones, Erik","Hassner, Pierre;","Adams, Laura L.;","Tichý, Lukáš;","Tichý, Lukáš;","Tichy, L.;"),
+   Title=c("Book Reviews","Book Reviews","Book Reviews","Can Iran Reduce EU Dependence on Russian Gas?","Can Iran Reduce EU Dependence on Russian Gas?","can iran reduce eu dependence onrussian gas")
+ )
> 
> # Converting the special characters
> zotero$Author_new <- iconv(zotero$Author, from = '', to = "ASCII//TRANSLIT")
> zotero$Author_new <- tolower(zotero$Author_new)
> zotero$Author_new <- gsub("[[:punct:]]", "", zotero$Author_new)
> 
> # Removing punctuation making it lowercase
> zotero$Title_new <- gsub("[[:punct:]]", "", zotero$Title)
> zotero$Title_new <- tolower(zotero$Title_new)
> 
> # Removing exact duplicates
> dups <- duplicated(zotero[,c("Title_new", "Author_new", "Year")])
> zotero <- zotero[!dups,]
> zotero
  Year           Author                                         Title     Author_new
1 2016      Jones, Erik                                  Book Reviews     jones erik
2 2016 Hassner, Pierre;                                  Book Reviews hassner pierre
3 2010 Adams, Laura L.;                                  Book Reviews  adams laura l
4 2010    Tichý, Lukáš; Can Iran Reduce EU Dependence on Russian Gas?    tichy lukas
6 2010       Tichy, L.;   can iran reduce eu dependence onrussian gas        tichy l
                                     Title_new Title_dist Author_dist
1                                 book reviews          0           9
2                                 book reviews          0           9
3                                 book reviews          0           9
4 can iran reduce eu dependence on russian gas          0           0
6  can iran reduce eu dependence onrussian gas          1           4
>
> # Creating a distance measure for your title, author, and year
> zotero$Title_dist <- sapply(zotero$Title_new, function(x) sort(levenshteinDist(x, zotero$Title_new))[2])
> zotero$Author_dist <- sapply(zotero$Author_new, function(x) sort(levenshteinDist(x, zotero$Author_new))[2])
>
> # Filter here

从那里您可以使用距离变量来创建标准和过滤器。例如，如果一篇文章的作者距离为 2，标题距离为 5，您可能会觉得删除很舒服。

编辑以阐明过滤示例。您需要在查看数据后进行调整。开始保守总是好的

> library(dplyr)
> zotero <- zotero %>%
+   group_by(Year) %>%
+   filter(!between(Title_dist, 1, 5) | 
+          !between(Author_dist, 1, 5))
> zotero
# A tibble: 4 x 7
# Groups:   Year [2]
   Year Author       Title                     Author_new   Title_new                   Title_dist Author_dist
  <dbl> <fct>        <fct>                     <chr>        <chr>                            <int>       <int>
1  2016 Jones, Erik  Book Reviews              jones erik   book reviews                         0           9
2  2016 Hassner, Pi~ Book Reviews              hassner pie~ book reviews                         0           9
3  2010 Adams, Laur~ Book Reviews              adams laura~ book reviews                         0           9
4  2010 Tichý, Luká~ Can Iran Reduce EU Depen~ tichy lukas  can iran reduce eu depende~          0           0