如何通过从随机化中排除那些包含特定单词列表的段落来从语料库中随机选择段落？

Question

我有一个语料库。我想从这个语料库中随机提取段落。但是，随机化练习必须确保不能对具有特定字词的段落进行抽样。

这是一个例子：

txt <- c("PAGE 1. A single sentence.  Short sentence. Three word sentence. \n\n Quarentine is hard",
         "PAGE 2. Very short! Shorter.\n\n quarantine is very very hard",
         "Very long sentence, with three parts, separated by commas.  PAGE 3.\n\n quarantine it's good tough to focus on paper.",
         "Fiscal policy is a bad thing. \n\n SO is a great place where skilled people solve coding problems.",
         "Fiscal policy is not as good as people may think",
         "Economics is fun. \n\n I prefer Macro.")
corp <- corpus(txt, docvars = data.frame(serial = 1:6))

没有任何限制地做它是直截了当的：

reshape = corpus_reshape(corp, "paragraphs")
sample = corpus_sample(reshape, 4)

# Result

[1] "Economics is fun."                                "Fiscal policy is not as good as people may think"
[3] "Fiscal policy is a bad thing."                    "Quarentine is hard"

如您所见，随机选择了包含财政政策的“段落”。我希望通过排除出现财政政策的段落/句子来对语料库进行抽样。

我可以在采样之前删除原始数据集中与这个词相关的句子吗？你会怎么做？

请注意，在真实数据集中，我需要排除包含不止一个或两个关键字的句子。所以，请提出一些可以轻松扩展为多个单词的内容。

非常感谢！

Answer 1

如果要排除包含“财政政策”的段落/句子，则需要先将文本重新整形为段落，然后过滤掉包含排除短语的术语，然后样本。

如果您在创建语料库之前过滤文本，您将从还包含过滤词组的输入文本中排除非过滤词组段落。

library("quanteda")
## Package version: 2.0.1
set.seed(10)

txt <- c(
  "PAGE 1. A single sentence.  Short sentence. Three word sentence. \n\n Quarentine is hard",
  "PAGE 2. Very short! Shorter.\n\n quarantine is very very hard",
  "Very long sentence, with three parts, separated by commas.  PAGE 3.\n\n quarantine it's good tough to focus on paper.",
  "Fiscal policy is a bad thing. \n\n SO is a great place where skilled people solve coding problems.",
  "Fiscal policy is not as good as people may think",
  "Economics is fun. \n\n I prefer Macro."
)

corp <- corpus(txt, docvars = data.frame(serial = 1:6)) %>%
  corpus_reshape(to = "paragraphs")
tail(corp)
## Corpus consisting of 6 documents and 1 docvar.
## text3.2 :
## "quarantine it's good tough to focus on paper."
## 
## text4.1 :
## "Fiscal policy is a bad thing."
## 
## text4.2 :
## "SO is a great place where skilled people solve coding proble..."
## 
## text5.1 :
## "Fiscal policy is not as good as people may think"
## 
## text6.1 :
## "Economics is fun."
## 
## text6.2 :
## "I prefer Macro."

现在我们可以根据模式匹配进行子集化。

corp2 <- corpus_subset(corp, !grepl("fiscal policy", corp, ignore.case = TRUE))
tail(corp2)
## Corpus consisting of 6 documents and 1 docvar.
## text2.2 :
## "quarantine is very very hard"
## 
## text3.1 :
## "Very long sentence, with three parts, separated by commas.  ..."
## 
## text3.2 :
## "quarantine it's good tough to focus on paper."
## 
## text4.2 :
## "SO is a great place where skilled people solve coding proble..."
## 
## text6.1 :
## "Economics is fun."
## 
## text6.2 :
## "I prefer Macro."

corpus_sample(corp2, size = 4)
## Corpus consisting of 4 documents and 1 docvar.
## text6.2 :
## "I prefer Macro."
## 
## text1.2 :
## "Quarentine is hard"
## 
## text2.2 :
## "quarantine is very very hard"
## 
## text3.2 :
## "quarantine it's good tough to focus on paper."

包含“财政政策”的段落不见了。

请注意，这里我使用了grepl()，但一个全面的高级替代品是来自 stringi（或等效的 stringr 包装器）的str_detect()。这些还使您能够更好地控制能够使用更快的固定匹配，同时还可以控制是否匹配大小写。

all.equal(
  grepl("fiscal policy", txt, ignore.case = TRUE),
  stringi::stri_detect_fixed(txt, "fiscal policy", case_insensitive = TRUE),
  stringr::str_detect(txt, fixed("fiscal policy"), case_insensitive = TRUE)
)
## [1] TRUE

Answer 2

如果您有文本，那么您可以在创建语料库之前使用带有grepl 的子集来省略“财政政策”（或任何其他词）。

txt2 <- txt[!grepl("fiscal policy|I am groot", tolower(txt))]
txt2

[1] "PAGE 1. A single sentence.  Short sentence. Three word sentence. \n\n Quarentine is hard"                             
[2] "PAGE 2. Very short! Shorter.\n\n quarantine is very very hard"                                                        
[3] "Very long sentence, with three parts, separated by commas.  PAGE 3.\n\n quarantine it's good tough to focus on paper."
[4] "Economics is fun. \n\n I prefer Macro."

第 4 项和第 5 项尚未被选中。现在进行采样。

如果你只有语料库，那么提取文本然后使用上面的代码。

txt <- texts(corp)