我正在使用字典来识别语料库中特定单词集的用法。我在字典中包含了多词模式，但是，我认为 dfm_lookup （来自 quanteda 包）与多词表达式不匹配。有谁知道如何使用包含多词表达式的字典执行与 dfm_lookup 相同的操作？
library(quanteda)
BritainEN <-
dictionary(list(identity=c("British", "Great Britain")))
British <- dfm_lookup(debate_dfm,
BritishEN,case_insensitive=T)
【问题讨论】：
valuetype = "regex"参数。
标签： r dictionary nlp pattern-matching quanteda
是的 - 在形成 dfm 之前，您需要在令牌上使用 tokens_lookup()。一旦您对单个单词进行了标记，它们就不再作为您需要匹配字典中的多单词值的有序序列存在。所以 1) 形成标记对象，2) 使用 tokens_lookup() 将字典应用于标记，然后 3) 形成 dfm。
library("quanteda")
#> Package version: 1.5.2
BritainEN <-
dictionary(list(identity = c("British", "Great Britain")))
txt <- c(doc1 = "Great Britain is a country.",
doc2 = "British citizens live in Great Britain.")
tokens(txt) %>%
tokens_lookup(dictionary = BritainEN, exclusive = FALSE)
#> tokens from 2 documents.
#> doc1 :
#> [1] "IDENTITY" "is" "a" "country" "."
#>
#> doc2 :
#> [1] "IDENTITY" "citizens" "live" "in" "IDENTITY" "."
tokens(txt) %>%
tokens_lookup(dictionary = BritainEN) %>%
dfm()
#> Document-feature matrix of: 2 documents, 1 feature (0.0% sparse).
#> 2 x 1 sparse Matrix of class "dfm"
#> features
#> docs identity
#> doc1 1
#> doc2 2
已添加
为了回答额外的评论问题并扩展 @phiver 对此非常有用的答案，还有一个 nested_scope 参数设计用于可能在另一个 MWE 字典键的值中发生的匹配。
例子：
library("quanteda")
## Package version: 1.5.2
Ireland_nested <- dictionary(list(
ie_alone = "Ireland",
ie_nested = "Northern Ireland"
))
txt <- c(
doc1 = "Northern Ireland is a country.",
doc2 = "Some citizens of Ireland live in Northern Ireland."
)
toks <- tokens(txt)
tokens_lookup(toks, dictionary = Ireland_nested, exclusive = FALSE)
## Tokens consisting of 2 documents.
## doc1 :
## [1] "IE_NESTED" "IE_ALONE" "is" "a" "country" "."
##
## doc2 :
## [1] "Some" "citizens" "of" "IE_ALONE" "live" "in"
## [7] "IE_NESTED" "IE_ALONE" "."
tokens_lookup(toks,
dictionary = Ireland_nested, nested_scope = "dictionary",
exclusive = FALSE
)
## Tokens consisting of 2 documents.
## doc1 :
## [1] "IE_NESTED" "is" "a" "country" "."
##
## doc2 :
## [1] "Some" "citizens" "of" "IE_ALONE" "live" "in"
## [7] "IE_NESTED" "."
第一个匹配 both 键，因为嵌套级别正好在键内，但嵌套模式出现在两个不同的键中。 （在@phiver 中，模式嵌套在键中，在我的示例中它们不是。）当nested_scope = "dictionary" 时，它会在整个字典中查找嵌套模式匹配，而不仅仅是在键中，因此在我的示例中不会重复。
您选择哪一种取决于您的目的。我们将 quanteda 设计为具有大多数用户想要和期望的默认设置，但为有特定需求的用户添加了类似的附加选项。 （通常这些需求首先由 Kohei 或我在处理我们自己的特定用例时表达！）
【讨论】：
在评论中回答你的问题：
如果字典中包含一个单词，那么它是如何工作的 出现在字典中的多词表达式中
如果文本包含“Northern Ireland”并且字典同时包含“Northern Ireland”和“Ireland”，则只会计算一次，但ONLY IF两个值都在同一个字典分组中，例如在肯的回答中的英国例子中。
请参阅下面的示例了解差异。
组合字典示例：
library("quanteda")
Ireland_combined <-
dictionary(list(identity = c("Ireland", "Northern Ireland")))
txt <- c(doc1 = "Northern Ireland is a country.",
doc2 = "Some citizens of Ireland live in Northern Ireland.")
tokens(txt) %>%
tokens_lookup(dictionary = Ireland_combined , exclusive = FALSE)
# tokens from 2 documents.
# doc1 :
# [1] "IDENTITY" "is" "a" "country" "."
#
# doc2 :
# [1] "Citizens" "of" "IDENTITY" "live" "in" "IDENTITY" "."
tokens(txt) %>%
tokens_lookup(dictionary = Ireland_combined ) %>%
dfm()
# Document-feature matrix of: 2 documents, 1 feature (0.0% sparse).
# 2 x 1 sparse Matrix of class "dfm"
# features
# docs identity
# doc1 1
# doc2 2
单独的字典条目示例：
Ireland_seperated <-
dictionary(list(identity1 = c("Ireland"),
identity2 = "Northern Ireland"))
tokens(txt) %>%
tokens_lookup(dictionary = Ireland_seperated , exclusive = FALSE)
# tokens from 2 documents.
# doc1 :
# [1] "IDENTITY2" "IDENTITY1" "is" "a" "country" "."
#
# doc2 :
# [1] "Citizens" "of" "IDENTITY1" "live" "in" "IDENTITY2" "IDENTITY1" "."
tokens(txt) %>%
tokens_lookup(dictionary = Ireland_seperated ) %>%
dfm()
# Document-feature matrix of: 2 documents, 2 features (0.0% sparse).
# 2 x 2 sparse Matrix of class "dfm"
# features
# docs identity1 identity2
# doc1 1 1
# doc2 2 1
【讨论】：