为文本中的单词列表替换下划线空格的最快方法

Question

给定 10,000,000,000 行，每行大约 20-50 个单词，例如：

Anarchism is often defined as a political philosophy which holds the state to be undesirable , unnecessary , or harmful .
However , others argue that while anti-statism is central , it is inadequate to define anarchism .
Therefore , they argue instead that anarchism entails opposing authority or hierarchical organization in the conduct of human relations , including , but not limited to , the state system .
Proponents of anarchism , known as " anarchists " , advocate stateless societies based on non - hierarchical free association s. As a subtle and anti-dogmatic philosophy , anarchism draws on many currents of thought and strategy .
Anarchism does not offer a fixed body of doctrine from a single particular world view , instead fluxing and flowing as a philosophy .
There are many types and traditions of anarchism , not all of which are mutually exclusive .
Anarchist schools of thought can differ fundamentally , supporting anything from extreme individualism to complete collectivism .
Strains of anarchism have often been divided into the categories of social and individualist anarchism or similar dual classifications .
Anarchism is often considered a radical left-wing ideology , and much of anarchist economics and anarchist legal philosophy reflect anti-authoritarian interpretations of communism , collectivism , syndicalism , mutualism , or participatory economics .
Anarchism as a mass social movement has regularly endured fluctuations in popularity .
The central tendency of anarchism as a social movement has been represented by anarcho-communism and anarcho-syndicalism , with individualist anarchism being primarily a literary phenomenon which nevertheless did have an impact on the bigger currents and individualists have also participated in large anarchist organizations .
Many anarchists oppose all forms of aggression , supporting self-defense or non-violence ( anarcho-pacifism ) , while others have supported the use of some coercive measures , including violent revolution and propaganda of the deed , on the path to an anarchist society .
Etymology and terminology The term derives from the ancient Greek ἄναρχος , anarchos , meaning " without rulers " , from the prefix ἀν - ( an - , " without " ) + ἀρχός ( arkhos , " leader " , from ἀρχή arkhē , " authority , sovereignty , realm , magistracy " ) + - ισμός ( - ismos , from the suffix - ιζειν , - izein " - izing " ) . "
Anarchists " was the term adopted by Maximilien de Robespierre to attack those on the left whom he had used for his own ends during the French Revolution but was determined to get rid of , though among these " anarchists " there were few who exhibited the social revolt characteristics of later anarchists .
There would be many revolutionaries of the early nineteenth century who contributed to the anarchist doctrines of the next generation , such as William Godwin and Wilhelm Weitling , but they did not use the word " anarchist " or " anarchism " in describing themselves or their beliefs .
Pierre-Joseph Proudhon was the first political philosopher to call himself an anarchist , making the formal birth of anarchism the mid-nineteenth century .
Since the 1890s from France , the term " libertarianism " has often been used as a synonym for anarchism and was used almost exclusively in this sense until the 1950s in the United States ; its use as a synonym is still common outside the United States .
On the other hand , some use " libertarianism " to refer to individualistic free-market philosophy only , referring to free-market anarchism as " libertarian anarchism " .

假设我有一个由一个或多个单词组成的字典术语列表，例如：

clinical anatomy
clinical psychology
cognitive neuroscience
cognitive psychology
cognitive science
comparative anatomy
comparative psychology
compound morphology
computational linguistics
correlation
cosmetic dentistry
cosmography
cosmology
craniology
craniometry
criminology
cryobiology
cryogenics
cryonics
cryptanalysis
crystallography
curvilinear correlation
cybernetics
cytogenetics
cytology
deixis
demography
dental anatomy
dental surgery
dentistry
philosophy
political philosophy

我需要找到所有包含这些术语的句子，然后将术语中单词之间的空格替换为下划线。

比如文中有这样一句话：

Anarchism is often defined as a political philosophy which holds the state to be undesirable , unnecessary , or harmful .

文本中有字典术语political philosophy。所以这句话的输出需要是：

Anarchism is often defined as a political_philosophy which holds the state to be undesirable , unnecessary , or harmful .

我可以这样做：

dictionary = sort(dictionary, key=len) # replace the longest terms first.
for line in text:
   for term in dictionary: 
       if term in line:
           line = line.replace(term, term.replace(' ', '_'))

假设我有 10,000 个字典术语 (D) 和 10,000,000,000 个句子 (S)，使用简单方法的复杂度将是 O(D*S)，对吗？ 是否有更快、更简单的方法来获得相同的结果？

有没有办法用每行下划线的术语替换所有带空格的术语？这将有助于避免内部循环。

如果先使用whoosh 之类的东西对文本进行索引，然后查询索引并替换术语会更有效？我仍然需要O(1*S) 之类的东西来进行替换，对吧？

解决方案不需要在 Python 中，即使是 grep/sed/awk 之类的 Unix 命令技巧也可以，只要 subprocess.Popen-able 即可。

如果我错了，请纠正我的复杂性假设，请原谅我的菜鸟。

---

给定一句话：

这是一个包含多个短语的句子，我需要 替换为带下划线的短语，例如社会政治 哲学分支下的政治哲学哲学 和一些计算语言学，其中认知语言学和 心理认知语言学伴随语言学出现

假设我有字典：

cognitive linguistics
psycho cognitive linguistics
socio political philosophy
political philosophy
computational linguistics
linguistics
philosophy
social political philosophy 

输出应如下所示：

这是一个包含多个短语的句子，我需要 替换为带下划线的短语，例如 分支下的social_political_philosophy和political_philosophy 哲学和一些计算语言学 认知语言学和心理认知语言学与 语言学

我们的目标是使用 100 亿行的文本文件和 10-100k 短语的字典来做到这一点。

Answer 1

拆分单词可能会更好，将单词从短语的开头映射到完整的短语，如果您需要最大的，而不是检查字典中的每个项目，您只需对出现的短语进行排序长度：

from collections import defaultdict

def get_phrases(fle):
    phrase_dict = defaultdict(list)
    with open(fle) as ph:
        for line in map(str.rstrip, ph):
            k, _, phr = line.partition(" ")
            phrase_dict[k].append(line)
        return phrase_dict

from itertools import chain


def replace(fle, dct):
    with open(fle) as f:
        for line in f:
            phrases = sorted(chain.from_iterable(dct[word] for word in line.split() 
                             if word in dct) ,reverse=1, key=len)
            for phr in phrases:
                  line = line.replace(phr, phr.replace(" ", "_"))
            yield line

输出：

In [10]: cat out.txt
This is a sentence that contains multiple phrases that I need to replace with phrases with underscores, e.g. social political philosophy with political philosophy under the branch of philosophy and some computational linguistics where the cognitive linguistics and psycho cognitive linguistics appears with linguistics
In [11]: cat phrases.txt
cognitive linguistics
psycho cognitive linguistics
socio political philosophy
political philosophy
computational linguistics
linguistics
philosophy
social political philosophy
In [12]: list(replace("out.txt",get_phrases("phrases.txt")))
Out[12]: ['This is a sentence that contains multiple phrases that I need to replace with phrases with underscores, e.g. social_political_philosophy with political_philosophy under the branch of philosophy and some computational_linguistics where the cognitive_linguistics and psycho_cognitive_linguistics appears with linguistics']

其他几个版本：

def repl(x):
    if x:
        return x.group().replace(" ", "_")
    return x


def replace_re(fle, dct):
    with open(fle) as f:
        for line in f:
            spl = set(line.split())
            phrases = chain.from_iterable(dct[word] for word in spl if word in dct)
            line = re.sub("|".join(phrases), repl, line)
            yield line


def replace_re2(fle, dct):
    cached = {}
    with open(fle) as f:
        for line in f:
            phrases = tuple(chain.from_iterable(dct[word] for word in set(line.split()) if word in dct))
            if phrases not in cached:
                r = re.compile("|".join(phrases))
                cached[phrases] = r
                line = r.sub(repl, line)
            else:
                line = cached[phrases].sub(repl, line)
            yield line

Answer 2

我会为您的字典创建一个正则表达式以匹配数据。
然后在替换方面，使用回调将空格替换为_。

我估计完成整个过程需要不到 3 个小时。

幸运的是，有一个 三元工具（字典） 正则表达式生成器。

要生成正则表达式以及如下所示，您需要试用版
RegexFormat 7的版本

部分链接：
Screenshot of tool
TernaryTool(Dictionary) - Text version Dictionary samples
A 175,000 word Dictionary Regex

您基本上可以生成自己的字典
通过放入您要查找的字符串，然后按 Generate 按钮。

然后您所要做的就是读取 5 MB 块并使用
进行查找/替换 正则表达式，然后将其附加到新文件中。冲洗重复。
真的很简单。

根据您的样本（上图），这是对所需时间的估计
完成100亿条线路。

此分析基于使用生成的正则表达式（如下）在您的示例输入上运行的基准。

19 lines  (@ 3600 chars)

Completed iterations:   50  /  50     ( x 1000 )
Matches found per iteration:   5
Elapsed Time:    4.03 s,   4034.28 ms,   4034278 µs

////////////////////////////
3606 chars
x 50,000
------------
180,300,000  (chars)

or 

20 lines
x 50,000
------------
1,000,000  (lines)
=========================
10,000,000,000 lines
/
1,000,000  (lines) per 4 seconds
-----------------------------------------
40,000 seconds
/
3600 secs per hour
-------------------------
11 hours
////////////////////////////

但是，如果您读入并处理 5 兆字节的块
（作为单个字符串）它将减少引擎开销
并将时间缩短到 1-3 小时。

这是为您的示例字典（压缩）生成的正则表达式：

\b(?:c(?:linical[ ](?:anatomy|psychology)|o(?:gnitive[ ](?:neuroscience|psychology|science)|mp(?:arative[ ](?:anatomy|psychology)|ound[ ]morphology|utational[ ]linguistics)|rrelation|sm(?:etic[ ]dentistry|o(?:graphy|logy)))|r(?:anio(?:logy|metry)|iminology|y(?:o(?:biology|genics|nics)|ptanalysis|stallography))|urvilinear[ ]correlation|y(?:bernetics|to(?:genetics|logy)))|de(?:ixis|mography|nt(?:al[ ](?:anatomy|surgery)|istry))|p(?:hilosophy|olitical[ ]philosophy))\b

(请注意，空格分隔生成为每个空格[ ]。
如果您想将其更改为量化类，只需运行一个
找到 (?:\[ \])+ 并替换为您想要的任何内容。
例如\s+ 或[ ]+)

---

这里是格式化的：

 \b 
 (?:
      c
      (?:
           linical [ ] 
           (?: anatomy | psychology )
        |  o
           (?:
                gnitive [ ] 
                (?: neuroscience | psychology | science )
             |  mp
                (?:
                     arative [ ] 
                     (?: anatomy | psychology )
                  |  ound [ ] morphology
                  |  utational [ ] linguistics
                )
             |  rrelation
             |  sm
                (?:
                     etic [ ] dentistry
                  |  o
                     (?: graphy | logy )
                )
           )
        |  r
           (?:
                anio
                (?: logy | metry )
             |  iminology
             |  y
                (?:
                     o
                     (?: biology | genics | nics )
                  |  ptanalysis
                  |  stallography
                )
           )
        |  urvilinear [ ] correlation
        |  y
           (?:
                bernetics
             |  to
                (?: genetics | logy )
           )
      )
   |  de
      (?:
           ixis
        |  mography
        |  nt
           (?:
                al [ ] 
                (?: anatomy | surgery )
             |  istry
           )
      )
   |  p
      (?: hilosophy | olitical [ ] philosophy )
 )
 \b 

添加 10,000 个短语非常容易，并且正则表达式不大于
短语中的字节数加上交错的一些开销
正则表达式。

最后一点。您可以通过仅生成
短语上的正则表达式..那只是由水平空格分隔的单词。

并且，一定要预编译正则表达式。只需执行一次。