将整个列表与文件的每一行匹配

Question

我写了一段代码，基本上是从文本文件的列表中执行查找和替换。

因此，它将整个列表映射到字典中。然后从文本文件中处理每一行，并与字典中的整个列表匹配，如果在该行中的任何位置找到匹配项，它将替换为列表（字典）中的相应值。

代码如下：

import sys
import re

#open file using open file mode
fp1 = open(sys.argv[1]) # Open file on read mode
lines = fp1.read().split("\n") # Create a list containing all lines
fp1.close() # Close file


fp2 = open(sys.argv[2]) # Open file on read mode
words = fp2.read().split("\n") # Create a list containing all lines
fp2.close() # Close file

word_hash = {}

for word in words:
    #print(word)
    if(word != ""):
        tsl = word.split("\t")
        word_hash[tsl[0]] = tsl[1]

#print(word_hash)
keys = word_hash.keys()
#skeys = sorted(keys, key=lambda x:x.split(" "),reverse=True)
#print(keys)
#print (skeys)

for line in lines:
    if(line != ""):
        for key in keys:
            #my_regex = key + r"\b"
            my_regex = r"([\"\( ])" + key + r"([ ,\.!\"।)])"
            #print(my_regex)
            if((re.search(my_regex, line, re.IGNORECASE|re.UNICODE))):
                line = re.sub(my_regex, r"\1" + word_hash[key]+r"\2",line,flags=re.IGNORECASE|re.UNICODE|re.MULTILINE)
                #print("iam :1",line)
            if((re.search(key + r"$", line, re.IGNORECASE|re.UNICODE))):
                line = re.sub(key+r"$", word_hash[key],line,flags=re.IGNORECASE|re.UNICODE|re.MULTILINE)
                #print("iam :2",line)
            if((re.search(r"^" + key, line, re.IGNORECASE|re.UNICODE))):
                #print(line)
                line = re.sub(r"^" + key, word_hash[key],line,flags=re.IGNORECASE|re.UNICODE|re.MULTILINE)
                #print("iam :",line)
        print(line)
    else:
        print(line)

这里的问题是当列表大小增加时，执行速度会变慢，因为文本文件的所有行都与列表中的每个键匹配。那么我在哪里可以改进这段代码的执行呢。

列表文件：

word1===>替换word1

word2===>替换word2

.....

列表是制表符分隔的。为了便于理解，这里我使用了 ===>。

输入文件：

hello word1 I am here.

word2. how are you word1?

预期输出：

hello replaceword1 I am here.

replaceword2. how are you replaceword1?

Answer 1

如果您的单词列表足够小，您可以通过匹配和替换过程实现的最佳加速是使用单个大正则表达式并使用函数re.sub

这样你就可以对优化函数进行一次调用。

编辑：为了保留替换顺序（这可能导致链替换，不知道是否预期行为）我们可以按批次执行替换，而不是在一次运行中执行替换，其中批次顺序尊重文件顺序和每个批次由不相交的可能字符串匹配组成。

代码如下

import sys
import re


word_hashes = []

def insert_word(word, replacement, hashes):
    if not hashes:
        return [{word: replacement}]
    for prev_word in hashes[0]:
        if word in prev_word or prev_word in word:
            return [hashes[0]] + insert_word(word, replacement, hashes[1:])
    hashes[0][word] = replacement
    return hashes

with open(sys.argv[2]) as fp2: # Open file on read mode
    words = fp2.readlines()
    for word in [w.strip() for w in words if w.strip()]:
        tsl = word.split("\t")
        word_hashes = insert_word(tsl[0],tsl[1], word_hashes)

#open file using open file mode
lines = []
with open(sys.argv[1]) as fp1:
    content = fp1.read()
    for word_hash in word_hashes: 
        my_regex = r"([\"\( ])(" + '|'.join(word_hash.keys()) + r")([ ,\.!\"।)])"
        content = re.sub(my_regex, lambda x: x.group(1) + word_hash[x.group(2)] + x.group(3) ,content,flags=re.IGNORECASE|re.UNICODE|re.MULTILINE)

print(content)

我们获得了示例数据的链式替换。比如用下面的词来代替

roses are red==>flowers are blue
are==>is

要解析的文本

roses are red and beautiful
flowers are yellow

输出

roses is red and beautiful
flowers is yellow

Answer 2

你为什么不把整个文件的内容读成一个字符串，然后string.replace。例如。

def find_replace():

    txt = ''
    #Read text from the file as a string
    with open('file.txt', 'r') as fp:
        txt = fp.read()

    dct = {"word1":"replaceword1","word2":"replaceword2"}

    #Find and replace characters
    for k,v in dct.items():
        txt = txt.replace(k,v)

    #Write back the modified string
    with open('file.txt', 'w') as fp:
        fp.write(txt)

如果输入文件是：

hello word1 I am here.

word2. how are you word1?

输出将是：

hello replaceword1 I am here.

replaceword2. how are you replaceword1?