【发布时间】:2019-03-15 17:01:36
【问题描述】:
我正在寻找更好的解决方案来解决这个问题:
我想要做的是自动连接由换行符分隔的书的单词。我试过的代码是:
import nltk
from nltk.tokenize import word_tokenize
import re
with open ('Fr-dictionary.txt') as fr: #opens the dictionary
dic = word_tokenize(fr.read().lower()) #stores the first dictionary
pat=re.compile(r'[.?\-",:;.?!»’()quls\d]+|\w+(?:-\w+)+') #pattern for
punctuation, digits and words separated by hyphens (-)
reg= list(filter(pat.match, text))
with open ('fr-text.txt') as tx2: #opening text containing the
separated words
text_input = word_tokenize(tx2.read().lower()) #stores the input
text
words_it = iter(text_input)
out_file1=open("finaltext.txt","w") #defining name of output file
valid_words1=[ ] #empty list to append the existing words
invalid_words1=[ ] #empty list to append the invalid(non-existing)words
for w in words_it: #looping through the tokenized text
if w in dic:
valid_words1.append(w)
elif w in reg:
valid_words1.append(w)#appending the valid items
else:
try:
concatenated = w + next(words_it) #concatenating strings
if concatenated in dic:
valid_words1.append(concatenated)#append if valid
except StopIteration:
pass
else:
invalid_words1.append(w) #appending the invalid_words
a1=' '.join(valid_words1) #converting list into a string
out_file1.write(a1) #writing the output to a file
out_file1.close()
print(a1) #print list converted into text
print(invalid_words1)
print(len(invalid_words)
使用这段代码我已经:
a) 标记文本(到列表中)并在整个列表中循环检查每个项目是否存在于字典中(包括标点符号) b)如果不是,我尝试连接单词的两个部分, c) 检查字典中是否存在连接的输出,并且, d) 如果是,附加到相同的有效单词列表中,但是 e) 如果不附加到带有无效单词的另一个列表。
问题:问题是有时要连接的单词的第一部分是一个现有/有效的单词(存在于字典中),然后程序会忽略它并且不与它的第二部分连接,从而导致文本与这些错误。任何想法来解决这个问题?我认为解决方案可能是:循环并附加所有存在的单词,当出现不存在的单词时,程序可以返回上一个,连接,签入 dic,然后继续......怎么做?
【问题讨论】:
-
在我看来这不是迭代错误,而是处理错误...
标签: python string list concatenation word