如何使用正则表达式 python 在大文件中查找地名

Question

所以我想从 country.txt 文件中找到确切的单词，该文件定义了带有以下描述文件的地点名称：

这里是 country.txt 的示例

Pic de Font Blanca
Roc Mélé
Pic des Langounelles
Pic de les Abelletes
Estany de les Abelletes
Port Vieux de la Coume d’Ose
Port de la Cabanette
Port Dret
Costa de Xurius
Font de la Xona

这是一个 description.csv description file

描述文件是包含文章标题和描述的数据列表。我要做的是从带有 country.txt 文件的描述文件中找到确切的地名单词

code.py

import csv
import time
import re

allCities = open('country.txt', encoding="utf8").readlines()
timestr = time.strftime("%Y-%m-%d-(%H-%M-%S)")

with open('description.csv') as descriptions,open('desc_place7---' + str(timestr) + '.csv', 'w', newline='', encoding='utf-8') as output:
    descriptions_reader = csv.DictReader(descriptions)
    fieldnames = ['title', 'description', 'place']
    output_writer = csv.DictWriter(output, delimiter='|', fieldnames=fieldnames)
    output_writer.writeheader()
    line=0
    pattern = r'|'.join(r'\b{}\b'.format(re.escape(city.strip())) for city in sorted(allCities, key=len, reverse=True))

    for eachRow in descriptions_reader:
        title = eachRow['row']
        description = eachRow['desc']
        citiesFound = set()
        found = re.findall(pattern, description, re.IGNORECASE | re.MULTILINE)
        citiesFound.update(found)
        if len(citiesFound)==0:
            output_writer.writerow({'title': title, 'description': description, 'place': " - "})

        else:
            output_writer.writerow({'title': title, 'description': description, 'place': " , ".join(citiesFound)})
        line += 1
        print(line)

预期输出： output

但是因为 country.txt(185.94MB) 是一个大文件，所以我的代码不能完全运行。它使我的笔记本电脑冻结。 有没有好的方法来处理这个？我认为这也是因为我的模式线性能低下，但我还需要一个正则表达式来找到准确的单词

Answer 1

这是针对您的问题的第一个实现，您需要根据您的特定需求对其进行改进。

首先将您所有的descriptions 保存到pandas DataFrame，如下所示：

import pandas as pd
descriptions = pd.read_csv('description.csv')

然后不要将所有文件行读入内存。您可以逐行读取country 文件并在descriptions 数据中查找匹配项。使用以下内容：

 with open('country.txt', encoding="utf8") as cities_file, open('desc_place7---' + str(timestr) + '.csv', 'w', newline='', encoding='utf-8') as output:
    fieldnames = ['title', 'description', 'place']
    output_writer = csv.DictWriter(output, delimiter='|', fieldnames=fieldnames)
    output_writer.writeheader()
    line = 0        
    for city in cities_file:
        pattern = r'\b{}\b'.format(re.escape(city.strip())
        for index, row in descriptions.iterrows():
            title = row['row']
            description = row['desc']
            citiesFound = set()            
            found = re.findall(pattern, description, re.IGNORECASE | re.MULTILINE)
            citiesFound.update(found)
            if len(citiesFound)==0:
                output_writer.writerow({'title': title, 'description': description, 'place': " - "})
            else:
                output_writer.writerow({'title': title, 'description': description, 'place': " , ".join(citiesFound)})
            line += 1
            print(line)