改进我的代码以在大列表 python 中对相同的单词进行分组并与其他代码进行比较

Question

我一直在阅读与分组相似词相关的其他一些链接（What is a good strategy to group similar words? 和 Fuzzy Group By, Grouping Similar Words）。我很好奇（1）是否有人可以指导我在第二个链接中找到的一种算法如何工作，以及（2）编程风格与我自己的“幼稚”方法相比如何？

如果你甚至可以回答 1 或 2，我会给你一个赞成票。

(1) 有人可以帮我看看这里发生了什么吗？

class Seeder:
    def __init__(self):
        self.seeds = set()
        self.cache = dict()
    def get_seed(self, word):
        LIMIT = 2
        seed = self.cache.get(word,None)
        if seed is not None:
            return seed
        for seed in self.seeds:
            if self.distance(seed, word) <= LIMIT:
                self.cache[word] = seed
                return seed
        self.seeds.add(word)
        self.cache[word] = word
        return word

    def distance(self, s1, s2):
        l1 = len(s1)
        l2 = len(s2)
        matrix = [range(zz,zz + l1 + 1) for zz in xrange(l2 + 1)]
        for zz in xrange(0,l2):
            for sz in xrange(0,l1):
                if s1[sz] == s2[zz]:
                    matrix[zz+1][sz+1] = min(matrix[zz+1][sz] + 1, matrix[zz][sz+1] + 1, matrix[zz][sz])
                else:
                    matrix[zz+1][sz+1] = min(matrix[zz+1][sz] + 1, matrix[zz][sz+1] + 1, matrix[zz][sz] + 1)
        return matrix[l2][l1]

import itertools

def group_similar(words):
    seeder = Seeder()
    words = sorted(words, key=seeder.get_seed)
    groups = itertools.groupby(words, key=seeder.get_seed)

(2) 在我的方法中，我有一个要分组的字符串列表，称为 residentyList 并使用默认字典。

Array(['Psychiatry', 'Radiology Medicine-Prelim',
       'Radiology Medicine-Prelim', 'Medicine', 'Medicine',
       'Obstetrics/Gynecology', 'Obstetrics/Gyncology',
       'Orthopaedic Surgery', 'Surgery', 'Pediatrics',
       'Medicine/Pediatrics',])

我努力分组。我基于 uniqueResList，即 np.unique(residencyList)

d = collections.defaultdict(int)
for i in residencyList:
    for x in uniqueResList:
        if x ==  i:
            if not d[x]:
                #print i, x
                d[x] = i  
                #print d
            if d[x]:
                d[x] = d.get(x, ()) + ', ' + i
        else:
            #print 'no match'
            continue

Answer 1

远距离“忍者数学”的简短解释：

 # this is just the edit distance (Levenshtein) between the two words
    def distance(self, s1, s2):
        l1 = len(s1) # length of first word
        l2 = len(s2) # length of second word
        matrix = [range(zz,zz + l1 + 1) for zz in xrange(l2 + 1)] 
           # make an l2 + 1 by l1 + 1 matrix where the first row and column count up from
           # 0 to l1 and l2 respectively (these will be the costs of
           # deleting the letters that came before that element in each word)
        for zz in xrange(0,l2):
            for sz in xrange(0,l1):
                if s1[sz] == s2[zz]: # if the two letters are the same then we
                       # don't have to change them so take the 
                       # cheapest path from the options of
                       # matrix[zz+1][sz] + 1 (delete the letter in s1)
                       # matrix[zz][sz+1] + 1 (delete the letter in s2)
                       # matrix[zz][sz] (leave both letters)
                    matrix[zz+1][sz+1] = min(matrix[zz+1][sz] + 1, matrix[zz][sz+1] + 1, matrix[zz][sz])
                else: # if the two letters are not the same then we
                         # have to change them so take the 
                         # cheapest path from the options of
                         # matrix[zz+1][sz] + 1 (delete the letter in s1)
                         # matrix[zz][sz+1] + 1 (delete the letter in s2)
                         # matrix[zz][sz] + 1 (swap a letter)
                    matrix[zz+1][sz+1] = min(matrix[zz+1][sz] + 1, matrix[zz][sz+1] + 1, matrix[zz][sz] + 1)
        return matrix[l2][l1] # the value at the bottom of the matrix is equal to the cheapest set of edits

Answer 2

我将尝试回答第一部分。 Seeder 类尝试查找单词的seeds。假设两个相似的词具有相同的种子，相似度由参数LIMIT（在本例中为 2）控制，该参数测量两个词之间的距离。计算String distance 的方法有很多种，而您的班级在distance 函数中使用某种忍者数学来计算，坦率地说，这比我高。

def __init__(self):
    self.seeds = set()
    self.cache = dict()

将种子初始化为 set 以跟踪迄今为止唯一的种子，并以 cache 加速查找，以防我们已经看到该词（以节省计算时间）。

对于任何单词，get_seed 函数都会返回其种子。

def get_seed(self, word):
    #Set the acceptable distance
    LIMIT = 2
    #Have we seen this word before? 
    seed = self.cache.get(word,None)
    if seed is not None:
        #YES. Return from the cache
        return seed
    for seed in self.seeds:
        #NO. For each pre-existing seed, find the distance of this word from that seed
        if self.distance(seed, word) <= LIMIT:
            #This word is similar to the seed
            self.cache[word] = seed
            #We found this word's seed, cache it and return
            return seed
    #No we couldn't find a matching word in seeds. This is a new seed
    self.seeds.add(word)
    #Cache this word for future
    self.cache[word] = word
    #And return the seed (=word)
    return word

然后你按它们的种子对有问题的单词列表进行排序。这可确保具有相同种子的单词彼此相邻出现。这对于您用来根据种子组成词组的group by 很重要。

distance 函数看起来很复杂，可能会被 Levenshtein 之类的东西代替。