如何优化 2 个元组列表的组合并删除它们的重复项？答案

【问题标题】：How to optimize the combination of 2 lists of tuples and remove their duplicates?如何优化 2 个元组列表的组合并删除它们的重复项？
【发布时间】：2013-03-03 08:38:22
【问题描述】：

从这里，How do I remove element from a list of tuple if the 2nd item in each tuple is a duplicate?，我可以从 1 个元组列表中删除一个元组中第二个元素的重复项。

假设我有 2 个元组列表：

alist = [(0.7897897,'this is a foo bar sentence'),
(0.653234, 'this is a foo bar sentence'),
(0.353234, 'this is a foo bar sentence'),
(0.325345, 'this is not really a foo bar'),
(0.323234, 'this is a foo bar sentence'),]

blist = [(0.64637,'this is a foo bar sentence'),
(0.534234, 'i am going to foo bar this sentence'),
(0.453234, 'this is a foo bar sentence'),
(0.323445, 'this is not really a foo bar')]

如果第二个元素相同（score_from_alist * score_from_blist），我需要合并分数并获得所需的输出：

clist = [(0.51,'this is a foo bar sentence'), # 0.51 = 0.789 * 0.646
(0.201, 'this is not really a foo bar')] # 0.201  = 0.325 * 0.323

目前，我正在通过这样做来实现 clist，但是当我的 alist 和 blist 有大约 5500 多个元组时，它需要 5 多秒，其中第二个元素每个大约有 20-40 个单词。有什么方法可以让下面的函数更快？

def overlapMatches(alist, blist):
    start_time = time.time()
    clist = []
    overlap = set()
    for d in alist:
        for dn in blist:
            if d[1] == dn[1]:
                score = d[0]*dn[0]
                overlap.add((score,d[1]))
    for s in sorted(overlap, reverse=True)[:20]:
        clist.append((s[0],s[1]))
    print "overlapping matches takes", time.time() - start_time 
    return clist

【问题讨论】：

曾经考虑过字典吗？

标签： python list duplicates tuples

【解决方案1】：

我会使用字典/集合来消除重复并提供快速查找：

alist = [(0.7897897,'this is a foo bar sentence'),
(0.653234, 'this is a foo bar sentence'),
(0.353234, 'this is a foo bar sentence'),
(0.325345, 'this is not really a foo bar'),
(0.323234, 'this is a foo bar sentence'),]

blist = [(0.64637,'this is a foo bar sentence'),
(0.534234, 'i am going to foo bar this sentence'),
(0.453234, 'this is a foo bar sentence'),
(0.323445, 'this is not really a foo bar')]

bdict = {k:v for v,k in reversed(blist)}
clist = []
cset = set()
for v,k in alist:
   if k not in cset:
      b = bdict.get(k, None)
      if b is not None:
        clist.append((v * b, k))
        cset.add(k)
print(clist)

这里，blist 用于消除每个句子的第一次出现以外的所有内容，并提供逐句快速查找。

如果您不关心clist 的顺序，您可以稍微简化结构：

bdict = {k:v for v,k in reversed(blist)}
cdict = {}
for v,k in alist:
   if k not in cdict:
      b = bdict.get(k, None)
      if b is not None:
        cdict[k] = v * b
print(list((k,v) for v,k in cdict.items()))

【讨论】：

这里有一个问题：bdict = {k:v for v,k in reversed(blist)} 当'this is a foo bar sentence' 有两句话时，那么只有最新的一个会在dict 中，它的值，在dict-comprehension 或construction 中你不能为同一个键分配多个值，您需要迭代元组列表并追加，最好使用defaultdict(list)
@InbarRose：保留第一个（由于reversed()）并且是设计使然。如果我正确理解了 OP，他/她只希望使用第一个条目。
因为在 k,v => v,k 转换之前已经被 reversed() 预排序了。
我在 OP 的任何地方都没有看到这种愿望，只是应该添加来自 Alist 和 Blist 的重复句子。
设法将此功能降低到 0.015 秒。感谢您提供有关使用 dict/set 而不是列表的提示，它是通过列表的顺序循环导致大量浪费循环。

【解决方案2】：

假设具有最高第一项的元组在单个列表中存在重复项，假设它按元组中的第一项按降序排序，如果对应的第二项在一个列表中，则合并两个列表的分数元组是一样的：

# remove duplicates (take the 1st item among duplicates)
a, b = [{sentence: score for score, sentence in reversed(lst)}
        for lst in [alist, blist]]

# merge (leave only tuples that have common 2nd items (sentences))
clist = [(a[s]*b[s], s) for s in a.viewkeys() & b.viewkeys()]
clist.sort(reverse=True) # sort by (score, sentence) in descending order
print(clist)

输出：

[(0.510496368389, 'this is a foo bar sentence'),
 (0.10523121352499999, 'this is not really a foo bar')]

【讨论】：