python - 用于池映射的字典迭代器

Question

我正在处理一组frozensets。我正在尝试为字典“输出”中的每个冻结集找到最小集。我有 70k freezesets，所以我正在制作这个frozenset 字典并并行化这个任务。当我尝试将此字典作为输入传递给我的函数时，只发送密钥，所以我收到错误，有人可以帮我找出问题所在。

output => {frozenset({'rfid', 'zone'}): 0, frozenset({'zone'}): 0, frozenset({'zone', 'time'}): 0}

def reduce(prob,result,output):
    print(output)
    for k in output.keys():
    #Function to do something


def reducer(prob,result,output):
    print(output)
    p = Pool(4) #number of processes = number of CPUs
    func2 = partial(reduce,prob,result)
    reduced_values= p.map( func2,output,chunksize=4)
    p.close() # no more tasks
    p.join()  # wrap up current tasks
    return reduced_values

if __name__ == '__main__':
    final = reducer(prob,result,output)

{frozenset({'rfid', 'zone'}): 0, frozenset({'zone'}): 0, frozenset({'zone', 'time'}): 0}
frozenset({'rfid', 'zone'}) 
Error : AttributeError: 'frozenset' object has no attribute 'keys'

已按要求更新

from multiprocessing import Pool
from functools import partial
import itertools

output = {frozenset({'rfid', 'zone'}): 0, frozenset({'zone'}): 0, frozenset({'zone', 'time'}): 0}
prob = {'3': 0.3, '1': 0.15, '2': 0.5, '4': 0.05}
result = {'2': {frozenset({'time', 'zone'}), frozenset({'time', 'rfid'})}, '3': {frozenset({'time', 'rfid'}), frozenset({'rfid', 'zone'})}}

def reduce(prob,result,output):
    print(output)
    for k in output.keys():
        for ky,values in result.items():
            if any(k>=l for l in values):
                output[k] += sum((j for i,j in prob.items() if i == ky))
    return output


def reducer(prob,result,output):
    print(output)
    p = Pool(4) #number of processes = number of CPUs
    func2 = partial(reduce,prob,result)
    reduced_values= p.map( func2,output,chunksize=4)
    p.close() # no more tasks
    p.join()  # wrap up current tasks
    return reduced_values

if __name__ == '__main__':
    final = reducer(prob,result,output)


{frozenset({'zone', 'rfid'}): 0, frozenset({'zone'}): 0, frozenset({'time', 'zone'}): 0}
    for k in output.keys():
AttributeError: 'frozenset' object has no attribute 'keys'
frozenset({'zone', 'rfid'})

来自控制台的完整错误详情：

{frozenset({'zone', 'time'}): 0, frozenset({'zone', 'rfid'}): 0, frozenset({'zone'}): 0}
frozenset({'zone', 'time'})
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "F:\Python34\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "F:\Python34\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Users\Dell\workspace\key_mining\src\variable.py", line 16, in reduce
    for k in output.keys():
AttributeError: 'frozenset' object has no attribute 'keys'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\***\variable.py", line 33, in <module>
    final = reducer(prob,result,output)
  File "C:\***\variable.py", line 27, in reducer
    reduced_values= p.map( func2,output,chunksize=4)
  File "F:\Python34\lib\multiprocessing\pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "F:\Python34\lib\multiprocessing\pool.py", line 599, in get
    raise self._value
AttributeError: 'frozenset' object has no attribute 'keys'

Answer 1

问题是您将dict 对象传递给map。当map 迭代output 中的项目时，它会这样做：

for key in output:  # When you iterate over a dictionary, you just get the keys.
    func2(key)

因此，每次调用 func2 时，output 中包含的所有内容都是字典中的一个键（frozenset）。

根据您上面的 cmets，您似乎想将整个字典传递给 func2，但如果您这样做，您实际上根本没有并行执行任何操作。我想也许你认为这样做

pool.map(func2, output, chunksize=4)

将导致output 字典被拆分为四个字典，每个块被传递给func2 的一个实例。但事实并非如此。相反，字典中的每个键都是单独发送的 func2。

chunksize 只是用来告诉pool 有多少output 的元素一次通过进程间通信发送给每个子进程。它仅用于内部目的；无论您使用什么chunksize，func2 都只会被output 的单个元素调用。

如果你想真正传递字典的块，你需要做这样的事情：

# Break the output dict into 4 lists of (key, value) pairs
items = list(output.items())
chunksize = 4
chunks = [items[i:i + chunksize ] for i in range(0, len(items), chunksize)]
reduced_values= p.map(func2, chunks)

这会将(key, value) 元组的列表从output 字典传递到func2。然后，在func2 中，您可以将列表转回字典：

def reduce(prob,result,output):
    output = dict(item for item in output)  # Convert back to a dict
    print(output)
    ...

Answer 2

问题是，您正试图在output.keys() 中访问frozenset.keys。

set 和frozenset 用于执行设置操作，如UNION、INTERSECTION 等。

当您在 frozenset 中传递 arg 时，它会将其转换为 frozenset 对象，并且当您在 reducer you passed output it might have onlyoutput 中传递 output 时，frozenset 没有您试图访问 @for k in output.keys(): 的方法 for k in output.keys(): .keyslike[frozenset({...}), frozenset({...})], when you try to accessoutput.keys()meansfrozenset({...}).keys`它给出错误。