在python中跨多个进程同步

Question

我有一个 python 应用程序，它产生一个单独的进程来做一些工作（由于 GIL（全局解释器锁），我在使用线程时遇到了性能问题）。 现在我在 python 中的跨进程同步共享资源的方法是什么？

我将数据移动到一个队列中，当它从该队列接收数据时，生成进程会完成工作。但我需要能够保证数据以有序的方式输出，与复制的顺序相同，因此我需要保证任何时候只有一个进程可以读取/写入队列。 我怎样才能做到最好？

谢谢， 罗恩

Answer 1

我认为您需要一个信号量，请查看此示例代码：

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        pool.acquire()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)
        pool.release()


pool = threading.BoundedSemaphore(value=1)


for i in range(10):
        t = ThreadClass()
        t.start()

有这个输出：

Thread-1 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-2 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-3 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-4 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-5 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-6 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-7 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-8 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-9 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-10 says hello, World! at time: 2013-05-20 18:57:47.609000

在哪里：

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)




for i in range(10):
        t = ThreadClass()
        t.start()

有这个输出：

Thread-1 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-2 says hello, World! at time: 2013-05-20 18:58:05.
531000

 Thread-4 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-3 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-6 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-5 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-8 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-7 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-10 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-9 says hello, World! at time: 2013-05-20 18:58:0
5.531000

因此，使用 python 中的 BoundedSemaphore，您可以确保在任何人写入您的队列之前，他们必须拥有该信号量。但是，这并不能确保您的结果以正确的顺序添加到队列中。

编辑：

如果您要这样做并保持秩序，您将需要这样的东西：

import multiprocessing
import datetime
import random
import time

def funfun(number):
    time.sleep(random.randint(0,10))
    now = datetime.datetime.now()
    return "%s says hello, World! at time: %s"  % (number,now)

if __name__ == "__main__":
    pool = multiprocessing.Pool(10)
    for item in pool.imap(funfun,[i for i in range(10)]):
        print item

哪个打印：

0 says hello, World! at time: 2013-05-21 00:38:48.546000
1 says hello, World! at time: 2013-05-21 00:38:55.562000
2 says hello, World! at time: 2013-05-21 00:38:47.562000
3 says hello, World! at time: 2013-05-21 00:38:51.578000
4 says hello, World! at time: 2013-05-21 00:38:50.578000
5 says hello, World! at time: 2013-05-21 00:38:48.593000
6 says hello, World! at time: 2013-05-21 00:38:52.593000
7 says hello, World! at time: 2013-05-21 00:38:48.593000
8 says hello, World! at time: 2013-05-21 00:38:50.593000
9 says hello, World! at time: 2013-05-21 00:38:51.609000

因此，您可以按正确的顺序追加到队列中，作业将等待轮到他们添加到队列中。