在多处理代码中使用 pyodbc

Question

我正在使用pyodbc 构建一个应用程序，该应用程序可以通过多进程方法访问和操作SQL Server 2008 R2。

我首先设计了一个DataEngine类如下：

class DataEngine:
    def __init__(self, serverName, databaseName):
        self.serverName = serverName
        self.databaseName = databaseName
        self.cn = pyodbc.connect("driver={SQL Server};server="+ serverName + ";database=" + databaseName + ";trusted_connection=true") 
        self.cursor = self.cn.cursor()

然后是DataMapper 类：

class DataMapper:
    def __init__(self, dataEngine):
        self.dataEngine = dataEngine    
        self.cursor = dataEngine.get_cursor()    

我已经定义了一个函数，它使用参数调用 .exe 文件：

def func(dataEngine, dataMapper, buildRunId, queueNumber, ServerName, DatabaseName, ExecutableFilePath, DLLFilePath):
    os.system(ExecutableFilePath + " " + str(inputRowId) + " " + str(ServerName) + " " + str(DatabaseName) +" " + DLLFilePath)

那么函数func已经被包装成四个独立的进程：

queue_0 = Process(target = func, args = (dataEngine, dataMapper, buildRunId, 0, ServerName, DatabaseName))
queue_1 = Process(target = func, args = (dataEngine, dataMapper, buildRunId, 1, ServerName, DatabaseName))
queue_2 = Process(target = func, args = (dataEngine, dataMapper, buildRunId, 2, ServerName, DatabaseName))
queue_3 = Process(target = func, args = (dataEngine, dataMapper, buildRunId, 3, ServerName, DatabaseName))

queue_0.start()
queue_1.start()
queue_2.start()
queue_3.start()

queue_0.join()
queue_1.join()
queue_2.join()
queue_3.join()

但是，当上述代码运行时，我收到以下错误消息：

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 347, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 1083, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: object.__new__(pyodbc.Cursor) is not safe, use pyodbc.Cursor.__new__()
Exception AttributeError: "DataEngine instance has no attribute 'cursor'" in <bound method DataEngine.__del__ of <dataengine.DataEngine instance at 0x000000000451FC48>> ignored

我可以知道如何处理这个问题吗？

谢谢。

Answer 1

您需要为每个multiprocessing.Process 实例建立单独的数据库连接。您存储在每个 DataEngine 实例中的连接和游标对象是不可挑选的，因此您不能在进程之间发送它们。最简单的做法是调用在子进程中创建dataEngine 实例的函数，然后调用MatlabBuildIndex：

def create_conn_and_call_buildindex(run_id, x, server_name, database_name):
    data_engine = DataEngine(server_name, database_name)
    data_mapper = DataMapper(data_engine)
    MatlabBuildIndex(data_engine, data_mapper, run_id, x, server_name, database_name)

...

queue_0 = Process(target=create_conn_and_call_buildindex, args = (buildRunId, 0, ServerName, DatabaseName))
...