如何在 python 中对 SQL 查询进行多线程处理，以便获得所有查询的结果

Question

有没有办法使用线程同时执行 SQL 查询，这样我就可以减少下面代码的处理时间？有没有更好的方法可以在不使用 pandas 模块的情况下执行与以下相同的结果？鉴于我正在使用的数据集的大小，我无法将整个数据集存储在内存中，并且我发现循环遍历 SELECT * FROM 语句的行并将它们与我正在查询的列表进行比较会增加处理时间。

# DATABASE layout
#  _____________________________________________________________
# |     id      |         name       |        description       |
# |_____________|____________________|__________________________|
# |        1    |         John       |       Credit Analyst     |
# |        2    |         Jane       |          Doctor          |
# |      ...    |          ...       |            ...           |
# |  5000000    |       Mohammed     |         Dentist          |
# |_____________|____________________|__________________________|

import sqlite3


SEARCH_IDS = [x for x in range(15000)]
DATABASE_NAME = 'db.db'

def chunks(wholeList, chunkSize=999):
    """Yield successive n-sized chunks from wholeList."""
    for i in range(0, len(wholeList), chunkSize):
        yield wholeList[i:i + chunkSize] 

def search_database_for_matches(listOfIdsToMatch):
    '''Takes a list of ids and returns the rows'''
    conn = sqlite3.connect(DATABASE_NAME)
    cursor = conn.cursor()
    sql = "SELECT id, name, description FROM datatable WHERE id IN ({})".format(', '.join(["?" for x in listOfIdsToMatch]))
    cursor.execute(sql,tuple(listOfIdsToMatch))
    rows = cursor.fetchall()
    return rows

def arrange(orderOnList,listToBeOrdered,defaultReturnValue='N/A'):
    '''Takes a list of ids in the desired order and list of tuples which have ids as the first items.
       the list of tuples is aranged into a new list corresponding to the order of the source list'''
    from collections import OrderedDict
    resultList=[defaultReturnValue for x in orderOnList]
    indexLookUp = OrderedDict( [ ( value , key )   for   key , value   in enumerate( orderOnList ) ] )
    for item in listToBeOrdered:
        resultList[indexLookUp[item[0]]]=item
    return resultList


def main():
    results=[]
    for chunk in chunks(SEARCH_IDS,999):
        results += search_database_for_matches(chunk)
    results = arrange(SEARCH_IDS,results)
    print(results)


if __name__ == '__main__': main()

Answer 1

一些建议：

您应该使用分页，而不是使用迭代器通过卡盘读取记录。

查看以下问题：

• Efficient paging in SQLite with millions of records
• Sqlite LIMIT / OFFSET query

如果您使用多线程/多处理，请确保您的数据库可以支持它。 见：SQLite And Multiple Threads

要实现你想要的，你可以使用一个工作在每个块上的工人池。请参阅 Python 文档中的 Using a pool of workers。

例子：

Import multiprocessing 

with multiprocessing.pool.Pool(process = 4) as pool:
    result = pool.map(search_database_for_match, [for chunk in chunks(SEARCH_IDS,999)])