我可以将 executemany 用于带有 sqlite3 的大型批处理吗？答案

【问题标题】：Can I use executemany for a large batch process with sqlite3?我可以将 executemany 用于带有 sqlite3 的大型批处理吗？
【发布时间】：2021-11-05 18:28:54
【问题描述】：

我有一个相当长的 python 进程，它旨在重新处理一个表中的大量数据，提取必要的部分，分配适当的值并将其重新输入到另一个表中。这个过程很好！除了它非常耗时。我想知道是否有办法使用 executemany 方法来修改它？这些是整个过程的 sn-ps，但描述了我希望修改为 executemany 的位置

以下代码运行一系列 if/then 语句，以将适当的值分配给检索到的信息。

    # Current crawl begin date
    cur.execute('SELECT Crawl_Begin_Date FROM Crawl WHERE Crawl_ID=?', current_crawl_ID)
    current_crawl_begin_date = cur.fetchone()
    current_crawl_begin_date = current_crawl_begin_date[0]
    
    # Current crawl end date
    cur.execute('SELECT Crawl_End_Date FROM Crawl WHERE Crawl_ID=?', current_crawl_ID)
    current_crawl_end_date = cur.fetchone()
    current_crawl_end_date = current_crawl_end_date[0]
    
    # URL_Crawl table where Crawl_ID == current crawl
    sql = 'SELECT URL_Crawl_ID, Last_Updated, Last_Published, Date_of_HTML, Unique_URL_ID FROM URL_Crawl WHERE Crawl_ID=%s'
    current_crawl = pd.read_sql_query(sql %(current_crawl_ID), con=db)

    # num keeps track of number of rows read through from current_crawl (number of nodes in current crawl)
    num = 1

    # For every unique url in the current crawl
    for row in current_crawl.itertuples():
        
        # Calculate max date .......................................................
        if ((row.Last_Updated == None) | (row.Last_Updated == '')):
            last_updated = '0'
        else:
            last_updated = row.Last_Updated
        if ((row.Last_Published == None) | (row.Last_Published == '')):
            last_published = '0'
        else:
            last_published = row.Last_Published
        if ((row.Date_of_HTML == None) | (row.Date_of_HTML == '')):
            date_of_html = '0'
        else:
            date_of_html = row.Date_of_HTML
            
        if ((last_updated >= last_published) & (last_updated >= date_of_html)):
            max_date = last_updated
        elif ((last_published >= last_updated) & (last_published >= date_of_html)):
            max_date = last_published
        elif ((date_of_html >= last_updated) & (date_of_html >= last_published)):
            max_date = date_of_html
        # ..........................................................................
        
        # Set remaining variables from current_crawl dateframe
        url_crawl_id = row.URL_Crawl_ID
        unique_url_id = row.Unique_URL_ID

        # Initialize starting and end dates/statuses with None
        starting_date = None
        starting_date_status = None
        ending_date = None
        ending_date_status = None
        
        # URL_Crawl table up until (but not including) current crawl
        sql2 = 'SELECT URL_Crawl_ID, Last_Updated, Last_Published, Date_of_HTML, Unique_URL_ID FROM URL_Crawl WHERE Crawl_ID<%s'
        previous_crawls = pd.read_sql_query(sql2 %(current_crawl_ID), con=db)

        # If row's unique_url_id exists in previous crawls (not a new node)
        if (unique_url_id in (previous_crawls['Unique_URL_ID']).tolist()):
            
            # Situation B ...................................................
            
            # Finding most recent lifetime of existing node
            existing = previous_crawls[previous_crawls['Unique_URL_ID'] == unique_url_id]
            existing_url_crawl_ids = (existing.URL_Crawl_ID).tolist()
            
            existing_in_lifetime = pd.DataFrame()
            
            for i in existing_url_crawl_ids:
                sql3 = 'SELECT * FROM Lifetime WHERE URL_Crawl_ID=%d'
                exist_in_lt = pd.read_sql_query(sql3 %(i), con=db)
                existing_in_lifetime = existing_in_lifetime.append(exist_in_lt, ignore_index=True)
            
            most_recent_lifetime = existing_in_lifetime[existing_in_lifetime.Lifetime_ID == existing_in_lifetime.Lifetime_ID.max()]
                
            # Dates/statuses from most recent lifetime - convert to Strings
            most_recent_starting_date = ((most_recent_lifetime.Starting_Date).tolist())[0]
            most_recent_starting_date_status = ((most_recent_lifetime.Starting_Date_Status).tolist())[0]
            most_recent_ending_date = ((most_recent_lifetime.Ending_Date).tolist())[0]
            most_recent_ending_date_status = ((most_recent_lifetime.Ending_Date_Status).tolist())[0]
            most_recent_lifetimeID = ((most_recent_lifetime.Lifetime_ID).tolist())[0]
                
            if (max_date != '0'):
                if ((max_date >= current_crawl_begin_date) & (max_date <= current_crawl_end_date)):
                    # Situation B.2
                    ending_date = max_date
                    ending_date_status = "Exact"
                    cur.execute("""UPDATE Lifetime SET Ending_Date=?, Ending_Date_Status=? 
                                WHERE Lifetime_ID=?""", (ending_date, ending_date_status, most_recent_lifetimeID))
                    starting_date = max_date
                    starting_date_status = "Exact"
                    ending_date = None
                    ending_date_status = None
                    cur.execute("""INSERT INTO Lifetime VALUES (null, ?, ?, ?, ?, ?)
                                """, (starting_date, ending_date, starting_date_status, ending_date_status, url_crawl_id))
                elif ((max_date < current_crawl_begin_date) & (max_date > most_recent_starting_date)):
                    # Situation B.3
                    ending_date = max_date
                    ending_date_status = "Exact"
                    cur.execute("""UPDATE Lifetime SET Ending_Date=?, Ending_Date_Status=? 
                                WHERE Lifetime_ID=?""", (ending_date, ending_date_status, most_recent_lifetimeID))
                    starting_date = max_date
                    starting_date_status = "Exact"
                    ending_date = current_crawl_begin_date
                    ending_date_status = "Estimated"
                    cur.execute("""INSERT INTO Lifetime VALUES (null, ?, ?, ?, ?, ?)
                                """, (starting_date, ending_date, starting_date_status, ending_date_status, url_crawl_id))
                elif (max_date == most_recent_starting_date):
                    # Situation B.4
                    ending_date = current_crawl_begin_date
                    ending_date_status = "Estimated"
                    cur.execute("""UPDATE Lifetime SET Ending_Date=?, Ending_Date_Status=? 
                                WHERE Lifetime_ID=?""", (ending_date, ending_date_status, most_recent_lifetimeID))
                elif ((max_date > current_crawl_end_date) | (max_date < most_recent_starting_date)):
                    # Situation B.1
                    max_date = '0'
            if (max_date == '0'):
                # Situation B.5
                ending_date = current_crawl_begin_date
                ending_date_status = "Estimated"
                cur.execute("""UPDATE Lifetime SET Ending_Date=?, Ending_Date_Status=? 
                            WHERE Lifetime_ID=?""", (ending_date, ending_date_status, most_recent_lifetimeID))
                    
        # If row's unique_url_id is a new node (not seen in previous crawls)
        else:
            
            # Situation A ...................................................
            
            if (max_date != '0'):
                if ((max_date >= current_crawl_begin_date) & (max_date <= current_crawl_end_date)):
                    # Situation A.2
                    starting_date = max_date
                    starting_date_status = "Exact"
                elif (max_date < current_crawl_begin_date):
                    # Situation A.3
                    starting_date = max_date
                    starting_date_status = "Exact"
                    ending_date = current_crawl_begin_date
                    ending_date_status = "Estimated"
                elif (max_date > current_crawl_end_date):
                    # Situation A.1
                    max_date = '0'
            if (max_date == '0'):
                # Situation A.4
                starting_date = current_crawl_end_date
                starting_date_status = "Estimated"
        
            cur.execute("""INSERT INTO Lifetime VALUES (null, ?, ?, ?, ?, ?)
                        """, (starting_date, ending_date, starting_date_status, ending_date_status, url_crawl_id))

executemany 可以用于此功能吗？如果是这样，我不知道 executemany 的适当语法 - 我已经尝试了一些还没有奏效的东西。数据库是 SQLite，程序是基于 python 的。

【问题讨论】：

executemany 对数据序列进行操作。你能指出那个顺序应该是什么吗？
将 executemany 与 UPDATE 一起使用 -- stackoverflow.com/questions/33793952/…
抱歉，我添加了更多显示序列的更多代码，我相信 - 该过程贯穿数据集中的每个 url_crawl_id。
@KlausD。我刚刚再次更新它，最初的部分并不是真正的问题所在 - 这是它遍历所有 URL_ID 和生命周期 ID 的部分。
似乎使用了自动提交，以便每个 INSERT 或 UPDATE 都是它自己的事务。仅在 e 之后使用 BEGIN 和 COMMIT。 G。每 1000 次更改应该会快得多。

标签： python sqlite executemany

【解决方案1】：

如果不完全理解您的代码，很难给出准确的答案。我不太明白你在哪里迭代 urls/ids/etc。您将需要在循环外创建一个更新列表和一个插入列表，然后在其相应列表中累积参数序列。最后，在循环之后，您将使用要执行的固定 SQL 将每个列表传递给 executemany。

这应该让您了解它如何与循环/迭代一起工作。


    #...

    # These are each a list of tuples/lists
    # ie. [(param0, ..., paramN), ..., (param0, ..., paramN)]
    params_to_update = []
    params_to_insert = []

    # For every unique url in the current crawl
    for row in current_crawl.itertuples():

        #...

            if (max_date != '0'):
                if ((max_date >= current_crawl_begin_date) & (max_date <= current_crawl_end_date)):
                    # Situation B.2
                    ending_date = max_date
                    ending_date_status = "Exact"
                    params_to_update.append((ending_date, ending_date_status, most_recent_lifetimeID))
                    starting_date = max_date
                    starting_date_status = "Exact"
                    ending_date = None
                    ending_date_status = None
                    params_to_insert.append((starting_date, ending_date, starting_date_status, ending_date_status, url_crawl_id))
                elif ((max_date < current_crawl_begin_date) & (max_date > most_recent_starting_date)):
                    # Situation B.3
                    ending_date = max_date
                    ending_date_status = "Exact"
                    params_to_update.append((ending_date, ending_date_status, most_recent_lifetimeID))
                    starting_date = max_date
                    starting_date_status = "Exact"
                    ending_date = current_crawl_begin_date
                    ending_date_status = "Estimated"
                    params_to_insert.append((starting_date, ending_date, starting_date_status, ending_date_status, url_crawl_id))

    # After for loop is done.
    # Call UPDATE for each sequence of params in this list.
    UPDATE_SQL = """UPDATE Lifetime SET Ending_Date=?, Ending_Date_Status=? WHERE Lifetime_ID=?"""
    cur.executemany(UPDATE_SQL, params_to_update)
    # Call INSERT for each sequence of params in this list.
    INSERT_SQL = """INSERT INTO Lifetime VALUES (null, ?, ?, ?, ?, ?)"""
    cur.executemany(INSERT_SQL, params_to_insert)

stackoverflow.com：using-executemany-to-update-entries-in-an-existing-sqlite3-database-using-pyt

docs.python.org：python doc executemany example

【讨论】：

感谢您的回复！我想我现在更好地理解它了。我不想发布整个代码，因为它很麻烦。但是，我修改了最初的帖子以包含整个内容。有了这个，那么，如果代码在多个场景中运行，那么每个场景在迭代之后是否应该是它自己的更新/插入语句？
@MeredithAbrams 看起来你会在这个循环中积累参数序列——for row in current_crawl.itertuples():。在该循环之后，每个唯一语句都需要单独执行。在您的示例中，两个插入和两个更新似乎是相同的（在 if max_date ... elif 块中），所以我将它们分组。如果语句相同，您可以对更多块执行此操作。
似乎在您的所有块中，您的INSERTs 和您的UPDATEs 都相同。所以你可以继续追加到相应的列表，然后在最后将它传递给 executemany。
我明白了，这真的很有帮助！谢谢！从理论上讲，这应该加快进程，对吗？目前平均每分钟只有大约 30 条记录。
感谢您抽出宝贵时间！我能够将该段移出循环，它现在运行得更快。