Pandas 将 DataTable 加入 SQL 表以防止内存错误答案

【问题标题】：Pandas Join DataTable to SQL Table to Prevent Memory ErrorsPandas 将 DataTable 加入 SQL 表以防止内存错误
【发布时间】：2020-04-01 21:31:05
【问题描述】：

所以我每个表有大约 4-5 百万行数据。我有大约 10-15 张这样的桌子。我创建了一个表，它将根据一些 ID 和快照日期将 30,000 行连接到其中的几百万行。

有没有办法将我现有的数据表写入 SQL 查询，它会为我过滤结果，这样我就不必将整个表加载到内存中？

目前我一直在一次加载每个表，然后释放内存。但是，它仍然会占用我计算机上 100% 的内存。

    for table in tablesToJoin:
        if df is not None:
            print("DF LENGTH", len(df))

        query = """SET NOCOUNT ON; SELECT * FROM """ + table + """ (nolock) where snapshotdate = '"""+ date +"""'"""
        query += """ SET NOCOUNT OFF;"""

        start = time.time()
        loadedDf = pd.read_sql_query(query, conn)
        if df is None:
            df = loadedDf
        else:
            loadedDf.info(verbose=True, null_counts=True)
            df.info(verbose=True, null_counts=True)
            df = df.merge(loadedDf, how='left', on=["MemberID", "SnapshotDate"])
            #df = df.fillna(0)
            print("DATA AFTER ALL MERGING", len(df))
        print("Length of data loaded:", len(loadedDf))
        print("Time to load data from sql", (time.time() - start))

【问题讨论】：

标签： python sql pandas

【解决方案1】：

我曾经遇到过和你一样的问题。我的解决方案是尽可能在 SQL 层进行过滤。因为我没有你的代码和你的数据库，所以我在下面写的是未经测试的代码，很可能包含错误。您必须根据需要进行更正。

我们的想法是尽可能少地从数据库中读取。 pandas 并非旨在分析数百万行的帧（至少在典型计算机上）。为此，请将过滤条件从 df 传递给您的数据库调用：

from sqlalchemy import MetaData, and_, or_

engine = ... # construct your SQL Alchemy engine. May correspond to your `conn` object
meta = MetaData()
meta.reflect(bind=engine, only=tablesToJoin)


for table in tablesToJoin:
    t = meta[table]
    # Building the WHERE clause. This is equivalent to:
    #     WHERE     ((MemberID = <MemberID 1>) AND (SnapshotDate = date))
    #            OR ((MemberID = <MemberID 2>) AND (SnapshotDate = date))
    #            OR ((MemberID = <MemberID 3>) AND (SnapshotDate = date))
    cond = _or(**[and_(t.c['MemberID'] == member_id, t.c['SnapshotDate'] == date) for member_id in df['MemberID'] ])

    # Be frugal here: only get the columns that you need, or you will blow your memory
    # If you specify None, it's equivalent to a `SELECT *`
    statement = t.select(None).where(cond)

    # Note that it's `read_sql`, not `read_sql_query` here
    loadedDf = pd.read_sql(statement, engine)

    # loadedDf should be much smaller now since you have already filtered it at the DB level
    # Now do your joins...

【讨论】：