Python 3.5 多处理池 && 'numpy.int64 没有属性 .loc'

Question

我正在尝试了解多处理和池来处理我在 MySQL 数据库中获得的一些推文。这是代码和错误消息。

import multiprocessing
import sqlalchemy
import pandas as pd
import config
from nltk import tokenize as token

q = multiprocessing.Queue()
engine = sqlalchemy.create_engine(config.sqlConnectionString)


def getRow(pandasSeries):
    df = pd.DataFrame()
    tweetTokenizer = token.TweetTokenizer()
    print(pandasSeries.loc['BODY'], "\n", type(pandasSeries.loc['BODY']))
    for tokens in tweetTokenizer.tokenize(pandasSeries.loc['BODY']):
        df = df.append(pd.Series(data=[pandasSeries.loc['ID'], tokens, pandasSeries.loc['AUTHOR'],
                                  pandasSeries.loc['RETWEET_COUNT'], pandasSeries.loc['FAVORITE_COUNT'],
                                  pandasSeries.loc['FOLLOWERS_COUNT'], pandasSeries.loc['FRIENDS_COUNT'],
                                  pandasSeries.loc['PUBLISHED_AT']],
                                 index=['id', 'tweet', 'author', 'retweet', 'fav', 'followers', 'friends',
                                        'published_at']), ignore_index=True)
    df.to_sql(name="tweet_tokens", con=engine, if_exists='append')


if __name__ == '__main__':
    ##LOADING SQL INTO DATAFRAME##
    databaseData = pd.read_sql_table(config.tweetTableName, engine)

    pool = multiprocessing.Pool(6)
    for row in databaseData.iterrows():
        print(row)
        pool.map(getRow, row)
    pool.close()
    q.close()
    q.join_thread()


"""
            OUPUT

C:\Users\Def\Anaconda3\python.exe C:/Users/Def/Dropbox/Dissertation/testThreadCopy.py
(0, ID                                                              3247
AUTHOR                                             b'Elon Musk News'
RETWEET_COUNT                                                      0
FAVORITE_COUNT                                                     0
FOLLOWERS_COUNT                                                20467
FRIENDS_COUNT                                                  14313
BODY               Elon Musk Takes an Adorable 5th Grader's Idea ...
PUBLISHED_AT                                     2017-03-03 00:00:01
Name: 0, dtype: object)
Elon Musk Takes an Adorable 5th Grader's 
 <class 'str'>
multiprocessing.pool.RemoteTraceback: 

Traceback (most recent call last):
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Users\Def\Dropbox\Dissertation\testThreadCopy.py", line 16, in getRow
    print(pandasSeries.loc['BODY'], "\n", type(pandasSeries.loc['BODY']))
AttributeError: 'numpy.int64' object has no attribute 'loc'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:/Users/Def/Dropbox/Dissertation/testThreadCopy.py", line 34, in <module>
    pool.map(getRow, row)
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 608, in get
    raise self._value
AttributeError: 'numpy.int64' object has no attribute 'loc'

Process finished with exit code 1
"""

我不明白为什么它会打印出第一个系列然后崩溃？为什么它说 pandasSeries.loc['BODY'] 是 numpy.int64 类型，而打印出来的却是字符串类型？我敢肯定我在其他一些地方出错了，如果你能看到，请你指出来。 谢谢。

Answer 1

当我构造一个简单的数据框时：

frame
   0  1   2   3
0  0  1   2   3
1  4  5   6   7
2  8  9  10  11

并迭代两次我得到：

for row in databaseData.iterrows():
    for i in row:
        print(i, type(i))

该内部循环产生 2 个项目、一个行索引/标签和一个带有值的系列。

0 <class 'numpy.int64'>
0    0
1    1
2    2
3    3
Name: 0, dtype: int32 <class 'pandas.core.series.Series'>

您的map 也这样做，将数字索引发送到一个进程（产生错误），并将一个系列发送到另一个。

---

如果我使用 pool.map 而不使用 for row：

pool.map(getRow, databaseData.iterrows())

然后getRow 接收到一个 2 元素元组。

def getRow(aTuple):
    rowlbl, rowSeries = aTuple
    print(rowSeries)   
    ...

---

你的print(row) 显示了这个元组；因为系列部分是多行的，所以更难看到。如果我添加一个 \n 可能会更清晰

(0,                                   # row label
ID                               3247       # multiline Series
AUTHOR                           b'Elon Musk News'
RETWEET_COUNT                    0
....
Name: 0, dtype: object)