Pandas Dataframe：将一列拆分为多列

Question

我有一个熊猫 dataframe，其列名称为：（列类型为 Object）

1. x_id
2. y_id
3. Sentence1
4. Sentences2
5. Label

我想将sentence1 和sentence2 分成同一个dataframe 中的多个列。

这是一个示例：dataframe 名称为 df

x_id     y_id     Sentence1          Sentence2          Label
0        2        This is a ball     I hate you         0
1        5        I am a boy         Ahmed Ali          1
2        1        Apple is red       Rose is red        1
3        9        I love you so much Me too             1

将列 [Sentence1,Sentence2] 用 ' ' Space 拆分后，dataframe 看起来像：

x_id     y_id     1     2     3    4     5      6      7     8      Label
0        2        This  is    a    ball  NONE   I      hate  you    0
1        5        I     am    a    boy   NONE   Ahmed  Ali   NONE   1
2        1        Apple is    red  NONE  NONE   Rose   is    red    1
3        9        I     love  you  so    much   Me     too   NONE   1

如何在python 中拆分这样的列？如何使用pandas dataframe 做到这一点？

Answer 1

In [26]: x = pd.concat([df.pop('Sentence1').str.split(expand=True),
    ...:                df.pop('Sentence2').str.split(expand=True)],
    ...:               axis=1)
    ...:

In [27]: x.columns = np.arange(1, x.shape[1]+1)

In [28]: x
Out[28]:
       1     2    3     4     5      6     7     8
0   This    is    a  ball  None      I  hate   you
1      I    am    a   boy  None  Ahmed   Ali  None
2  Apple    is  red  None  None   Rose    is   red
3      I  love  you    so  much     Me   too  None

In [29]: df = df.join(x)

In [30]: df
Out[30]:
   x_id  y_id  Label      1     2    3     4     5      6     7     8
0     0     2      0   This    is    a  ball  None      I  hate   you
1     1     5      1      I    am    a   boy  None  Ahmed   Ali  None
2     2     1      1  Apple    is  red  None  None   Rose    is   red
3     3     9      1      I  love  you    so  much     Me   too  None

Answer 2

单热编码标注方案：

In [14]: df.Sentence1 += ' ' + df.pop('Sentence2')

In [15]: df
Out[15]:
   x_id  y_id                  Sentence1  Label
0     0     2  This is a ball I hate you      0
1     1     5       I am a boy Ahmed Ali      1
2     2     1   Apple is red Rose is red      1
3     3     9  I love you so much Me too      1

In [16]: from sklearn.feature_extraction.text import CountVectorizer

In [17]: vect = CountVectorizer()

In [18]: X = vect.fit_transform(df.Sentence1.fillna(''))

X - 是一个稀疏（节省内存）矩阵：

In [23]: X
Out[23]:
<4x17 sparse matrix of type '<class 'numpy.int64'>'
        with 19 stored elements in Compressed Sparse Row format>

In [24]: type(X)
Out[24]: scipy.sparse.csr.csr_matrix

In [19]: X.toarray()
Out[19]:
array([[0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1],
       [1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 2, 1, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1]], dtype=int64)

大部分 sklearn 方法都接受稀疏矩阵。

如果你想“解包”它：

In [21]: r = pd.DataFrame(X.toarray(), columns=vect.get_feature_names())

In [22]: r
Out[22]:
   ahmed  ali  am  apple  ball  boy  hate  is  love  me  much  red  rose  so  this  too  you
0      0    0   0      0     1    0     1   1     0   0     0    0     0   0     1    0    1
1      1    1   1      0     0    1     0   0     0   0     0    0     0   0     0    0    0
2      0    0   0      1     0    0     0   2     0   0     0    2     1   0     0    0    0
3      0    0   0      0     0    0     0   0     1   1     1    0     0   1     0    1    1

Answer 3

这是Sentence1列中的句子的操作方法。 Sentence2 列的想法是相同的。

splits = df.Sentence1.str.split(' ')
longest = splits.apply(len).max()

注意longest是最长句子的长度。现在制作 Null 列：

for j in range(1,longest+1):
    df[str(j)] = np.nan

最后，检查拆分后的值并分配它们：

for j in splits.values:
    for k in range(1,longest+1):
        try:
            df.loc[str(j), k] = j[k]
        except:
            pass

`

Answer 4

这看起来像是一个机器学习问题。以这种方式从 1 col 转换为 max words 列可能效率不高。

另一个（可能更有效）的解决方案是将每个单词转换为整数，然后填充到最长的句子。 Tensorflow 作为工具。