带有熊猫数组的hstack csr矩阵

Question

我正在做一个关于亚马逊评论的练习，下面是代码。 基本上我无法将列（熊猫数组）添加到应用 BoW 后得到的 CSR 矩阵。 即使两个矩阵中的行数匹配，我也无法通过。

import sqlite3
import pandas as pd
import numpy as np
import nltk
import string
import matplotlib.pyplot as plt
import seaborn as sns
import scipy
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import confusion_matrix
from sklearn import metrics
from sklearn.metrics import roc_curve, auc
from nltk.stem.porter import PorterStemmer
from sklearn.manifold import TSNE

#Create Connection to sqlite3
con = sqlite3.connect('C:/Users/609316120/Desktop/Python/Amazon_Review_Exercise/database/database.sqlite')

filtered_data = pd.read_sql_query("""select * from Reviews where Score != 3""", con)
def partition(x):
    if x < 3:
       return 'negative'
    return 'positive'

actualScore = filtered_data['Score']
actualScore.head()
positiveNegative = actualScore.map(partition)
positiveNegative.head(10)
filtered_data['Score'] = positiveNegative
filtered_data.head(1)
filtered_data.shape

display = pd.read_sql_query("""select * from Reviews where Score !=3 and Userid="AR5J8UI46CURR" ORDER BY PRODUCTID""", con)

sorted_data = filtered_data.sort_values('ProductId', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')

final=sorted_data.drop_duplicates(subset={"UserId","ProfileName","Time","Text"}, keep='first', inplace=False)

final.shape

display = pd.read_sql_query(""" select * from reviews where score != 3 and id=44737 or id = 64422 order by productid""", con)

final=final[final.HelpfulnessNumerator<=final.HelpfulnessDenominator]

final['Score'].value_counts()

count_vect = CountVectorizer()

final_counts = count_vect.fit_transform(final['Text'].values)

final_counts.shape

type(final_counts)

positive_negative = final['Score']

#Below is giving error
final_counts = hstack((final_counts,positive_negative))

Answer 1

即使是稀疏矩阵，我也面临同样的问题。您可以通过todense() 将 CSR 矩阵转换为密集矩阵，然后您可以使用 np.hstack((dataframe.values,converted_dense_matrix))。它会正常工作。您无法使用 numpy.hstack
处理稀疏矩阵 然而，对于非常大的数据集，转换为密集矩阵并不是一个好主意。在您的情况下，scipy hstack 将不起作用，因为 hstack(int,object) 中的数据类型不同。 尝试 positive_negative = final['Score'].values 和 scipy.sparse.hstack 它。如果它不起作用，你能给我你的 positive_negative.dtype 的输出

Answer 2

Used Below but still getting error

merged_data = scipy.sparse.hstack((final_counts, scipy.sparse.coo_matrix(positive_negative).T))

Below is the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sparse' is not defined
>>> merged_data = scipy.sparse.hstack((final_counts, sparse.coo_matrix(positive_
negative).T))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sparse' is not defined
>>> merged_data = scipy.sparse.hstack((final_counts, scipy.sparse.coo_matrix(pos
itive_negative).T))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python34\lib\site-packages\scipy\sparse\construct.py", line 464, in h
stack
    return bmat([blocks], format=format, dtype=dtype)
  File "C:\Python34\lib\site-packages\scipy\sparse\construct.py", line 600, in b
mat
    dtype = upcast(*all_dtypes) if all_dtypes else None
  File "C:\Python34\lib\site-packages\scipy\sparse\sputils.py", line 52, in upca
st
    raise TypeError('no supported conversion for types: %r' % (args,))
TypeError: no supported conversion for types: (dtype('int64'), dtype('O'))

Answer 3

sparse.hstack 将输入的coo 格式矩阵组合成一个新的coo 格式矩阵。

final_counts 是一个csr 矩阵，所以sparse.coo_matrix(final_counts) 转换很简单。

positive_negative 是 DataFrame 的一列。看看

 sparse.coo_matrix(positive_negative)

它可能是一个 (1,n) 稀疏矩阵。但是要将它与final_counts 结合起来，它需要是 (1,n) 形的。

尝试创建稀疏矩阵，并将其转置：

sparse.hstack((final_counts, sparse.coo_matrix(positive_negative).T))