持久化 sklearn.feature_extraction.text.TfidfVectorizer 出错答案

【问题标题】：error with persisted sklearn.feature_extraction.text.TfidfVectorizer持久化 sklearn.feature_extraction.text.TfidfVectorizer 出错
【发布时间】：2016-08-16 01:53:27
【问题描述】：

我使用模块 joblib 持久化了一个 TfidfVectorizer。我通过 fit_transform 方法运行的对象是一个字符串列表。生成的矩阵的维度为 263744 列。

我正在通过 transform 方法运行一个字符串列表，我收到以下错误。

有什么线索吗？

File "/usr/local/lib/python2.7/dist-      packages/sklearn/feature_extraction/text.py", 
line 1334, in transform
return self._tfidf.transform(X, copy=False)
File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", 
line 1037, in transform
X = X * self._idf_diag

File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", line    
318, in __mul__
return self._mul_sparse_matrix(other)
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py",
line 487, in _mul_sparse_matrix
other = self.__class__(other)  # convert to this format
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py",
line 31, in __init__
arg1 = arg1.asformat(self.format)
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", 
line 219, in asformat
return getattr(self,'to' + format)()
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/dia.py", 
line 241, in tocsr
return self.tocoo().tocsr()
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/dia.py", 
line 249, in tocoo

num_offsets, offset_len = self.data.shape
AttributeError: 'NDArrayWrapper' object has no attribute 'shape'

【问题讨论】：

标签： python scikit-learn joblib

【解决方案1】：

假设您将经过训练的转换器或管道持久化到磁盘，然后在看到错误之前重新加载它，您可以：

尝试使用compress关键字参数参数将原始（工作）对象保存为joblib.dump，并使用大于0的整数值：
```
_ = joblib.dump(python_object, persisted_file_name, compress=3)
```
如果要将持久化文件移动到新位置，请确保复制所有文件片段。如果它很大，joblib 将拆分它，例如：
```
persisted_model.joblib.pkl
persisted_model.joblib.pkl_01.npy
persisted_model.joblib.pkl_02.npy
```

joblib docs

【讨论】：