Python - 管道的 Numpy 问题

Question

我已经构建了一个神经网络，它可以很好地处理较小的数据集，例如 300,000 个已知良好行和 70,000 个可疑行。我决定将已知良好的大小增加到 650 万行，但在内存方面遇到了一些错误，所以我决定尝试使用管道并运行数据帧。我有 2 个分类变量和 1 和 0 的因变量列。开始数据集如下所示：

DBF2
   ParentProcess                   ChildProcess               Suspicious
0  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
1  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
2  C:\Windows\System32\svchost.exe                      ...            1
3  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
4  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
5  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0

这是可行的，但是当我的数组变得太大时，它超出了内存：

X = DBF2.iloc[:, 0:2].values
y = DBF2.iloc[:, 2].values
#Encoding categorical data
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
#Label Encode destUserName
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
#Label Encode Parent Process
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])

#Create dummy variables
onehotencoder = OneHotEncoder(categorical_features = [0,1])
X = onehotencoder.fit_transform(X).toarray()

由于巨大的稀疏矩阵而得到这个内存错误：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py", line 947, in toarray
    out = self._process_toarray_args(order, out)
  File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", line 1184, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
 MemoryError

所以我做了一些研究，发现你可以使用管道（How to perform OneHotEncoding in Sklearn, getting value error），并尝试实现：

第二次编辑

>>> from sklearn.preprocessing import LabelEncoder, OneHotEncoder
>>> labelencoder_X_1 = LabelEncoder()
>>> X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
>>> labelencoder_X_2 = LabelEncoder()
>>> X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])

>>> onehotencoder = OneHotEncoder(categorical_features = [0,1])
>>> X = onehotencoder.fit_transform(X)

>>> X
<7026504x7045 sparse matrix of type '<type 'numpy.float64'>'
    with 14053008 stored elements in Compressed Sparse Row format>

#Avoid the dummy variable trap by deleting 1 from each categorical variable
>>> X = np.delete(X, [2038], axis=1)
>>> X = np.delete(X, [0], axis=1)

>>> from sklearn.model_selection import train_test_split
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)


#ERROR
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/sklearn/model_selection/_split.py", line 2031, in train_test_split
    arrays = indexable(*arrays)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 229, in indexable
check_consistent_length(*result)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 200, in check_consistent_length
    lengths = [_num_samples(X) for X in arrays if X is not None]
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 119, in _num_samples
" a valid collection." % x)
TypeError: Singleton array array(<7026504x7045 sparse matrix of type '<type 'numpy.float64'>'
with 14053008 stored elements in Compressed Sparse Row format>,
  dtype=object) cannot be considered a valid collection.

>>> from sklearn.preprocessing import StandardScaler
>>> sc = StandardScaler()
>>> X_train = sc.fit_transform(X_train)
#ERROR

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'X_train' is not defined

>>> X_test = sc.transform(X_test)
#ERROR

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'X_test' is not defined

Answer 1

你为什么首先在 OneHotEncoder 的输出上做toarray()？大多数 scikit 估计器将能够很好地处理稀疏矩阵。您的管道部分正在做同样的事情，您在内存错误上方所做的事情。

首先，你已经做到了：

X = DBF2.iloc[:, 0:2].values

这里，DBF2 是pandas DataFrame，它有values 属性来获取底层的numpy 数组。

所以现在X 是numpy array。你不能再做X.values了。这就是你第一个错误的原因。你现在已经更正了。

现在谈论警告，它与X 无关，但与y 有关。 （这只是一个警告，无需担心） 你这样做了：

y = DBF2.iloc[:, 2].values

所以，y 是一个形状为 (n_samples, 1) 的 numpy 数组。 1 因为您只选择了单列。但大多数 scikit 估计器需要形状为 (n_samples, ) 的 y。观察逗号后面的空值。

所以你需要这样做：

y = DBF2.iloc[:, 2].values.ravel()

更新：

X 是一个稀疏矩阵，因此您不能对其使用 numpy 操作 (np.delete)。改为这样做：

index_to_drop = [0, 2038]      #<=== Just add all the columns to drop here
to_keep = list(set(xrange(X.shape[1]))-set(index_to_drop))    
X = X[:,to_keep]

# Your other code here