使用 Sklearn 仅标准化 Pandas DataFrame 中的数字列时出现 SettingWithCopy 警告 [重复]

Question

执行以下操作时，我从 Pandas 获得了SettingWithCopyWarning。我了解警告的含义，我知道我可以关闭警告，但我很好奇我是否使用 pandas 数据框错误地执行了这种类型的标准化（我将数据与分类和数字列混合在一起）。检查后我的数字看起来不错，但我想清理我的语法以确保我正确使用Pandas。

我很好奇在处理像这样的混合数据类型的数据集时，这种类型的操作是否有更好的工作流程。

我的过程如下，一些玩具数据：

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from typing import List

# toy data with categorical and numeric data
df: pd.DataFrame = pd.DataFrame([['0',100,'A', 10],
                                ['1',125,'A',15],
                                ['2',134,'A',20],
                                ['3',112,'A',25],
                                ['4',107,'B',35],
                                ['5',68,'B',50],
                                ['6',321,'B',10],
                                ['7',26,'B',27],
                                ['8',115,'C',64],
                                ['9',100,'C',72],
                                ['10',74,'C',18],
                                ['11',63,'C',18]], columns = ['id', 'weight','type','age'])
df.dtypes
id        object
weight     int64
type      object
age        int64
dtype: object

# select categorical data for later operations
cat_cols: List = df.select_dtypes(include=['object']).columns.values.tolist()
# select numeric columns for later operations
numeric_cols: List = df.columns[df.dtypes.apply(lambda x: np.issubdtype(x, np.number))].values.tolist()

# prepare data for modeling by splitting into train and test
# use only standardization means/standard deviations from the TRAINING SET only 
# and apply them to the testing set as to avoid information leakage from training set into testing set
X: pd.DataFrame = df.copy()
y: pd.Series = df.pop('type')
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

# perform standardization of numeric variables using the mean and standard deviations of the training set only
X_train_numeric_tmp: pd.DataFrame = X_train[numeric_cols].values
X_train_scaler = preprocessing.StandardScaler().fit(X_train_numeric_tmp)
X_train[numeric_cols]: pd.DataFrame = X_train_scaler.transform(X_train[numeric_cols])
X_test[numeric_cols]: pd.DataFrame = X_train_scaler.transform(X_test[numeric_cols])


<ipython-input-15-74f3f6c70f6a>:10: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

Answer 1

我尝试解释 pd.get_dummies() 和 OneHotEncoder() 将分类数据转换为虚拟列。但我确实建议使用OneHotEncoder() 转换器，因为它是一个 sklearn 转换器，如果您愿意，您可以稍后在Pipeline 中使用它。

第一个OneHotEncoder()：它和pandas 的pd.get_dummies 函数做同样的工作，但是这个类的返回是一个Numpy ndarray 或者一个稀疏数组。你可以阅读更多关于这门课的内容here:

from sklearn.preprocessing import OneHotEncoder

X_train_cat = X_train[["type"]]
cat_encoder = OneHotEncoder(sparse=False)
X_train_cat_1hot = cat_encoder.fit_transform(X_train) #This is a numpy ndarray!
#If you want to make a DataFrame again, you can do so like below:
#X_train_cat_1hot = pd.DataFrame(X_train_cat_1hot, columns=cat_encoder.categories_[0])
#You can also concatenate this transformed dataframe with your numerical transformed one.

第二种方法，pd.get_dummies()：

df_dummies = pd.get_dummies(X_train[["type"]])
X_train = pd.concat([X_train, df_dummies], axis=1).drop("type", axis=1)

Answer 2

您的X_train、X_test 仍然是原始数据帧的切片。修改切片会触发警告并且通常不起作用。

您可以在train_test_split 之前进行转换，或者在拆分后进行X_train = X_train.copy()，然后再进行转换。

第二种方法可以防止代码中注释的信息泄漏。所以是这样的：

# these 2 lines don't look good to me
# X: pd.DataFrame = df.copy()    # don't you drop the label?
# y: pd.Series = df.pop('type')  # y = df['type']

# pass them directly instead
features = [c for c in df if c!='type']
X_train, X_test, y_train, y_test = train_test_split(df[features], df['type'], 
                                                    test_size = 0.2, 
                                                    random_state = 0)

# now copy what we want to transform
X_train = X_train.copy()
X_test = X_test.copy()

## Code below should work without warning
############
# perform standardization of numeric variables using the mean and standard deviations of the training set only
# you don't need copy the data to fit
# X_train_numeric_tmp: pd.DataFrame = X_train[numeric_cols].values
X_train_scaler = preprocessing.StandardScaler().fit(X_train[numeric_cols)

X_train[numeric_cols]: pd.DataFrame = X_train_scaler.transform(X_train[numeric_cols])
X_test[numeric_cols]: pd.DataFrame = X_train_scaler.transform(X_test[numeric_cols])