递归特征选择可能不会产生更高的性能？

Question

我正在尝试分析以下数据，首先使用逻辑回归对其进行建模，然后进行预测，计算准确性和 auc;然后进行递归特征选择并再次计算accuracy & auc，以为accuracy和auc会更高，但实际上在递归特征选择后它们都较低，不确定是否符合预期？还是我错过了什么？

数据： https://github.com/amandawang-dev/census-training/blob/master/census-training.csv

---------- 对于逻辑回归，准确度：0.8111649491571692；曲线下面积：0.824896256487386

递归特征选择后，准确率：0.8130075752405651；曲线下面积：0.7997315631730443

---

import pandas as pd
import numpy as np
from sklearn import preprocessing, metrics
from sklearn.model_selection import train_test_split


train=pd.read_csv('census-training.csv')
train = train.replace('?', np.nan)
for column in train.columns:
    train[column].fillna(train[column].mode()[0], inplace=True)
x['Income'] = x['Income'].str.contains('>50K').astype(int)
x['Gender'] = x['Gender'].str.contains('Male').astype(int)

obj = train.select_dtypes(include=['object']) #all features that are 'object' datatypes
le = preprocessing.LabelEncoder()
for i in range(len(obj.columns)):
    train[obj.columns[i]] = le.fit_transform(train[obj.columns[i]])#TODO  #Encode input data

train_set, test_set = train_test_split(train, test_size=0.3, random_state=42)

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report, roc_curve, roc_auc_score
from sklearn.metrics import accuracy_score


log_rgr = LogisticRegression(random_state=0)


X_train=train_set.iloc[:, 0:9]
y_train=train_set.iloc[:, 9:10]


X_test=test_set.iloc[:, 0:9]
y_test=test_set.iloc[:, 9:10]

log_rgr.fit(X_train, y_train)

y_pred = log_rgr.predict(X_test)

lr_acc = accuracy_score(y_test, y_pred)

probs = log_rgr.predict_proba(X_test)
preds = probs[:,1]
print(preds)
from sklearn.preprocessing import label_binarize
y = label_binarize(y_test, classes=[0, 1]) #note to myself: class need to have only 0,1
fpr, tpr, threshold = metrics.roc_curve(y, preds)

roc_auc = roc_auc_score(y_test, preds)

print("Accuracy: {}".format(lr_acc))
print("AUC: {}".format(roc_auc))

---

from sklearn.feature_selection import RFE


rfe = RFE(log_rgr, 5)
fit = rfe.fit(X_train, y_train)

X_train_new = fit.transform(X_train)
X_test_new = fit.transform(X_test)

log_rgr.fit(X_train_new, y_train)
y_pred = log_rgr.predict(X_test_new)

lr_acc = accuracy_score(y_test, y_pred)

probs = rfe.predict_proba(X_test)
preds = probs[:,1]
y = label_binarize(y_test, classes=[0, 1]) 

fpr, tpr, threshold = metrics.roc_curve(y, preds)
roc_auc =roc_auc_score(y_test, preds)

print("Accuracy: {}".format(lr_acc))
print("AUC: {}".format(roc_auc))

Answer 1

不保证任何类型的特征选择（向后、向前、递归 - 随便你）实际上都会带来更好的性能。一个都没有。这些工具只是为了方便而存在——它们可能有效，也可能无效。最好的指导和最终的判断永远是实验。

除了线性或逻辑回归中的一些非常特殊的情况，最值得注意的是 Lasso（并非巧合，实际上来自统计数据），或具有太多特征的极端情况（又名 维度的诅咒），即使它有效（或无效），也没有必要解释为什么（或为什么不）。