使用 Python 的随机森林特征重要性图

Question

我正在使用 Python 中的 RandomForestRegressor，我想创建一个图表来说明特征重要性的排名。这是我使用的代码：

from sklearn.ensemble import RandomForestRegressor

MT= pd.read_csv("MT_reduced.csv") 
df = MT.reset_index(drop = False)

columns2 = df.columns.tolist()

# Filter the columns to remove ones we don't want.
columns2 = [c for c in columns2 if c not in["Violent_crime_rate","Change_Property_crime_rate","State","Year"]]

# Store the variable we'll be predicting on.
target = "Property_crime_rate"

# Let’s randomly split our data with 80% as the train set and 20% as the test set:

# Generate the training set.  Set random_state to be able to replicate results.
train2 = df.sample(frac=0.8, random_state=1)

#exclude all obs with matching index
test2 = df.loc[~df.index.isin(train2.index)]

print(train2.shape) #need to have same number of features only difference should be obs
print(test2.shape)

# Initialize the model with some parameters.

model = RandomForestRegressor(n_estimators=100, min_samples_leaf=8, random_state=1)

#n_estimators= number of trees in forrest
#min_samples_leaf= min number of samples at each leaf


# Fit the model to the data.
model.fit(train2[columns2], train2[target])
# Make predictions.
predictions_rf = model.predict(test2[columns2])
# Compute the error.
mean_squared_error(predictions_rf, test2[target])#650.4928

功能重要性

features=df.columns[[3,4,6,8,9,10]]
importances = model.feature_importances_
indices = np.argsort(importances)

plt.figure(1)
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), features[indices])
plt.xlabel('Relative Importance')

此功能重要性代码是根据http://www.agcross.com/2015/02/random-forests-in-python-with-scikit-learn/ 上的示例更改的

当我尝试使用我的数据复制代码时收到以下错误：

  IndexError: index 6 is out of bounds for axis 1 with size 6

此外，在我的图表上，只有一个没有标签的功能具有 100% 的重要性。

任何帮助解决此问题以便我可以创建此图表将不胜感激。

Answer 1

这是一个使用 iris 数据集的示例。

>>> from sklearn.datasets import load_iris
>>> iris = load_iris()
>>> rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=42)
>>> rnd_clf.fit(iris["data"], iris["target"])
>>> for name, importance in zip(iris["feature_names"], rnd_clf.feature_importances_):
...     print(name, "=", importance)

sepal length (cm) = 0.112492250999
sepal width (cm) = 0.0231192882825
petal length (cm) = 0.441030464364
petal width (cm) = 0.423357996355

绘制特征重要性

>>> features = iris['feature_names']
>>> importances = rnd_clf.feature_importances_
>>> indices = np.argsort(importances)

>>> plt.title('Feature Importances')
>>> plt.barh(range(len(indices)), importances[indices], color='b', align='center')
>>> plt.yticks(range(len(indices)), [features[i] for i in indices])
>>> plt.xlabel('Relative Importance')
>>> plt.show()

Answer 2

在 spies006 的上述代码中，“feature_names”对我不起作用。一个通用的解决方案是使用 name_of_the_dataframe.columns。

Answer 3

y 刻度不正确。要修复它，它应该是

plt.yticks(range(len(indices)), [features[i] for i in indices])

Answer 4

来自 spies006 的这段代码不起作用：plt.yticks(range(len(indices)), features[indices]) 所以你必须将其更改为 plt.yticks(range(len(indices)),features.columns[indices])

Answer 5

将特征重要性加载到按列名索引的 pandas 系列中，然后使用它的 plot 方法。例如对于使用 df 训练的 sklearn RF 分类器/回归器 model：

feat_importances = pd.Series(model.feature_importances_, index=df.columns)
feat_importances.nlargest(4).plot(kind='barh')

Answer 6

条形图会非常有用以可视化功能的重要性 。

使用这个（使用 Iris 数据集的示例）：

from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets
import numpy as np
import matplotlib.pyplot as plt

# Load data
iris = datasets.load_iris()
X = iris.data
y = iris.target

# Create decision tree classifer object
clf = RandomForestClassifier(random_state=0, n_jobs=-1)
# Train model
model = clf.fit(X, y)

# Calculate feature importances
importances = model.feature_importances_
# Sort feature importances in descending order
indices = np.argsort(importances)[::-1]

# Rearrange feature names so they match the sorted feature importances
names = [iris.feature_names[i] for i in indices]

# Barplot: Add bars
plt.bar(range(X.shape[1]), importances[indices])
# Add feature names as x-axis labels
plt.xticks(range(X.shape[1]), names, rotation=20, fontsize = 8)
# Create plot title
plt.title("Feature Importance")
# Show plot
plt.show()

Answer 7

您尝试应用的方法是使用随机森林的内置特征重要性。这种方法有时更喜欢数值特征而不是分类特征，并且可能更喜欢高基数的分类特征。有关详细信息，请参阅此article。还有其他两种方法可以获得特征重要性（但也有它们的优缺点）。

基于排列的特征重要性

在来自版本0.22 的scikit-learn 中有方法：permutation_importance。它与模型无关。如果它们遵循scikit-learn 接口，它甚至可以与其他包中的算法一起使用。完整代码示例：

import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
import shap
from matplotlib import pyplot as plt

# prepare the data
boston = load_boston()
X = pd.DataFrame(boston.data, columns=boston.feature_names)
y = boston.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=12)

# train the model
rf = RandomForestRegressor(n_estimators=100)
rf.fit(X_train, y_train)

# the permutation based importance
perm_importance = permutation_importance(rf, X_test, y_test)

sorted_idx = perm_importance.importances_mean.argsort()
plt.barh(boston.feature_names[sorted_idx], perm_importance.importances_mean[sorted_idx])
plt.xlabel("Permutation Importance")

基于排列的重要性在计算上可能会很昂贵，并且可能会忽略高度相关的重要特征。

基于 SHAP 的重要性

可以使用 Shapley 值计算特征重要性（您需要 shap 包）。

import shap
explainer = shap.TreeExplainer(rf)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test, plot_type="bar")

一旦计算出 SHAP 值，就可以绘制其他图：

计算 SHAP 值的计算成本可能很高。计算随机森林特征重要性的 3 种方法的完整示例可以在我的 blog post 中找到。

Answer 8

from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets
import numpy as np
import matplotlib.pyplot as plt

# Load data
iris = datasets.load_iris()
X = iris.data
y = iris.target

# Create decision tree classifer object
clf = RandomForestClassifier(random_state=0, n_jobs=-1)
# Train model
model = clf.fit(X, y)

feat_importances = pd.DataFrame(model.feature_importances_, index=iris.feature_names, columns=["Importance"])
feat_importances.sort_values(by='Importance', ascending=False, inplace=True)
feat_importances.plot(kind='bar', figsize=(8,6))

print(feat_importances)

我们得到：

                   Importance
petal width (cm)     0.489820
petal length (cm)    0.368047
sepal length (cm)    0.118965
sepal width (cm)     0.023167