SHAP 异常：TreeExplainer 中的可加性检查失败

Question

我正在尝试为单行创建 shap 值以进行本地解释，但我一直收到此错误。我尝试了各种方法，但仍然无法修复它们。

到目前为止我做过的事情 -

创建了随机决策树模型 -

from sklearn.ensemble import ExtraTreesRegressor
extra_tree = ExtraTreesRegressor(random_state=42)
extra_tree.fit(X_train, y_train)

然后尝试计算shap值-

# create a explainer object
explainer = shap.Explainer(extra_tree)    
explainer.expected_value
array([15981.25812347])

#calculate shap value for a single row
shap_values = explainer.shap_values(pd.DataFrame(X_train.iloc[9274]).T)

这给了我这个错误 -

Exception: Additivity check failed in TreeExplainer! Please ensure the data matrix you passed to the explainer is the same shape that the model was trained on. If your data shape is correct then please report this on GitHub. Consider retrying with the feature_perturbation='interventional' option. This check failed because for one of the samples the sum of the SHAP values was 25687017588058.968750, while the model output was 106205.580000. If this difference is acceptable you can set check_additivity=False to disable this check.

训练的形状和我传的单行列数一样

X_train.shape
(421570, 164)
(pd.DataFrame(X_train.iloc[9274]).T).shape
(1, 164)

我不认为，这应该会导致任何问题。但为了确保，我也尝试使用 reshape 方法带来正确的形状。

shap_values = explainer.shap_values(X_train.iloc[9274].values.reshape(1, -1))

X_train.iloc[9274].values.reshape(1, -1).shape
(1, 164)

这也不能解决问题。所以，我想也许我还需要匹配行数。所以我创建了一个小数据框并尝试对其进行测试。

train = pd.concat([X_train, y_train], axis="columns")
train_small = train.sample(n=500, random_state=42)
X_train_small = train_small.drop("Weekly_Sales", axis=1).copy()
y_train_small = train_small["Weekly_Sales"].copy()

# train a randomized decision tree model
from sklearn.ensemble import ExtraTreesRegressor
extra_tree_small = ExtraTreesRegressor(random_state=42)
extra_tree_small.fit(X_train_small, y_train_small)

# create a explainer object
explainer = shap.Explainer(extra_tree_small)
shap_values = explainer.shap_values(X_train_small)

# I also tried to add the y value like this 
shap_values = explainer.shap_values(X_train_small, y_train_small)

但没有任何效果。

GitHub 上的一个人建议卸载和reinstall 来自 GitHub 的 shap 的最新版本 -

pip install git+https://github.com/slundberg/shap.git

也试过了，还是不行。

有人知道如何解决这个问题吗？

Answer 1

我仍然不确定您为什么要转置或尝试更改输入的形状，因为这不在 the examples 中，但我认为下面的解决方案应该举例说明使用 sklearn's ExtraTreeRegressor 和使用 SHAP。请注意，我无权访问您的数据，所以我必须generate my own data。

注意——我最初将它设置为 1000 个样本，以便它运行得很快。后来我把它设置为 10000 并且它运行了，只是慢了一点。

如果您有任何问题，请告诉我：

# Import statements
import shap, matplotlib.pyplot as plt, pandas as pd, numpy as np
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

## Generating data since none was provided
X, y = make_regression(n_samples=1000, n_features=50, n_informative=45, noise=1, random_state=8)
# Convert data to pandas dataframe as in question
X = pd.DataFrame(data=X, columns=["Feature_{}".format(i) for i in range(X.shape[1])])
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=8, test_size=0.2)

## Creating model per question
extra_tree = ExtraTreesRegressor(random_state=42, verbose=2)
extra_tree.fit(X_train, y_train)

"""
Provided code:
explainer = shap.Explainer(extra_tree) 
"""
# Our Code
explainer = shap.TreeExplainer(extra_tree) 

# Visualize one value
single_shap_value = explainer(X_test.sample(n=1))
shap.summary_plot(single_shap_value, feature_names=X_test.columns, plot_type='bar')
plt.show()

# Visualize all values
shap_values = explainer(X_test)
shap.summary_plot(shap_values, feature_names=X_test.columns)
plt.show()

这会产生如下图像：

我相信您的问题与我关于您的数据形状的评论有关。如果你保持原样，你应该没问题。

一些注意事项：

python -V
3.8.8
print(sklearn.__version__)
print(shap.__version__)
0.24.1
0.39.0

Answer 2

尝试直接调用解释器

explainer = shap.Explainer(model)
shap_values = explainer(X)

这里 X 是你的行。