计算 silhouette_score 时出现 MemorError答案

【问题标题】：MemorError while calculating silhouette_score计算 silhouette_score 时出现 MemorError
【发布时间】：2023-03-03 00:47:01
【问题描述】：

我在形状为 (190868,35) 的矩阵上运行 KMeans 聚类算法。我正在运行以下代码：

for n_clusters in range(3,10):
kmeans = KMeans(init='k-means++',n_clusters=n_clusters,n_init=30)
kmeans.fit(matrix)
clusters = kmeans.predict(matrix)
silhouette_avg=silhouette_score(matrix,clusters)
print("For n_clusters =",n_clusters,"The avg silhouette_score is :",silhouette_avg)

我遇到以下错误

Traceback (most recent call last):

  File "<ipython-input-6-be918e90030a>", line 5, in <module>
    silhouette_avg=silhouette_score(matrix,clusters)

  File "C:\Users\arindam\Anaconda3\lib\site-packages\sklearn\metrics\cluster\unsupervised.py", line 101, in silhouette_score
    return np.mean(silhouette_samples(X, labels, metric=metric, **kwds))

  File "C:\Users\arindam\Anaconda3\lib\site-packages\sklearn\metrics\cluster\unsupervised.py", line 169, in silhouette_samples
    distances = pairwise_distances(X, metric=metric, **kwds)

  File "C:\Users\arindam\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py", line 1247, in pairwise_distances
    return _parallel_pairwise(X, Y, func, n_jobs, **kwds)

  File "C:\Users\arindam\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py", line 1090, in _parallel_pairwise
    return func(X, Y, **kwds)

  File "C:\Users\arindam\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py", line 246, in euclidean_distances
    distances = safe_sparse_dot(X, Y.T, dense_output=True)

  File "C:\Users\arindam\Anaconda3\lib\site-packages\sklearn\utils\extmath.py", line 140, in safe_sparse_dot
    return np.dot(a, b)

MemoryError

如果有人知道任何解决方案，请提出建议。我尝试指定 sample_size = 70000，代码运行并消耗所有内存并且系统冻结。我有一台配备 16GB RAM 和 i7 处理器的联想 Thinkpad。

【问题讨论】：

标签： python-3.x scikit-learn k-means

【解决方案1】：

MemoryError 表示在执行silhouette_score 时内存不足以分配numpy 数组。所以解决办法是少用内存或者增加内存空间：

解决方案1.通过将sample_size设置为silhouette_score来分配更少的内存空间

参考：https://stackoverflow.com/a/16425008/1229868

如何找到最合适的sample_size？

def eval_silhouette_score(matrix, clusters, sample_size):
    try:
        silhouette_avg = metrics.silhouette_score(matrix, clusters, sample_size = sample_size)
        return silhouette_avg
    except MemoryError:
        return None

div_factor = 1.
silhouette_avg = None
while silhouette_avg == None:
    sample_size = int(len(clusters) / div_factor)
    silhouette_avg = eval_silhouette_score(matrix, clusters, sample_size)
    div_factor += 1.

解决方案 2. 安装更多物理内存 :)

【讨论】：