优化循环以查找具有十分位分桶的两个数组之间的差异以用于热图图

Question

代码是在任何计划将其用于许多数据文件之前编写的 - 缺乏可扩展性

问题：我有两个数组，其中填充了 0 到 1 之间的预测分数。我想在 10x10 热图中比较两个不同模型输出之间的差异。我从 NNC 模型中得到一个分数，看看对应的实例在 FLC 中的距离有多远，计算这些差异的数量和分布，然后进行绘图。

我正在考虑/其他人建议的事情：

1) 将数组转换为 pandas 数据帧，批量操作可能更快。也许对每个十分位子集使用渐进式分支列数据框

2) 动态创建 10 个单独的数组，以便在循环之前将值分区到十分位桶中

3) 将所有文件合并到一个数组中，所以仍然需要很长时间，但不是一夜之间

4) 用定义函数替换一些内联数学运算

对于一个文件，完成过程大约需要 80 秒，这对于一个数据集是可以的，但不是 600，除非我想在一夜之间运行。这是耗时最长的单元格： （代码做了一些修改，让它自己运行）

import time
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

startTime = time.time()

fromNNC = 0
toNNC = 0.1
fromDC = 0
toDC = 0.1

comparisonNNC = np.random.rand(14044, 1)
comparisonFLC = np.random.rand(14044, 1)

diffGridC = np.array([])
diffGridCounterC = 0

thisDiffC = 0

for yaxis in range(10):        
    for xaxis in range(10):
        for eachScore in range(len(comparisonNNC)):
            if(comparisonNNC[eachScore] > fromNNC and comparisonNNC[eachScore] < toNNC):
                thisDiffC = (abs(comparisonNNC[eachScore] - comparisonFLC[eachScore]))
                #print(thisDiff)
                if(thisDiffC > fromDC and thisDiffC < toDC):
                    diffGridCounterC = diffGridCounterC + 1

        diffGridC = np.append(diffGridC, diffGridCounterC)
        diffGridCounterC = 0

        fromDC = fromDC + 0.1
        #print(fromNN)
        toDC = toDC + 0.1
        #print(toNN)    

    fromDC = 0.0
    toDC = 0.1

    fromNNC = fromNNC + 0.1
    toNNC = toNNC + 0.1
    print(fromNNC)    


diffGridC = diffGridC.reshape(10, 10)
diffGridC = diffGridC.astype(int)
print(diffGridC.shape)

diffMapC = sns.heatmap(diffGridC, annot=True, fmt='d', cmap="OrRd")
diffMapC.set(xlabel='Diff', ylabel='NN')
plt.xticks(range(10), ['0.0-0.1', '0.1-0.2', '0.2-0.3', '0.3-0.4', '0.4-0.5',
                       '0.5-0.6', '0.6-0.7', '0.7-0.8', '0.8-0.9', '0.9-1.0'], rotation=50)
plt.yticks(range(10), ['0.0-0.1', '0.1-0.2', '0.2-0.3', '0.3-0.4', '0.4-0.5',
                       '0.5-0.6', '0.6-0.7', '0.7-0.8', '0.8-0.9', '0.9-1.0'], rotation=0)
plt.show()

#diffGridDFC = pd.DataFrame(diffGridC)
#diffGridDFC.to_csv('difference grid correct.csv')  

endTime = time.time()
print(endTime - startTime)

输出如下：Heatmap

编辑：尝试压缩数组，根本没有提高速度

for eachScoreNNC, eachScoreFLC in zip(comparisonNNC, comparisonFLC):
    if(eachScoreNNC > fromNNC and eachScoreNNC < toNNC):
        thisDiffC = (abs(eachScoreNNC - eachScoreFLC))
        #print(thisDiff)

这里的专家有什么建议吗？

Answer 1

[已解决]，感谢 /r/learnpython 上的 /u/two_bob

为要比较的两个数组创建存储桶，然后使用 groupby 计算交叉点 - 然后使用 dropin 方法进入热图网格，而不是循环/搜索所有 100 个单元格以进行匹配。

    correctNNFL['NNbuckets'] = pd.cut(correctNNFL['Score'], np.linspace(0, 1, 11), labels=['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'])
    correctNNFL['difference'] = abs(correctNNFL['Score'] - correctNNFL['Conf. Perc Uncreditworthy'])
    correctNNFL['diffBuckets'] = pd.cut(correctNNFL['difference'], np.linspace(0, 1, 11), labels=['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'])

    correctNNFL.groupby(['diffBuckets']).count()
    diffCoords = np.array(correctNNFL[['NNbuckets', 'diffBuckets']].groupby(['diffBuckets', 'NNbuckets'])['diffBuckets'].count().index.tolist())
    diffIntersections = correctNNFL[['NNbuckets', 'diffBuckets']].groupby(['diffBuckets', 'NNbuckets'])['diffBuckets'].count().as_matrix()

    diffGridC = np.zeros(100)
    #diffGridC = np.arange(100)
    diffGridC = diffGridC.reshape(10, 10)
    diffGridC = diffGridC.astype(int)

    for eachBucket in range(len(diffCoords)):
        diffGridC[int(diffCoords[eachBucket][1])][int(diffCoords[eachBucket][0])] = int(diffIntersections[eachBucket])
    print(diffGridC)
    diffMapC = sns.heatmap(diffGridC, annot=True, fmt='d', cmap="OrRd")
    diffMapC.set(xlabel='Diff', ylabel='NN')
    plt.show()