加快内核估计的采样

Question

这是我正在使用的更大代码的MWE。基本上，它对位于某个阈值以下的所有值在 KDE (kernel density estimate) 上执行蒙特卡洛积分（顺便说一句，在这个问题上建议了积分方法：Integrate 2D kernel density estimate）。

import numpy as np
from scipy import stats
import time

# Generate some random two-dimensional data:
def measure(n):
    "Measurement model, return two coupled measurements."
    m1 = np.random.normal(size=n)
    m2 = np.random.normal(scale=0.5, size=n)
    return m1+m2, m1-m2

# Get data.
m1, m2 = measure(20000)
# Define limits.
xmin = m1.min()
xmax = m1.max()
ymin = m2.min()
ymax = m2.max()

# Perform a kernel density estimate on the data.
x, y = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]
values = np.vstack([m1, m2])
kernel = stats.gaussian_kde(values)

# Define point below which to integrate the kernel.
x1, y1 = 0.5, 0.5

# Get kernel value for this point.
tik = time.time()
iso = kernel((x1,y1))
print 'iso: ', time.time()-tik

# Sample from KDE distribution (Monte Carlo process).
tik = time.time()
sample = kernel.resample(size=1000)
print 'resample: ', time.time()-tik

# Filter the sample leaving only values for which
# the kernel evaluates to less than what it does for
# the (x1, y1) point defined above.
tik = time.time()
insample = kernel(sample) < iso
print 'filter/sample: ', time.time()-tik

# Integrate for all values below iso.
tik = time.time()
integral = insample.sum() / float(insample.shape[0])
print 'integral: ', time.time()-tik

输出看起来像这样：

iso:  0.00259208679199
resample:  0.000817060470581
filter/sample:  2.10829401016
integral:  4.2200088501e-05

这显然意味着 filter/sample 调用几乎占用了代码运行所用的所有时间。我必须重复运行这段代码几千次，这样会非常耗时。

有什么方法可以加快过滤/采样过程？

---

添加

这是我的实际代码的一个稍微更真实的MWE，其中写入了 Ophion 的多线程解决方案：

import numpy as np
from scipy import stats
from multiprocessing import Pool

def kde_integration(m_list):

    m1, m2 = [], []
    for item in m_list:
        # Color data.
        m1.append(item[0])
        # Magnitude data.
        m2.append(item[1])

    # Define limits.
    xmin, xmax = min(m1), max(m1)
    ymin, ymax = min(m2), max(m2)

    # Perform a kernel density estimate on the data:
    x, y = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]
    values = np.vstack([m1, m2])
    kernel = stats.gaussian_kde(values)

    out_list = []

    for point in m_list:

        # Compute the point below which to integrate.
        iso = kernel((point[0], point[1]))

        # Sample KDE distribution
        sample = kernel.resample(size=1000)

        #Create definition.
        def calc_kernel(samp):
            return kernel(samp)

        #Choose number of cores and split input array.
        cores = 4
        torun = np.array_split(sample, cores, axis=1)

        #Calculate
        pool = Pool(processes=cores)
        results = pool.map(calc_kernel, torun)

        #Reintegrate and calculate results
        insample_mp = np.concatenate(results) < iso

        # Integrate for all values below iso.
        integral = insample_mp.sum() / float(insample_mp.shape[0])

        out_list.append(integral)

    return out_list


# Generate some random two-dimensional data:
def measure(n):
    "Measurement model, return two coupled measurements."
    m1 = np.random.normal(size=n)
    m2 = np.random.normal(scale=0.5, size=n)
    return m1+m2, m1-m2

# Create list to pass.
m_list = []
for i in range(60):
    m1, m2 = measure(5)
    m_list.append(m1.tolist())
    m_list.append(m2.tolist())

# Call KDE integration function.
print 'Integral result: ', kde_integration(m_list)

Ophion 提供的解决方案在我提供的原始代码上运行良好，但在此版本中失败并出现以下错误：

Integral result: Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

我尝试移动 calc_kernel 函数，因为这个问题中的一个答案 Multiprocessing: How to use Pool.map on a function defined in a class? 声明 “您提供给 map() 的函数必须可以通过导入您的模块来访问”时间>;但我仍然无法让这段代码工作。

任何帮助将不胜感激。

---

添加 2

实施Ophion 的建议以删除calc_kernel 函数并简单地使用：

results = pool.map(kernel, torun)

努力摆脱PicklingError，但现在我看到，如果我创建的初始m_list 包含超过62-63 个项目，我会收到此错误：

Traceback (most recent call last):
  File "~/gauss_kde_temp.py", line 67, in <module>
    print 'Integral result: ', kde_integration(m_list)
  File "~/gauss_kde_temp.py", line 38, in kde_integration
    pool = Pool(processes=cores)
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 161, in __init__
    self._result_handler.start()
  File "/usr/lib/python2.7/threading.py", line 494, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

由于我在实际执行此代码时的实际列表最多可包含 2000 个项目，因此此问题会导致代码无法使用。线38是这个：

pool = Pool(processes=cores)

显然这与我使用的核心数量有关？

这个问题"Can't start a new thread error" in Python 建议使用：

threading.active_count()

检查我收到该错误时的线程数。我检查了一下，当它到达374 线程时它总是崩溃。我该如何解决这个问题？

---

这是处理最后一期的新问题：Thread error: can't start new thread

Answer 1

可能最简单的加速方法是并行化kernel(sample)：

获取此代码片段：

tik = time.time()
insample = kernel(sample) < iso
print 'filter/sample: ', time.time()-tik
#filter/sample:  1.94065904617

改成使用multiprocessing:

from multiprocessing import Pool
tik = time.time()

#Create definition.
def calc_kernel(samp):
    return kernel(samp)

#Choose number of cores and split input array.
cores = 4
torun = np.array_split(sample, cores, axis=1)

#Calculate
pool = Pool(processes=cores)
results = pool.map(calc_kernel, torun)

#Reintegrate and calculate results
insample_mp = np.concatenate(results) < iso

print 'multiprocessing filter/sample: ', time.time()-tik
#multiprocessing filter/sample:  0.496874094009

仔细检查他们是否返回相同的答案：

print np.all(insample==insample_mp)
#True

在 4 核上提高了 3.9 倍。不确定您在运行什么，但是在大约 6 个处理器之后，您的输入数组大小不足以获得可观的收益。例如，使用 20 个处理器，它的速度只有大约 5.8 倍。

Answer 2

本文的 cmets 部分（链接如下）中的声明是

“SciPy 的 gaussian_kde 不使用 FFT，而有一个 statsmodels 实现”

...这是观察到的性能不佳的可能原因。它继续报告使用 FFT 的数量级改进。请参阅@jseabold 的回复。

http://slendrmeans.wordpress.com/2012/05/01/will-it-python-machine-learning-for-hackers-chapter-2-part-1-summary-stats-and-density-estimators/

免责声明：我没有使用 statsmodels 或 scipy 的经验。