【问题标题】:Replacing for loops with function call inside with broadcasting/vectorized solution用广播/矢量化解决方案替换内部函数调用的循环
【发布时间】:2020-06-13 21:06:55
【问题描述】:

问题:

当使用广播而不是广播标量来匹配数组时,向量化函数会出于某种原因将数组缩小为标量。

MWE:

下面是 MWE。它包含一个双 for 循环。我无法编写不使用 for 循环而是使用广播/矢量化 numpy 的更快代码。

import numpy as np

def OneD(x, y, z):
    ret = np.exp(x)**(y+1) / (z+1)
    return ret 

def ThreeD(a,b,c):
    value = OneD(a[0],b[0], c)
    value *= OneD(a[1],b[1], c)
    value *= OneD(a[2],b[2], c)

    return value

M_1 = M_2 = [[0,0,0],[0,0,1], [1,1,1], [1,0,2]] 
scales0 = scales1 = [1.1, 2.2, 3.3, 4.4]
cc0 = cc1 = 1.77   
results = np.zeros((4,4))
for s0, n0, in enumerate(M_1):
    for s1, n1, in enumerate(M_2):
        v = ThreeD(n0, n1, s1)
        v *= cc0 * cc1 * scales0[s0] * scales1[s1]
        results[s0, s1] += v

虽然我想删除两个 for 循环,但为了简单起见,我首先尝试摆脱内部循环。不过,请随意回答这两个问题。

尝试失败:

这是我改变循环的方式

rr = [0,1,2,3]
myfun = np.vectorize(ThreeD)
for s0, n0, in enumerate(M_1):
    #for s1, n1, in enumerate(M_2):
    v = myfun(n0, M_2, rr)
    v *= cc0 * cc1 * scales0[s0] * scales1[rr]
    results[s0, rr] += v

错误信息:

Traceback (most recent call last):                                                                                                                                               
  File "main.py", line 36, in <module>                                                                                                                                           
    v = myfun(n0, M_2, rr)                                                                                                                                                       
  File "/usr/lib/python3/dist-packages/numpy/lib/function_base.py", line 1573, in __call__                                                                                       
    return self._vectorize_call(func=func, args=vargs)                                                                                                                           
  File "/usr/lib/python3/dist-packages/numpy/lib/function_base.py", line 1633, in _vectorize_call                                                                                
    ufunc, otypes = self._get_ufunc_and_otypes(func=func, args=args)                                                                                                             
  File "/usr/lib/python3/dist-packages/numpy/lib/function_base.py", line 1597, in _get_ufunc_and_otypes                                                                          
    outputs = func(*inputs)                                                                                                                                                      
  File "main.py", line 18, in ThreeD                                                                                                                                             
    value = OneD(a[0],b[0], c)                                                                                                                                                   
IndexError: invalid index to scalar variable.

我还需要矢量化OneD 函数吗?我希望通过对ThreeD 函数进行矢量化处理,它会进行适当的记账。

【问题讨论】:

  • 我开始怀疑使用 np.vectorize 是否会使其更快,即使它确实有效......
  • np.vectorize 承诺什么样的加速?
  • 没有,这就是问题所在。所以我完全不知道如何加快速度。
  • np.vectorize 确实将输入相互广播,将标量值传递给函数。您的 OneD 已经完全“矢量化”。只要“广播”,它将在任何维度的x,y,z 上工作。如果不仔细检查,我怀疑ThreeD 可能类似于OneD(a, b, c).sum(axis=0)
  • 我已经尝试过对ThreeD 进行矢量化,但它实际上减慢了它的速度。在调用OneD 之前,我使用了np.vectorize(OneD)np.prod(OneD)。不过没关系。如果可以的话,我主要关心的是完全摆脱双重for loops。这需要广播,而我的大脑没有……广播。

标签: python-3.x performance numpy vectorization array-broadcasting


【解决方案1】:

在您的循环中,n0n1 是嵌套的 M_ 列表的元素,每个列表包含 3 个元素。

In [78]: ThreeD(np.arange(3),np.arange(3),3)                                                                    
Out[78]: 46.577468547527005

OneD 与数组一起使用,因此可以获得完整的n 列表/数组:

In [79]: OneD(np.arange(3), np.arange(3),3)                                                                     
Out[79]: array([  0.25      ,   1.84726402, 100.85719837])
In [80]: np.prod(_)                                                                                             
Out[80]: 46.577468547527005

并且产品匹配ThreeD

只看双循环的ThreeD 部分:

In [81]: for s0, n0, in enumerate(M_1): 
    ...:     for s1, n1, in enumerate(M_2): 
    ...:         print(n0,n1,s1, ThreeD(n0, n1, s1)) 
    ...:                                                                                                        
[0, 0, 0] [0, 0, 0] 0 1.0
[0, 0, 0] [0, 0, 1] 1 0.125
[0, 0, 0] [1, 1, 1] 2 0.037037037037037035
[0, 0, 0] [1, 0, 2] 3 0.015625
[0, 0, 1] [0, 0, 0] 0 2.718281828459045
...
[1, 0, 2] [1, 0, 2] 3 46.577468547527005

从列表中创建数组:

In [82]: M1 = np.array(M_1); M2 = np.array(M_2)                                                                 
In [83]: M1.shape                                                                                               
Out[83]: (4, 3)

我通过这个广播呼叫复制了那些 ThreeD 结果:

In [87]: np.prod(OneD(M1[:,None,:], M2[None,:,:], np.arange(4)[None,:,None]), axis=2)                           
Out[87]: 
array([[1.00000000e+00, 1.25000000e-01, 3.70370370e-02, 1.56250000e-02],
       [2.71828183e+00, 9.23632012e-01, 2.73668744e-01, 3.13836514e-01],
       [2.00855369e+01, 6.82476875e+00, 1.49418072e+01, 6.30357490e+00],
       [2.00855369e+01, 1.85516449e+01, 1.49418072e+01, 4.65774685e+01]])

我将 (4,1,3)、(1,4,3) 和 (1,4,1) 数组传递给 OneD。结果是 (4,4,3),然后在最后一个轴上相乘得到 (4,4)。

剩下的计算是:

In [88]: (cc0*cc1*np.array(scales0)[:,None]*np.array(scales1)[None,:])                                          
Out[88]: 
array([[ 3.790809,  7.581618, 11.372427, 15.163236],
       [ 7.581618, 15.163236, 22.744854, 30.326472],
       [11.372427, 22.744854, 34.117281, 45.489708],
       [15.163236, 30.326472, 45.489708, 60.652944]])

In [89]: _87*_88        # multiplying these two 4x4 arrays                                                                           
Out[89]: 
array([[3.79080900e+00, 9.47702250e-01, 4.21201000e-01, 2.36925563e-01],
       [2.06089744e+01, 1.40052502e+01, 6.22455564e+00, 9.51755427e+00],
       [2.28421302e+02, 1.55228369e+02, 5.09773834e+02, 2.86747781e+02],
       [3.04561737e+02, 5.62605939e+02, 6.79698445e+02, 2.82506059e+03]])

匹配`结果:

In [90]: results                                                                                                
Out[90]: 
array([[3.79080900e+00, 9.47702250e-01, 4.21201000e-01, 2.36925563e-01],
       [2.06089744e+01, 1.40052502e+01, 6.22455564e+00, 9.51755427e+00],
       [2.28421302e+02, 1.55228369e+02, 5.09773834e+02, 2.86747781e+02],
       [3.04561737e+02, 5.62605939e+02, 6.79698445e+02, 2.82506059e+03]])

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2021-11-10
    • 2020-07-09
    • 1970-01-01
    • 2021-03-17
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多