numpy中具有相同数组的数组中的项目差异

Question

我有一个数组a。

rnd = np.random.default_rng(12345)
a = rnd.uniform(0, -50, 5)
# array([-11.36680112, -15.83791699, -39.86827287, -33.81273354,
#       -19.55547753])

我想找出数组与同一数组中每个元素的差异。示例输出为：

[array([ 0.        ,  4.47111586, 28.50147174, 22.44593241,  8.18867641]),
 array([-4.47111586,  0.        , 24.03035588, 17.97481655,  3.71756054]),
 array([-28.50147174, -24.03035588,   0.        ,  -6.05553933,
        -20.31279534]),
 array([-22.44593241, -17.97481655,   6.05553933,   0.        ,
        -14.25725601]),
 array([-8.18867641, -3.71756054, 20.31279534, 14.25725601,  0.        ])]

我的第一种方法是使用列表理解 [i - a for i in a]。但是，由于我的原始数组 a 非常庞大，并且我有数千个这样的 as 我需要执行相同的操作，因此整个过程变得非常缓慢并且内存不足，以至于 jupyter 内核死亡。

有什么方法可以加快速度吗？

Answer 1

最简单的方法是使用broadcasting：

import numpy as np
rnd = np.random.default_rng(12345)
a = rnd.uniform(0, -50, 5)
a[:, None] - a 

哪个输出：

array([[  0.        ,   4.47111586,  28.50147174,  22.44593241,
          8.18867641],
       [ -4.47111586,   0.        ,  24.03035588,  17.97481655,
          3.71756054],
       [-28.50147174, -24.03035588,   0.        ,  -6.05553933,
        -20.31279534],
       [-22.44593241, -17.97481655,   6.05553933,   0.        ,
        -14.25725601],
       [ -8.18867641,  -3.71756054,  20.31279534,  14.25725601,
          0.        ]])

Answer 2

有两种方法，一种是只使用 numpy vectos

1. 内存效率低，（在这种情况下 numpy 更快）。但如果数组大小较小，仍然应该是您尝试的第一种方法

a[:, None] - a 

1. 使用 numba + numpy，它具有 llvm 优化，因此它可以在速度方面发挥作用，您还可以使用 parallel = True 选项来调整速度。对于超大的阵列，这应该去。或者 c++

对于 40000 大小，在没有并行性的情况下在 3 秒内完成，而在我的 12 核并行机上则在 0.6 秒内完成

import numpy as np
import numba as nb

rnd = np.random.default_rng(12345)
a = rnd.uniform(0, -50, 5)

# return type nb.float64[:, :]
# input argument type nb.float64[:, :]
# By specifying these you can do eager compilation instead of lazy
# also you can add parallel = True, cache=True
# if you are using python threading then nogil=True
# you can do lots of stuff
# numba has SIMD vectorization, which just means it shall not loose to numpy on performance grounds if coded properly
@nb.njit(nb.float64[:, :](nb.float64[:]))
def speed(a):
    # empty to prevent unnecessary initializations
    b = np.empty((a.shape[0], a.shape[0]), dtype=a.dtype)

    # nb.prange needed to tell numba this for loop can be parallelized
    for i in nb.prange(a.shape[0]):
        for j in range(a.shape[0]):
            b[i][j] = a[i] - a[j]
    return b

speed(a)

性能

import numpy as np
import numba as nb
import sys
import time


@nb.njit(nb.float64[:, :](nb.float64[:]))
def f1(a):
    b = np.empty((a.shape[0], a.shape[0]), dtype=a.dtype)
    for i in nb.prange(a.shape[0]):
        for j in range(a.shape[0]):
            b[i][j] = a[i] - a[j]
    return b

@nb.njit(nb.float64[:, :](nb.float64[:]), parallel=True, cache=True)
def f2(a):
    b = np.empty((a.shape[0], a.shape[0]), dtype=a.dtype)
    for i in nb.prange(a.shape[0]):
        for j in range(a.shape[0]):
            b[i][j] = a[i] - a[j]
    return b

def f3(a):
    return a[:, None] - a

if __name__ == '__main__':
    s0 = time.time()
    rnd = np.random.default_rng(12345)
    a = rnd.uniform(0, -50, int(sys.argv[2]))
    b = eval(sys.argv[1] + '(a)')
    print(time.time() - s0)

(base) xxx:~$ python test.py f1 40000
3.0324509143829346
(base) xxx:~$ python test.py f2 40000
0.6196465492248535
(base) xxx:~$ python test.py f3 40000
2.4126882553100586

我遇到了类似的限制，我需要快速的东西。仅通过解决内存使用和 numba 问题，我在没有并行性的情况下获得了大约 50 倍的速度 Why are np.hypot and np.subtract.outer very fast?