在 numpy.array 或 pandas 列中查找重复序列

Question

例如，我有一个这样的数组：

([  1,  5,  7,  9,  4,  6,  3,  3,  7,  9,  4,  0,  3,  3,  7,  8,  1, 5 ])

我需要一一找到所有重复的序列，而不是值，而是至少有两个值的序列。

结果应该是这样的：

of length 2: [1, 5] with indexes (0, 16);
of length 3: [3, 3, 7] with indexes (6, 12); [7,  9,  4] with indexes (2, 8)

如果不重复，则应排除长序列。 ([5, 5, 5, 5]) 不应该被视为索引 (0, 1, 2) 上的 [5, 5]！这不是重复序列，而是一个长序列。

我可以用 pandas.apply 函数做到这一点，但它计算太慢，更快并没有帮助我。

在现实生活中，我需要找到所有这些，长度从 10 到 100 个值在数据库中一一对应，有 1500 列，每列有 700 000 个值。所以我真的需要一个向量化的决定。

是否有一个矢量化的决策来一次查找所有内容？或者至少只找到 10 值序列？还是只有 4 值序列？有什么会被完全矢量化的吗？

Answer 1

一种可能的实现（虽然不是完全矢量化），可以找到所有出现多次的大小为n 的序列，如下所示：

import numpy as np

def repeated_sequences(arr, n):
    Na = arr.size
    r_seq = np.arange(n)
    n_seqs = arr[np.arange(Na - n + 1)[:, None] + r_seq]
    unique_seqs = np.unique(n_seqs, axis=0)
    comp = n_seqs == unique_seqs[:, None]
    M = np.all(comp, axis=-1)

    if M.any():
        matches = np.array(
            [np.convolve(M[i], np.ones((n), dtype=int)) for i in range(M.shape[0])]
        ) 
        repeated_inds = np.count_nonzero(matches, axis=-1) > n
        repeated_matches = matches[repeated_inds]
        idxs = np.argwhere(repeated_matches > 0)[::n]
        grouped_idxs = np.split(
            idxs[:, 1], np.unique(idxs[:, 0], return_index=True)[1][1:]
        )
    else:
        return [], []

    return unique_seqs[repeated_inds], grouped_idxs

理论上，你可以替换

matches = np.array(
    [np.convolve(M[i], np.ones((n), dtype=int)) for i in range(M.shape[0])]
) 

与

matches = scipy.signal.convolve(
    M, np.ones((1, n), dtype=int), mode="full"
).astype(int)

这将使整个事情“完全矢量化”，但我的测试表明这比 for 循环慢 3 到 4 倍。所以我会坚持下去。或者简单地说，

matches = np.apply_along_axis(np.convolve, -1, M, np.ones((n), dtype=int))

它没有任何显着的加速，因为它基本上是一个隐藏循环（参见this）。

这是基于@Divakar's answer here 处理的一个非常相似的问题，其中提供了要查找的序列。我只是做了它，以便它可以对所有可能的大小为n 的序列执行此过程，这些序列在函数内部以n_seqs = arr[np.arange(Na - n + 1)[:, None] + r_seq]; unique_seqs = np.unique(n_seqs, axis=0) 找到。

例如，

>>> a = np.array([1, 5, 7, 9, 4, 6, 3, 3, 7, 9, 4, 0, 3, 3, 7, 8, 1, 5])
>>> repeated_seqs, inds = repeated_sequences(a, n)
>>> for i, seq in enumerate(repeated_seqs[:10]):
...:    print(f"{seq} with indexes {inds[i]}")
...:
    [3 3 7] with indexes [ 6 12]
    [7 9 4] with indexes [2 8]

---

免责声明

如果不重复，则应排除长序列。 ([5, 5, 5, 5]) 不应在索引 (0, 1, 2) 上视为 [5, 5]！这不是重复序列，而是一个长序列。

这没有被直接考虑在内，序列[5, 5]会根据这个算法出现不止一次。你可以根据@Paul's answer here 做这样的事情，但它涉及一个循环：

import numpy as np

repeated_matches = np.array([[0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0],
                             [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]])

idxs = np.argwhere(repeated_matches > 0)
grouped_idxs = np.split(
    idxs[:, 1], np.unique(idxs[:, 0], return_index=True)[1][1:]
)

>>> print(grouped_idxs)
    [array([ 6,  7,  8, 12, 13, 14], dtype=int64), 
     array([ 7,  8,  9, 10], dtype=int64)]  

# If there are consecutive numbers in grouped_idxs, that means that there is a long 
# sequence that should be excluded. So, you'd have to check for consecutive numbers
filtered_idxs = []
for idx in grouped_idxs:
    if not all((idx[1:] - idx[:-1]) == 1):
        filtered_idxs.append(idx)
        
>>> print(filtered_idxs)
    [array([ 6,  7,  8, 12, 13, 14], dtype=int64)]

---

一些测试：

>>> n = 3
>>> a = np.array([1, 5, 7, 9, 4, 6, 3, 3, 7, 9, 4, 0, 3, 3, 7, 8, 1, 5])
>>> %timeit repeated_sequences(a, n)
    414 µs ± 5.88 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> n = 4
>>> a = np.random.randint(0, 10, (10000,))
>>> %timeit repeated_sequences(a, n)
    3.88 s ± 54 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
>>> result, _ = repeated_sequences(a, n)
>>> result.shape
    (2637, 4)

到目前为止，这并不是最有效的实现，但它可以作为 2D 方法使用。另外，如果没有任何重复的序列，它会返回空列表。

---

编辑：全面实施

我将我在 免责声明 部分中添加的例程向量化，作为长序列问题的可能解决方案，结果如下：

import numpy as np

# Taken from:
# https://stackoverflow.com/questions/53051560/stacking-numpy-arrays-of-different-length-using-padding
def stack_padding(it):
    def resize(row, size):
        new = np.array(row)
        new.resize(size)
        return new

    row_length = max(it, key=len).__len__()
    mat = np.array([resize(row, row_length) for row in it])
    return mat

def repeated_sequences(arr, n):
    Na = arr.size
    r_seq = np.arange(n)
    n_seqs = arr[np.arange(Na - n + 1)[:, None] + r_seq]
    unique_seqs = np.unique(n_seqs, axis=0)
    comp = n_seqs == unique_seqs[:, None]
    M = np.all(comp, axis=-1)

    repeated_seqs = []
    idxs_repeated_seqs = []
    if M.any():
        matches = np.apply_along_axis(np.convolve, -1, M, np.ones((n), dtype=int))
        repeated_inds = np.count_nonzero(matches, axis=-1) > n

        if repeated_inds.any():
            repeated_matches = matches[repeated_inds]
            idxs = np.argwhere(repeated_matches > 0)
            grouped_idxs = np.split(
                idxs[:, 1], np.unique(idxs[:, 0], return_index=True)[1][1:]
            )

            # Additional routine
            # Pad this uneven array with zeros so that we can use it normally
            grouped_idxs = np.array(grouped_idxs, dtype=object)
            padded_idxs = stack_padding(grouped_idxs) 

            # Find the indices where there are padded zeros
            pad_positions = padded_idxs == 0 

            # Perform the "consecutive-numbers check" (this will take one
            # item off the original array, so we have to correct for its shape).
            idxs_to_remove= np.pad(
                (padded_idxs[:, 1:] - padded_idxs[:, :-1]) == 1,
                [(0, 0), (0, 1)],
                constant_values=True,
            ) 
            pad_positions = np.argwhere(pad_positions)
            i = pad_positions[:, 0]
            j = pad_positions[:, 1] - 1 # Shift by one (shape correction)
            idxs_to_remove[i, j] = True # Masking, since we don't want pad indices

            # Obtain a final mask (boolean opposite of indices to remove)
            final_mask = ~idxs_to_remove.all(axis=-1)
            grouped_idxs = grouped_idxs[final_mask] # Filter the long sequences

            repeated_seqs = unique_seqs[repeated_inds][final_mask]

            # In order to get the correct indices, we must first limit the
            # search to a shape (on axis=1) of the closest multiple of n.
            # This will avoid taking more indices than we should to show where 
            # each repeated sequence begins
            to = padded_idxs.shape[1] & (-n) 

            # Build the final list of indices (that goes from 0 - to with
            # a step of n
            idxs_repeated_seqs = [
                grouped_idxs[i][:to:n] for i in range(grouped_idxs.shape[0])
            ]

    return repeated_seqs, idxs_repeated_seqs

例如，

n = 2

examples = [
    # First example is your original example array.
    np.array([1, 5, 7, 9, 4, 6, 3, 3, 7, 9, 4, 0, 3, 3, 7, 8, 1, 5]),

    # Second example has a long sequence of 5's, and since there aren't
    # any [5, 5] anywhere else, it's not taken into account and therefore
    # should not come out.
    np.array([1, 5, 5, 5, 5, 6, 3, 3, 7, 9, 4, 0, 3, 3, 7, 8, 1, 5]),

    # Third example has the same long sequence but since there is a [5, 5]
    # later, then it should take it into account and this sequence should
    # be found.
    np.array([1, 5, 5, 5, 5, 6, 5, 5, 7, 9, 4, 0, 3, 3, 7, 8, 1, 5]),

    # Fourth example has a [5, 5] first and later it has a long sequence of 
    # 5's which are uneven and the previous implementation got confused with
    # the indices to show as the starting indices. In this case, it should be
    # 1, 13 and 15 for [5, 5].
    np.array([1, 5, 5, 9, 4, 6, 3, 3, 7, 9, 4, 0, 3, 5, 5, 5, 5, 5]),
]

for a in examples:
    print(f"\nExample: {a}")
    repeated_seqs, inds = repeated_sequences(a, n)
    for i, seq in enumerate(repeated_seqs):
        print(f"\t{seq} with indexes {inds[i]}")

输出（如预期）：

Example: [1 5 7 9 4 6 3 3 7 9 4 0 3 3 7 8 1 5]
    [1 5] with indexes [0 16]
    [3 3] with indexes [6 12]
    [3 7] with indexes [7 13]
    [7 9] with indexes [2 8]
    [9 4] with indexes [3 9]

Example: [1 5 5 5 5 6 3 3 7 9 4 0 3 3 7 8 1 5]
    [1 5] with indexes [0 16]
    [3 3] with indexes [6 12]
    [3 7] with indexes [7 13]

Example: [1 5 5 5 5 6 5 5 7 9 4 0 3 3 7 8 1 5]
    [1 5] with indexes [ 0 16]
    [5 5] with indexes [1 3 6]

Example: [1 5 5 9 4 6 3 3 7 9 4 0 3 5 5 5 5 5]
    [5 5] with indexes [ 1 13 15]
    [9 4] with indexes [3 9]

您可以通过更多示例和更多案例自行测试。请记住，这是我从您的免责声明中了解到的。如果您想将长序列计为一个，即使其中有多个序列（例如，[5, 5] 在[5, 5, 5, 5] 中出现两次），这对您不起作用，您必须想出一些办法否则。