有效测试二维 numpy 数组中是否存在字符串答案

【问题标题】：Efficiently testing if strings are present in 2D numpy arrays有效测试二维 numpy 数组中是否存在字符串
【发布时间】：2015-12-09 21:41:01
【问题描述】：

我有一个要转换为 Pandas MultiIndex 的句子列表（但不用担心，这个问题可能完全可以使用 numpy 来回答）。例如，假设句子是：

sentences = ['she went', 'I went', 'she and I']

要创建索引，我首先需要获取所有句子中单词的唯一列表。每个单词都会成为一个索引。这样的结果应该是：

words = ['she', 'went', 'I', 'and']

然后要计算出每行的索引值，我需要一个 2d 布尔数组。制作这个数组是主要问题，因为我希望它尽可能高效，并且希望完全不依赖 python 数据操作。这个二维数组可以是任一种两种不同的格式：

元组数组。每个元组都包含布尔值以指示给定单词在行中的存在。这将传递给pandas.MultiIndex.from_tuples() 例如：

tuples = [
    #"She went" contains "she" and "went", but not "I" or "and"
    (True, True, False, False), 
    #"I went" contains "I" and "went", but not "she" or "and"
    (False, True, True, False),
    #"She and I" contains "she", "I" and "and", but not "went"
    (True, False, True, True),
]

一个数组数组，每个单词都有一个内部数组。这将传递给pandas.MultiIndex.from_array()。例如：

arrays = [
    # 'she' is in the first and third sentences
    [True, False, True],
    # 'went' is in the first and second sentences
    [True, True, False],
    # 'I' is in the second and third sentences
    [True, False, True],
    # 'and' is in the first sentence only
    [True, False, False],
]

理想情况下，该解决方案会将句子转换为 np 数组并从那时起使用它。到目前为止，我的幼稚实现是这样的。不幸的是，我不确定如何在没有列表理解的情况下使用 numpy 来做到这一点

import pandas as pd

sentences = ['she went', 'I went', 'she and I']

# Can this be done using numpy?
split_sentences = [sentence.split(" ") for sentence in sentences]

words = list(set(sum(split_sentences, [])))

# Is there a built in way of doing this with numpy, for example np.intersect?
tuples = [
    [True if word in sentence_words else False for word in words]
    for sentence_words in split_sentences
]

index = pd.MultiIndex.from_tuples(tuples, names=words)

【问题讨论】：

你确定这条线有效 - list(set(split_sentences.sum()))？或者可能是 Python 3.x 的东西，我在 Python 2.7 上。
哦，是的，我的错。这是 split_sentences 是一个 numpy 数组时的遗留问题。我会改用sum(split_sentences, [])。
您是否关心words 是否从输入sentences 维护了订单？
这不重要，不

标签： python performance python-3.x numpy pandas

【解决方案1】：

这是一种矢量化寻找交叉点问题的症结的方法 -

# Split setences
split_sentences = [sentence.split(" ") for sentence in sentences]

# Get group lengths
grplens = np.array([len(item) for item in split_sentences])

# ID each word
unq_words,ID = np.unique(np.concatenate((split_sentences)),return_inverse=True)

# Get lengths
N = len(ID)
Nunq = len(unq_words)
Nsent = len(sentences)

# Get ID shift positions and thus row shifts for final 2D array output
row_shifts = np.zeros(N,dtype=int)
row_shifts[grplens.cumsum()[:-1]] = 1

# Finally get output boolean array using ID and row_shift IDs
out = np.zeros((Nsent,Nunq),dtype=bool)
out[row_shifts.cumsum(),ID] = 1

示例运行 -

In [494]: sentences
Out[494]: ['she went', 'I went', 'she and I', 'with dog', 'dog and she']

In [495]: unq_words
Out[495]: 
array(['I', 'and', 'dog', 'she', 'went', 'with'], 
      dtype='|S4')

In [496]: out
Out[496]: 
array([[False, False, False,  True,  True, False],
       [ True, False, False, False,  True, False],
       [ True,  True, False,  True, False, False],
       [False, False,  True, False, False,  True],
       [False,  True,  True,  True, False, False]], dtype=bool)

【讨论】：