将多个numpy文件的数据聚合为一个

Question

我有一个列表，其中包含一个数据集的 6 个不同的子数据集。我想进行 6 折交叉验证。因此，在每次 6 个步骤的 for 循环中，将我的数据集分成 2 组（将包含 5 个子数据集和测试集的训练 - 包含留下一个子数据集）。我的代码如下：

编辑（考虑到 cmets）：

sets = ['datasets/1.pickle', 'datasets/2.pickle', ..., 'datasets/6.pickle']
for i in range(0,7):
  train_set = sets[:i]+sets[i+1:]
  test_data, test_lbls = crossValidFiles(sets[i]) # returns the data for a specific sub-samlpe, returns two numpy arrays.
  for item in train_set:
      train_set = [(train_data, train_lbls) for crossValidFiles(item) in train_set]
      train_data = np.concatenate([a for (a,b) in train_set], axis = 0)
      train_lbls = np.concatenate([b for (a,b) in train_set], axis = 0)
      #train_data, train_lbls = crossValidFiles(item) # that returns one file at time.

如何汇总我为训练集返回的文件？

Answer 1

你可以使用np.concatenate()：np concatenate

例如

import numpy as np
t1 = np.array([[1,2,3],[4,5,6]])
t2 = np.array([[7,8,9],[10,11,12]])
train array = np.concatenate((t1,t2), axis=0)

为了处理您的文件，我将提取 train_data 和 train_lbls 以获取您的数据，然后将每个列表连接起来。例如：

import numpy as np
t1 = [np.array([[1,2,3],[4,5,6]]), np.array(['train_lbls'])]
t2 = [np.array([[7,8,9],[10,11,12]]), np.array(['train_lbls'])]
train_set = [t1,t2]
train_set = [(train_data, train_lbls) for crossValidFiles(item) in train_set]
train_data = np.concatenate([a for (a,b) in train_set], axis=0)
train_lbls = np.concatenate([b for (a,b) in train_set], axis=0)

Answer 2

作为Mason's 答案的替代方案，您可以在 crossValidFiles 函数中使用 np.concatenate ，以便在聚合的测试数据上运行其中的任何代码。

import numpy as np

def crossValidFiles(input_file):
    data, labels = some_load_function(input_file)
    return data, labels

def some_load_function(input_file):
    # Check if the input file is a string or list-like
    if isinstance(input_file, str):
        train_array = some_load_function_2(input_file)
    else:
        train_array = np.concatenate([some_load_function_2(f) for f in input_file], axis=0)

    # rest of your code to create variables 'data' and 'labels'
    return data, labels

Link.