使用 CNN 训练海量数据

Question

作为机器学习的新手，我尝试搜索了很多用于训练分类器以及如何使数据可读的方法。到目前为止，我所知道的标签是分类器最重要的东西，这是显而易见的。我的问题是，我在其他文件夹中有超过 300000 个图像和标签的庞大数据集，解释了每个图像的侧面和边界框，我还有其他数据信息，这些信息不在标签文件夹中，而是在 misc 文件夹中.mat 文件和每个 .mat 文件都包含图像中汽车的品牌或型号名称。因为，到目前为止，我正在将图像和标签数据转换为 numpy 格式，并将它们一起附加到 training_data 中，我应该如何处理 misc 数据，以便也可以使用标签和图像数据对其进行训练。

我们将非常感谢您的回答。

我将在下面详细解释文件夹，以便大家对数据有更多了解。我只需要理论上的答案步骤，如果你们能做到的话。

数据说明

文件夹和文件说明如下：

-image:
  Stores all full car images in the path format 'make_id/model_id/released_year/image_name.jpg'.
-label:
  Stores all labels to the full car images in the path format 'make_id/model_id/released_year/image_name.txt'. Each label file has three lines. The first line is a number which is the viewpoint annotation (-1 - uncertain, 1 - front, 2 - rear, 3 - side, 4 - front-side, 5 - rear-side). The second line is the number of the bounding boxes, which is all '1' in the current release. The third line is the coordinates of the bounding box in the format 'x1 y1 x2 y2' in pixels, where 1 <= x1 < x2 <= image_width, and 1 <= y1 < y2 <= image_height. 
-misc:
  -attributes.txt:
    Each line is the attribute annotation for one model which is in the format 'model_id maximum_speed displacement door_number seat_number type'. For car types, a number from 1~12 corresponds to a specific type, which is described in 'car_type.mat'. Unavailable attributes are denoted by '0' or '0.0'.
  -make_model_name.mat
    Cell array 'make_names' provides the projections from 'make_id' to make names, and cell array 'model_names' provides the projections from 'model_id' to model names.
-part:
  Stores all part images in the path format 'make_id/model_id/released_year/part_id/image_name.jpg'. The correspondance of 'part_id' and part names are: 1 - headlight, 2 - taillight, 3 - fog light, 4 - air intake, 5 - console, 6 - steering wheel, 7 - dashboard, and 8 - gear lever.
-train_test_split:
  This folder generally provides all the train/test subsets used in the paper.
  -classification
    Stores the train/test lists for the classification task with full car images in the paper.
  -part:
    Stores the train/test lists for the classification task with car part in the paper.
  -verification:
    'verification_train.txt' is the image list for training the verification models which is also for testing attribute prediction. 'verification_pairs_easy.txt', 'verification_pairs_medium.txt', and 'verification_pairs_hard.txt' are the three sets with different difficulties for testing car verification models. Each line of 'verification_pairs_XXX.txt' is in the format of 'path_to_image_1 path_to_image_2 label' where label is '1' for positive pairs and is '0' for negative pairs.

谢谢。

Answer 1

您可以做的是将 misc 数据设为数字。

假设您有 3 种不同型号的汽车：Ferrari、Tesla 和 Lambo。你可以定义如果汽车是Ferrari，它的号码是0。如果是Tesla，它的号码是1，如果它是Lambo，它的号码是2。现在，当您加载标签时，还加载 misc 文件夹数据并按照上面的定义进行交换：Ferrari = 0, Tesla = 1, Lambo = 2 现在您可以将其附加到标签向量并教网络预测 0、1 或 @987654333 @ 代表汽车模型。当网络做出预测时，您可以断言预测是否为真（例如，如果 NN 预测为1，则意味着图像中的汽车是Tesla）。

此方法可以应用于 misc 文件夹中的任何其他非数字特征。这就是所谓的embedding - 将非数值转换为数值。