在 C++ 中向 Tensorflow lite 模型提供一批输入答案

【问题标题】：Feeding a batch of inputs to a Tensorflow lite model in C++在 C++ 中向 Tensorflow lite 模型提供一批输入
【发布时间】：2020-12-09 15:47:37
【问题描述】：

我使用 Python 训练了一个 TensorFlow 模型，并将其导出为 TF lite 模型。输入是 17 个浮点数的特征向量，输出是单个浮点分数。当我在 C++ 中使用 Tensor Flow Lite 导入模型时，我得到输入形状是 (1, 17, 1)，而输出形状为 (1,1)。

因此，在 C++ 中，我可以输入大小为 17 的输入向量并获得结果。在生产中，我有多个特征向量，我想将它们中的许多提供给 TF，做一些类似于 Python 中存在的 Model.predict() 方法的事情。是否可以？
有没有人尝试过这样做？

到目前为止，我尝试重塑输入和输出张量，就像下面粘贴的代码一样。我无法显示整个代码，因为它依赖于内部库。下面的代码尝试添加新的张量，将它们设置为输入和输出，并重塑它们。问题是：当我需要形状时，结果不是我所期望的。

最后，当我尝试向它们提供随机值（未显示）时，程序崩溃了，因为我写出了它分配的张量。欢迎任何想法。
谢谢

//---------------------------------------------------------------------------------------------
constexpr size_t NumFeatures{ 17};
constexpr size_t NumInputSamples{ 2};

const std::string& modelFilepath{ cliArgs.tfModelFilepath()};

std::unique_ptr<tflite::FlatBufferModel> model= tflite::FlatBufferModel::BuildFromFile(modelFilepath.c_str());
if (!model)
{
  TRACE_ERR1("BuildFromFile() failed.");
  return;
}

tflite::ops::builtin::BuiltinOpResolver resolver;
tflite::InterpreterBuilder builder(*model, resolver);
std::unique_ptr<tflite::Interpreter> interpreter;
builder(&interpreter);
if (!interpreter)
{
  TRACE_ERR1("Invalid interpreter.");
  return;
}

if (kTfLiteOk != interpreter->AllocateTensors())
{
  TRACE_ERR1("Failed at allocating tensors.");
  return;
}

TRACE_DBG2("Number of tensors:", interpreter->tensors_size());
TRACE_DBG2("Number of nodes:", interpreter->nodes_size());
TRACE_DBG2("Num Inputs:", interpreter->inputs().size());
TRACE_DBG2("Input(0) name:", interpreter->GetInputName(0));
TRACE_DBG2("Num Outputs:", interpreter->outputs().size());

for (size_t indxInput=0; indxInput<interpreter->inputs().size(); ++indxInput)
{
  for (int indxDim=0; indxDim<interpreter->input_tensor(indxInput)->dims->size; ++indxDim)
  {
    TRACE_VAR2(indxDim, interpreter->input_tensor(indxInput)->dims->data[indxDim]);
  }
}
for (size_t indxOutput=0; indxOutput<interpreter->outputs().size(); ++indxOutput)
{
  for (int indxDim=0; indxDim<interpreter->output_tensor(indxOutput)->dims->size; ++indxDim)
  {
    TRACE_VAR3(indxOutput, indxDim, interpreter->output_tensor(indxOutput)->dims->data[indxDim]);
  }
}

std::vector<int> inDims( {NumInputSamples,17,1});
std::vector<int> outDims;
outDims.push_back(NumInputSamples);
TfLiteQuantizationParams quant;
quant.scale=1.0f;
quant.zero_point=0;

std::vector<int> myIndexInTensors{};
myIndexInTensors.resize(NumInputSamples);

std::vector<int> myIndexOutTensors{};
myIndexOutTensors.resize(NumInputSamples);

THROW_OPERATION_FAILED_IFX( kTfLiteOk!=interpreter->AddTensors(NumInputSamples, myIndexInTensors.data()),
                            "AddTensors() failed.");
THROW_OPERATION_FAILED_IFX( kTfLiteOk!=interpreter->SetInputs(myIndexInTensors), "SetInputs() failed.");

for (const auto indxInput: myIndexInTensors)
{
  TRACE_VAR1(indxInput);
  THROW_OPERATION_FAILED_IFX( kTfLiteOk!=interpreter->SetTensorParametersReadWrite( indxInput,
                                                                                    kTfLiteFloat32,
                                                                                    "input",
                                                                                    inDims.size(),
                                                                                    inDims.data(),
                                                                                    quant),
                              "SetTensorParametersReadWrite() failed.");
}

THROW_OPERATION_FAILED_IFX( kTfLiteOk!=interpreter->AddTensors(NumInputSamples, myIndexOutTensors.data()),
                            "AddTensors() failed.");
THROW_OPERATION_FAILED_IFX( kTfLiteOk!=interpreter->SetOutputs(myIndexOutTensors), "SetOutputs() failed.");
for (const auto indxOutput: myIndexOutTensors)
{
  TRACE_VAR1(indxOutput);
  THROW_OPERATION_FAILED_IFX( kTfLiteOk!=interpreter->SetTensorParametersReadWrite( indxOutput,
                                                                                    kTfLiteFloat32,
                                                                                    "output",
                                                                                    outDims.size(),
                                                                                    outDims.data(),
                                                                                    quant),
                              "SetTensorParametersReadWrite() failed.");
}

if (kTfLiteOk != interpreter->AllocateTensors())
{
  TRACE_ERR1("Failed at allocating tensors.");
  return;
}

TRACE_DBG1("Requiring the shape................................................................");
TRACE_VAR1(interpreter->inputs().size());
for (size_t indxInput=0; indxInput<interpreter->inputs().size(); ++indxInput)
{
  for (int indxDim=0; indxDim<interpreter->input_tensor(indxInput)->dims->size; ++indxDim)
  {
    TRACE_VAR3(indxInput,indxDim, interpreter->tensor(indxInput)->dims->data[indxDim]);
  }
}

TRACE_VAR1(interpreter->outputs().size());
for (size_t indxOutput=0; indxOutput<interpreter->outputs().size(); ++indxOutput)
{
  for (int indxDim=0; indxDim<interpreter->output_tensor(indxOutput)->dims->size; ++indxDim)
  {
    TRACE_VAR3(indxOutput, indxDim, interpreter->tensor(indxOutput)->dims->data[indxDim]);
  }
}
  
//--------------------------------------------------------------------------------------------------
 OUTPUT:

[DBG] runInferenceOnRandomFeatures(): Number of tensors: 46 

[DBG] runInferenceOnRandomFeatures(): Number of nodes: 18 

[DBG] runInferenceOnRandomFeatures(): Num Inputs: 1 

[DBG] runInferenceOnRandomFeatures(): Input(0) name: conv1d_input 

[DBG] runInferenceOnRandomFeatures(): Num Outputs: 1 

[DBG] runInferenceOnRandomFeatures(): indxDim= 0 interpreter->input_tensor(indxInput)->dims->data[indxDim]= 1 

[DBG] runInferenceOnRandomFeatures(): indxDim= 1 interpreter->input_tensor(indxInput)->dims->data[indxDim]= 17 

[DBG] runInferenceOnRandomFeatures(): indxDim= 2 interpreter->input_tensor(indxInput)->dims->data[indxDim]= 1 

[DBG] runInferenceOnRandomFeatures(): indxOutput= 0 indxDim= 0 interpreter->output_tensor(indxOutput)->dims->data[indxDim]= 1 

[DBG] runInferenceOnRandomFeatures(): indxOutput= 0 indxDim= 1 interpreter->output_tensor(indxOutput)->dims->data[indxDim]= 1 

[DBG] runInferenceOnRandomFeatures(): indxInput= 46 

[DBG] runInferenceOnRandomFeatures(): indxInput= 0 

[DBG] runInferenceOnRandomFeatures(): indxOutput= 48 

[DBG] runInferenceOnRandomFeatures(): indxOutput= 0 

[DBG] runInferenceOnRandomFeatures(): Requiring the shape................................................................ 

[DBG] runInferenceOnRandomFeatures(): interpreter->inputs().size()= 2 

[DBG] runInferenceOnRandomFeatures(): indxInput= 0 indxDim= 0 interpreter->tensor(indxInput)->dims->data[indxDim]= 2 

[DBG] runInferenceOnRandomFeatures(): indxInput= 0 indxDim= 1 interpreter->tensor(indxInput)->dims->data[indxDim]= 0 

[DBG] runInferenceOnRandomFeatures(): indxInput= 0 indxDim= 2 interpreter->tensor(indxInput)->dims->data[indxDim]= 0 

[DBG] runInferenceOnRandomFeatures(): indxInput= 1 indxDim= 0 interpreter->tensor(indxInput)->dims->data[indxDim]= 32 

[DBG] runInferenceOnRandomFeatures(): interpreter->outputs().size()= 2 

[DBG] runInferenceOnRandomFeatures(): indxOutput= 0 indxDim= 0 interpreter->tensor(indxOutput)->dims->data[indxDim]= 2 

[DBG] runInferenceOnRandomFeatures(): indxOutput= 1 indxDim= 0 interpreter->tensor(indxOutput)->dims->data[indxDim]= 32

【问题讨论】：

我回答了你的问题还是有什么不清楚的地方？

标签： c++ tensorflow tensorflow-lite

【解决方案1】：

当您需要更改输入时，TFLite 解释器有下一个流程：

更改输入形状
分配张量
设置输入（和输出）源
运行推理
收集结果

我看到你有：

if (kTfLiteOk != interpreter->AllocateTensors())
{
  TRACE_ERR1("Failed at allocating tensors.");
  return;
}

TRACE_DBG1("Requiring the shape................................................................");
TRACE_VAR1(interpreter->inputs().size());
for (size_t indxInput=0; indxInput<interpreter->inputs().size(); ++indxInput)
{
  for (int indxDim=0; indxDim<interpreter->input_tensor(indxInput)->dims->size; ++indxDim)
  {
    TRACE_VAR3(indxInput,indxDim, interpreter->tensor(indxInput)->dims->data[indxDim]);
  }
}

分配先前的输入更改。在输入/输出大小更改后进行分配，你会没事的。见details

【讨论】：