在 Keras 模型中保存 SentencepieceTokenizer 抛出 TypeError: Failed to convert elements of [None, None] to Tensor答案

【问题标题】：Saving SentencepieceTokenizer in Keras model throws TypeError: Failed to convert elements of [None, None] to Tensor在 Keras 模型中保存 SentencepieceTokenizer 抛出 TypeError: Failed to convert elements of [None, None] to Tensor
【发布时间】：2022-08-02 18:15:17
【问题描述】：

我正在尝试保存使用SentencepieceTokenizer 的 Keras 模型。

到目前为止一切正常，但我无法保存 Keras 模型。

在训练了sentencepiece 模型之后，我正在创建 Keras 模型，首先用一些示例调用它，然后尝试像这样保存它：

proto = tf.io.gfile.GFile(model_path, \"rb\").read()
model = Model(tokenizer=proto)
embed = model(examples)
assert embed.shape[0] == len(examples)
model.save(\"embed_model\")

该模型本身很简单，如下所示：

class Model(keras.Model):

    def __init__(self, tokenizer: spm.SentencePieceProcessor, embed_size: int = 32, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.tokenizer = tf_text.SentencepieceTokenizer(model=tokenizer, nbest_size=1)
        self.embeddings = layers.Embedding(input_dim=self.tokenizer.vocab_size(), output_dim=embed_size)

    def call(self, inputs, training=None, mask=None):
        x = self.tokenizer.tokenize(inputs)
        if isinstance(x, tf.RaggedTensor):
            x = x.to_tensor()

        x = self.embeddings(x)
        return x

我得到的错误是：

TypeError: Failed to convert elements of [None, None] to Tensor. 
Consider casting elements to a supported type. 
See https://www.tensorflow.org/api_docs/python/tf/dtypes for supported TF dtypes.

在我看来，就好像模型在调用model.save() 之后实际上被调用了model([None, None])。

准确地说，错误似乎发生在ragged_tensor.convert_to_tensor_or_ragged_tensor(input)：

E                       TypeError: Exception encountered when calling layer \"model\" (type Model).
E                       
E                       in user code:
E                       
E                           File \"/home/sfalk/workspaces/technical-depth/ris-ml/tests/ris/ml/text/test_tokenizer.py\", line 20, in call  *
E                               x = self.tokenizer.tokenize(inputs)
E                           File \"/home/sfalk/miniconda3/envs/ris-ml/lib/python3.10/site-packages/tensorflow_text/python/ops/sentencepiece_tokenizer.py\", line 133, in tokenize  *
E                               input_tensor = ragged_tensor.convert_to_tensor_or_ragged_tensor(input)
E                       
E                           TypeError: Failed to convert elements of [None, None] to Tensor. Consider casting elements to a supported type. See https://www.tensorflow.org/api_docs/python/tf/dtypes for supported TF dtypes.
E                       
E                       
E                       Call arguments received by layer \"model\" (type Model):
E                         • inputs=[\'None\', \'None\']
E                         • training=False
E                         • mask=None

/tmp/__autograph_generated_file99ftv9jw.py:22: TypeError

标签： tensorflow keras sentencepiece

【解决方案1】：

也许尝试为call 方法定义一个input_signature。也调用self.tokenizer.vocab_size().numpy() 而不是self.tokenizer.vocab_size()，因为急切的张量不可序列化：

import tensorflow as tf
import tensorflow_text as tf_text
import requests

url = "https://github.com/tensorflow/text/blob/master/tensorflow_text/python/ops/test_data/test_oss_model.model?raw=true"
sp_model = requests.get(url).content

class Model(tf.keras.Model):

    def __init__(self, tokenizer, embed_size: int = 32, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.tokenizer = tf_text.SentencepieceTokenizer(model=tokenizer, nbest_size=1)
        self.embeddings = tf.keras.layers.Embedding(input_dim=self.tokenizer.vocab_size().numpy(), output_dim=embed_size)

    @tf.function(input_signature=(tf.TensorSpec([None], tf.string), tf.TensorSpec([None], tf.int32)))
    def call(self, inputs, mask=None):
        x = self.tokenizer.tokenize(inputs)
        if isinstance(x, tf.RaggedTensor):
            x = x.to_tensor()
        x = self.embeddings(x)
        return x

model = Model(sp_model)
embed = model(["What you know you can't explain, but you feel it."], training=False, mask=[1, 1, 1, 1, 0])
model.save("embed_model")

请注意，我从call 方法中删除了training 参数，因为它已经退出。此外，如果您可以在构造函数中设置self.built=True，那么您就不必在实际数据上调用您的模型，但这取决于您：

class Model(tf.keras.Model):

    def __init__(self, tokenizer, embed_size: int = 32, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.tokenizer = tf_text.SentencepieceTokenizer(model=tokenizer, nbest_size=1)
        self.embeddings = tf.keras.layers.Embedding(input_dim=self.tokenizer.vocab_size().numpy(), output_dim=embed_size)
        self.built = True

    @tf.function(input_signature=(tf.TensorSpec([None], tf.string), tf.TensorSpec([None], tf.int32)))
    def call(self, inputs, mask=None):
        ...
        return x

model = Model(sp_model)
model.save("embed_model")

【讨论】：