如何用 CharCNN 替换使用 LSTM 的字符嵌入？答案

【问题标题】：How to replace Character Embedding using LSTM with CharCNN?如何用 CharCNN 替换使用 LSTM 的字符嵌入？
【发布时间】：2020-10-03 09:03:54
【问题描述】：

我正在处理这个 repo https://github.com/Franck-Dernoncourt/NeuroNER 它使用 LSTM 进行字符级嵌入，我想为此使用 CNN。

Link 使用 LSTM 进行字符级嵌入

我尝试像这样使用多个 CharCNN 实现

self.character_embedding_weights = tf.get_variable(
                "character_embedding_weights",
                shape=[dataset.alphabet_size, parameters['character_embedding_dimension']],
                initializer=initializer)
            embedded_characters = tf.nn.embedding_lookup(self.character_embedding_weights,
                                                         self.input_token_character_indices, name='embedded_characters')
            if self.verbose:
                print("embedded_characters: {0}".format(embedded_characters))
            utils_tf.variable_summaries(self.character_embedding_weights)
            s = tf.shape(embedded_characters)
            print("dimension-",s)
            char_embeddings = tf.reshape(embedded_characters, shape=[-1, 25, 20])

            # Conv #1
            conv1 = tf.layers.conv1d(
                inputs=char_embeddings,
                filters=30,
                kernel_size=3,
                padding="valid",
                activation=tf.nn.relu)

            # Conv #2
            conv2 = tf.layers.conv1d(
                inputs=conv1,
                filters=30,
                kernel_size=3,
                padding="valid",
                activation=tf.nn.relu)
            pool2 = tf.layers.max_pooling1d(inputs=conv2, pool_size=2, strides=2)
            #
            # # Dense Layer
            character_cnn_output = tf.layers.dense(inputs=pool2, units=32, activation=tf.nn.relu)

当我将 Char 嵌入与 Word Embedding 连接时出现此问题。

Traceback (most recent call last):
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1607, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 3 but is rank 2 for 'concatenate_token_and_character_vectors/token_lstm_input' (op: 'ConcatV2') with input shapes: [?,10,32], [?,100], [].
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\neuroner\neuromodel.py", line 483, in __init__
    self.model = EntityLSTM(self.modeldata, self.parameters)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\neuroner\entity_lstm.py", line 176, in __init__
    axis=1, name='token_lstm_input')
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\ops\array_ops.py", line 1420, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\ops\gen_array_ops.py", line 1257, in concat_v2
    "ConcatV2", values=values, axis=axis, name=name)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1770, in __init__
    control_input_ops)
  File "C:\Users\DeLL\Desktop\NLP\NeuroNER-master\venv\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1610, in _create_c_op
    raise ValueError(str(e))
ValueError: Shape must be rank 3 but is rank 2 for'concatenate_token_and_character_vectors/token_lstm_input' (op: 'ConcatV2') with input shapes:[?,10,32], [?,100], [].**

这是我连接 CharCNN 的输出和导致错误的词嵌入的代码片段。

        if not parameters['use_character_lstm']:
            with tf.variable_scope("concatenate_token_and_character_vectors"):

                if self.verbose: 
                    print('embedded_tokens: {0}'.format(embedded_tokens))
                
//This is the line where the error begins
                token_lstm_input = tf.concat([character_cnn_output, embedded_tokens], 
                    axis=1, name='token_lstm_input')

                if self.verbose: 
                    print("token_lstm_input: {0}".format(token_lstm_input))
        else:
            token_lstm_input = embedded_tokens

【问题讨论】：

您好，欢迎来到 stackoverflow。也请在发生错误的地方发布该行
@MichaelJanz 完成编辑。这是代码 sn-p 中的第 5 行。谢谢
@KUSHALVIJAY 您能否在问题本身中准确指出哪一行产生了错误？您可以使用代码样式的注释来执行此操作，例如//以下行产生错误
@Nick 完成编辑
你基本上得到这个错误，因为形状不匹配。什么是 character_cnn_output，embedding_tokens 来自哪里？

标签： python tensorflow keras nlp conv-neural-network

【解决方案1】：

只需展平您的 character_cnn_output 以获得与令牌嵌入相似的维度。

在密集层输出之后添加这一行。

character_cnn_output = tf.layers.Flatten()(character_cnn_output)

谢谢

【讨论】：

【解决方案2】：

正如错误消息所示，concat 操作的张量等级不同，因此会导致错误。

下面是重现错误的简单代码。

重现错误的代码 -

%tensorflow_version 1.x
import tensorflow as tf

t1 = tf.constant([[[1, 2]], [[2, 3]], [[4, 4]], [[5, 3]]])
print(t1.shape)
t2 = tf.constant([[7, 4], [8, 4], [2, 10], [15, 11]])
print(t2.shape)

tf.concat([t1, t2], 1)

输出 -

TensorFlow 1.x selected.
(4, 1, 2)
(4, 2)
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs)
   1606   try:
-> 1607     c_op = c_api.TF_FinishOperation(op_desc)
   1608   except errors.InvalidArgumentError as e:

InvalidArgumentError: Shape must be rank 3 but is rank 2 for 'concat' (op: 'ConcatV2') with input shapes: [4,1,2], [4,2], [].

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
9 frames
/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs)
   1608   except errors.InvalidArgumentError as e:
   1609     # Convert to ValueError for backwards compatibility.
-> 1610     raise ValueError(str(e))
   1611 
   1612   return c_op

ValueError: Shape must be rank 3 but is rank 2 for 'concat' (op: 'ConcatV2') with input shapes: [4,1,2], [4,2], [].

注意：Tensorflow 2.x 中的错误消息发生了变化。

要修复错误，请将两个张量设为相同的形状并传递给concat 操作。

固定代码 -

%tensorflow_version 1.x
import tensorflow as tf

t1 = tf.constant([[1, 2], [2, 3], [4, 4], [5, 3]])
print(t1.shape)
t2 = tf.constant([[7, 4], [8, 4], [2, 10], [15, 11]])
print(t2.shape)

tf.concat([t1, t2], 1)

输出 -

(4, 2)
(4, 2)
<tf.Tensor 'concat_1:0' shape=(4, 4) dtype=int32>

【讨论】：