如何在 Keras 中读取 RNN 的初始状态？

Question

我想查看任意 Keras 模型使用的初始条件。在下面的示例中，我在第一批中将第一个单元的初始状态设置为 0.123。但是当我调用rnn(...) 函数时的输出在一个时间步之后开始（当单元活动由于一的偏差而处于 1.123 时）并且不返回初始条件。

有没有办法让它显示初始状态？

示例代码如下：

SEED=42
tf.random.set_seed(SEED)
np.random.seed(SEED)

timesteps = 3
embedding_dim = 4
units = 2
num_samples = 5

input_shape = (num_samples, timesteps, embedding_dim)
model = Sequential([
SimpleRNN(units, stateful=True, batch_input_shape=input_shape, return_sequences=True, activation="linear", 
          recurrent_initializer="identity", bias_initializer="ones"), 
Dense(1)])

some_initial_state = np.ones((num_samples, units))
some_initial_state[0,0] = 0.123
rnn = model.layers[0]
rnn.reset_states(states=some_initial_state)

some_initial_state, rnn(np.zeros((num_samples, timesteps, embedding_dim)))

谁的输出是：

(array([[0.123, 1.   ],
    [1.   , 1.   ],
    [1.   , 1.   ],
    [1.   , 1.   ],
    [1.   , 1.   ]]),
 <tf.Tensor: shape=(5, 3, 2), dtype=float32, numpy=
 array([[[1.123    , 2.       ],
     [2.1230001, 3.       ],
     [3.1230001, 4.       ]],

    [[2.       , 2.       ],
     [3.       , 3.       ],
     [4.       , 4.       ]],

    [[2.       , 2.       ],
     [3.       , 3.       ],
     [4.       , 4.       ]],

    [[2.       , 2.       ],
     [3.       , 3.       ],
     [4.       , 4.       ]],

    [[2.       , 2.       ],
     [3.       , 3.       ],
     [4.       , 4.       ]]], dtype=float32)>)

请注意，我使用rnn.reset_states(...) 设置网络的初始状态，如Setting the initial state of an RNN represented as a Keras sequential model 中所述

Answer 1

我认为您可以通过执行 RNN 单元的逆计算来获得初始状态。在这种情况下，您使用的是SimpleRNNCell，并且正向计算在call method 中实现。这是一个可能的实现：

def get_rnn_initial_state(rnn_layer, rnn_input=None, rnn_output=None):

    # Get the kernel, recurrent_kernel, and bias variables
    kernel = rnn_layer.variables[0]
    recurrent_kernel = rnn_layer.variables[1]
    bias = rnn_layer.variables[2]

    if rnn_input is None:
        rnn_input = np.zeros(rnn_layer.input_shape)

    if rnn_output is None:
        rnn_output = rnn_layer(rnn_input)

    # Calculate hidden state
    h = rnn_input @ kernel.numpy() + bias.numpy()

    # Calculate pre-activation using the inverse of the activation function
    if rnn_layer.activation == tf.keras.activations.linear:
        pre_activation = rnn_output
    # inverse for other activation functions... 

    # Calculate the inverse of `output = h + backend.dot(prev_output, self.recurrent_kernel)`
    prev_output = (pre_activation - h) @ tf.linalg.inv(recurrent_kernel)

    # Get the first timesteps for each sample
    initial_state = prev_output[:, 0, :]    
    return initial_state

some_input = np.zeros((num_samples, timesteps, embedding_dim))
initial_state = get_rnn_initial_state(rnn, some_input, rnn_output)
assert np.allclose(some_initial_state, initial_state)
initial_state

输出：

<tf.Tensor: shape=(5, 2), dtype=float32, numpy=
array([[0.12300003, 1.        ],
       [1.        , 1.        ],
       [1.        , 1.        ],
       [1.        , 1.        ],
       [1.        , 1.        ]], dtype=float32)>

由于您使用的是 有状态 RNN，因此您需要提供与初始状态相对应的输入/输出对。如果您只想获取当前状态，则可以使用states 属性：

state = rnn.states