Tensorflow 2.2.0 强化学习 - 模型参数的梯度为无

Question

我正在尝试创建一个基本的深度 Q 学习神经网络，用于玩两人单挑德州扑克。

对于给定的状态，模型必须生成一组可能动作的概率分布，在本示例中，这些动作已被简化为弃牌、过牌或激进选项（跟注/下注/加注）。我计划如何训练模型的细节与这个问题的目的无关。

由于模型的权重会随机初始化，因此模型有时会尝试执行非法移动（例如在对手加注时尝试“检查”，而不是跟注或弃牌）。发生这种情况时，我想惩罚网络并让它更新其权重，以便从模型的曲目中“过滤掉”非法动作。

在这种情况下，我将为模型分配 1.0 的损失，并尝试计算损失和模型参数之间的梯度。但出于某种原因，每当我尝试这样做时，渐变总是以“无”告终。这是什么意思，我做错了什么？

当涉及到 tensorflow 中的很多东西时，我比较幼稚，所以如果你不能假设太多的 tensorflow 知识并简化更高级的概念，这对我真的很有帮助。

这是我正在谈论的示例的代码：

from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras import Sequential, Input
from tensorflow.keras.optimizers import Adam
import tensorflow as tf
import numpy as np


# Creating a linear model with a simple structure. The input size represents a combination of vectors that
## will be used to create the input state.
model = Sequential()
model.add(Input(shape=(52 + 52 + 2 + 3 + 3), batch_size=1))
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dense(3))
# The model will produce probability values for choosing from three output actions (fold, check, call/bet/raise.) 
model.add(Activation('softmax'))
model.compile(loss="mse", optimizer=Adam(lr=0.001), metrics=['accuracy'])

# Bot hand is a 1-D boolean array of size 52. Each index represents a unique card, and a value of 1 in that index
## indicates the card is present in the hand.
bot_hand = np.zeros((52))

# For this example, we'll assume cards with ids 0 and 1 are in the bot's hand.
bot_hand[:2] = 1

# The same applies to the cards on the table; a 1-D boolean array of size 52.
table_cards = np.zeros((52))

# In this example, the bot has bet $2.
bot_bet_size = np.array([2])

# The opponent has bet $4.
opponent_bet_size = np.array([4])

# The last action the bot has ever made is represented as a 1-D boolean vector of size 3. A value of 1 in an 
## index represents which action has been taken. 0=fold, 1=check, 2=bet/raise.
bot_previous_action = np.zeros((3))

# The opponent's previous action is represented in the same way.
opponent_previous_action = np.zeros((3))

# In this example, the opponent's last action was to raise to $4.
opponent_previous_action[2] = 2

# The 'state' is represented by all of the above vectors.
bot_state = np.concatenate((bot_hand, table_cards, bot_bet_size, opponent_bet_size, bot_previous_action, 
                           opponent_previous_action), axis=0)

# The action taken is determined by which of the Softmax output nodes produces the highest input.
model_output = model.predict(bot_state.reshape(1, 52 + 52 + 2 + 3 + 3))
bot_decision = np.argmax(model_output)
                         
# Let's assume in this case the bot chose action '1' (check), which is an illegal move, since the opponents has
## raised. I am attempting to punish the network by assigning it a loss value of 1, and backpropagating to
## update the weights.
with tf.GradientTape() as t:
    # Creating dummy output 
    correct_output = model_output - 1
    # Ensuring that the loss is equal to 1.
    loss = tf.keras.losses.mse(correct_output, model_output)

gradients = t.gradient(loss, model.trainable_variables)

“渐变”变量总是看起来像 [None, None, None, None, None, None]

我非常感谢有关如何解决此问题或如何解决非法移动问题的建议 不同的方式。

Answer 1

解决了。有多个问题导致此问题，我将在此处列出所有问题以及代码解决方案。

1. 需要将watch() 方法添加到GradientTape 块中，以便专门告诉GradientTape 跟踪model.trainable_variables 的梯度。
2. 模型的输出需要是model(input)得到的张量，而不是model.predict(input)得到的numpy数组。
3. 模型的输出还需要在 GradientTape 内计算，而不是在其外计算。
4. correct_output 对象需要是张量而不是 numpy 数组。
5. MSE 损失函数不适用于此模型，因为它具有 Softmax 输出层，并且在架构上更接近分类网络。而是使用了 categorical_crossentropy 损失函数。

完整代码：

with tf.GradientTape(watch_accessed_variables=False) as t:
    t.watch(model.trainable_variables)
    model_output = model(bot_state.reshape(1, 52 + 52 + 2 + 3 + 3))
    # Creating dummy output - action 1 is illegal
    correct_output = tf.convert_to_tensor([[0.5, 0, 0.5]])
    # Calculating loss
    loss = tf.keras.losses.categorical_crossentropy(correct_output, model_output)

gradients = t.gradient(loss, model.trainable_variables)