Chrome T-Rex-Game 强化学习没有改善答案

【问题标题】：Chrome T-Rex-Game Reinforcement learning showing no improvementChrome T-Rex-Game 强化学习没有改善
【发布时间】：2020-06-14 01:26:51
【问题描述】：

我想为 Chrome-No-Internet-Dino-Game 创建一个 AI。因此我调整了这个Github-Repository 来满足我的需要。我使用以下公式计算新的 Q：

来源：https://en.wikipedia.org/wiki/Q-learning

我现在的问题是，即使经过大约 2.000.000 次迭代，我的游戏分数也没有增加。

您可以在这里找到游戏文件：https://pastebin.com/XrwQ0suJ

QLearning.py：

import pickle
import Game_headless
import Game
import numpy as np
from collections import defaultdict

rewardAlive = 1
rewardKill = -10000
alpha = 0.2  # Learningrate
gamma = 0.9  # Discount

Q = defaultdict(lambda: [0, 0, 0])  # 0 = Jump / 1 = Duck / 2 = Do Nothing

oldState = None
oldAction = None

gameCounter = 0
gameScores = []


def paramsToState(params):
    cactus1X = round(params["cactus1X"] / 10) * 10
    cactus2X = round(params["cactus2X"] / 10) * 10
    cactus1Height = params["cactus1Height"]
    cactus2Height = params["cactus2Height"]
    pteraX = round(params["pteraX"] / 10) * 10
    pteraY = params["pteraY"]
    playerY = round(params["playerY"] / 10) * 10
    gamespeed = params["gamespeed"]

    return str(cactus1X) + "_" + str(cactus2X) + "_" + str(cactus1Height) + "_" + \
           str(cactus2Height) + "_" + str(pteraX) + "_" + str(pteraY) + "_" + \
           str(playerY) + "_" + str(gamespeed)


def shouldEmulateKeyPress(params):  # 0 = Jump / 1 = Duck / 2 = Do Nothing

    global oldState
    global oldAction

    state = paramsToState(params)
    oldState = state
    estReward = Q[state]
    action = estReward.index(max(estReward))
    if oldAction is None:
        oldAction = action
        return action

    # Previous action was successful
    # -> Update Q
    prevReward = Q[oldState]
    prevReward[oldAction] = (1 - alpha) * prevReward[oldAction] + \
                            alpha * (rewardAlive + gamma * max(estReward))
    Q[oldState] = prevReward
    oldAction = action
    return action


def onGameOver(score):
    # Previous action was NOT successful
    # -> Update Q
    global oldState
    global oldAction
    global gameCounter
    global gameScores

    gameScores.append(score)

    if gameCounter % 10000 == 0:
        print(f"{gameCounter} : {np.mean(gameScores[-100:])}")

    prevReward = Q[oldState]
    prevReward[oldAction] = (1 - alpha) * prevReward[oldAction] + \
                            alpha * rewardKill
    Q[oldState] = prevReward

    oldState = None
    oldAction = None

    if gameCounter % 10000 == 0:
        with open("Q\\" + str(gameCounter) + ".pickle", "wb") as file:
            pickle.dump(dict(Q), file)

    gameCounter += 1


Game_headless.main(shouldEmulateKeyPress, onGameOver)

在每一帧上，来自Game_headless.py 的gameplay() 函数调用shouldEmulateKeyPress()。然后，该函数返回 0 表示 Jump，1 表示鸭子，2 表示空。我尝试调整常量，但没有显示任何效果。如果您有任何问题，请不要犹豫，问我！提前谢谢！

【问题讨论】：

标签： python machine-learning deep-learning reinforcement-learning q-learning

【解决方案2】：

我能够解决问题，但我真的不知道错误是什么。我在游戏功能的末尾添加了一个返回语句，现在不知何故它可以工作了。

【讨论】：