使用 Tensorflowjs 对上传的图像进行预测时，React 状态总是落后一步

Question

我正在开发一个 React 应用程序，使用 Tensorflow.js 根据图像对 Pokemon 进行分类。

• 我想要什么 - 上传口袋妖怪的图片，为同一个口袋妖怪生成预测。

• 实际发生的情况 - 当我上传一张图片进行预测时，输出始终是前一张图片。所以，第一个预测总是垃圾（随机口袋妖怪）。我上传第二个口袋妖怪时得到的预测总是针对第一个口袋妖怪。上传后的第 3 个 Pokemon 会预测第 2 个 Pokemon，依此类推。

请参阅问题的底部，了解它在哪些地方有效和在哪些地方无效。

这里是相关的代码-

1. 我首先检查模型是否存在于indexeddb，如果存在，我将其加载到状态model。如果没有，我从服务器获取它并将其存储在状态中。这就是第一个 useEffect 在第一次渲染页面时所做的事情。

2. 我使用另一个useEffect，只要findState.uploadedImage 更改，它就会运行。这种状态存在于 Redux-toolkit 中。

这是问题的简短演示 => https://youtu.be/MX70zbupNWQ

这是应用网址 => https://poke-zoo.herokuapp.com/

这是 Github 存储库 => https://github.com/theairbend3r/poke-zoo/tree/master/frontend/src/features/find

这是文件SearchOutput.js。这会获取模型并进行预测。

const SearchOutput = () => {
  const findState = useSelector(selectorFind)
  const dispatch = useDispatch()
  const imageRef = useRef(null)

  const [model, setModel] = useState(null)
  const [predictions, setPredictions] = useState([])

  const MODEL_HTTP_URL = "api/pokeml/classify"
  const MODEL_INDEXEDDB_URL = "indexeddb://poke-model"

  useEffect(() => {
    async function fetchModel() {
      try {
        const localClassifierModel = await tf.loadLayersModel(
          MODEL_INDEXEDDB_URL
        )

        setModel(localClassifierModel)
        console.log("Model loaded from IndexedDB")
      } catch (e) {
        const classifierModel = await tf.loadLayersModel(MODEL_HTTP_URL)
        setModel(classifierModel)

        await classifierModel.save(MODEL_INDEXEDDB_URL)

        console.error(e)
      }
    }
    fetchModel()
  }, [])

  const getTopKPred = (pred, k) => {
    const predIdx = []
    const predNames = []

    const topkPred = [...pred].sort((a, b) => b - a).slice(0, k)

    topkPred.map(i => predIdx.push(pred.indexOf(i)))
    predIdx.map(i => predNames.push(idx2class[i]))

    return predNames
  }

  useEffect(() => {
    async function makePredictions() {
      if (imageRef && model) {
        try {
          const imgTensor = tf.browser
            .fromPixels(imageRef.current)
            .resizeNearestNeighbor([160, 160])
            .toFloat()
            .sub(127.5)
            .div(127.5)
            .expandDims()

          const y_pred = await model.predict(imgTensor).dataSync()
          const topkPredNames = getTopKPred(y_pred, 5)

          console.log(topkPredNames)
          return topkPredNames
        } catch (e) {
          console.log("Unable to run predictions.")
        }
      }
    }
    makePredictions()
  }, [findState.uploadedImage])

  return (
     <div>
        {findState.uploadedImage && (
          <img
            ref={imageRef}
            tw="border border-purple-700 p-1 rounded shadow-lg"
            src={findState.uploadedImage}
            width={600}
            height={600}
          />
        )}


        <div>
           {findState.matchesFound.length === 6 &&
             findState.matchesFound.map(poke => (
               <PokemonCardML
                 key={`key-${poke.id}`}
                 pokemonId={poke.id}
                 pokemonName={poke.name}
                 pokemonType={poke.type}
                 pokemonHeight={poke.height}
                 pokemonWeight={poke.weight}
                 pokemonBaseExperience={poke.baseExperience}
                 pokemonSprite={poke.sprites}
               />
             ))}
       </div>
     </div>
)
}

这是文件findSlice.js，它将输入图像存储到redux状态。

import { createSlice } from "@reduxjs/toolkit"
import axios from "axios"

const initialState = {
  uploadedImage: "",
  model: null,
  matchesFound: [],
}

export const findSlice = createSlice({
  name: "find",
  initialState: initialState,
  reducers: {
    storeInputImage: (state, action) => {
      state.uploadedImage = action.payload.uploadedImage
    },
    setModel: (state, action) => {
      state.model = action.payload.model
    },
  },
})

export const selectorFind = state => state.find
export const { storeInputImage, setModel } = findSlice.actions
export default findSlice.reducer

---

有关问题的详细信息。

### Desktop

#### Table

|         |   Ubuntu    |     Windows      |      MacOS       |
| ------- | :---------: | :--------------: | :--------------: |
| Firefox | not working |   not working    |   not working    |
| Chrome  | not working | somewhat working | somewhat working |
| Safari  |     NA      |        NA        | somewhat working |

#### Comments

|         |                                    Ubuntu                                    |                                   Windows                                    |                                    MacOS                                     |
| :-----: | :--------------------------------------------------------------------------: | :--------------------------------------------------------------------------: | :--------------------------------------------------------------------------: |
| Firefox | Predictions are always one step behin for both Captured and Uploaded images. | Predictions are always one step behin for both Captured and Uploaded images. | Predictions are always one step behin for both Captured and Uploaded images. |
| Chrome  | Works only on Captured Images. Uploaded images give same predictions always. | Works only on Captured Images. Uploaded images give same predictions always. | Works only on Captured Images. Uploaded images give same predictions always. |
| Safari  |                                      NA                                      |                                      NA                                      | Works only on Captured Images. Uploaded images give same predictions always. |

### Mobile

#### Table

|         |     Android      |       iOS        |
| ------- | :--------------: | :--------------: |
| Firefox | somewhat working |   not working    |
| Chrome  | somewhat working |   not working    |
| Safari  |        NA        | somewhat working |

#### Comments

|         | Android                                                                     | iOS                                                                          |
| :-----: | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Firefox | No predictions load for a captured image. Works on uploaded images only.    | Camera does not load. Uploaded images give same predictions always.          |
| Chrome  | Works on capture images only. Uploaded images give same predictions always. | Camera does not load. Uploaded images give same predictions always.          |
| Safari  | NA                                                                          | Works only on Captured Images. Uploaded images give same predictions always. |

---

编辑：基于以下建议。这并没有解决问题。放在这里供参考。

const SearchOutput = () => {
  const findState = useSelector(selectorFind)
  const [imageRef, setImageRef] = useState(null)

  const onChangeRef = useCallback(node => {
    setImageRef(node)
  }, [])

  const [model, setModel] = useState(null)
  const [predictions, setPredictions] = useState([])

  const MODEL_HTTP_URL = "api/pokeml/classify"
  const MODEL_INDEXEDDB_URL = "indexeddb://poke-model"


  useEffect(() => {
    async function fetchModel() {
      try {
        const localClassifierModel = await tf.loadLayersModel(
          MODEL_INDEXEDDB_URL
        )

        setModel(localClassifierModel)
        console.log("Model loaded from IndexedDB")
      } catch (e) {
        try {
          const classifierModel = await tf.loadLayersModel(MODEL_HTTP_URL)
          setModel(classifierModel)

          await classifierModel.save(MODEL_INDEXEDDB_URL)
          console.log("Model saved to IndexedDB")
        } catch (e) {
          console.log("Unable to load model at all: ", e)
        }
      }
    }
    fetchModel()
  }, [])

  useEffect(() => {
    async function makePredictions() {
      if (imageRef && model) {
        console.log(
          "Uploaded Image from inside the useEffect",
          findState.uploadedImage
        )
        console.log("ImageRef from inside the useEffect", imageRef.current)
        try {
          const imgTensor = tf.browser
            .fromPixels(imageRef.current)
            .resizeNearestNeighbor([160, 160])
            .toFloat()
            .sub(127)
            .div(127)
            .expandDims()

          const y_pred = await model.predict(imgTensor).data()
          console.log(y_pred)
          console.log(pokemonState)

          const topkPredNames = getTopKPredPokeObj(y_pred, 6, pokemonState)

          dispatch(storePredictions({ predictions: topkPredNames }))

          console.log(topkPredNames)

          return topkPredNames
        } catch (e) {
          console.log("Unable to run predictions.", e)
        }
      }
    }
    makePredictions()
  }, [findState.uploadedImage])

  return (
    <div>
          {findState.uploadedImage && (
            <img
              ref={onChangeRef}
              src={findState.uploadedImage}
              width="600"
              height="600"
            />
          )}
    </div>
  )
}

export default SearchOutput

Answer 1

我之前已经阅读过这个问题，我自己没有遇到过这个问题，但这应该会对你有所帮助。

useRef 钩子可以成为你的自定义钩子的陷阱，如果你把它结合起来的话 使用跳过渲染的 useEffect。你的第一直觉是 将 ref.current 添加到 useEffect 的第二个参数，所以它会更新 一旦 ref 改变。但是 ref 直到你之后才会更新 组件已渲染——意思是任何跳过渲染的 useEffect， 在下一次渲染之前不会看到对 ref 的任何更改。

正如您所遇到的行为所见，useEffect 中的 imageRef.current 没有返回它正在记录先前值的更新对象。

解决这个问题的一种合乎逻辑的方法是：

  useEffect(() => {
    async function makePredictions() {
      //...
    }
    makePredictions()
  }, [findState.uploadedImage, imageRef.current])

但问题在于 imageRef.current 中的更改不会触发 React 中的渲染。因此，根据 React 关于如何测量 DOM 节点的文档，您应该使用 useCallback 而不是 useRef。

这样的事情应该可以工作：

  const [imageRef, setImageRef] = useState(null);
  const onChangeRef = useCallback(node => {
    // ref value changed to node
    setImageRef(node); // e.g. change ref state to trigger re-render
    if (node === null) { 
      // node is null, if DOM node of ref had been unmounted before
    } else {
      // ref value exists
    }
  }, []);
  
  useEffect(() => {
    async function makePredictions() {
      if (imageRef && model) {
        try {
          const imgTensor = tf.browser
            .fromPixels(imageRef)
            .resizeNearestNeighbor([160, 160])
            .toFloat()
            .sub(127.5)
            .div(127.5)
            .expandDims()

          const y_pred = await model.predict(imgTensor).dataSync()
          const topkPredNames = getTopKPred(y_pred, 5)

          console.log(topkPredNames)
          return topkPredNames
        } catch (e) {
          console.log("Unable to run predictions.")
        }
      }
    }
    makePredictions()
  }, [findState.uploadedImage, imageRef])

您应该使用<img ref={onChangeRef} />，而不是<img ref={imageRef} />。

参考资料：

Ref objects inside useEffect Hooks
How can I measure a DOM node?

Answer 2

乍一看，原因可能是包含 makePredictions 函数的 useEffect 中缺少“模型”的依赖项。由于您使用的是模型，并且仅在更新 findState.uploadedImage 时才会触发效果。 imageRef 是一个参考，因此它不必是一个依赖项。 “模型”必须是，因为它在您的流程中是一个不断变化的状态。

  useEffect(() => {
    async function makePredictions() {
      if (imageRef && model) {
        try {
          const imgTensor = tf.browser
            .fromPixels(imageRef)
            .resizeNearestNeighbor([160, 160])
            .toFloat()
            .sub(127.5)
            .div(127.5)
            .expandDims()

          const y_pred = await model.predict(imgTensor).data()
          const topkPredNames = getTopKPred(y_pred, 5)

          console.log(topkPredNames)
          return topkPredNames
        } catch (e) {
          console.log("Unable to run predictions.", e)
        }
      }
    }
    makePredictions()
  }, [findState.uploadedImage, model]) // added model and remove imageRef

我希望这会有所帮助。我没有完全尝试了解到底发生了什么，但这引起了我的注意并且通常会导致此类错误。