prometheus 告警 gpu 卡的ecc错误

prometheus有两个指标可以告警gpu卡的错误

# HELP dcgm_ecc_dbe_volatile_total Total number of double-bit volatile ECC errors.  双位易失性ECC错误的总数
# TYPE dcgm_ecc_dbe_volatile_total counter
dcgm_ecc_dbe_volatile_total{gpu="0",uuid="GPU-4d52e430-b8c7-a0b9-7fda-4aa825af5c97"} 0


# HELP dcgm_ecc_sbe_volatile_total Total number of single-bit volatile ECC errors.   单位易失性ECC错误的总数
# TYPE dcgm_ecc_sbe_volatile_total counter
dcgm_ecc_sbe_volatile_total{gpu="0",uuid="GPU-4d52e430-b8c7-a0b9-7fda-4aa825af5c97"} 0

2021-06-10
2021-08-20
2021-07-09
2021-10-01
2023-02-23
2021-12-18
2022-12-23
2022-12-23