【问题标题】:Can I use cuda-memcheck on a python program我可以在 python 程序上使用 cuda-memcheck
【发布时间】:2023-04-21 02:48:01
【问题描述】:

我有很多 cuda 内核要测试,所以我希望能够通过从 python 程序执行它们来测试它们(python 程序调用一个启动 cuda 内核的库),即类似于cuda-memcheck --leak-check full -- "python my_test_program.py"。与编写 C++ 程序来做同样的事情相比,这将节省时间。这是否与测试内核形成 *.cc 或 *.cu 程序的方式相同?

【问题讨论】:

  • 我同时使用 cuda-memchecknvprof 与 python + numba/cuda 和 python + pycuda。

标签: python cuda nvidia


【解决方案1】:

是的,您可以在 python 程序中使用 cuda-memcheck 和 nvprof 等工具。例如,考虑以下(故意)有缺陷的 Numba 示例:

$ cat numbatest.py
from __future__ import division
from numba import cuda
import numpy
import math

# CUDA kernel
@cuda.jit
def my_kernel(io_array):
    pos = cuda.grid(1)
    if pos > io_array.size:
        io_array[pos] *= 2 # do the computation

# Host code   
data = numpy.ones(62)
threadsperblock = 64
blockspergrid = math.ceil(data.shape[0] / threadsperblock)
my_kernel[blockspergrid, threadsperblock](data)
print(data)

这个内核在运行时会产生越界的内存访问。使用 cuda-memcheck 执行此操作表明情况确实如此:

$ cuda-memcheck python numbatest.py 
========= CUDA-MEMCHECK
========= Invalid __global__ read of size 8
=========     at 0x000000b8 in cudapy::__main__::my_kernel$241(Array<double, int=1, C, mutable, aligned>)
=========     by thread (63,0,0) in block (0,0,0)
=========     Address 0x7040601f8 is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so (cuLaunchKernel + 0x2cd) [0x2486ed]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call_unix64 + 0x4c) [0x6adc]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call + 0x1f2) [0x6282]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (_ctypes_callproc + 0x2ce) [0x12e4e]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so [0x13885]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
=========     Host Frame:python [0x19642e]
=========     Host Frame:python [0x196fc1]
=========     Host Frame:python [0x19ccf5]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
=========     Host Frame:python [0x196d8b]
=========     Host Frame:python [0x19ccf5]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
=========     Host Frame:python [0x196124]
=========     Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
=========     Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
=========     Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python [0x171127]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
=========     Host Frame:python [0x19642e]
=========     Host Frame:python [0x196fc1]
=========     Host Frame:python [0x19ccf5]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
=========     Host Frame:python [0x196124]
=========     Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
=========     Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
=========     Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python [0x171127]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
=========     Host Frame:python [0x196124]
=========     Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
=========     Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
=========     Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python [0x171127]
=========     Host Frame:python (_PyObject_FastCallDict + 0x8b) [0x1135bb]
=========     Host Frame:python [0x19cd6e]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
=========     Host Frame:python (PyEval_EvalCodeEx + 0x329) [0x197ad9]
=========     Host Frame:python (PyEval_EvalCode + 0x1c) [0x19887c]
=========     Host Frame:python [0x219074]
=========     Host Frame:python (PyRun_FileExFlags + 0xa1) [0x219471]
=========     Host Frame:python (PyRun_SimpleFileExFlags + 0x1c3) [0x219673]
=========     Host Frame:python (Py_Main + 0x650) [0x21d1b0]
=========     Host Frame:python (main + 0xee) [0xe4b4e]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21f45]
=========     Host Frame:python [0x1c61a8]
=========
========= Program hit CUDA_ERROR_LAUNCH_FAILED (error 719) due to "unspecified launch failure" on CUDA API call to cuMemcpyDtoH_v2. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so (cuMemcpyDtoH_v2 + 0x173) [0x2436a3]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call_unix64 + 0x4c) [0x6adc]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call + 0x1f2) [0x6282]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (_ctypes_callproc + 0x2ce) [0x12e4e]
=========     Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so [0x13885]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
=========     Host Frame:python (PyEval_EvalCodeEx + 0x966) [0x198116]
=========     Host Frame:python [0x1988f4]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
=========     Host Frame:python [0x196124]
=========     Host Frame:python [0x196fc1]
=========     Host Frame:python [0x19ccf5]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
=========     Host Frame:python (PyEval_EvalCodeEx + 0x329) [0x197ad9]
=========     Host Frame:python [0x198a06]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
=========     Host Frame:python [0x1962c6]
=========     Host Frame:python [0x196fc1]
=========     Host Frame:python [0x19ccf5]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
=========     Host Frame:python [0x19642e]
=========     Host Frame:python [0x196fc1]
=========     Host Frame:python [0x19ccf5]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
=========     Host Frame:python [0x19642e]
=========     Host Frame:python [0x196fc1]
=========     Host Frame:python [0x19ccf5]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
=========     Host Frame:python [0x196124]
=========     Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
=========     Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
=========     Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python [0x171127]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
=========     Host Frame:python [0x196124]
=========     Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
=========     Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
=========     Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
=========     Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
=========     Host Frame:python [0x171127]
=========     Host Frame:python (_PyObject_FastCallDict + 0x8b) [0x1135bb]
Traceback (most recent call last):
=========     Host Frame:python [0x19cd6e]
=========     Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
  File "numbatest.py", line 17, in <module>
=========     Host Frame:python (PyEval_EvalCodeEx + 0x329) [0x197ad9]
=========     Host Frame:python (PyEval_EvalCode + 0x1c) [0x19887c]
=========     Host Frame:python [0x219074]
=========     Host Frame:python (PyRun_FileExFlags + 0xa1) [0x219471]
=========     Host Frame:python (PyRun_SimpleFileExFlags + 0x1c3) [0x219673]
=========     Host Frame:python (Py_Main + 0x650) [0x21d1b0]
=========     Host Frame:python (main + 0xee) [0xe4b4e]
=========
    my_kernel[blockspergrid, threadsperblock](data)
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 704, in __call__
    cfg(*args)
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 484, in __call__
    sharedmem=self.sharedmem)
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 586, in _kernel_call
    wb()
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 601, in <lambda>
    retr.append(lambda: devary.copy_to_host(val, stream=stream))
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context
    return fn(*args, **kws)
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 241, in copy_to_host
    _driver.device_to_host(hostary, self, self.alloc_size, stream=stream)
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1606, in device_to_host
    fn(host_pointer(dst), device_pointer(src), size, *varargs)
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 288, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 323, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [719] Call to cuMemcpyDtoH results in CUDA_ERROR_LAUNCH_FAILED
========= ERROR SUMMARY: 2 errors

这将适用于任何使用 CUDA 加速库(Tensorflow、PyTorch 等)或生成 CUDA 代码并运行它(Numba、PyCUDA)的 Python 进程。

【讨论】: