【发布时间】：2018-11-10 22:07:28
【问题描述】：
我在 Windows 10 上的 matlab 2018b 中运行更快的 R-CNN。当我增加训练项目的数量或增加 MaxEpoch 时，我遇到了异常 CUDA_ERROR_ILLEGAL_ADDRESS。
以下是我gpuDevice的资料
CUDADevice with properties:
Name: 'GeForce GTX 1050'
Index: 1
ComputeCapability: '6.1'
SupportsDouble: 1
DriverVersion: 9.2000
ToolkitVersion: 9.1000
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 4.2950e+09
AvailableMemory: 3.4635e+09
MultiprocessorCount: 5
ClockRateKHz: 1493000
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1
这是我的代码
latest_index =0;
for i=1:6
load (strcat('newDataset', int2str(i), '.mat'));
len =length(vehicleDataset.imageFilename);
for j=1:len
filename = vehicleDataset.imageFilename{j};
latest_index=latest_index+1;
fulldata.imageFilename{latest_index} = filename;
fulldata.vehicle{latest_index} = vehicleDataset.vehicle{j};
end
end
trainingDataTable = table(fulldata.imageFilename', fulldata.vehicle');
trainingDataTable.Properties.VariableNames = {'imageFilename','vehicle'};
data.trainingDataTable = trainingDataTable;
trainingDataTable(1:4,:)
% Split data into a training and test set.
idx = floor(0.6 * height(trainingDataTable));
trainingData = trainingDataTable(1:idx,:);
testData = trainingDataTable(idx:end,:);
% Create image input layer.
inputLayer = imageInputLayer([32 32 3]);
% Define the convolutional layer parameters.
filterSize = [3 3];
numFilters = 64;
% Create the middle layers.
middleLayers = [
convolution2dLayer(filterSize, numFilters, 'Padding', 1)
reluLayer()
convolution2dLayer(filterSize, numFilters, 'Padding', 1)
reluLayer()
maxPooling2dLayer(3, 'Stride',2)
];
finalLayers = [
fullyConnectedLayer(128)
% Add a ReLU non-linearity.
reluLayer()
fullyConnectedLayer(width(trainingDataTable))
% Add the softmax loss layer and classification layer.
softmaxLayer()
classificationLayer()
];
layers = [
inputLayer
middleLayers
finalLayers
];
% Options for step 1.
optionsStage1 = trainingOptions('sgdm', ...
'MaxEpochs', 2, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);
% Options for step 2.
optionsStage2 = trainingOptions('sgdm', ...
'MaxEpochs', 2, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);
% Options for step 3.
optionsStage3 = trainingOptions('sgdm', ...
'MaxEpochs', 2, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);
% Options for step 4.
optionsStage4 = trainingOptions('sgdm', ...
'MaxEpochs', 2, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);
options = [
optionsStage1
optionsStage2
optionsStage3
optionsStage4
];
doTrainingAndEval = true;
if doTrainingAndEval
% Set random seed to ensure example training reproducibility.
rng(0);
% Train Faster R-CNN detector. Select a BoxPyramidScale of 1.2 to allow
% for finer resolution for multiscale object detection.
detector = trainFasterRCNNObjectDetector(trainingData, layers, options, ...
'NegativeOverlapRange', [0 0.3], ...
'PositiveOverlapRange', [0.6 1], ...
'BoxPyramidScale', 1.2);
data.detector= detector;
else
% Load pretrained detector for the example.
detector = data.detector;
end
save mix_data data
if doTrainingAndEval
% Run detector on each image in the test set and collect results.
resultsStruct = struct([]);
for i = 1:height(testData)
% Read the image.
I = imread(testData.imageFilename{i});
% Run the detector.
[bboxes, scores, labels] = detect(detector, I);
% Collect the results.
resultsStruct(i).Boxes = bboxes;
resultsStruct(i).Scores = scores;
resultsStruct(i).Labels = labels;
end
% Convert the results into a table.
results = struct2table(resultsStruct);
data.results = results;
save mix_data data
else
% Load results from disk.
results = data.results;
end
% Extract expected bounding box locations from test data.
expectedResults = testData(:, 2:end);
% Evaluate the object detector using Average Precision metric.
[ap, recall, precision] = evaluateDetectionPrecision(results, expectedResults);
% Plot precision/recall curve
figure
plot(recall,precision)
xlabel('Recall')
ylabel('Precision')
grid on
title(sprintf('Average Precision = %.2f', ap))
首先它会多次打印警告并抛出以下异常
警告：CUDA 执行期间发生意外错误。 CUDA 错误是： CUDA_ERROR_ILLEGAL_ADDRESS 在 trainFasterRCNNObjectDetector 中（第 320 行） 在 rcnn_trail（第 184 行）
错误使用 - CUDA 执行期间发生意外错误。 CUDA 错误是： CUDA_ERROR_ILLEGAL_ADDRESS
vision.internal.cnn.layer.SmoothL1Loss/backwardLoss 中的错误（第 156 行） idx = (X > -one) & (X
nnet.internal.cnn.DAGNetwork 中的错误>@()efficientBackProp(i)（第 661 行） @() 高效BackProp(i), ...
nnet.internal.cnn.util.executeWithStagedGPUOOMRecovery 中的错误（第 11 行） [ varargout{1:nOutputs} ] = computeFun();
nnet.internal.cnn.DAGNetwork 中的错误>iExecuteWithStagedGPUOOMRecovery（第 1195 行） [varargout{1:nargout}] = nnet.internal.cnn.util.executeWithStagedGPUOOMRecovery(varargin{:});
nnet.internal.cnn.DAGNetwork/computeGradientsForTraining 中的错误（第 660 行） 这些梯度 = iExecuteWithStagedGPUOOMRecovery( ...
nnet.internal.cnn.Trainer/computeGradients 中的错误（第 184 行） [梯度、预测、状态] = net.computeGradientsForTraining(X, Y, 需要状态训练，传播状态）；
nnet.internal.cnn.Trainer/train 中的错误（第 85 行） [梯度、预测、状态] = this.computeGradients(net, X, response, 需要状态训练，传播状态）；
vision.internal.cnn.trainNetwork 中的错误（第 47 行） trainedNet = trainer.train(trainedNet, trainingDispatcher);
fastRCNNObjectDetector.train 中的错误（第 190 行） [网络，信息] = vision.internal.cnn.trainNetwork(ds, lgraph, opts, mapping, checkpointSaver);
trainFasterRCNNObjectDetector 中的错误（第 410 行） [stage2Detector, fastRCNN, ~, info(2)] = fastRCNNObjectDetector.train(trainingData, fastRCNN, options(2), iStageTwoParams(params), checkpointSaver);
rcnn_trail 中的错误（第 184 行） 检测器 = trainFasterRCNNObjectDetector(trainingData, 层, 选项, ...
【问题讨论】：
标签： matlab deep-learning