Keras / Theano 异常被掩盖答案

【问题标题】：Keras / Theano exceptions are getting maskedKeras / Theano 异常被掩盖
【发布时间】：2018-04-02 15:38:48
【问题描述】：

我正在使用进化算法为用 Keras/Theano 编写的 CNN 找到令人满意的超参数。这种方法的随机性意味着有时会尝试一种病态配置，这将产生异常。在这些情况下，我想捕捉异常，以便分配适当的低适应度。不幸的是，当 Theano 抛出异常时，它似乎在到达我的 try/catch 块之前被屏蔽了。也就是说，在某些时候异常会被捕获并且不会重新引发，这意味着它永远不会向上传播到堆栈以到达我的 try/catch 块。

我已经在 Keras Slack 工作区询问是否有一些配置我必须在 Keras 中处理以取消屏蔽这些异常，但我被告知问题不在 Keras 级别，而是与 Theano 相关.而且，我来了。

我希望能解决问题的相应 theanorc 文件顶部有以下配置设置：

[config]
on_opt_error = raise
on_shape_error = raise
numpy.seterr_all = raise
compute_test_value = raise

而且，这些是我看到的例外情况：

ERROR (theano.gof.opt): SeqOptimizer apply <theano.tensor.opt.ShapeOptimizer object at 0x2aaae03674a8>
ERROR (theano.gof.opt): Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/gof/opt.py", line 235, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/gof/opt.py", line 83, in optimize
    self.add_requirements(fgraph)
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/tensor/opt.py", line 1482, in add_requirements
    fgraph.attach_feature(ShapeFeature())
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/gof/fg.py", line 541, in attach_feature
    attach(self)
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/tensor/opt.py", line 1299, in on_attach
    self.on_import(fgraph, node, reason='on_attach')
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/tensor/opt.py", line 1362, in on_import
    self.set_shape(r, s)
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/tensor/opt.py", line 1151, in set_shape
    shape_vars.append(self.unpack(s[i], r))
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/theano/tensor/opt.py", line 1073, in unpack
    raise ValueError(msg)
ValueError: There is a negative shape in the graph!
Backtrace when that variable is created:

  File "/ccs/proj/geo121/mcoletti/dl-4-settlement-mapping/eadl/train_cnn.py", line 218, in <module>
    validation_accuracy = train_cnn(data_dir=args.data_dir, kernel_sizes=args.kernel_sizes, max_epoch=args.epoch, batch_sizes=args.batch_size)
  File "/ccs/proj/geo121/mcoletti/dl-4-settlement-mapping/eadl/train_cnn.py", line 193, in train_cnn
    model = create_cnn(kernel_sizes=kernel_sizes)
  File "/ccs/proj/geo121/mcoletti/dl-4-settlement-mapping/eadl/train_cnn.py", line 52, in create_cnn
    model.add(Conv2D(256, kernel_size=kernel_sizes[3], activation="relu", kernel_initializer="normal"))
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/keras/models.py", line 475, in add
    output_tensor = layer(self.outputs[0])
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/keras/engine/topology.py", line 602, in __call__
    output = self.call(inputs, **kwargs)
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/keras/layers/convolutional.py", line 164, in call
    dilation_rate=self.dilation_rate)
  File "/ccs/proj/geo121/python3.5-packages/dl4sm/keras/backend/theano_backend.py", line 1890, in conv2d
    filter_dilation=dilation_rate)

而且，如果您想看看 try/catch 块，就是这样：

   try:
        validation_accuracy = train_cnn(data_dir=args.data_dir, kernel_sizes=args.kernel_sizes, max_epoch=args.epoch, batch_sizes=args.batch_size)
   except Exception as e:
        print(socket.gethostname(), ', Caught exception while training:', str(e) )

我的直觉是，这可能是非常非常简单的事情。也许我需要向 THEANORC 文件添加更多选项？

【问题讨论】：

添加这行可能已经完成了这项工作：theano.config.compute_test_value = 'raise'

标签： keras theano

【解决方案1】：

设置theano.config.compute_test_value = 'raise' 似乎有效。

奇怪的是，compute_test_value 应该是从 Theano 配置文件中设置的，这表明它没有被正确读取和解析。当我在配置文件中明确设置它时，我不应该以编程方式设置它。

【讨论】：