add_loss() 用于规范层活动/权重不再适用于 tensorflow.keras 2.0 更新

Question

我之前在预训练网络上的 Tensorflow.keras 中添加了激活和/或内核的正则化，使用层上的循环：

if regul_what is 'kernel':
    for layer in model.layers:
        if isinstance(layer, DepthwiseConv2D):
                layer.add_loss(regularizers.l1_l2(l1,l2)(layer.depthwise_kernel))
        elif isinstance(layer, layers.Conv2D) or isinstance(layer, layers.Dense):
                layer.add_loss(regularizers.l1_l2(l1,l2)(layer.kernel))
if regul_what is 'activity':
    for layer in model.layers:
        if isinstance(layer, Activation):
            layer.add_loss(regularizers.l1_l2(l1,l2)(layer.output))

在升级到 tensorflow 2.0 之前它曾经可以工作（据我测试）。

现在我需要将整个框架更新到 tensorflow 2.0。前面的代码在执行时，在应用 add_loss() 时返回以下错误：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-123-cc0c5783731e> in <module>
      3         if ('_relu' in layer.name): #isinstance(layer, Activation):
      4             #layer.activity_regularizer = regularizers.l1_l2(l1,l2)
----> 5             layer.add_loss(regularizers.l1_l2(l1,l2)(layer.output))

~/miniconda/envs/l1l2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in add_loss(self, losses, inputs)
   1119     if eager_losses and not in_call_context:
   1120       raise ValueError(
-> 1121           'Expected a symbolic Tensors or a callable for the loss value. '
   1122           'Please wrap your loss computation in a zero argument `lambda`.')
   1123 

ValueError: Expected a symbolic Tensors or a callable for the loss value. Please wrap your loss computation in a zero argument `lambda`.

因此，我尝试引入零参数 lambda 函数，如下所示：

if regul_what is 'kernel':
    for layer in model.layers:
        if isinstance(layer, DepthwiseConv2D):
                layer.add_loss(lambda: regularizers.l1_l2(l1,l2)(layer.depthwise_kernel))
        elif isinstance(layer, layers.Conv2D) or isinstance(layer, layers.Dense):
                layer.add_loss(lambda: regularizers.l1_l2(l1,l2)(layer.kernel))
if regul_what is 'activity':
    for layer in model.layers:
        if isinstance(layer, Activation):
            layer.add_loss(lambda: regularizers.l1_l2(l1,l2)(layer.output))

随着 lambda 的引入，add_loss 循环没有错误地通过，但是当训练开始时我得到了错误：

File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 1297, in fit_generator
 steps_name='steps_per_epoch')
File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_generator.py", line 295, in model_iteration
 progbar.on_batch_end(step, batch_logs)
File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/tensorflow_core/python/keras/callbacks.py", line 760, in on_batch_end
 self.progbar.update(self.seen, self.log_values)
File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/tensorflow_core/python/keras/utils/generic_utils.py", line 440, in update
 avg = np.mean(self._values[k][0] / max(1, self._values[k][1]))
File "<__array_function__ internals>", line 6, in mean
File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 3257, in mean
 out=out, **kwargs)
File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/numpy/core/_methods.py", line 135, in _mean
 arr = asanyarray(a)
File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/numpy/core/_asarray.py", line 138, in asanyarray
 return array(a, dtype, copy=False, order=order, subok=True)
File "~/miniconda/envs/l1l2/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 736, in __array__
 " array.".format(self.name))
NotImplementedError: Cannot convert a symbolic Tensor (truediv:0) to a numpy array.

我不知道如何解决这个问题...提前感谢您的帮助！

Answer 1

看来问题实际上是因为TF2.0中默认的eager execution。

通过在脚本开头禁用急切执行：

import tensorflow as tf
tf.compat.v1.disable_eager_execution()

.. 原始版本（没有 lambda）运行顺畅。

这里也提到了 Eager execution 和 add_loss() 的不兼容：https://github.com/tensorflow/compression/issues/9