relu6 = min(max(features, 0), 6)
This is useful in making the networks ready for fixed-point inference. If you unbound the upper limit, you lose too many bits to the Q part of a Q.f number.
Keeping the ReLUs bounded by 6 will let them take a max of 3 bits (upto 8) leaving 4/5 bits for .f
this used to be called a "hard sigmoid"
paper：Convolutional Deep Belief Networks on CIFAR-10
In our tests, this encourages the model to learn sparse features earlier. In the formulation of [8], this is equivalent to imagining that each ReLU unit consists of only 6 replicated bias-shifted Bernoulli units, rather than an innute amount. We will refer to ReLU units capped at n as ReLU-n units