TensorFlow

Implementing super-convergence for deep neural network training in Tensorflow 2 with the 1Cycle learning rate policy.

Implementing the technique in Tensorflow 2 is straightforward. Start from a low learning rate, increase the learning rate and record the loss. Stop when a very high learning rate is reached. Plot the losses and learning rates choosing a learning rate where the loss is decreasing at a rapid rate.