How to optimize DNN
Initial Weight Value
- It is very important to select the initial value of weights and biases.
- Until now, in several example codes, the initial values of weights and biases are chosen randomly. Therefore, results are varied at every training.
- Still, how to set the initial values of weights and biases is researched and developed, but there is strict anti-pattern for the initialization.
0 for Weight
- 0 should not be chosen for weights and biases.
- If one of weights or biases is 0, the result of forward propagation is 0. Also, for back propagation, 0 will prevent other weights and biases from updating.
RBM Initialization
- Many approaches to select good initial values are proposed until now.
- One of popular approaches is Restricted Boltzmann Machine with Deep Belief Network. However, it requires high computation power to select initial values. - Wiki
Xavier/He initialization
- Another popular approach is Xavier initialization. It focuses on the numbers of inputs and outputs.
W = np.random.randn(fan_in, fan_out) / np.sqrt(fan_in)
- It provides fast training speed and high accuracy, but there was an optimal point.
- Xavier's optimal version for ReLU is He initialization.
W = np.random.randn(fan_in, fan_out) / np.sqrt(fan_in / 2)
- The MNIST performances of typical standard deviation method, Xaliver and He are compared in the below graph
Image 1. Performance comparison among Xaliver, std and He
- For std and Xavier, sigmoid is used as activation function, and He uses ReLU.
- He shows smaller cost during training, and it learns faster than Xavier.
Drop Out
- Drop out is one of powerful technique to avoid overfitting.
- Randomly set some neurons to zero in the forward pass.
- In Drop out graph, grey nodes are dropped out, and grey lines are not linked practically. Therefore, only black nodes and lines are working.
- Drop out technique should be applied during training, not inference. For inference, DNN should use all nodes and links.
import numpy an np
class Dropout:
def __init__(self, ratio=0.5):
self._ration = ration
self._mask = None
def foward(self, X, isTrain=True):
if isTrain:
self._mask = np.random.rand(*X.shape) > self._ratio
return X * self._mask
else:
return X
def backward(self, d):
return d * self._mask
Image 2. MNIST training without Drop out (300 epochs)
Image 3. MNIST training with Drop out (300 epochs
Image 3. MNIST training with Drop out (600 epochs
- As a result, the difference between train and test become reduced, and it requires more training time for high accuracy.
Ensemble
- Ensemble is to inference with combination of independently trained multiple DNN machine.
- Its concept is similar to Drop out. Training multiple machines is similar to dropping out nodes randomly, and inference with combination of multiple machines is similar to using all nodes in drop out technique.
COMMENTS