Why multi layer neural network is necessary
TOC
Single Layer Neural Network
- Until now, only single layer neural network is considered, not multi layer.
- Single layer neural network has only one hidden layer between input layer and output layer. - WikiBooks
Image 1. Single layer neural network
- Even though single layer neural network is not complicated, it shows good capability for linear regression, binary regression and softmax regression.
- Single layer neural network finds optimal weights and bias for a line which divides True group and False group. Therefore, Single layer neural network can do logical operation well, such as AND, OR, NAND and NOR problem.
Logical Operations with Single Layer Neural Network
- Weights and bias are coefficients to divide area of red and blue dots in this examples.
AND
- Truth table
| x1 | x2 | Y |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
Table 1. AND
import numpy as np
import matplotlib.pyplot as plt
def AND(x1, x2):
i = np.array([x1, x2])
# Wegith and threshold is defined manually for correct AND
w = np.array([0.5, 0.5])
b = -0.7
tmp = np.sum(i*w) + b
if tmp <= 0:
return 0
else:
return 1
X = [[0,0], [0,1], [1,0], [1,1]]
for x in X:
y = AND(x[0], x[1])
color = None
if y == 0:
color = "b"
else:
color = "r"
plt.plot(x[0], x[1], "o"+color)
# Dividing line
eq = lambda _x : -_x + 1.2
x1 = [i*0.1 for i in range(-4, 14)]
y1 = list(map(eq, x1))
plt.plot(x1, y1, "g")
plt.grid()
plt.xlim(-0.2, 1.2)
plt.ylim(-0.2, 1.2)
plt.title("AND")
plt.xlabel("x1")
plt.ylabel("x2")
plt.show()
Image 2. AND
OR
- Truth table
| x1 | x2 | Y |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 1 |
Table 2. OR
import numpy as np
import matplotlib.pyplot as plt
def OR(x1, x2):
i = np.array([x1, x2])
# Wegith and threshold is defined manually for correct AND
w = np.array([0.5, 0.5])
b = -0.2
tmp = np.sum(i*w) + b
if tmp <= 0:
return 0
else:
return 1
X = [[0,0], [0,1], [1,0], [1,1]]
for x in X:
y = OR(x[0], x[1])
color = None
if y == 0:
color = "b"
else:
color = "r"
plt.plot(x[0], x[1], "o"+color)
# Dividing line
eq = lambda _x : -_x + 0.4
x1 = [i*0.1 for i in range(-4, 14)]
y1 = list(map(eq, x1))
plt.plot(x1, y1, "g")
plt.grid()
plt.xlim(-0.2, 1.2)
plt.ylim(-0.2, 1.2)
plt.title("OR")
plt.xlabel("x1")
plt.ylabel("x2")
plt.show()
Image 3. OR
NAND
- Truth table
| x1 | x2 | Y |
|---|---|---|
| 0 | 0 | 1 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
Table 3. NAND
import numpy as np
import matplotlib.pyplot as plt
def NAND(x1, x2):
i = np.array([x1, x2])
# Wegith and threshold is defined manually for correct AND
w = np.array([-0.5, -0.5])
b = 0.7
tmp = np.sum(i*w) + b
if tmp <= 0:
return 0
else:
return 1
X = [[0,0], [0,1], [1,0], [1,1]]
for x in X:
y = NAND(x[0], x[1])
color = None
if y == 0:
color = "b"
else:
color = "r"
plt.plot(x[0], x[1], "o"+color)
# Dividing line
eq = lambda _x : -_x + 1.4
x1 = [i*0.1 for i in range(-4, 14)]
y1 = list(map(eq, x1))
plt.plot(x1, y1, "g")
plt.grid()
plt.xlim(-0.2, 1.2)
plt.ylim(-0.2, 1.2)
plt.title("NAND")
plt.xlabel("x1")
plt.ylabel("x2")
plt.show()
Image 4. NAND
XOR Problem
- However, XOR cannot be solved with single layer neural network.
- Truth table
| x1 | x2 | Y |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
Table 4. XOR
import numpy as np
import matplotlib.pyplot as plt
X = [[0,0], [0,1], [1,0], [1,1]]
Y = [0, 1, 1, 0]
for i in range(len(X)):
x = X[i]
y = Y[i]
color = None
if y == 0:
color = "b"
else:
color = "r"
plt.plot(x[0], x[1], "o"+color)
plt.grid()
plt.xlim(-0.2, 1.2)
plt.ylim(-0.2, 1.2)
plt.title("XOR")
plt.xlabel("x1")
plt.ylabel("x2")
plt.show()
Image 5. XOR
- It is impossible to divide red and blue dots by one line. - University of Torino
- This problem can be solved with multi layer neural network.
Multi Layer Neural Network
- Multi layer neural network mean there are several overlapped hidden layers. - HackerNoon
Image 6. Multi layer neural network
- If there are many hidden layers, the neural network is Deep Neural Network.
XOR Problem with Multi Layer Neural Network
- XOR can be represented by combination of AND, OR and NAND. - Wiki
Image 7. XOR gate
- In the graph, NAND and OR are placed in the same layer, so XOR can be solved with two layer neural network.
import numpy as np
import matplotlib.pyplot as plt
def NAND(x1, x2):
i = np.array([x1, x2])
# Wegith and threshold is defined manually for correct AND
w = np.array([-0.5, -0.5])
b = 0.7
tmp = np.sum(i*w) + b
if tmp <= 0:
return 0
else:
return 1
def OR(x1, x2):
i = np.array([x1, x2])
# Wegith and threshold is defined manually for correct AND
w = np.array([0.5, 0.5])
b = -0.2
tmp = np.sum(i*w) + b
if tmp <= 0:
return 0
else:
return 1
def AND(x1, x2):
i = np.array([x1, x2])
# Wegith and threshold is defined manually for correct AND
w = np.array([0.5, 0.5])
b = -0.7
tmp = np.sum(i*w) + b
if tmp <= 0:
return 0
else:
return 1
def layer1(x1, x2):
s1 = NAND(x1, x2)
s2 = OR(x1, x2)
return s1, s2
def layer2(x1, x2):
return AND(x1, x2)
X = [[0,0], [0,1], [1,0], [1,1]]
for x in X:
s1, s2 = layer1(x[0], x[1])
y = layer2(s1, s2)
color = None
if y == 0:
color = "b"
else:
color = "r"
plt.plot(x[0], x[1], "o"+color)
plt.grid()
plt.xlim(-0.2, 1.2)
plt.ylim(-0.2, 1.2)
plt.title("XOR")
plt.xlabel("x1")
plt.ylabel("x2")
plt.show()
Image 8. XOR
Training Multi Layer Neural Network
- Single layer neural network is trained with gradient descent algorithm.
- Multi layer neural network is also trained with gradient descent algorithm, but it is more complicated.
- For multi layer neural network, cost calculated in output layer should be passed to the first hidden layer. Furthermore, it is necessary to know how much each neurons affect the cost. These processes demand lots of computing power.
- Fortunately, there is a way to reduce computations dramatically, Back propagation. This will be introduced in the next post.
COMMENTS