Cost function for linear regression
TOC
Cost(Loss) Function
- To evaluate the quantity of error for hypothesis.
- It is the goal of machine learning to find values of W and b minimizing the error.
- Usually, MSE(Mean Square Error) or CEE(Cross Entropy Error) are used as a cost function.
Cost Function for Linear Regression
- Cost function for linear regression is MSE
$$ cost(W, b) = \frac{1}{m} \sum_{i=1}^m (H(x_{i})-y_{i})^2 $$
- Simplified Hypothesis will be used for further explanations
$$ H(x) = W * x $$
$$ cost(W) = \frac{1}{m} \sum_{i=1}^m ((W * x_{i})-y_{i})^2 $$
Mean Square Error(MSE)
- As mentioned before, the goal of machine learning is to find values for W and b minimizing the error.
- In mathematics, if the value of derivative at particular point is 0, that point is a candidate of minimum or maximum value.
- MSE is 2-dimensional function, so its derivative has only one single minimum point.
import numpy as np
import matplotlib.pyplot as plt
# number of points
num_points = 300
# For the sample data
# Answer values
x_ans = [i * 0.01 for i in range(-200, 200)]
W_ans = 0.1
# Generate normal random values for input and output
x = [np.random.normal(0.0, 0.55) for i in range(num_points + 1)]
y = [W_ans * _x + np.random.normal(0.0, 0.03) for _x in x]
# Draw input, output and answer line
plt.plot(x, y, "ro")
plt.plot(x_ans, [W_ans * _x for _x in x_ans], label="W=0.1")
plt.xlabel("X")
plt.ylabel("Y")
plt.legend()
plt.title("y = 0.1 * x")
plt.show()
# For the Mean Square Error
# Weight from 0 to 0.2
W = [i * 0.001 for i in range(0, 201)]
# Cost
cost = []
for w in W:
# hypothesis = W * x
hypo = [w * _x for _x in x]
diffSqrts = list(map(lambda _hypo, _answer : \
(_hypo - _answer) ** 2, hypo, y))
sumDiffSqrt= sum(diffSqrts)
cost.append(1 / (len(W)) * sumDiffSqrt)
# Draw cost function
plt.plot(W, cost)
plt.title("Cost Function")
plt.xlabel("W")
plt.ylabel("Cost(W)")
plt.xlim(0.00, 0.20)
plt.show()
Image 1. Sample data and answer line
Image 2. Cost function
COMMENTS