Normalization

What is the normalization formula used for?

Normalization is useful in statistics for creating a common scale to compare data sets with very different values.

Deep Learning view?

1. 학습의 안정화: Gradient vanising/exploding 문제를 해결할 수 있음
2. 학습시간의 단축: learning rate를 크게 할 수 있음
3. 성능 개선: local optimum에서 빨리 빠져나올 수 있음

Min-Max Normalization

Method

normalization formula to [0,1] xnormalized = (x-xmin) / (xmax-xmin)

• if x==xmin, xnormalized = 0
• if x==xmax, xnormalized = 1

Normalization formula for custom ranges(a,b) xnormalized = a + (((x-xmin)*(b-a)) / (xmax-xmin))

• if x==xmin, xnormalized = a
• if x==xmax, xnormalized = a+(b-a) = b

Summary

• guaranteed to reshape both of our features to be between 0 and 1
• MinMax Scaling is that it is highly influenced by the maximum and minimum values in our data so if our data contains outliers it is going to be biased
• might compress all inliers in a narrow range if affected by outliers 
  • Normalizing fixed the squishing problem on the y-axis, but the x-axis is still problematic. Now if we were to compare these points, the y-axis would dominate; the y-axis can differ by 1, but the x-axis can only differ by 0.4.

Standardization

Method

scales the features so that they have μ=0 and σ=1

• μ : mean value of the feature
• σ : standard deviation of the feature

Z-Score Normalization

normalization formula xnormalized = (x-μ)/σ

• if x==mean of the all the values of the feature, xnormalized = 0
• if x < mean of the all the values of the feature, xnormalized < 0

Summary

• avoids this outlier issue
• The size of those negative, positive numbers is determined by the standard deviation of the feature. (if large standard deviation, the normalized values will be closer to 0)

Pytorch Normalization (Standardization)

• 각 channel별로 mean, std 값 할당

transform = torchvision.transforms.Compose([
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(
        mean=[mean_1, mean_2, mean_3],
        std=[std_1, std_2, std_3],
    ),
])

 

 

reference

https://www.codecademy.com/article/normalization

https://www.indeed.com/career-advice/career-development/normalization-formula