Bias correction in exponentially weighted averages

Loading...

Do curso por deeplearning.ai

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

23504 classificações

deeplearning.ai

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

23504 classificações

Curso 2 de 5 em Specialization Deep Learning

This course will teach you the "magic" of getting deep learning to work well. Rather than the deep learning process being a black box, you will understand what drives performance, and be able to more systematically get good results. You will also learn TensorFlow. After 3 weeks, you will: - Understand industry best-practices for building deep learning applications. - Be able to effectively use the common neural network "tricks", including initialization, L2 and dropout regularization, Batch normalization, gradient checking, - Be able to implement and apply a variety of optimization algorithms, such as mini-batch gradient descent, Momentum, RMSprop and Adam, and check for their convergence. - Understand new best-practices for the deep learning era of how to set up train/dev/test sets and analyze bias/variance - Be able to implement a neural network in TensorFlow. This is the second course of the Deep Learning Specialization.

Na lição

Optimization algorithms

Mini-batch gradient descent11:28

Understanding mini-batch gradient descent11:18

Exponentially weighted averages5:58

Understanding exponentially weighted averages9:41

Bias correction in exponentially weighted averages4:11

Gradient descent with momentum9:20

Adam optimization algorithm7:07

Learning rate decay6:44

The problem of local optima5:23

Conheça os instrutores

Andrew Ng
Co-founder, Coursera; Adjunct Professor, Stanford University; formerly head of Baidu AI Group/Google Brain
Head Teaching Assistant - Kian Katanforoosh
Lecturer of Computer Science at Stanford University, deeplearning.ai, Ecole CentraleSupelec
Teaching Assistant - Younes Bensouda Mourri
Mathematical & Computational Sciences, Stanford University, deeplearning.ai

You've learned how to implement exponentially weighted averages.

There's one technical detail called biased correction

that can make you computation of these averages more accurately.

Let's see how that works.

In a previous video, you saw this figure for beta = 0.9.

This figure for beta = 0.98.

But it turns out that if you implement the formula as written here,

you won't actually get the green curve when, say, beta = 0.98.

You actually get the purple curve here.

And you notice that the purple curve starts off really low.

So let's see how it affects that.

When you're implementing a moving average, you initialize it with v0 = 0,

and then v1 = 0.98 V0 + 0.02 theta 1.

But V0 is equal to 0 so that term just goes away.

So V1 is just 0.02 times theta 1.

So that's why if the first day's temperature is, say 40 degrees Fahrenheit,

then v1 will be 0.02 times 40, which is 8.

So you get a much lower value down here.

So it's not a very good estimate of the first day's temperature.

v2 will be 0.98 times v1 + 0.02 times theta 2.

And if you plug in v1, which is this down here and multiply it out,

then you find that v2 is actually equal to 0.98 times

0.02 times theta 1 + 0.02 times theta 2.

And that 0.0 196 theta1 + 0.02 theta2.

So again, assuming theta1 and theta2 are positive numbers,

when you compute this v2 will be much less than theta1 or theta2.

So v2 isn't a very good estimate of the first two days' temperature of the year.

So it turns out that there is a way to modify this estimate that makes it much

better, that makes it more accurate,

especially during this initial phase of your estimate.

Which is that, instead of taking Vt, take Vt divided by 1-

Beta to the power of t where t is the current data here on.

So let's take a concrete example.

When t = 2, 1- beta to the power

of t is 1- 0.98 squared and

it urns out that this is 0.0396.

And so your estimate of the tempature on day 2

becomes v2 divided by 0.0396 and

this is going to be 0.0196 times theta 1 + 0.02 theta 2.

You notice that these two things adds up to the denominator 0.03 and 6.

And so this becomes a weighted average of theta 1 and theta 2 and

this removes this bias.

So you notice that as t becomes large, beta to the t will

approach 0 which is why when t is large enough,

the bias correction makes almost no difference.

This is why when t is large, the purple line and

the green line pretty much overlap.

But during this initial phase of learning when you're still warming up your

estimates when the bias correction can help you to obtain a better estimate of

this temperature.

And it is this bias correction that helps you go from the purple line

to the green line.

So in machine learning, for most implementations of the exponential

weighted average, people don't often bother to implement bias corrections.

Because most people would rather just wait that initial period and

have a slightly more biased estimate and go from there.

But if you are concerned about the bias during this initial phase,

while your exponentially weighted moving average is still warming up.

Then bias correction can help you get a better estimate early on.

So you now know how to implement exponentially weighted moving averages.

Let's go on and use this to build some better optimization algorithms.

Explore nosso catálogo

Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.

O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.

© 2018 Coursera Inc. Todos os direitos reservados.

Baixar na App Store

Baixar no Google Play