>>> By enrolling in this course you agree to the End User License Agreement as set out in the FAQ. Once enrolled you can access the license in the Resources area <<< This course, Applied Artificial Intelligence with DeepLearning, is part of the IBM Advanced Data Science Certificate which IBM is currently creating and gives you easy access to the invaluable insights into Deep Learning models used by experts in Natural Language Processing, Computer Vision, Time Series Analysis, and many other disciplines. We’ll learn about the fundamentals of Linear Algebra and Neural Networks. Then we introduce the most popular DeepLearning Frameworks like Keras, TensorFlow, PyTorch, DeepLearning4J and Apache SystemML. Keras and TensorFlow are making up the greatest portion of this course. We learn about Anomaly Detection, Time Series Forecasting, Image Recognition and Natural Language Processing by building up models using Keras one real-life examples from IoT (Internet of Things), Financial Marked Data, Literature or Image Databases. Finally, we learn how to scale those artificial brains using Kubernetes, Apache Spark and GPUs. IMPORTANT: THIS COURSE ALONE IS NOT SUFFICIENT TO OBTAIN THE "IBM Watson IoT Certified Data Scientist certificate". You need to take three other courses where two of them are currently built. The Specialization will be ready late spring, early summer 2018 Using these approaches, no matter what your skill levels in topics you would like to master, you can change your thinking and change your life. If you’re already an expert, this peep under the mental hood will give your ideas for turbocharging successful creation and deployment of DeepLearning models. If you’re struggling, you’ll see a structured treasure trove of practical techniques that walk you through what you need to do to get on track. If you’ve ever wanted to become better at anything, this course will help serve as your guide. Prerequisites: Some coding skills are necessary. Preferably python, but any other programming language will do fine. Also some basic understanding of math (linear algebra) is a plus, but we will cover that part in the first week as well. If you choose to take this course and earn the Coursera course certificate, you will also earn an IBM digital badge. To find out more about IBM digital badges follow the link ibm.biz/badging.
Neural Network Debugging with TensorBoard
Loading...
Do curso por IBM
Applied AI with DeepLearning
117 classificações
Na lição
deep learning frameworks
Conheça os instrutores
Romeo KienzlerChief Data Scientist, Course Lead
IBM Watson IoT
Niketan PansareSenior Software Engineer
IBM Research
Tom HanlonTraining Director
Skymind
Max PumperlaDeep Learning Engineer
Ilja RasinData Scientist
IBM Watson Health
Welcome to the TensorBoard intro [inaudible] to debug your Neural Networks.
So Neural Network often is some sort
of black box and it's very hard to see what's going on during training.
So usually people tend to print out all sorts of measures during
the Gradient Descent Loop in order to the debug and make sense of the training phase.
So what are the most important parameters to be looked at in training Neural Network?
So the most important is definitely the loss overtime.
So as you remember from the previous video,
loss defines how good or bad a Neural Network fits the data.
And by the way, this here is a quite good loss function
because it converges to a local optima quite fast.
The blue line instead is indicating a learning rate which is too small.
We are slowly converging against the optimum, but it takes too long.
So let's run this for 10,000 iterations and we obtained the following diagram.
It's hard to say, but chances are high,
that with a too low learning rate,
we never reach to decide local optimum.
Finally, if you have chosen the learning rate which is too high,
we notice a considerable amount of bouncing,
and the only reason why we are still conversion is that
the underlying model we're optimizing over is rather simple.
Another may be even more important measure is accuracy.
Here you see accuracy over training iterations.
And as loss goes down,
you should see accuracy going up.
So those are scalar times series,
but now let's have a look at the weight matrices.
This is far more interesting.
Those contain many values,
and we have to find a way to visualize them at once.
Fortunately, TensorBoard does a pretty good job in creating a summary,
overweight matrix of arbitrary shape.
You can find more on how these summaries are calculated in the video description.
But here, we see a pretty healthy distribution of weights.
Bad examples include cases where all values are very close to zero
or a uniform distribution resembling a random weight initialization.
This can be an indication that the actual layer hasn't done anything.
Besides monitoring weights, we can also monitor activations.
Note that the activation is the output of
a particular layer with the activation function applied.
In this simple case, the [inaudible] Y.
Remember, that Y is the output of the Softmax function.
So any K dimensional vector,
with value range between plus and minus infinity is squashed
into a K dimensional vector if value ranges between zero and one.
This is the default activation function for output layer and classification tasks.
So here we see what we expect in a healthy classifier.
Most of the values are close to zero because those are assigned part of probability,
a particular input is not in the class.
And we see a fair amount of values near to one.
Those are the cases where particular input is in the class.
Since we have 10 different classes.
It is obvious that we see more values close to zero than close to one.
So let's have a look at TensorBoard.
Instructions on how to access TensorBoard from
data science experience are given in the description section of this video.
TensorBoard can visualize multiple run simultaneously,
in order to compare among those.
But let's only have a look at a single run for now.
First, you have to look at the Scalar view.
So in this step, all summaries of scalars
and how they evolve over training time are displayed.
In our example, we've recorded the two most important measures.
Loss and Accuracy. As you can see,
they are in loss, which is actually a very good sign.
The lower the loss and therefore the arrow of the neural network is,
the better the accuracy.
We can maximize the plot and also adjust the smoothing parameter.
If we set it to one,
we obtain a line which isn't displayed here.
So let's decrease it slowly.
As we can see, this line,
more and more fits the actual trajectory and with 0.96
we can clearly see a trend without getting lost too much into details.
And here again, we can have a look at
those two important measures and see that they are inverse.
Remember that loss is based on a defined loss function.
Cross-Entropy in this case and accuracy tells us
the fraction of correctly classified examples over a total number of examples.
So now we have a look at the graph tab.
This graph should be read bottom up.
So we start with a weight matrix W and the placeholder X for your input data.
This is multiplied and then the bias where it will be is added.
This result is quashed using Softmax and the final result is thought in Y.
Note that Y underscore and Y are then used to
calculate accuracy by taking the Argmax of both.
By comparing those and taking the mean over this vector,
we obtain the accuracy.
The upper branch of the graph computes loss using the Cross-Entropy function.
Note that parts of the graph are hidden to us in an externalized sub-graph.
As you can see, the connection points of the sub-graph have turned red now.
You can include the sub-graph but then the graph becomes more complex.
This is because the gradient computation is reading values from variables and
placeholders all over the place and doesn't add more information at that point.
Therefore, let's remove it again.
Now, let's turn to histograms and have a look at the weight matrix.
This is a highly compressed view of what's going on inside the matrix during training.
So in the C axis,
training iterations are reflected.
The closer, the more recent they are.
The x axis informs us about the values,
elements in the matrix have been assigned to.
And the Y axis shows frequency.
So in this case,
we are looking at a pretty healthy chart.
It is important to understand how this condensed view is created.
Therefore, a link containing further explanations can be found below.
But intuitively, lacking the histogram in a pure mathematical sense,
the plot gives you an intuition but value ranges
at what frequencies are assigned to each element of the weight matrix.
So we see that the values are symmetrically centered around the mean of zero,
but although they are close to zero but not exactly zero.
This is important since gradients are correlating
with weights and with zero gradients we cannot work.
The other observation is,
that on the left and the right extremes,
close to minus one and plus one,
there's nothing much going on.
And therefore, we are definitely not over saturating.
Finally, this also definitely doesn't look like a uniform distribution.
Therefore, training worked pretty well.
Now we have a look at Y. Y in
the Softmax regression model corresponds to the newer activations of an output player.
You see that most of the values are close to
zero and some are close to one and in between.
That's nothing much going on.
And this is also a very good sign,
since this is exactly what Softmax should do in a mighty class classifier,
because class probabilities in the output vectors are either
zero for the wrong classes or one for the correct class.
Since there are more incorrect classes and correct ones,
nine versus one in this case,
you see much more zero values than one's.
There's much more to say about TensorBoard,
but we've covered the most important aspects for this course and we will see in
future lectures how those matrices can be
used in order to debug neural networks during training.
In the next module, we will recover automatic differentiation,
one of the key features of Tensor.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
