The full Neural Networks playlist, from the basics to deep learning, is here: th-cam.com/video/CqOfi41LfDw/w-d-xo.html Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
In neural network, does the gradient for parameters are calculated parallel? For example: when I start with finding gradient for all the 7 parameters, do I calculate all 7 parameters simultaneously by taking the previous iteration values or, first I calculate the bias gradient and get the new bias, then calculate predicted value by new bias and then calculate gradient for w3 ? And so on till w1 ?
I am watching all of these eventhough i already graduated with masters with focus on machine learning and deep learning. its actually amazing how much I am learning even as a intermediate student.
Had to watch 17:09 at least ten times. But now I get the most dificult part: the orange and blue curves, represent... the orange and blue curves. Without that, I would be completely lost 😆. Thanks for the hard work. Amazing series!
Thanks Josh for these videos, I passed my data mining exam by watching your videos, now preparing for the ML exam. Your explanation is brilliant, I learn topics of 3 lectures with these 18 minutes videos. Please continue to publish such valuable content, you save lives of many people like me.
Despite the simple explanations, these videos continuously make me doubt my mathematical abilities for about 15 minutes. But without fail, there’ll be a DOUBLE BAM! out of left field and suddenly everything’s clear in my head. Thank you! You’re doing God’s work.
I'm a student for CS at the Hebrew University of Jerusalem, study right now IML course. Your lectures so help me and my friends, and I really want to thank you. You're a great & funny teacher and your lessons are a perfect example to how to teach at the 21 century. Tnx again
BAM !!! I'm doing my PHD in this field, and it is the BEST serie of videos that I have watched since the bigenning of my study ! Thank you so much for that :D
What a great explanation to such complex topic. I can't imagine the amount of effort you put in to create such detailed videos along with spoken text. One of the best youtube channel I have ever come across ! Hats off to you .. Don't BAM me :)
One of these days I'd love to see you do a NN to watch the process you produced on these videos get lined up with some code, maybe python or R. It's incredible work you do, hell you are helping me survive my masters program. If you put it up, I'd trust the content. Thank you for all your hard work
Such hard work. Thank you Josh, you are helping generations with this + all your videos. Step by step learning with examples is the right way to learn anything !
wow! WoW! WOW!, I have always been scared of math, cus it took me hell lot of time to understand, but you just explain it as smooth as butter, Thanks a lot really!!
[Notes] 6:44 Notation for activation functions 2:50 Initialize weights using standard normal distribution. Q: Why N(0,1)? -- A: Just one of many ways to initialize weights. [ref. 9:50 of th-cam.com/video/GKZoOHXGcLo/w-d-xo.html&ab_channel=StatQuestwithJoshStarmer] Initialize bias with 0 since bias terms frequently start from 0. 4:334:48 plot SSR with respect to b3
These videos are so much better than 3blue1brown, he starts with complicated analogies and examples and then delves into heavy math whereas this simplifies the problem using simpler examples and works through all the small steps
thank you so much for these videos. i hated math back in high school, but now in my mid 20's i would rather do math than play video games. all thanks to your tutorials
that was exactly what I needed. It would be great if you could 'also' do an application through one of Python libraries in order to show a real application by scripting with using this knowledge.
This is just awesome. I had started learning machine learning algorithm from multiple sources until I found your youtube channel. And now I don't have to check for any other source for understanding any ML algorithm. Looking Forward for more deep learning videos as my area of interest is deep learning. Could you help me with a good book for deep learning? And thanks for such wonderful videos.
Love your videos man, very helpful at providing detail without sacrificing clarity. However I have noticed quite a few errors across the videos, generally small errors such as saying the wrong numbers or when calling up examples such as in this video at 9:26. input_3 would be 0, not 1. Again, it is not a major error and the information provided is nonetheless exemplary however it does make following along a tad challenging when trying to listen to the video rather than watching it like a hawk. Keep up the good work man, much appreciated x
I'm glad you like my videos. It is indeed unfortunate that a few of them have small "typos". However, the example you provide is not one of them. The inputs to the neural network are the x-axis coordinates, not the y-axis coordinates. The 3rd data point has an x-axis coordinate of 1 and a y-axis coordinate of 0. Thus, for the 3rd data point, the input to the neural network is 1 and the desired output is 0. So, not only is this not a major error, it's not an error at all.
To get a better understanding of how we determine this derivative, check out the StatQuest on The Chain Rule: th-cam.com/video/wl1myxrtQHQ/w-d-xo.html It will explain exactly where that -1 comes from.
As usual, your videos are totally awesome, I like them much and easy to understand. I wonder if you will make a video about spatial transcriptomic analysis please since you uploaded the scRNA three years ago considering the spatial analysis is now more famous?
Hi Josh, great video as always. One question, if I want to speak in epoch and batch terms for this video, is it correct to say that this video shows one epoch, which includes one batch that contains all 3 data points we have (Batch Gradient Descent process)? Thanks a lot !!!
Thanks for your great video as always! I have a question though after watching this video and the other SGD video you've made in the past. When calculating the gradients for each parameter with regular gradient descent, we are plugging in all of the samples into the derivative of the loss function w.r.t the current parameter; versus we will just randomly pick one sample in the same process with SGD being used. If that's the case, then what will be the purpose of looping through all the samples (with regular GD) in a complete epoch if we are already using all the samples when calculating the gradients? Thanks in advance!
I'm not sure I fully understand your question. The difference between "regular" and "stochastic" gradient descent in this context has to do with the summation. In "regular", the summation goes from 1 to 'n', where 'n' is the number of samples. In "stochastic", the summation goes from 1 to m, were 'm' is < 'n' and is the number of samples randomly selected for the iteration. Does that help?
@@statquest Thank you for the quick reply! Yes that’s helpful and I think I’m understanding that part. I was mixing the concept of Gradient Descent with epoch/batch numbers, but I guess whether the GD is stochastic or not has nothing to do with the general epoch/batching concept when running a neural network, as we would still need to go over all the samples in a full epoch.
@@statquest Thank you for answering even after 2 years! Funnily enough, while I wrote the elaboration of my question here, I stumbled upon the answer myself. Thank you again for your commitment. Let me tell you, that the work you do absolutely outclasses any learning material that I have stumbled across. I will definitely check out your book! Great work!
Hey just a question! Around 14:00, why are you taking the derivative of SSR with respect to w_3 and w_4 rather than y_2,i and y,1_i? What is the logic between choosing taking the derivative with respect to the weight rather than the functions themselves?
Ah nevermind, its because you want to optimize the weights w_3 and w_4, so you just take their derivative to get step size and so forth... im so dumb haha! Im assuming that in the next part then you will optimize the weights w_1 and w_2 by also connecting them to the derivative of the loss function with respect to the weights, so itll be a huge bonkers chain rule in action
I think for the sake of clarity and rigour, it should be noted that all of the differentials are partial. Otherwise, some people may wonder why implicit differentiation wasn't used in such cases where W2 was differentiated with respect to W1 or vice versa.
Great video and explanation. But I'm missing something simple. The blue and orange lines are added to render the green line, right? It appears (I'm squinting) that, after convergence, the middle dose (the 1/2 dose; actually, just to the left of it) value is 1 but the intersection of the blue and orange lines is at about -.5. Adding those together gives -1, not 1. What am I missing??
Hi Josh ! Love your videos, could you please explain why normal distribution is used to initialise w3 and w4 or else if you have already uploaded a video on normal distribution can you tag it
It's just a standard way to do it. However, you can use uniform distributions or other distributions if you would like. One thing people like about the normal distribution is that changing the standard deviation for each hidden layer can make it easier to train deeper models (models with lots of hidden layers).
Hey Josh, great video as always!! Can you also please point to some source with examples (with answers) which we can practice on our own? I know there are tons of them on internet, but you know, your selection will be really helpful as always!!
Small question: since we fiddle with all (or part) of the parameters at once, and for example bias is dependent on weights on the graph, does that mean they fight with each other? Can something be done about it? Like we calculate the derivatives for current forward pass, ok, but then changing all parameters at once to what the think is optimal might throw off everything, since they can't communicate in any way, how does it not explode?
In my video on gradient descent, I show how to optimize two parameters at the same here: th-cam.com/video/sDv4f4s2SB8/w-d-xo.html In that video, we're trying to fit a straight line to some data points and are using gradient descent to find the best values for two parameters, the y-axis intercept and the slope. If you watch, you'll see a fancy graph, where one axis represents different values for the y-axis intercept and another axis represents different values for the slope. When we optimize both at the same time, we take a step towards a better intercept on that axis and take a step towards a better slope on that axis, which is different, and doesn't affect the one the intercept is on. So the parameters don't fight each other because each one gets its own axis to work on. That being said, we can still get stuck in a local minimum, but it's like progress in one parameter can be negated by progress in another.
@@statquest Ah, this makes a lot of sense now, I think I know why it was misleading for me - in the end all you see a numerical value, the error, but behind the scenes the partial derivatives take apart the loss function in their own domains, so it's not just one number. Thank you for very descriptive response!
basically, taking derivatives of losses with respect to unknown terms to find how quickly the loss is changing if we change the parameters is the essence of this whole Machine learning thing.
BAM!! I finally understand but.... Am I correct to say that if I was optimizing 3 weights and biases at the same time i would do gradient descent in a function with 3 dimensions (1 for each weight and bias)??
Because it works just fine in this simple example. However, if you keep watching the series, you'll see how to do backpropagation with ArgMax and SoftMax and Cross Entropy. Here's the whole playlist: th-cam.com/video/CqOfi41LfDw/w-d-xo.html
Sir, one question I have. when you said we randomly select w3 and w4 from standard distrib in the first iteration, that is any values from standard distrib table or we select no's w.r.t. given dataset?
In this example I selected random value from a standard normal distribution. This is a normal distribution with mean = 0 and standard deviation = 1 and is completely independent of the data.
Why do you not average the derivative of the SSR (the gradient). What I mean by average is dividing the derivative of the SSR by the number of training examples. I read online that this is more common practice unless we are doing stochastic gradient descent. I was a little bit confused as this was not clarified. Thanks for the video though it really helped me understand the topic better.
As the video shows, it works just fine without averaging the SSR. However, we have a relatively small dataset and that keeps the derivative from getting out of hand. If we had tons and tons of data, the SSR alone might lead to a massive derivative that's too big to be helpful, and averaging could help with that.
@@statquest Basically I'm just confused about if the final curve approximating the 3 points is a "curve" as in a polynomial, or just a set of 3 points. Because when we add the two activation functions, you talked about adding them at each point as if we were adding the equations for the lines themselves, in order to get the final line. But it seems like instead we're just adding the y values at each input (the 3 given inputs) rather than a line itself..?
@@dahirou_harden The adding is done for all possible x-axis coordinates (or input values), and thus, we are adding the lines themselves, not just the 3 points. The points (or circles) on the lines are just to illustrate the concept of adding y-axis values, and do not to limit the adding to just those points.
In statistics, data science, machine learning and almost all programming languages, the default base for the log function is 'e', and that's what I use here.
We include the derivative of the SoftPlus activation function in the next video (part 2), when we optimize all of the weights and biases, including the ones to the left of the activation functions: th-cam.com/video/GKZoOHXGcLo/w-d-xo.html
The full Neural Networks playlist, from the basics to deep learning, is here: th-cam.com/video/CqOfi41LfDw/w-d-xo.html
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
In neural network, does the gradient for parameters are calculated parallel?
For example: when I start with finding gradient for all the 7 parameters, do I calculate all 7 parameters simultaneously by taking the previous iteration values or, first I calculate the bias gradient and get the new bias, then calculate predicted value by new bias and then calculate gradient for w3 ? And so on till w1 ?
@@sahanamd707 Everything is done at the same time.
Thank you
the way you explain things,so patiently and in depth, i feel 200% more confident in the topic afterwards
Awesome! :)
I'm writing from Brazil. This channel's playlist about neural networks is a masterpiece.
Muito obrigado! :)
join a cartel
Triple BAM (Explanation)+Triple BAM (Animations)......
You are a very great teacher Josh Starmer :) :)
Wow, thanks!
We all should buy his book, he deserves it given the quality of these videos!!
Thank you!!! :)
yes!
@@statquest
Hey man,
just bought your book,will be arriving in a few days via amazon.All these topics are covered right?
@@MultiSamarjit The basics of neural networks and backpropagation are covered. The other topics are listed here: statquest.org/statquest-store/
OMG THE HAPPINESS I WAS FEELING WHEN I UNDERSTOOD EVERYTHING, you seriously are a really good teacher.
Thank you!
I am watching all of these eventhough i already graduated with masters with focus on machine learning and deep learning. its actually amazing how much I am learning even as a intermediate student.
bam!
Had to watch 17:09 at least ten times. But now I get the most dificult part: the orange and blue curves, represent... the orange and blue curves. Without that, I would be completely lost 😆.
Thanks for the hard work. Amazing series!
I'm glad you figured it out! :)
hahahaha
Thanks Josh for these videos, I passed my data mining exam by watching your videos, now preparing for the ML exam. Your explanation is brilliant, I learn topics of 3 lectures with these 18 minutes videos. Please continue to publish such valuable content, you save lives of many people like me.
Thank you and good luck with your exam! Let me know how it goes.
@@statquest Josh, are you planning to make a video about batch normalization?
@@vusalaalakbarova7378 Not soon. Currently I'm working on a series of videos about how to build neural networks with pytorch and pytorch_lightning.
Despite the simple explanations, these videos continuously make me doubt my mathematical abilities for about 15 minutes. But without fail, there’ll be a DOUBLE BAM! out of left field and suddenly everything’s clear in my head. Thank you! You’re doing God’s work.
Bam! :)
I really like the medieval guitar sound you added when you said "fancy notation" , you effort can really be seen in the little details
Thanks!
I'm a student for CS at the Hebrew University of Jerusalem, study right now IML course. Your lectures so help me and my friends, and I really want to thank you. You're a great & funny teacher and your lessons are a perfect example to how to teach at the 21 century. Tnx again
Wow! Thank you very much! BAM! :)
How the heck do you have this much understanding in each concept...you are irreplaceable.
Thanks!
I wish I had this when I was first learning backpropagation! Can I "work my way backward" with this knowledge haha
BAM! :)
Pro tip: you can watch movies on flixzone. Me and my gf have been using it for watching lots of of movies lately.
@Roman Randall Definitely, have been using Flixzone for years myself :)
haha good one !!
@Roman Randall and @Amos Zahir are bots but nice one
BAM !!! I'm doing my PHD in this field, and it is the BEST serie of videos that I have watched since the bigenning of my study ! Thank you so much for that :D
Thanks and good luck!
The details are just out of this world. Amazing. Breath-taking and short of words.
Thanks!
BAMMMMMMM!
I like the animation in the last part and the music with Fan~cy notation lol
BAM! :)
You are One of the Best Content Creator I have ever Seen.
Wow, thanks!
What a great explanation to such complex topic. I can't imagine the amount of effort you put in to create such detailed videos along with spoken text. One of the best youtube channel I have ever come across ! Hats off to you .. Don't BAM me :)
Wow, thank you!
I agree 100%
Mega BAMM!! I really love the explanation. Awesome!
Thanks!
One of these days I'd love to see you do a NN to watch the process you produced on these videos get lined up with some code, maybe python or R. It's incredible work you do, hell you are helping me survive my masters program. If you put it up, I'd trust the content. Thank you for all your hard work
Thank you! And good luck with your masters degree.
Such hard work. Thank you Josh, you are helping generations with this + all your videos. Step by step learning with examples is the right way to learn anything !
Thank you!
wow! WoW! WOW!, I have always been scared of math, cus it took me hell lot of time to understand, but you just explain it as smooth as butter, Thanks a lot really!!
Thank you very much!
this playlist is excellent
Thank you!
This NN series is so underrated just 124K I mean come on
Thanks!
woah this was some amazing teaching skills sir, you're totally gifted with that
Thanks! 😃
[Notes]
6:44 Notation for activation functions
2:50 Initialize weights using standard normal distribution. Q: Why N(0,1)? -- A: Just one of many ways to initialize weights. [ref. 9:50 of th-cam.com/video/GKZoOHXGcLo/w-d-xo.html&ab_channel=StatQuestwithJoshStarmer]
Initialize bias with 0 since bias terms frequently start from 0.
4:33 4:48 plot SSR with respect to b3
bam!
Really loving these videos, thank you so much for your work Josh
Thank you!
These videos are so much better than 3blue1brown, he starts with complicated analogies and examples and then delves into heavy math whereas this simplifies the problem using simpler examples and works through all the small steps
Thank you!
I totally agree with you 👏
You make this stuff so accessible, well done!
Thank you!
Hats off to you Josh!!
So nicely explained ❤
Glad you liked it!
A Brief Indepth Explanation.Thank you Sir
Glad you liked it
Thank you for your clear explanations with the simple example! Great work, and very useful.
Glad it was helpful!
The little harpsichord really tickled me haha, love it
:)
That was the best explanation I had ever seen. Thank you very much.
Thank you! :)
This is simply beautiful!. You are the best.
Thank you!
You're a legend Doctor Starmer.
Thanks!
I am on vacation in Hawaii but I am watching your neural network video. This video is so entertaining to watch :) Tai
BAM! Have a great vacation! :)
@@statquest thank you! you too. have a nice day
thank you so much for these videos. i hated math back in high school, but now in my mid 20's i would rather do math than play video games. all thanks to your tutorials
Wow! That's awesome! Thank you!
Well you actually make complex things super easy.Hats off and of course BAAA...M!!!
Bam! :)
God bless this Good Man.
Thanks!
Notes: 2:31 6:14 15:57 the "y"s are calculated based on other weights (w1 and w2)
:)
Great explanation as usual. Thank you very much.
Thanks again!
I wish I could have taken your classes when I was back in high school.
bam! :)
that was exactly what I needed. It would be great if you could 'also' do an application through one of Python libraries in order to show a real application by scripting with using this knowledge.
Thanks! I would like to do that.
amazing video, thanks for all your hard work on this.
Glad you enjoyed it!
I love you Statquest
:)
Thank you so much for sharing your knowledge, it is really so increadibly helped me understand the basics of the NN.
Glad it was helpful!
thank you so much sir for clearly explaining everything
Thanks!
I love you man! You have a sense of humor about you that is rare in deez parts lol
Thank you!
i hate stats, and i hate statquest. But i keep finding myself on this channel again and again
noted
Best channel ever
Wow! Thank you! :)
Thanks. Great video again and again.
Thank you very much! :)
This is just awesome. I had started learning machine learning algorithm from multiple sources until I found your youtube channel. And now I don't have to check for any other source for understanding any ML algorithm. Looking Forward for more deep learning videos as my area of interest is deep learning. Could you help me with a good book for deep learning? And thanks for such wonderful videos.
This series ends (for now) with Convolutional Neural Networks, so just keep watching to learn about deep learning.
BAM! Great work!
Thank you!
Hi Josh! Love the videos. Do you have any posts for building models in R/Rstudio on neural networks? Thanks,Tina
Not yet!
Im gonna make a AI agent that create youtube bots that promotes your channel. You really deserve all kudos.
Bam!
So underrated
Glad you think so! :)
THIS MAN IS AN ANGEL! :D QUADRUPLE BAM!
Thank you! :)
Love your videos man, very helpful at providing detail without sacrificing clarity. However I have noticed quite a few errors across the videos, generally small errors such as saying the wrong numbers or when calling up examples such as in this video at 9:26. input_3 would be 0, not 1. Again, it is not a major error and the information provided is nonetheless exemplary however it does make following along a tad challenging when trying to listen to the video rather than watching it like a hawk. Keep up the good work man, much appreciated x
I'm glad you like my videos. It is indeed unfortunate that a few of them have small "typos". However, the example you provide is not one of them. The inputs to the neural network are the x-axis coordinates, not the y-axis coordinates. The 3rd data point has an x-axis coordinate of 1 and a y-axis coordinate of 0. Thus, for the 3rd data point, the input to the neural network is 1 and the desired output is 0. So, not only is this not a major error, it's not an error at all.
superb, thanks a lot.
Thanks!
YOU ARE THE BEST!
Thanks!
Hey Josh, this is dope. Hope you could do some videos about the Hessian and Jacobian matrices, Thanks.
I'll keep those topics in mind.
I can not thank you enough!
Thanks!
You are amazing!
Wow, thank you!
13:35 Do you mean the derivative of observed - predicted? Wouldn't that be a derivative of a single number? Or does it always just come out to be -1?
To get a better understanding of how we determine this derivative, check out the StatQuest on The Chain Rule: th-cam.com/video/wl1myxrtQHQ/w-d-xo.html It will explain exactly where that -1 comes from.
@@statquest Oh the derivative of the negative intercept? ok thanks
Yeaaaa finally new episodde
:)
As usual, your videos are totally awesome, I like them much and easy to understand. I wonder if you will make a video about spatial transcriptomic analysis please since you uploaded the scRNA three years ago considering the spatial analysis is now more famous?
I'll keep it in mind.
Hi Josh, great video as always. One question, if I want to speak in epoch and batch terms for this video, is it correct to say that this video shows one epoch, which includes one batch that contains all 3 data points we have (Batch Gradient Descent process)? Thanks a lot !!!
Yes, that is correct.
Thanks for your great video as always! I have a question though after watching this video and the other SGD video you've made in the past. When calculating the gradients for each parameter with regular gradient descent, we are plugging in all of the samples into the derivative of the loss function w.r.t the current parameter; versus we will just randomly pick one sample in the same process with SGD being used. If that's the case, then what will be the purpose of looping through all the samples (with regular GD) in a complete epoch if we are already using all the samples when calculating the gradients? Thanks in advance!
I'm not sure I fully understand your question. The difference between "regular" and "stochastic" gradient descent in this context has to do with the summation. In "regular", the summation goes from 1 to 'n', where 'n' is the number of samples. In "stochastic", the summation goes from 1 to m, were 'm' is < 'n' and is the number of samples randomly selected for the iteration. Does that help?
@@statquest Thank you for the quick reply! Yes that’s helpful and I think I’m understanding that part. I was mixing the concept of Gradient Descent with epoch/batch numbers, but I guess whether the GD is stochastic or not has nothing to do with the general epoch/batching concept when running a neural network, as we would still need to go over all the samples in a full epoch.
very informative ty
:)
Absolutely amazing! I‘ve got a question though. How do we know if we are going the right direction when calculating the new parameter.
The derivative tells us what direction to change the parameter. To see more details, see: th-cam.com/video/sDv4f4s2SB8/w-d-xo.html
@@statquest Thank you for answering even after 2 years!
Funnily enough, while I wrote the elaboration of my question here, I stumbled upon the answer myself.
Thank you again for your commitment. Let me tell you, that the work you do absolutely outclasses any learning material that I have stumbled across. I will definitely check out your book! Great work!
great content!
Thanks!
My regards to the friendly folks of the genetics department of University of North Carolina at Chapel Hill
Thanks!
Hey just a question! Around 14:00, why are you taking the derivative of SSR with respect to w_3 and w_4 rather than y_2,i and y,1_i? What is the logic between choosing taking the derivative with respect to the weight rather than the functions themselves?
Ah nevermind, its because you want to optimize the weights w_3 and w_4, so you just take their derivative to get step size and so forth... im so dumb haha! Im assuming that in the next part then you will optimize the weights w_1 and w_2 by also connecting them to the derivative of the loss function with respect to the weights, so itll be a huge bonkers chain rule in action
Yes! It will be totally bonkers with chain rule action. :)
this dude is him
:)
I think for the sake of clarity and rigour, it should be noted that all of the differentials are partial. Otherwise, some people may wonder why implicit differentiation wasn't used in such cases where W2 was differentiated with respect to W1 or vice versa.
noted
Great video and explanation. But I'm missing something simple. The blue and orange lines are added to render the green line, right? It appears (I'm squinting) that, after convergence, the middle dose (the 1/2 dose; actually, just to the left of it) value is 1 but the intersection of the blue and orange lines is at about -.5. Adding those together gives -1, not 1. What am I missing??
You forgot to add the bias term.
Hi Josh ! Love your videos, could you please explain why normal distribution is used to initialise w3 and w4 or else if you have already uploaded a video on normal distribution can you tag it
It's just a standard way to do it. However, you can use uniform distributions or other distributions if you would like. One thing people like about the normal distribution is that changing the standard deviation for each hidden layer can make it easier to train deeper models (models with lots of hidden layers).
Hey Josh, great video as always!!
Can you also please point to some source with examples (with answers) which we can practice on our own?
I know there are tons of them on internet, but you know, your selection will be really helpful as always!!
I don't have anything yet, but I will create a "how to do neural networks" video soon.
BAM! that was good!
Thanks!
Small question: since we fiddle with all (or part) of the parameters at once, and for example bias is dependent on weights on the graph, does that mean they fight with each other? Can something be done about it? Like we calculate the derivatives for current forward pass, ok, but then changing all parameters at once to what the think is optimal might throw off everything, since they can't communicate in any way, how does it not explode?
In my video on gradient descent, I show how to optimize two parameters at the same here: th-cam.com/video/sDv4f4s2SB8/w-d-xo.html In that video, we're trying to fit a straight line to some data points and are using gradient descent to find the best values for two parameters, the y-axis intercept and the slope. If you watch, you'll see a fancy graph, where one axis represents different values for the y-axis intercept and another axis represents different values for the slope. When we optimize both at the same time, we take a step towards a better intercept on that axis and take a step towards a better slope on that axis, which is different, and doesn't affect the one the intercept is on. So the parameters don't fight each other because each one gets its own axis to work on. That being said, we can still get stuck in a local minimum, but it's like progress in one parameter can be negated by progress in another.
@@statquest Ah, this makes a lot of sense now, I think I know why it was misleading for me - in the end all you see a numerical value, the error, but behind the scenes the partial derivatives take apart the loss function in their own domains, so it's not just one number. Thank you for very descriptive response!
@@madghostek3026 bam! Your question is actually a very good one and maybe one day I'll make a short video that explains it for everyone.
Many many thanks for your videos.
Glad you like them!
basically, taking derivatives of losses with respect to unknown terms to find how quickly the loss is changing if we change the parameters is the essence of this whole Machine learning thing.
yep
BAM!! I finally understand
but....
Am I correct to say that if I was optimizing 3 weights and biases at the same time i would do gradient descent in a function with 3 dimensions (1 for each weight and bias)??
Yes
@4:45 ,Hi Josh why sum of squares residual used classification problem
Because it works just fine in this simple example. However, if you keep watching the series, you'll see how to do backpropagation with ArgMax and SoftMax and Cross Entropy. Here's the whole playlist: th-cam.com/video/CqOfi41LfDw/w-d-xo.html
@@statquest ,Thank you Josh …I am watching your videos daily …Please make videos on RNN GAN LSTM and NLP ..
@@rafibasha1840 I plan on making those in the spring.
@@statquest ,Thank you Josh
Sir, one question I have. when you said we randomly select w3 and w4 from standard distrib in the first iteration, that is any values from standard distrib table or we select no's w.r.t. given dataset?
In this example I selected random value from a standard normal distribution. This is a normal distribution with mean = 0 and standard deviation = 1 and is completely independent of the data.
OMGGGGG, Is that you who sings at "big bang theory", S12, E24???!!!!!
I wish! :)
Please make a tutorial on Gaussian mixture model and EM algorithm
I'll keep that in mind.
@@statquest thanks..
It will be really helpful 🙂
Why do you not average the derivative of the SSR (the gradient). What I mean by average is dividing the derivative of the SSR by the number of training examples. I read online that this is more common practice unless we are doing stochastic gradient descent. I was a little bit confused as this was not clarified. Thanks for the video though it really helped me understand the topic better.
As the video shows, it works just fine without averaging the SSR. However, we have a relatively small dataset and that keeps the derivative from getting out of hand. If we had tons and tons of data, the SSR alone might lead to a massive derivative that's too big to be helpful, and averaging could help with that.
AMAZING BROOOOOOOOO
Thanks!
Just wanted to clarify. Is the output given at the end of each pass an actual function or just a set of 3 points (summed from y1 and y2)? Thanks!
What time point, minutes and seconds, are you asking about?
@@statquest Basically I'm just confused about if the final curve approximating the 3 points is a "curve" as in a polynomial, or just a set of 3 points. Because when we add the two activation functions, you talked about adding them at each point as if we were adding the equations for the lines themselves, in order to get the final line. But it seems like instead we're just adding the y values at each input (the 3 given inputs) rather than a line itself..?
@@statquest At 4:03 for example.
@@dahirou_harden The adding is done for all possible x-axis coordinates (or input values), and thus, we are adding the lines themselves, not just the 3 points. The points (or circles) on the lines are just to illustrate the concept of adding y-axis values, and do not to limit the adding to just those points.
In "d SSR/ d Predicted", is Predicted a single value like Predictedi (with index i ) or a collection of values as i can range from 1 to 3?
A collection of values. You can tell if you keep watching the video and see how it is used.
@@statquest thank you for the prompt reply, Josh! you are the best!
The nested chain rule.
:)
when i plug the value -1.43 into the equation log(1 + e**x) i get 0.093. should I use the base 10 for log or a different one?
In statistics, data science, machine learning and almost all programming languages, the default base for the log function is 'e', and that's what I use here.
@@statquest Thanks, this was very helpful.
♥
:)
BAM!
:)
Dw_3 = (observed-predicted) * y1. The output is also a softplus activation. Why isn’t this derivative in the chainrule? Thank you!
We include the derivative of the SoftPlus activation function in the next video (part 2), when we optimize all of the weights and biases, including the ones to the left of the activation functions: th-cam.com/video/GKZoOHXGcLo/w-d-xo.html
Triple BAM!!! That's what I said when I knew my girl, married her and got children :)
:)
The God! Please do NLP also
Noted