Why Dividing By N Underestimates the Variance
ฝัง
- เผยแพร่เมื่อ 28 ก.ย. 2024
- This is the follow up video to:
Statistics Fundamentals: The Mean, Variance and Standard Deviation
• Calculating the Mean, ...
In it, we show exactly why, when we estimate the variance, dividing by 'n' underestimates the value we are interested in. It also describes why we square each term instead of taking the absolute value. The visuals used in this StatQuest make it easy to remember why we should divide by n-1, and this will save us from falling into a very common pitfall.
If you'd like to support StatQuest, please consider...
Support StatQuest by buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumr...
Paperback - www.amazon.com...
Kindle eBook - www.amazon.com...
Patreon: / statquest
...or...
TH-cam Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshi...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer....
...or just donating to StatQuest!
www.paypal.me/...
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
Corrections:
3:23 I should have said "To understand why dividing by n underestimates the variation around the population mean".
3:40 The estimated mean was switched with the population mean.
#statquest #variance
Corrections:
3:23 I should have said "To understand why dividing by n underestimates the variation around the population mean".
3:40 The estimated mean was switched with the population mean.
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
BAM BUM hahah
Please Give Video on degrees of freedom please🙇
At around 8:35, you should've used asterisk '*' character instead of 'x' character for multiplication. I was a bit confused and thought you wrote 2*(x-v)*x-1 instead of 2*(x-v)*(-1). Great video by the way!
@@m3c4nyku43 noted
is there some sort of award we can give this guy? please?!
:)
I think we're encouraged to purchase a double dam t-shirt or sweatshirt, which is more of a financial incentive than an award but who doesn't like getting paid to be awesome? I'll probably pick one up this weekend
@@jacobmoore8734 Thanks! :)
@@jacobmoore8734 award + t-shirt = double bam! just ordered my own shirt. gonna wear it to my statistics test in 2 weeks
Really, your way is too unique. one of the best
Amazing Josh!!
I can't imagine how much hard work goes into simplifying the complex statistics concepts and coming up with these amazing videos. And on top of that your ingenious ideas of adding humor and musical creativity, taking the content to another level.
If there was an Oscar for tutoring you'd be the undisputed winner.
BAMMM !!- simply the best educator on TH-cam....
BAM! :)
Came to this video for "Why Dividing By N Underestimates the Variance" but got to know why absolute values are not used in Variance calculation. Literally cried, Prof. Josh.. Kudos to you. You are supporting me to understand the topics in statistics. I will support you regularly after I get a job soon. And I'm sure your teachings are required for many of the upcoming students in the coming decades. In India we have a concept called "Guru Kulam", and I see you as my guru (Not the term commonly known in the western world, this is more about respect)
Thank you so much!!! It means a lot to me.
I have nothing but admiration; this is the clearest explanation that I've seen so far that does not shy away from the underlying math, yet still keeping it understandable for those with minimal math background. I feel like a bit of a fool when I see the contrast between my own attempts to explain this correction factor and your explanation.
I'm glad you like the video so much. Thanks! :)
I rather get a clear and understanding explanation with "BAMS" like i'm five, than a 50 pages long explanation with words like "trivial" and abbreviations (q.e.d) and just feel depressed and left clueless. And an other very important thing: Only if you *really* understood the topic, you can explain it with easy words. Very well done, Josh! Thank you very much!
Thank you very much!!!! :)
the best accessible explanation I can find in the whole internet for this mystery. then just as I was about to say "aha! you missed out something!" towards the end of the video, you seemed to have read my mind and "p.s. if you are wondering why n-1 and not 0.5 or 2 .... " you are so so spot-on!
Thank you very much! :)
@@statquest I agree - best explanation i have found and i'm sharing this video with all my students. THANK YOU! So.... any chance that next video is coming out soon? (or has come out already?)
@@naysannaderi5135 I hope the next video will come out soon. Possibly in the next 4 months or so. I hope!
Kids today are so lucky they can review their stats online like this with great teachers.
:)
Yet another video from this channel that leaves me speechless. I've never really understood this concept until I've watched your video. Thank you very much, again.
Wow, thank you!
came from calculating the mean, variance and SD video. Did not expect a proof for why variance = x-bar. This is a really good in depth video i've ever watched for statistics. Thank you very much.
bam!
Thank you, 10 years of confusion made clear by this 15 mins of video.
Hooray! I'm glad the video was helpful. :)
I searched for this on a number of online resources, some mentioned "n" while others "n-1", leaving me confused. This is the best possible explanation to the problem you made it really easy for us to understand. Thanks a lot !!! Bammmm subscribed and shared with friends.
Awesome!!! Thank you very much for subscribing and sharing my videos with your friends. :)
I click the like button before I watch it because I'm always sure I'll love it! Thanks so much for making this series. You'll never know how helpful it has been in my life
Hooray!!! Thank you very much! :)
I have always hated statistics but I just today found this channel and this guy explains everything elegantly! ❤😊
Wow, thank you!
Amazing explanation of why we use the square of the errors instead of the absolute value! I always asked myself that and all the teachers said it was just to give a bigger weight to the errors! We need the statquest on expected value!
Thanks! I'm working on the expected value, but it still might be a few months before it's ready.
I think for even n there wouldn't even be a minimum point, rather a flat line between the 2 middle samples
Awesome and the best video with most simplified explaination.
Thank you! :)
"Future is nooow, BAM " - #LOL #respect #welldone #thanks
Thank you! :)
Thanks for explanation!
I understand that differences between the SAMPLE data and the sample mean are smaller than the differences between the SAMPLE data and the population mean. BUT! We are not interested in the difference between the SAMPLE data and the population mean, rather we are looking for the difference between the TRUE POPULATION data and the population mean (the population variance). And it's not clear why this value would be larger.
I mean sample data is centered around sample mean the same way population data is centered around population mean. Comparing sample data with population mean feels to be misleading.
The best estimate we can do is the estimate of the variance around the sample mean, which is probably an underestimate, but not always. So this is the best we can do.
8:22 `the way he said "Whaat" is so cute.. I'm in love
:)
This video is such a gem! Thanks for explaining the root of this concept which is not easy to find even in statistics books.
Glad it was helpful!
Thank you so much for the clear and simple explanation. This is an example for when showing the proof is better than only trying to give an intuition.
Thanks
this is literally what I was trying to get a clear understanding on in the last few days? what are the chances? no seriously what are the chances?
That's awesome! :)
Aah! Finally end. What a excellent work by you!! Statquest rocks ❤.. Thank you sir. You helped a lot in my carrier ❤.
Thanks!
Oh no... I'm falling deeper and deeper into this rabbit hole
:)
BAM! I have not seen this concept explained better anywhere else ever. Have you gotten around to making the follow-up video on 'expected values' ? Can't thank you enough for your channel
I've got the video on expected values th-cam.com/video/KLs_7b7SKi4/w-d-xo.html and th-cam.com/video/OSPr6G6Ka-U/w-d-xo.html , but there are still a few steps to go after that... :(
wow that n-1 has something to do with E(X)? Im waiting for it!
I don't even remember what I was confused about in particular, but I remember feeling very happy to see this video. Will revisit this in the following days. Psst, you're a gem ;)
Thank you very much! :)
Man, I always thought that statistics doesn’t make any sense at all and that people should just blindly chug into weird formulas without questioning, but this was absolutely mind opening. Not even khan academy could explain the proof!
Thanks!
I'm eagerly awaiting the expected values quest! Thank you so much for making these videos, I love watching them before sleep.
Awesome! It's on the to-do list, but it might not be done for awhile. :(
@@statquest That's cool, take your time to keep making awesome videos. I still have loads of your videos on my to-watch list!
You left in a cliff hanger of expected values :((
Love your videos tho, thanks for these!
I'm working on it, but everything I do takes longer than I would like. :)
baaammm! subscribed.
Awesome! :)
BAAM! me too
This is excellent, I am looking forward to the next one.
Thank you St Josh for this illuminating explanation :)
My pleasure!
im currently learning data analytics and trying to figure out ab testing and bam! here i am! thank you so much for making statistics fun and easy to understand! double bam!
Happy to help!
Good job Josh!! Waiting for StatQuest on Expected Values! I am the one wondering why not dividing by 'n-0.5' or 'n-2'
Thanks!
The most impressive explanation I've ever seen.
Thanks!
I can only have love for these videos, thank you Josh and all the team if you have any.
Thank you! It's just me doing all this.
Great video!
Where is the one about Expected Values?
I cannot wait with such a cliffhanger! GoT finale can wait...
Very funny! Yes, I have my work to do. I hope to get to expected values before too long.
@@statquest Thanks a lot for responding! ... and sorry, as I noticed after reading more comments, that you had already answered this question many times. Quest on!
8:21 -> I will watch a thousand times and I will laugh out loud a thousand times 😂
Hooray! :)
Greatest explanation so far!
Thank you! :)
This intuitively makes more sense to me now. If I take a sample, the sample mean may end up being larger or smaller than the population mean. But the sample variance can never be larger than the population variance, it might be equal to it, but most probably it will be smaller.
That's exactly right. :)
I think you have to be a little bit careful with what you mean by "sample variance" and "population variance". As long as you're comparing an estimated population variance using the sample data and actual population mean vs an estimated population variance using the sample data and sample mean. But, comparing the estimated population variance using the sample data and sample mean vs the actual population variance (all data and actual population mean) doesn't have the guarantee that sample variance will be lower than population variance.
"The future is now" I'm dying
BAM! :)
This is awesome explanation. Waiting for quest on 'Expected Values'....BAM!
Me too. Hopefully I can get to it soon.
Super interesting! Thanks for your work!
Thanks!
You are a very great teacher, i like your coaching style, keep going on!
Thank you! 😃
Nice! PS. Small typo at 8:42, you say -1 but write x-1.
That 'x' is a "times" symbol. So it's "times -1", not "x - 1"
@@statquest thx I was confused too - by the way - your videos among the very best one can find!!! Thank you so much!!!
I literally watch your videos for fun.
bam!
The minute you mentioned derivatives, my mind shut down!
:( They sound fancy, but they aren't that bad. If you want to learn more about them, see: th-cam.com/video/wl1myxrtQHQ/w-d-xo.html
15:46 it’s the god moment 👏🏻👏🏻👏🏻👏🏻
bam!
Your Voice is magical 🌹🌹🌹
Thank you!
I wish you were my stats teacher!! Amazing job!!!
Thank you! :)
@@statquest really waiting for the expected value video to get explanation of n-1. When can we expect it?
@@shouryanand456 Unfortunately, it might be a while. I've got a full plate until after the summer.
THX!!! Looking forward for the STATQUEST on expected Values ;))))
Me too!
Loved the video. But didn't understand something clearly. The variance is the least around the calculated mean. But that is only when the data x remains the same right? How can you compare it with the population variance which has a lot more data points and the summation is therefore different?
We are not comparing it to the population variance. We are simply comparing the variance of the data calculated around the sample mean compared to the variance of the data calculated around the population mean.
Amazing video!
I think that you should teach another subject. Maybe MathQuest? That would be amazing!
Maybe one day!
Perhaps an idiotic question, but one that has troubled my conscience for a few years now :); so here goes... why do we not divide the sum of squares by n**2 or (n-1)**2 for that matter, when arriving at the variance... for me, mean and std deviation (square root of variance) are of the same order, introducing the division by Square Root of the sample size somehow gives an impression that the std. deviation follows a different order .... where am I going wrong?
An awesome collection BTW... the explanations and the songs are absolutely priceless :)
Ultimately it has to do with something called "Expected Values" and I'll hopefully be able to do a video on that soon.
Este pibe tiene futuro.
Muchas gracias! :)
Hi Josh, where did you study about it, is it from Bessel's correction or Karl Pearson. I am interested to ready a bit about the history behind it. Can you please suggest a book or paper where the original discovery was made. Thanks in advance.
The idea for this came from Bessel's correction.
Sorry I am still getting confused. At 5:29, in the inequality, both left hand side and right hand side are using the same n, which is the number of samples.
You argued that the right hand side is greater so that we need to make the left hand side larger by dividing n-1.
However, the right hand side is not the actual population variance. The actual population variance should be using a much larger n to calculate.
What we are doing here is to estimate the population variance but not the right hand side.
Thinking to this point, all the linkage seems broken. How can I relate the right hand side to the population variance?
It is true that the inequality holds. But it does not mean also the population variance is always greater than the left hand side.
Thanks for your videos. They inspire me and teach me a lot.
Sometimes we know the population mean, but don't know the variance, so we sill have to estimate it. That is what is going on on the right side of the equation.
Thanks so much for the explanation, super clear as always
Glad it was helpful!
Very nice explanation, god bless you josh!
Thank you! :)
Can you provide the slides for all the statistics videos you used to explain the concepts
I have PDF study guides for some of my videos here: statquest.org/studyguides/
Awesome and Thank you!
No problem!
This is gold.
Thank you!
I think there is little problem here @8:47 where we use chain rule .......where that x-1 come I know -1 is derivative of -v but why there is x and where it goes later.
By the way nice explanation .
The little 'x' means "times" and the big "X" is a variable. Sorry for the confusion.
@@statquest Thank You so much for explanation .I get it now.
From 2:38 to 2:40, the symbol for estimated mean was switched to population mean. Great video anyway
Ooops. That's a typo. Thanks for pointing that out. I've updated the pinned comment with it.
Great video! Which book did you get this explanation from?
Ummm....I just did the math.
Is the statquest for expected values not out yet?
Not yet.
dang... why did I end up watching cool as ice (starring vanilla ice) instead of statquest!
:)
I need basic statistics lessons for data science. Is this playlist right for me?
I believe so.
Since sample mean is less than population mean, why don't we divide sample mean by (n-1) to inflate it? That way it will match population mean. And perhaps correction in sample SD may not be required? Just a thought
The sample mean is not always less than the population mean. About half of the time it is greater than the population mean. However, the sample mean will aways be closer to the data and have a smaller standard deviation.
Hey Josh, thanks for another nice video. I understand that dividing by N underestimates the population variance. But, why we are dividing with n-1 and not with n-2 or n-3, etc.
That has to do with expected values, which will be the subject of another StatQuest.
@@statquest Great!
Thanks!
Wow!!! Thank you so much for supporting StatQuest!!! :)
This is Amazing! BAM!!
Thanks!
I'm wondering why -1 T.T~~~ I can't wait for the statquest about expected value! haha it hasn't come out yet, right?
Not yet. :(
@@statquest Has it come out in the meantime? I watched your live session on the expected value but it wasn't in there yet
@@the-karabin Not yet. I'm working on neural networks right now.
mind blown!
BAM!
Thanks
:)
I do understand the way yo calculate min variance
But @ 14:54 How did you conclude in that "Thus valus around sample mean is always less than population mean" ?
Because the sample mean is the value that minimizes the variance, any other value will give you a larger variance.
BAM! Thank you !
:)
You are so awesome ! Thank you Josh :)
Thank you! :)
nice song 0:04
:)
Here you are considering that estimated mean will always be less than the true/population mean correct? if yes then how can you just consider that?
No, I'm not assuming that the estimated mean will always be less than the true/population mean. All I'm assuming is that it will be different from the true/population mean.
Perfect !!! thanks for posting.
Thanks!
1:47
Draw it on the graph how
Are you talking about that curvy line around the histogram?
The red line with arrows on each end represents the population standard deviation.
@@statquest yeah thanks alot that's what i meant
I don't understand 1 thing - In the actual standard deviation we are trying to measure what we are trying to get is sigma[(x-mu)^2]/n. In this the 'n' is the total population size right? Whereas in the sample that we take, the n is just the sample 'n' right? So didn't really understand how we could be comparing the variance with the same n. (For the actual standard deviation the min value will be gotten over the entire population at the actual mean itself right?) I mean are we making an apples to apples comparison?
In both cases, 'n' is the size of the sample, not the size of the population. It could be that we just "know" or "assume" the population mean equals something (like 0).
Your videos are so helpful
Thanks! :)
thank you and this is very helpful!!
I'm so glad!
I still dont really understand why showing V = X bar at minimal variance leads on to the conclusion that we always underestimate the population mean by dividing by n
What part of the video is confusing? Can you specify a minute and second time point?
Great as always! Thanks Prof.
Thanks! :)
@@statquest I know that you have some videos on quantiles on your channel. I was wondering if you could think about the possibility for a video on quantile regression and its implication possiblities in genetics, especially in population genetics. I have found some resources on youtube and through online which mainly focuses on social sciences. Anayways, thank you again.
Awesome! But why not divide by (n-2) or (n-0.5) or (n-10). Why (n-1) only?
16:35
tysm
Bam! :)
Sir,
How did you learn the things that you explain? From good teachers or by yourself by trying to derive what you are curious about?
Curiosity the background of any knowledge is a privilege.
I just try to figure things out on my own. I've had people inspire me, but not many good teachers.
i wish u was my math teacher , your awesome BAM BUM
Thanks! :)
@@statquest can u please make tutorials on deep learning and also tensor
@@Viralvlogvideos I'm working on them.
@@statquest thanks your awesome :)
You save me!!!! ( cfa learner
:)
16:38 , it's not resolved in the expected values video. How to know that prof.
Unfortunately I haven't had time to do the follow up video. The best I can do is give you this link for now: online.stat.psu.edu/stat415/lesson/1/1.3
@@statquest thanks for immediate response prof.😊
Coming from 'CLT', Can we have t-shirts with the double-bam saying "Even if you aren't Normal, the average is Normal" ?
That's such a good idea!!! I'll see what I can do. :)
Thanks for the great video! This was a super straightforward proof that /n will underestimate the variance, but I don’t think I caught why to use (n-1). Why not 8/9n? Or n-1.28362? Or n^(999/1000)? Is there a mathematical reason why (n-1) is standard?
Update: oh, it’s a degrees of freedom thing, I guess? I remember talking about those back in Biostats. I never fully understood them then. Time to go google around.
I'm glad you like the video. Dividing my n-1 is actually an expected value thing. I mention this at the end of the video ( at 16:35 ) and I'll do a video on that in the future. By the way, I'm sorry for the confusion on the subject matter. I tried to avoid confusion by naming this video "Why dividing by 'n' underestimates the variance" instead of "Why we divide by 'n-1' when estimating variance", but the difference is probably subtle.
@@statquest Oh, "you'll just have to wait for the statquest on expected values". I must have not been paying close enough attention at the end of the video. Thanks for putting up with silly comments, Josh. Looking forward to the next video, and thanks again for all your work on these! :)
MEGAAAA BAMMMM for the futureee
Thanks! :)
I think that is a little error in minute 8:48, great videos. have a good day.
I just checked and it looks ok to me, but I've seen it so many times I could be blind to the problem. Can you tell me what you think the error is?
@@statquest You give an explanation that the derivative of (x-v) is -1 but in the moment when I saw 2(x)x-1/n made me feel a little confused but the final is ok. have a great day
@@CarlosCardenas-pq4cq I see. Yes, that little 'x' is supposed to be a "times" symbol. However, I can see how it also looks like the variable 'X'.
@@statquest Ok, I can see the difference of 'X' and 'x', thanks a lot of I love your videos
If you just watched statistics playlist #5 video, start from 3:21 :)
Noted!
The future is now. BAM!
YES! :)