NOTE: This StatQuest was brought to you, in part, by a generous donation from TRIPLE BAM!!! members: M. Scola, N. Thomson, X. Liu, J. Lombana, A. Doss, A. Takeh, J. Butt. Thank you!!!! Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
This video makes me feel so happy. I've had a paper in review for a little while, the editor and one reviewer noticed in one of my figures (in which I had shown the raw data and CIs) that a single datum point was not in line with the rest (n=6 not great but better than most molecular biology papers!). They asked me to add an extra replicate to the experiment. I refused on the grounds of p-hacking and showed them a power analysis (+93% power including the variation of the point) and showed the point is a Grubbs outlier. I'm still waiting for the editor to get back to me ;)
@@karolgilbertosolanosuarez9094 I won ;) gurney et al 2020 microbiology Combinatorial quorum sensing in Pseudomonas aeruginosa allows for novel cheating strategies
Man josh i am trying to finish your playlist . Before this i had tried khan academy, brandon foltz, few books and i failed horribly. P value was something i could never grasp . I think the issue was that everyone talked about what is p value but never explained HOW it is calculated. You really go in the depth of how and why. And this one video right here has blown my mind. It just doesnt explain power analysis but also how you estimate population mean and how p values work . It's 5 am right now and i am still studying .Youve really blown my mind!
the other teachers cannot explain the concepts clearly because their concepts are not clear in the first place LOL. thats where our Mr BAM Sir rocks :D
Wow! I'm very impressed by your very clear explanations and calculations in this video. It was very easy to understand and follow. Big thanks from a PhD-student in Sweden!
I am so happy to see the growth rate of Josh Starmer's StatQuest. I still remember I was one of the few subscriber who joined your fan base when the subscribers where in thousands!. Great going happy learning
Thank you! For my paper my lecturer said we don’t have to include a power analysis, so it wasn’t taught… so glad I watched this because I was struggling to justify my sample size and now it all makes so much more sense.
Your videos are absolutely amazing! It would be fantastic if you could create (if it doesn't already exist) a video covering the entire hypothesis testing process, including all the steps. That would involve determining the sample size and addressing the temptations encountered along the way until reaching the final result.
Fantastic! I’m taking a statistical modeling class with machine learning and this particular topic just wasn’t sticking with me. As advertised, this was super clear and nailed home all the key points!
At 14:09, several assumptions and parameters are presented on the screen, creating the impression of the existence of "power analysis hacking." Furthermore, the concept of double-layered probabilities, where there is an 80% likelihood of correctly rejecting the Null Hypothesis and a 5% chance of randomly obtaining results below the threshold, is enough to make one's head spin. Nevertheless, this video provides the most exceptional and clear explanation.
Hey! After chatting for an hour with chatgpt about p-values, p-hacking and power analysis, I wanted to test my knowledge with a simple case and chatgpt suggested me a coffee shop experiment. The experiment is going to be about a new bean and if it improved the customer satisfaction or not. I'll try to prepare a dataset for this. If I can, I'll try to share! Also, thanks a lot for the content. I want to be a machine learning engineer and decided to learn the fundementals of statistics for that, but statistics basically changed the way I see the world ahahahah.
15:04 It seems tricky to find the right df for t score, for such small sample calculation, when n is unknown. Is it a trial and error process to find the right df?
Josh, thanks so much for your videos - so clearly explained relative to other content I've seen on inference. I do have a few questions. 1. In an A/B testing scenario, we don't know the mean and the standard deviation of the distribution of the treatment group before hand. We do know the mean and the standard deviation of the prior distribution. If I want to estimate sample size required for different effect sizes, holding the chosen p-value threshold constant at 0.05 and power at 0.8, how would we do that given that we don't have s2 or standard deviation of the second group? 2. Also since we don't know the mean of the treatment group's distribution, how would we calculate the estimated difference in means to plug it into the formula for effect size? 3. Do you have a video on How to pick the right test?
i have a question. so knowing statistical power, significance level, effect size, and sample size are all related through power analysis, then wouldn't choosing a statistical power (or even significance level, as we know 0.05 threshold is just for teaching demonstration and in reality it can be any different amount) also be consider p-hacking by effectively getting a desire sample size in "remote" in order to have it prove (or disprove) hypothesis at the researcher's discretion? or is there an objective way to determine the statistical power (and significance level)? most of the materials i read often says "commonly use power = 0.8 or alpha = 0.05" or even heard "amount chosen at researcher's discretion" but without giving sufficient reason for picking those amount.
There are a couple of things to say. 1) Because we select all of the thresholds (power, significance etc.) before doing the experiment, we end up with a sample size that should not be biased in terms of having a higher probability of giving us a false positive of false negative. The key is that we set those parameters before we do the experiment and then we have to live with the result no matter what it says. If we didn't get the result we wanted, and then we did it again, then that would be p-hacking. 2) The thresholds for significance and power are often "field specific" and depend on how easy or difficult it is to control things. In physics labs, they can control things relatively well, so they tend require stricter thresholds. However, when working with human data, where it's hard to control anything, then they tend to have more relaxed criteria. The thresholds also change depending on how serious the consequences are of getting things wrong. For example, if we are testing for a Ebola, we might want to error on the side of caution and allow more false positives so that we can minimize the false negatives.
Thank you Josh for being so careful and patient. I would like to ask you a question that I am not quite clear about. You used s when calculating pooled estimated standard deviations and represented s with a dotted line on the normal distribution. However, it seems that s is very wide in the figure, which looks like 2s. I want to make sure that this s is what we call s, right?
Thank you for the great video! I have one question though- 1:51 In the normal distribution graph, If the variable is day to recovery which can't be lower than 0, then would the estimated populatiln probability distribution be following poisson distribution..? Sorry if I may be confused with the concepts in advance!
Believe it or not, the normal distribution is often used to approximate the poisson distribution. The idea is that if the means are far enough from 0, then it's a good fit. This is due to the central limit theorem. For details, see: online.stat.psu.edu/stat414/lesson/28/28.2
Josh, in terms of the statement 7:49 «because we don’t have a lot of confidence in the estimated means, we’ll end up with a relatively large p-value», can we conclude statement like this: we’ll end up with relatively more both large and smaller p-values, ie p-value more «unstable»?
Hi Josh, thank you a lot for the awesome videos! I have two basic questions: 1. The denominator for the pooled estimated SD in a general condition is the number of the distributions (and not always 2), right? 2. What is the deal with statistics power calculator? isn't there simple formulas we can use ourselves? and does the googling we do to choose one, involve the nature of our experiment, or we just randomly choose one?
1) I'm not sure how you could have more than two distributions, but, presumably, if you did, then you would divide by that number. 2) Every single statistical test has a different formula for doing power calculations. And there are a lot of statistical tests, because there are a lot of different experiment types and data types. Going through every single statistical test would take forever and, to be honest, be pretty boring. So the best thing to do is to google "power calculator" and you'll find a page that has a pulldown menu where you select the test that you want to do (t-test? chi-square goodness of fit? K-S test? etc.) and then plug in your numbers. bam.
Excellent lesson on power analysis. I am immensely impressed by StatQuest’s instructional videos. One criticism here, however, is that the final computation of the necessary sample size to achieve the desired power could have been covered rather than telling a student to find an app on some university’s web site. I think most students could probably work out the math but it seems to go against the overall grain of the talk which careful walks the student through a well chosen toy example that nevertheless fully elucidates the process.
Thanks! I decided to not go over the math for calculating power because I did not want to give the illusion that there was a single equation that fit all situations. In reality, for each test, there is a different equation that we need to use to calculate power and, rather than worry about all of that, I thought it would be easier (and much more practical) to show how power analyses are done.
Thanks Josh. I wanted to ask what should you do if all your data comes from one giant experiment/repository. Your explanations usually assume we have done multiple experiments of small sample size and can generate pooled statistics. But if you did one big experiment with a very large sample size, should you artificially separate individual test into small groups so you can generate pooled statistics? Thanks again, I hope i make sense.
Most of the time people do one big experiment and collect all of the data at once. This is the standard procedure. However, in the video, I show what happens if you can repeat that same experiment a bunch of times to give you a sense of how whatever you are estimating (like the mean) has less variation when you have more samples. Thus, the repeated experiments in this video are just an illustration to give you confidence that a large sample size will be more accurate that a small one.
Thank you, very helpful! I am a new bioinformatician and planning to do differential gene expression with DESeq2 comparing a human sample size=1000. How could we apply power analysis for RNA seq differential expression?
I believe there are some standard RNS-seq power calculators out there that you can use. I remember using one, but that was 6 years ago and I bet there is something better now.
Thanks Josh! Just want to make sure, the green and red normal distributions are created from 1. collecting data_size=n from one group, calculate mean 2. repeat 1 R times 3. use R means to create green/red normal distribution correct?
Unfortunately that's not correct. The green and red distributions are the "population" distributions that we are trying to estimate with relatively small sample sizes. For details on population distributions and how they are different from samples, see: th-cam.com/video/vikkiwjQqfU/w-d-xo.html and th-cam.com/video/SzZ6GpcfoQY/w-d-xo.html
This is, as usual, excellent. I'm a little bit surprised that you do not show alpha and beta for a given test (as the surface below or above the critical value). It allows people to understand the main problem of the power analysis: you need to bet on the effect size as, by definition, it's unknown. It leads, as you know, to soo many problems as we alsways discover too late that the actual mean difference is smaller than expected (also known as the "curse of just a little bit too small sample". The most common solution being a ritual sacrifice of a random slave. Sorry, I want to mean a PhD student.
Thank you for the detailed explanation! But...I am having trouble finding a reliable power calculator online that allows me to input the Effect Size...perhaps you could recommend a few? What power calculator did you use to get the value of sample size 9? Regardless, I loved the first part of your video! Very detailed, very interactive, thank you for this!
Hi Josh! Thanks for your amazing explanation! Appreciate greatly! I have a questions. I read from a book, it says that large sample size can have unwanted implication. QUOTE ....'By “too high” we mean that by increasing sample size, smaller and smaller effects (e.g., correlations) will be found to be statistically significant. The researcher must always be aware that sample size can affect the statistical test either by making it insensitive (at small sample sizes) or overly sensitive (at very large sample sizes). UNQUOTE So does it mean that even the test we are conducting is from the same distribution(or null hypothesis is true) . It will some how reject the null hypothesis given large enough sample size? Thanks!
Regardless of the sample size, you always need to make sure that the effect size is large enough to have "meaning". For example, if taking an expensive medication increases life span by 2 seconds, then who cares if it is statistically significant or not - 2 seconds is not worth taking an expensive medication. So when ever you do statistics, don't just look at the p-value, also look at the effect size and make sure it is meaningful.
Thank you Josh for all your videos! they help make statistics look less deadly. Would look forward to some content on bayesian statistics and then maybe solving problems using both bayesian and frequentist methods :)
Hello, thank you so much for this video. I have a question as I want to try do a power analysis for my data however it has a very small sample size so I could not assume that the data was normally distributed. If you do do the Shapiro Wilks test most of the data appears to be normally distributed. In the actual study I compared medians and IQRs rather than means and SDs. Does this mean I cannot do a power calculation because I decided to use non-parametric tests for my data because I couldn't assume normality?
Thanks for the great video. In your example you said 9 measurements per a group is enough. What does group mean in correlation studies? Do we have more than one group in these studies?
@@Mostafaseyyedabadi If you're just trying to establish a correlation, you just have one group. You can determine the sample size using a sample size calculator for regression. For example, you could try this one: www.statskingdom.com/sample_size_regression.html
If the sample size is large enough, the distribution doesn't matter (per the Central Limit Theorem: th-cam.com/video/YAlJCEDH2uY/w-d-xo.html ). Alternatively you can use non-parametric methods. These do not depend on the distribution.
@@statquest Thank you very much! We have a really small sample size, so I don't think that is an option. Do you maybe have a link to wich non-parametric methods we can use?
@@nelhuens7590 Unfortunately, I don't. But if your data is continuous, instead of using a t-test, you can use a Mann-Whitney U test: en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
Love your videos, thanks! :) Quick question though - from this video it seems that we should google 'statistics power calculator' after we set up 3 things - alpha, power and effect size. We can easily choose alpha and power, but what about effect size? It seems to be the thing that you are able to calculate after the test finishes, you can't set it up beforehand. Or maybe you meant something like 'minimal effect size that is enough', e.g. from business perspective?
It depends on the area you are working in. For example, in biology, we often want a 2-fold effect size. In physics, a smaller effect size might be good enough. So you have to know what people want in the field to determine the value.
@@statquest thank you for the answer. And in case of AB testing? It would be nice to know beforehand what would be the sufficient sample size. Then what should we set up in place of effect size?
Thanks for the great video! Do I understand correctly that the sample size determined from the power analysis depends on the means and variances estimated from the first experiment? How do I deal with the fact that this first experiment could result in estimating means and variances far away from the population's?
Hi, very useful material and well explained. Thanks. What should be the power if i want to accept null hypothesis? How should be the overlap and sample size, in this case?
Unfortunately, traditional statistics is not designed to accept the null. The best we can do is to "fail to reject the null". You can make this failure relatively convincing if you select a relatively high power (like 95%) for rejecting the null and still fail.
@@statquest I have a subject interviewed at time 0 and at the same time, the subject has given rating for those questions asked via an app. Now, such measurements are repeated at an interval. I want to show that there is no statistical difference between interview rating and self-rating via app. I need to find out the sample number for this analysis. What should be the power , alpha and effect size to find out the sample number? I appreciate your help. Thank you
Thank for making these videos. One question on the beginning of this video. What is the actual method behind to "do a statistical test and compare the means and get a p-value=0.06"? From what I read, is T-test for two sample mean in this case? Thanks!
Thanks for this video I now get the 'effect size' true meaning I am unable to find the sample size calculator. I am learning stats to work on SPSS but was curious how you got the value 9 for the example
Thanks for another great video Josh! There's one thing I still fail to understand, though: Isn't it true that a p-value and Power measure the same thing, namely the chance of (correctly) rejecting the H0? And if this is the case, is a chance of 80% of correctly rejecting the H0 not a downgrade from having a 95% chance when using an alpha of .05?
No, p-values and power are fundamentally different. p-values assume that the null hypothesis is correct. Power assumes that the null hypothesis is not correct. This makes pretty much everything about them different.
It's the same - even with just one sample, the sample has a distribution that may or may not be the same as an ideal distribution that we are comparing it to.
Wonderful video, very clear as always. Often the terms 'type 1 error' and 'type 2 error' are used in these discussions, or α and β. Was there a reason for avoiding this terminology, aside from just generally avoiding jargon?
Great series of video again Josh!! I have one question tho, and I have tried to ask Chatgpt, and it didnt help much... Given we rejected the null, if we say alpha is the possibilities of falsely reject the null, lets say it is 0.05. And then the opposite would be correctly reject the null which is 1-0.05 = 0.95. Isn't this exactly the definition of Power? So given alpha = 0.05, the chance of correctly reject the null is 95%, and that is our power? I know it is somehow false, but i cannot figure out why
I actually have an entire video that answers your question: th-cam.com/video/Rsc5znwR5FA/w-d-xo.html (I'll give you a hint: 1-alpha is not what you think it is. 1-alpha is related to the probability that we will fail to reject the null).
@@statquest Thanks a lot Josh! I realized I made that mistake haha! 1-alpha is consist of two parts, 1.correctly accept the true null 2. falsely accept the false null and Power is the chance of correctly reject the false null. So they are not completely the same
I have a question: if the original data that are used for the sd and mean of the two distributions are not representative due to small sample size, does all of this power analysis breaks down. That is to say, power analysis is only based on previous large sample size data?
No, it doesn't break down because we give ourselves the potential to be wrong. When the "power" is set to 80%, that means we only have an 80% chance of correctly rejecting the null hypothesis. If we want a 90% chance of correctly rejecting the null, we need to increase the sample size.
Man, I'm totally hooked on your playlist (it deserves to be on Netflix!). When you mentioned having a sample size of 9, I got a bit confused. How many samples, each with a size of 9, should I collect? I always struggle with this. For instance, would it be the same if I had 3 samples, each with a size of 3? Could you clarify that?"
@@vladfarias It depends. For example, 1 person getting 9 measurements could be different than 3 people getting 3 measurements each because there could be variation in how the 3 different people collected measurements. Or if we use the same person, but collected 3 measurements on 3 different days, because there could be extra variation due to the different days. So your estimate of variation needs to take how the data will be collected into account.
Hi Josh, thanks for the amazing videos! There is one part of this that confuses me, how do you get the population means and population standard deviations to put in the power analysis calculator? Do you simply run the experiment once with many replicates for your positive control and negative control? Then is it okay to apply the resulting sample size to your conditions?
@@statquest I'm also struggling with this. In previous publications, they don't show the actual numbers for mean and standard deviation; only a graph. I guess I have to just guess based on the graph?
According to the video, calculating the effect size in the end using the difference of means, it is divided by the pooled estimated standard deviations. What standard deviation (std) is that? The std of samples, or the standard deviation of the estimated difference in means? (the std of the mean should be 1/sqrt(n) of the samples std) Does the effect size follow the same calculation as Cohen's d ?
Hi Josh, for a trustworthy result I would combine Power Analysis to determine the sample size and after that calculate P-values with the FDR correction. Is this right? Or is the FDR correction not more needed after Power Analysis? Many thanks for your response!
Thank you Josh! I have a question. Could you collect a sample from Drug A and Drug B and use the two samples to calculate an Estimated Mean and Standard Deviation in order to calculate the Sample Size required for future research?
Is it possible to calculate the sample size required even if the distributions can't be assumed to be normal distribution? I would think it is possible by using the Mann-Whitney U test but can't seem to figure out how. Thanks for the great explanation anyways!
Thanks for the video! One quick question: before we calculate the effect size and sample size, how do we get estimated µ and s for green and purple group in the first place?
Thank you for the excellent video. If you please, I would like to ask you a simple question. In order to calculate the effect size, you need to know S which depends on the sample size (in the video it is 10). Then you use this to estimate the "sample size" which is 9. What happens if they are so much different? Can this happen in 2 cases 1. the start sample size >> the calculated one and vice versa.
In theory, 's' (the standard deviation) does not actually depend on the sample size. We can get it from any source or just guess based on previous experience. Thus, if the output sample size is very different, that's OK.
Awesome video! There's really nothing like being able to visualise a concept. Is there a reason you never mentioned the standard error when you were explaining how larger sample sizes increase the confidence that your sample mean is close to the population mean? I mean standard error is literally the measure of that confidence right?
Yes, the standard error is the measure of that confidence. However, I omitted it because I was simply trying to convey the concept of power, rather than dive into the technical terminology.
What if I am trying to compare three groups? How would I calculate the effect size? What other key words should I google in addition to "statistics power calculator"? Thanks in advance for answer my questions, your videos are awesome!
I discovered these videos a while ago and they're really great for understanding various statistical concepts. I need to do some power analyses on non-parametric data, is that even a thing? Is there even a variability estimate for a Mann-Whitney two sample test? Thanks.
Hi Josh, Thoroughly enjoyed the video. One quick question though: if the two means are d1 and d2, then how do you to take the difference between them, should it be |d1-d2| or one can used either d1-d2 or d2-d1. On a side note my kids have started complaining - They say that I am always watching stat quest :)
The order doesn't matter (and we don't use the absolute value either). If you want to learn a very cool way to compare two means, check out my playlist on linear regression. Believe it or not, a test to compare means is a type of linear regression: th-cam.com/play/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU.html (Tell your kids that I apologize for making your watch more StatQuest! :)
Thanks for the video :) and I got 2 questions: 1. If I cannot find any published data for the effect size, what should be the sample size to generate a preliminary data? 2. is the threshold of significance (alpha) = p value?
Really enjoying these videos - I was recommend by a colleague to check out your channel. One question I wanted to ask you - Is the idea that we would collect 10 samples several times, and then get the mean of the estimated means and compare that to the mean of the estimated means from the other distribution using a t-test? Just curious how it all comes together but maybe that comes in some later videos...thank you
The idea is to collect 10 measurements 1 time and then do your tests. However, this video illustrates what would happen if we did it a bunch of times to give you a sense of how much variability there would be in different sample sizes. The larger the sample size, the lower the variability in the mean, and thus, the more confidence we can have in its value.
NOTE: This StatQuest was brought to you, in part, by a generous donation from TRIPLE BAM!!! members: M. Scola, N. Thomson, X. Liu, J. Lombana, A. Doss, A. Takeh, J. Butt. Thank you!!!!
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
This video makes me feel so happy. I've had a paper in review for a little while, the editor and one reviewer noticed in one of my figures (in which I had shown the raw data and CIs) that a single datum point was not in line with the rest (n=6 not great but better than most molecular biology papers!). They asked me to add an extra replicate to the experiment. I refused on the grounds of p-hacking and showed them a power analysis (+93% power including the variation of the point) and showed the point is a Grubbs outlier. I'm still waiting for the editor to get back to me ;)
BAM! :)
What did the editor reply?
@@raycyst-k9v Editor said: BAAM!
So... What happened? Do you have your DOI for us to read? 🎉🥳🤩
@@karolgilbertosolanosuarez9094 I won ;) gurney et al 2020 microbiology Combinatorial quorum sensing in Pseudomonas aeruginosa allows for novel cheating strategies
Man josh i am trying to finish your playlist . Before this i had tried khan academy, brandon foltz, few books and i failed horribly. P value was something i could never grasp . I think the issue was that everyone talked about what is p value but never explained HOW it is calculated. You really go in the depth of how and why. And this one video right here has blown my mind. It just doesnt explain power analysis but also how you estimate population mean and how p values work . It's 5 am right now and i am still studying .Youve really blown my mind!
Hooray!!! I'm glad my videos are helpful. :)
@@statquest HUGE BAM!
You've freaking done it again. I can't belive how simple this explanation was compared to the lecture I got in school. Thank you very much!
bam!
the other teachers cannot explain the concepts clearly because their concepts are not clear in the first place LOL. thats where our Mr BAM Sir rocks :D
Wow! I'm very impressed by your very clear explanations and calculations in this video. It was very easy to understand and follow. Big thanks from a PhD-student in Sweden!
Thank you very much! :)
I am so happy to see the growth rate of Josh Starmer's StatQuest. I still remember I was one of the few subscriber who joined your fan base when the subscribers where in thousands!. Great going happy learning
Thank you very much! :)
I learned more (and laughed more) in these 16 minutes than a whole semester at med school. Thank you!
bam! :)
Thank you! For my paper my lecturer said we don’t have to include a power analysis, so it wasn’t taught… so glad I watched this because I was struggling to justify my sample size and now it all makes so much more sense.
Awesome!!! :)
I've done so much stats but akways had a difficulty understanding power analysis. This is very clear, and practical.
Thanks!
One of the best channels out there to learn statistics.
Thank you!
Your videos are absolutely amazing! It would be fantastic if you could create (if it doesn't already exist) a video covering the entire hypothesis testing process, including all the steps. That would involve determining the sample size and addressing the temptations encountered along the way until reaching the final result.
Great suggestion!
Fantastic! I’m taking a statistical modeling class with machine learning and this particular topic just wasn’t sticking with me. As advertised, this was super clear and nailed home all the key points!
Glad it was helpful!
Thanks I haven’t slept this good in weeks
noted!
Josh in May 2020: "Imagine there is a virus"
2020: "Hold my beer"
:)
Thanks Josh, you are the best at storytelling when explaining statistics.
Wow, thanks!
You are a great teacher, Josh ! Thank you so much. For a long time, I had difficulty in understanding this concept. Now, it is crystal clear.. :-)
Thanks!
Simulation is really helpful to understand this.
At 14:09, several assumptions and parameters are presented on the screen, creating the impression of the existence of "power analysis hacking." Furthermore, the concept of double-layered probabilities, where there is an 80% likelihood of correctly rejecting the Null Hypothesis and a 5% chance of randomly obtaining results below the threshold, is enough to make one's head spin. Nevertheless, this video provides the most exceptional and clear explanation.
Thanks!
Thank you Professor. I will use it for my class. It is so well explained, I learnt how to explain complex concepts.
Glad it was helpful!
Totally loving the series
Hooray! :)
Thank you for making learning more interesting. Double bammm for the knowledge and funny remarks
Thank you! :)
The interesting examples really gave me a new perspective on the problem. Appreciate this! Thank you so much!
Glad it was helpful!
I'm happy my fate made me study statistics after you made this playlist , Thank you so much for this amazing explanation
Happy to help!
Hey! After chatting for an hour with chatgpt about p-values, p-hacking and power analysis, I wanted to test my knowledge with a simple case and chatgpt suggested me a coffee shop experiment. The experiment is going to be about a new bean and if it improved the customer satisfaction or not. I'll try to prepare a dataset for this. If I can, I'll try to share! Also, thanks a lot for the content. I want to be a machine learning engineer and decided to learn the fundementals of statistics for that, but statistics basically changed the way I see the world ahahahah.
I have to learn how to do those statistical tests though :)) I'll watch the quest
bam!
15:04 It seems tricky to find the right df for t score, for such small sample calculation, when n is unknown. Is it a trial and error process to find the right df?
df for a t-test is directly related to the sample size. So if you can solve for 'n', you can solve for the degrees of freedom.
i swear to god that you are a life saver man
:)
Josh, thanks so much for your videos - so clearly explained relative to other content I've seen on inference. I do have a few questions.
1. In an A/B testing scenario, we don't know the mean and the standard deviation of the distribution of the treatment group before hand. We do know the mean and the standard deviation of the prior distribution. If I want to estimate sample size required for different effect sizes, holding the chosen p-value threshold constant at 0.05 and power at 0.8, how would we do that given that we don't have s2 or standard deviation of the second group?
2. Also since we don't know the mean of the treatment group's distribution, how would we calculate the estimated difference in means to plug it into the formula for effect size?
3. Do you have a video on How to pick the right test?
1) Do a pilot study to get a sense of the mean and standard dev.
2) Same
3) Not yet.
i have a question. so knowing statistical power, significance level, effect size, and sample size are all related through power analysis, then wouldn't choosing a statistical power (or even significance level, as we know 0.05 threshold is just for teaching demonstration and in reality it can be any different amount) also be consider p-hacking by effectively getting a desire sample size in "remote" in order to have it prove (or disprove) hypothesis at the researcher's discretion? or is there an objective way to determine the statistical power (and significance level)? most of the materials i read often says "commonly use power = 0.8 or alpha = 0.05" or even heard "amount chosen at researcher's discretion" but without giving sufficient reason for picking those amount.
There are a couple of things to say. 1) Because we select all of the thresholds (power, significance etc.) before doing the experiment, we end up with a sample size that should not be biased in terms of having a higher probability of giving us a false positive of false negative. The key is that we set those parameters before we do the experiment and then we have to live with the result no matter what it says. If we didn't get the result we wanted, and then we did it again, then that would be p-hacking.
2) The thresholds for significance and power are often "field specific" and depend on how easy or difficult it is to control things. In physics labs, they can control things relatively well, so they tend require stricter thresholds. However, when working with human data, where it's hard to control anything, then they tend to have more relaxed criteria. The thresholds also change depending on how serious the consequences are of getting things wrong. For example, if we are testing for a Ebola, we might want to error on the side of caution and allow more false positives so that we can minimize the false negatives.
This tutorial is great and make a lot of common sense.
Thank you!
Thanks
Wow!!! THANK YOU VERY MUCH!!!!
Dude, you have awesome explanatory superPower!
Thanks! 😃
watching a night before my finals for STaT! thanks josh
Good luck!!
Nice way to start a remote workday.
Niila Saarinen yes right?
Hooray! :)
Hope your boss is not the subscriber :P
This explanation is amazing!!! Thank you so very much!!
Thank you!
excellent video series in general, and particularly good video on Power Analysis! Thank you for making and sharing!
Thank you!
Thank you Josh for being so careful and patient. I would like to ask you a question that I am not quite clear about. You used s when calculating pooled estimated standard deviations and represented s with a dotted line on the normal distribution. However, it seems that s is very wide in the figure, which looks like 2s.
I want to make sure that this s is what we call s, right?
's' looks pretty accurate to me. About 65% of the area under each curve should be under the dotted line.
Thank you for the great video! I have one question though- 1:51 In the normal distribution graph, If the variable is day to recovery which can't be lower than 0, then would the estimated populatiln probability distribution be following poisson distribution..?
Sorry if I may be confused with the concepts in advance!
Believe it or not, the normal distribution is often used to approximate the poisson distribution. The idea is that if the means are far enough from 0, then it's a good fit. This is due to the central limit theorem. For details, see: online.stat.psu.edu/stat414/lesson/28/28.2
@@statquest I understand the explanation, thank you!!
so clear and understandable !!!! Thank you very much!!!
Glad it helped!
Fantastic explanation, very useful and awesome graphics, high educational value!! CLAP CLAP
Thank you very much! :)
Josh, in terms of the statement 7:49 «because we don’t have a lot of confidence in the estimated means, we’ll end up with a relatively large p-value», can we conclude statement like this: we’ll end up with relatively more both large and smaller p-values, ie p-value more «unstable»?
That sounds like a reasonable statement. However, how "stable" or "unstable" the p-value is probably depends on the underlying distributions.
Hi Josh, thank you a lot for the awesome videos!
I have two basic questions:
1. The denominator for the pooled estimated SD in a general condition is the number of the distributions (and not always 2), right?
2. What is the deal with statistics power calculator? isn't there simple formulas we can use ourselves? and does the googling we do to choose one, involve the nature of our experiment, or we just randomly choose one?
1) I'm not sure how you could have more than two distributions, but, presumably, if you did, then you would divide by that number.
2) Every single statistical test has a different formula for doing power calculations. And there are a lot of statistical tests, because there are a lot of different experiment types and data types. Going through every single statistical test would take forever and, to be honest, be pretty boring. So the best thing to do is to google "power calculator" and you'll find a page that has a pulldown menu where you select the test that you want to do (t-test? chi-square goodness of fit? K-S test? etc.) and then plug in your numbers. bam.
Excellent lesson on power analysis. I am immensely impressed by StatQuest’s instructional videos. One criticism here, however, is that the final computation of the necessary sample size to achieve the desired power could have been covered rather than telling a student to find an app on some university’s web site. I think most students could probably work out the math but it seems to go against the overall grain of the talk which careful walks the student through a well chosen toy example that nevertheless fully elucidates the process.
Thanks! I decided to not go over the math for calculating power because I did not want to give the illusion that there was a single equation that fit all situations. In reality, for each test, there is a different equation that we need to use to calculate power and, rather than worry about all of that, I thought it would be easier (and much more practical) to show how power analyses are done.
14:55 If the calculator suggests a sample size of 9, is it safe to increase the sample size or should I just stick to the calculated sample size?
You can always collect and use more measurements than the minimum.
I love it when you're saying "BAM!" lol
BAM! :)
Thanks Josh. I wanted to ask what should you do if all your data comes from one giant experiment/repository. Your explanations usually assume we have done multiple experiments of small sample size and can generate pooled statistics. But if you did one big experiment with a very large sample size, should you artificially separate individual test into small groups so you can generate pooled statistics? Thanks again, I hope i make sense.
Most of the time people do one big experiment and collect all of the data at once. This is the standard procedure. However, in the video, I show what happens if you can repeat that same experiment a bunch of times to give you a sense of how whatever you are estimating (like the mean) has less variation when you have more samples. Thus, the repeated experiments in this video are just an illustration to give you confidence that a large sample size will be more accurate that a small one.
This lecture was really excellent 😊
Glad you liked it!
Thank you, very helpful! I am a new bioinformatician and planning to do differential gene expression with DESeq2 comparing a human sample size=1000. How could we apply power analysis for RNA seq differential expression?
I believe there are some standard RNS-seq power calculators out there that you can use. I remember using one, but that was 6 years ago and I bet there is something better now.
Thanks Josh! Just want to make sure, the green and red normal distributions are created from
1. collecting data_size=n from one group, calculate mean
2. repeat 1 R times
3. use R means to create green/red normal distribution
correct?
Unfortunately that's not correct. The green and red distributions are the "population" distributions that we are trying to estimate with relatively small sample sizes. For details on population distributions and how they are different from samples, see: th-cam.com/video/vikkiwjQqfU/w-d-xo.html and th-cam.com/video/SzZ6GpcfoQY/w-d-xo.html
Awesome channel. Thank you for what you do
Glad you enjoy it!
This is, as usual, excellent. I'm a little bit surprised that you do not show alpha and beta for a given test (as the surface below or above the critical value). It allows people to understand the main problem of the power analysis: you need to bet on the effect size as, by definition, it's unknown. It leads, as you know, to soo many problems as we alsways discover too late that the actual mean difference is smaller than expected (also known as the "curse of just a little bit too small sample". The most common solution being a ritual sacrifice of a random slave. Sorry, I want to mean a PhD student.
Thank you for the detailed explanation!
But...I am having trouble finding a reliable power calculator online that allows me to input the Effect Size...perhaps you could recommend a few? What power calculator did you use to get the value of sample size 9?
Regardless, I loved the first part of your video! Very detailed, very interactive, thank you for this!
www.statskingdom.com/32test_power_t_z.html
You're the best, boss!
Wow, thanks!
Hi Josh! Thanks for your amazing explanation! Appreciate greatly! I have a questions.
I read from a book, it says that large sample size can have unwanted implication.
QUOTE
....'By “too high” we mean that by increasing sample size, smaller and smaller effects (e.g., correlations) will be found to be statistically significant. The researcher must always be aware that sample size can affect the statistical test either by making it insensitive (at small sample sizes) or overly sensitive (at very large
sample sizes). UNQUOTE
So does it mean that even the test we are conducting is from the same distribution(or null hypothesis is true) . It will some how reject the null hypothesis given large enough sample size? Thanks!
Regardless of the sample size, you always need to make sure that the effect size is large enough to have "meaning". For example, if taking an expensive medication increases life span by 2 seconds, then who cares if it is statistically significant or not - 2 seconds is not worth taking an expensive medication. So when ever you do statistics, don't just look at the p-value, also look at the effect size and make sure it is meaningful.
Excellent explanation...
Thank you! :)
followed you on bilibili haha, cant wait to your new vedios, so I came here :)
bam! :)
Thank you Josh for all your videos! they help make statistics look less deadly. Would look forward to some content on bayesian statistics and then maybe solving problems using both bayesian and frequentist methods :)
Will do!
Hello, thank you so much for this video. I have a question as I want to try do a power analysis for my data however it has a very small sample size so I could not assume that the data was normally distributed. If you do do the Shapiro Wilks test most of the data appears to be normally distributed. In the actual study I compared medians and IQRs rather than means and SDs. Does this mean I cannot do a power calculation because I decided to use non-parametric tests for my data because I couldn't assume normality?
There are plenty of power calculators out there for non-parametric tests.
Thanks for the great video. In your example you said 9 measurements per a group is enough. What does group mean in correlation studies? Do we have more than one group in these studies?
What time point in the video, minutes and seconds, are you asking about?
@@statquest thanks for your quick reply. I asked about minute: 14 and second: 59
@@Mostafaseyyedabadi If you're just trying to establish a correlation, you just have one group. You can determine the sample size using a sample size calculator for regression. For example, you could try this one: www.statskingdom.com/sample_size_regression.html
great explanation. Thanks!
Thanks!
Great explanation! What do you do if you want to compare a normal distribution with a non-normal distribution?
If the sample size is large enough, the distribution doesn't matter (per the Central Limit Theorem: th-cam.com/video/YAlJCEDH2uY/w-d-xo.html ). Alternatively you can use non-parametric methods. These do not depend on the distribution.
@@statquest Thank you very much! We have a really small sample size, so I don't think that is an option. Do you maybe have a link to wich non-parametric methods we can use?
@@nelhuens7590 Unfortunately, I don't. But if your data is continuous, instead of using a t-test, you can use a Mann-Whitney U test: en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
Love your videos, thanks! :) Quick question though - from this video it seems that we should google 'statistics power calculator' after we set up 3 things - alpha, power and effect size. We can easily choose alpha and power, but what about effect size? It seems to be the thing that you are able to calculate after the test finishes, you can't set it up beforehand. Or maybe you meant something like 'minimal effect size that is enough', e.g. from business perspective?
It depends on the area you are working in. For example, in biology, we often want a 2-fold effect size. In physics, a smaller effect size might be good enough. So you have to know what people want in the field to determine the value.
@@statquest thank you for the answer. And in case of AB testing? It would be nice to know beforehand what would be the sufficient sample size. Then what should we set up in place of effect size?
@@jakubmazur1159 Again, this is domain specific.
Thanks for the great video! Do I understand correctly that the sample size determined from the power analysis depends on the means and variances estimated from the first experiment? How do I deal with the fact that this first experiment could result in estimating means and variances far away from the population's?
You have to start with some general sense of the variation in the data. It doesn't have to be perfect, but it's the best you can do.
Supereb sir... Thank you..
Thank you!
Excelent video, thank you very much.
Glad you liked it!
Great explanation! Thank you.
Thanks!
Hi, very useful material and well explained. Thanks. What should be the power if i want to accept null hypothesis? How should be the overlap and sample size, in this case?
Unfortunately, traditional statistics is not designed to accept the null. The best we can do is to "fail to reject the null". You can make this failure relatively convincing if you select a relatively high power (like 95%) for rejecting the null and still fail.
@@statquest I have a subject interviewed at time 0 and at the same time, the subject has given rating for those questions asked via an app. Now, such measurements are repeated at an interval. I want to show that there is no statistical difference between interview rating and self-rating via app. I need to find out the sample number for this analysis. What should be the power , alpha and effect size to find out the sample number? I appreciate your help. Thank you
every time when I watch this series it reminds me of Moss in the IT Crowd :D
Nice!
As easy as it can be! Thanks...
You're welcome!
Thank for making these videos. One question on the beginning of this video. What is the actual method behind to "do a statistical test and compare the means and get a p-value=0.06"? From what I read, is T-test for two sample mean in this case? Thanks!
Yes, in this case I'm using a t-test.
Thanks for this video
I now get the 'effect size' true meaning
I am unable to find the sample size calculator. I am learning stats to work on SPSS but was curious how you got the value 9 for the example
Just google "sample size calculator" and you should be good to go.
Hi, there are different power calculators online and they require different inputs. How do you call the one that needs: power, threshold, effect size?
You have to know what kind of test you want to use - once you figure that out, the inputs will all make sense.
It was very very very easy explanation
Thanks!
Thanks for another great video Josh! There's one thing I still fail to understand, though: Isn't it true that a p-value and Power measure the same thing, namely the chance of (correctly) rejecting the H0? And if this is the case, is a chance of 80% of correctly rejecting the H0 not a downgrade from having a 95% chance when using an alpha of .05?
No, p-values and power are fundamentally different. p-values assume that the null hypothesis is correct. Power assumes that the null hypothesis is not correct. This makes pretty much everything about them different.
@@statquest thanks a lot for replying! I clearly still have some work to do😁
@@baksteen2420 To learn more about p-values, try: th-cam.com/video/vemZtEM63GY/w-d-xo.html and th-cam.com/video/JQc3yx0-Q9E/w-d-xo.html
Thanks for your illustration. but to understand it more, you explained it on an example of two sample test. What if we run one sample test ?
It's the same - even with just one sample, the sample has a distribution that may or may not be the same as an ideal distribution that we are comparing it to.
Thank you, great explanation
Thanks!
Wonderful video, very clear as always. Often the terms 'type 1 error' and 'type 2 error' are used in these discussions, or α and β. Was there a reason for avoiding this terminology, aside from just generally avoiding jargon?
I really don't like that terminology. I'd rather people say "false positive" or "false negative" instead of "type 1" or "type 2".
Top-notch! Thank you!!!!!!
Thanks!
Great, easy, and simple video. Could you provide a reference for the formula you used to calculate Cohen's (d)? Thanks!
en.wikipedia.org/wiki/Effect_size NOTE: The pooled standard deviation is simplified to assume the same number of measurements from both categories.
Great series of video again Josh!! I have one question tho, and I have tried to ask Chatgpt, and it didnt help much...
Given we rejected the null, if we say alpha is the possibilities of falsely reject the null, lets say it is 0.05. And then the opposite would be correctly reject the null which is 1-0.05 = 0.95. Isn't this exactly the definition of Power? So given alpha = 0.05, the chance of correctly reject the null is 95%, and that is our power? I know it is somehow false, but i cannot figure out why
I actually have an entire video that answers your question: th-cam.com/video/Rsc5znwR5FA/w-d-xo.html (I'll give you a hint: 1-alpha is not what you think it is. 1-alpha is related to the probability that we will fail to reject the null).
@@statquest Thanks a lot Josh! I realized I made that mistake haha! 1-alpha is consist of two parts, 1.correctly accept the true null 2. falsely accept the false null and Power is the chance of correctly reject the false null. So they are not completely the same
I have a question: if the original data that are used for the sd and mean of the two distributions are not representative due to small sample size, does all of this power analysis breaks down. That is to say, power analysis is only based on previous large sample size data?
No, it doesn't break down because we give ourselves the potential to be wrong. When the "power" is set to 80%, that means we only have an 80% chance of correctly rejecting the null hypothesis. If we want a 90% chance of correctly rejecting the null, we need to increase the sample size.
Man, I'm totally hooked on your playlist (it deserves to be on Netflix!). When you mentioned having a sample size of 9, I got a bit confused. How many samples, each with a size of 9, should I collect? I always struggle with this. For instance, would it be the same if I had 3 samples, each with a size of 3? Could you clarify that?"
What time point, minutes and seconds, in the video are you referring to?
@@statquest 14:56. Anyway, would it be the same? (One sample size 9 and 3 samples size 3)
@@vladfarias It depends. For example, 1 person getting 9 measurements could be different than 3 people getting 3 measurements each because there could be variation in how the 3 different people collected measurements. Or if we use the same person, but collected 3 measurements on 3 different days, because there could be extra variation due to the different days. So your estimate of variation needs to take how the data will be collected into account.
Great video, I’m still a bit confused on how having different sample size for the two groups would affect the calculations
Here's something that might help: stats.stackexchange.com/questions/108079/can-i-do-a-t-test-power-analysis-for-unequal-size-groups-which-produces-2-differ
@@statquest thanks!! that was a useful link
Hi Josh, thanks for the amazing videos! There is one part of this that confuses me, how do you get the population means and population standard deviations to put in the power analysis calculator? Do you simply run the experiment once with many replicates for your positive control and negative control? Then is it okay to apply the resulting sample size to your conditions?
You have to estimate the population parameters with preliminary data, or data from other sources (like publications) or just take an educated guess.
@@statquest Thanks very much for your reply!
@@statquest I'm also struggling with this. In previous publications, they don't show the actual numbers for mean and standard deviation; only a graph. I guess I have to just guess based on the graph?
@@somenerdyblonde Email the authors of the publication.
According to the video, calculating the effect size in the end using the difference of means, it is divided by the pooled estimated standard deviations. What standard deviation (std) is that? The std of samples, or the standard deviation of the estimated difference in means? (the std of the mean should be 1/sqrt(n) of the samples std)
Does the effect size follow the same calculation as Cohen's d ?
It's the standard deviation of the samples.
Hi Josh,
for a trustworthy result I would combine Power Analysis to determine the sample size and after that calculate P-values with the FDR correction. Is this right?
Or is the FDR correction not more needed after Power Analysis?
Many thanks for your response!
If you do multiple tests, you should always use FDR.
Thank you Josh! I have a question. Could you collect a sample from Drug A and Drug B and use the two samples to calculate an Estimated Mean and Standard Deviation in order to calculate the Sample Size required for future research?
Yep!
@@statquest BAM!!! Thanks, Josh!
Is it possible to calculate the sample size required even if the distributions can't be assumed to be normal distribution?
I would think it is possible by using the Mann-Whitney U test but can't seem to figure out how.
Thanks for the great explanation anyways!
You can definitely calculate power for a mann-whitney U test. www.benchmarksixsigma.com/calculators/sample-size-calculator-for-mann-whitney-test/
Gpower is very good software for sample size calculation.
Thanks for the video! One quick question: before we calculate the effect size and sample size, how do we get estimated µ and s for green and purple group in the first place?
I answer this question at 13:57
@@statquest BAM!
@@stonewatertx :)
Thank you for the excellent video. If you please, I would like to ask you a simple question. In order to calculate the effect size, you need to know S which depends on the sample size (in the video it is 10). Then you use this to estimate the "sample size" which is 9. What happens if they are so much different? Can this happen in 2 cases 1. the start sample size >> the calculated one and vice versa.
In theory, 's' (the standard deviation) does not actually depend on the sample size. We can get it from any source or just guess based on previous experience. Thus, if the output sample size is very different, that's OK.
@@statquest Thank you very much for your answer.
What a way to explain the things😍
Thank you! :)
Awesome video! There's really nothing like being able to visualise a concept. Is there a reason you never mentioned the standard error when you were explaining how larger sample sizes increase the confidence that your sample mean is close to the population mean? I mean standard error is literally the measure of that confidence right?
Yes, the standard error is the measure of that confidence. However, I omitted it because I was simply trying to convey the concept of power, rather than dive into the technical terminology.
What if I am trying to compare three groups? How would I calculate the effect size? What other key words should I google in addition to "statistics power calculator"? Thanks in advance for answer my questions, your videos are awesome!
Add "ANOVA" to your search (which compares 3 or more groups). In general, if you have a specific test in mind, just add that to your search.
I discovered these videos a while ago and they're really great for understanding various statistical concepts. I need to do some power analyses on non-parametric data, is that even a thing? Is there even a variability estimate for a Mann-Whitney two sample test? Thanks.
Power estimates for non-parametric models exist. Just google "mann whitney u test power calculation"
Hi Josh, Thoroughly enjoyed the video. One quick question though: if the two means are d1 and d2, then how do you to take the difference between them, should it be |d1-d2| or one can used either d1-d2 or d2-d1. On a side note my kids have started complaining - They say that I am always watching stat quest :)
The order doesn't matter (and we don't use the absolute value either). If you want to learn a very cool way to compare two means, check out my playlist on linear regression. Believe it or not, a test to compare means is a type of linear regression: th-cam.com/play/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU.html (Tell your kids that I apologize for making your watch more StatQuest! :)
You are a Genius! Super
Wow, thanks!
Thanks for the video :) and I got 2 questions:
1. If I cannot find any published data for the effect size, what should be the sample size to generate a preliminary data?
2. is the threshold of significance (alpha) = p value?
1) Even if there is no published data, there is probably a standard sample size used within your field. Or just take a guess.
2) Yes
Really enjoying these videos - I was recommend by a colleague to check out your channel. One question I wanted to ask you - Is the idea that we would collect 10 samples several times, and then get the mean of the estimated means and compare that to the mean of the estimated means from the other distribution using a t-test? Just curious how it all comes together but maybe that comes in some later videos...thank you
The idea is to collect 10 measurements 1 time and then do your tests. However, this video illustrates what would happen if we did it a bunch of times to give you a sense of how much variability there would be in different sample sizes. The larger the sample size, the lower the variability in the mean, and thus, the more confidence we can have in its value.
14:43 damn just the part I needed and you guys decide to skip it.
The fact is, every single test has a different way to calculate power.
Hi!! this is such a great video thank you :) Is it the same if one wants to perform the power analysis for a repeated (test-retest) measure?
I don't know that off the top of my head, but presumably it is...(but I'm not sure).