The p-value is the likelihood that the result was due to random chance. A p-value of 5% is considered statistically significant, but there's still a 5% chance it was only due to random chance and not a real effect. If you do 40 studies on whether mobile phone masts cause cancer you expect to find one statistically significant study saying they do, one statistically significant study saying they reduce cancer risk and 38 studies saying they don't make a statistically significant difference. If you only publish the one study out of 40 saying they increase cancer risk you've now hacked the p-value by not publishing the 39 studies that didn't show the result you wanted. Taken together those 40 studies would have a p-value closer to 50%, but you only looked at the one with the p-value you wanted. If you also publish the study saying masts statistically significantly reduce cancer risk, you now have two conflicting outliers as your only results and people will be tempted to come up with some other factor to explain why one group had positive and another had negative health effects.
When you're doing an experiment, there are chances that the results don't represent the reality by pure chance. For example, if you wanted to know what is the male to female ratio in the population, you could go somewhere and count the number of male and female. In theory, that ratio should be close to 1 : 1. However, supposing you chose the perfect place, there is still a chance that the population is, say, majoritarily women for no specific reason other than luck. Let's say you wanted to prove that there were twice as many men as women. You could go to a ton of places until one of them gives you the ratio you expect, and only report that result. That's the idea behind p-hacking. In reality, to be efficient, you could do the experiment by gathering lots of different datapoints for a specific experiment. If one of these datapoints gives a significant unexpected result, only report that one.
@@EnteiFire4 I did kinda understand the entire p-hacking thing but I want to ask how to avoid or reduce it. Like if we collect data from different points do we just report the unusual data or report all the data.
2:00 Not quite: The P value is the probability you'd have gotten a result _at least as extreme_ as you did, _just due to chance,_ assuming the null hypothesis is true.
I think I can provide a fairly simple explanation to convey what this video tried to do. Imagine you are trying to see if eating snow cones causes mind-reading. Your partner thinks of a number from 1-5, and you eat a different colored snowcone each experiment. Just based on probabilities you will guess it 1 out of 5 times on average, so, you eat 5 different colored snow cones before you guess the number right. This is the data: Orange snowcone - No Red snowcone - No Green snowcone - No Blue snowcone - No Yellow snowcone - Yes Therefore, you publish the last data result showing that eating yellow snowcones leads to mind-reading. Of course, this is a simplification but it conveys the general idea that by increasing the data collected and ignoring certain results, you can prove something just by sheer probability. Had we repeated the yellow snow cone experiment 5 times, we would have seen it was luck but by neglecting the other data we were able to make a false conclusion.
@@sammarrasheed3000 Because they're eirher getting paid by a company or corporation to do so and come out with specific results that'll favor them, or because they have a personal bias that'll benefit themselves and/or whatever their cause might be.
Honestly I think unlike previous ted videos, this one is throwing some random words without detailed explanation (like null hypothesis, appeared here and there, and explained partially each times)
@@ericw2391 honestly,this was a total waste of time...99% people didn't learn a single thing and that is the whole point of watching ted ed video to learn something new....so they should be careful about topics which we can understand..
This kind of method which "proves" almost anything is widely used by corporates especially in consumer products to justify their "product's" credibility from research.
I agree, but to avoid this issue, researchers who "pre-register" a plan for the experiment cannot manipulate the data and analysis. As a result, " This Method" would defend the corporates' results with validity and reliability concerning their "products."
This was honestly not a very good explanation. Even as someone who somewhat (!) gets the concept of p-hacking I was pretty confused throughout most of the video and couldn't follow the logic. For example I've watched the part at 1:25 and after several times to try to understand what they're trying to say and I think it's that if she sorts the cups correctly and the p-value is low then there is a good reason to believe that she was actually able to taste the difference. But something so simple is worded so poorly that I had to infer that from my own knowledge and reasoning and I feel like if this was the first video someone watched about this topic it wouldn't be very helpful. Or maybe I'm just exceptionally slow, that's possible as well...
Yes exactly. The basic idea is we want to see how likely something happened purely by chance versus because of some actual cause. Say your friend claimed they have a supernatural ability to predict a coin flip. After you skeptically tested them, you find that they got around 50 out of 100 correct. That's not really impressive enough for you to reject your belief that they have no special ability: since we expect anybody to get around 50% just by pure chance. If however you found that they got 99 out of 100 correct, you have more evidence to assume they might have some special ability since this is way better than pure chance. It would be a highly unlikely event to happen under the assumption that they have no ability.
There is a slightly more subtle way in which this happens. Even when an experiment is planned in advance there is likely to be a vested interest in proving the hypothesis. So studies that do not do this are not finally published (studies that prove hypotheses are more cited for example) so when one looks for the number of studies that support a hypothesis (rather than the quality) it is easy to conclude that a certain hypothesis is true just because there are more documented cases.
Video recap: P-value cannot tell us if the reverse of null hypothesis is true when the null hypothesis is rejected. Instead, using p-value can only point in the right direction that one possibility is ruled out because of how (i.e., the terms) the null hypothesis is stated. Meaning, if the p-value rejected the null hypothesis, it just puts a “not” (i.e., alternative hypothesis) in front of the null hypothesis thus the reverse is still technically not true based on the phrasing of the null hypothesis. Thus, this result would only narrow down towards the reverse possibly being true. So, the pair of hypotheses statements (null and alternative) would need to be rephrased to prove if the reverse is true. Even when true, the tests need to be repeated for repeated “reverse-proven” results. Moreover, the irony here increased testing means increasing the chance of false negatives. So, there is no perfect p-value but still use the p-value to narrow down results.
I learned about this in stats! I'm so surprised that I can kinda understand this because the terms "null hypothesis" and "p-value" and the bell chart are familiar to me! :D
I feel like everyone is high watching this except for me: the stories are so clever, clear to understand and clean-cut animation, but videos like this blow my mind. I’ll never stop watching!
Fascinating stuff! This is analogous to optimization bias in machine learning, or even the idea of a standard of proof (e.g. "beyond a reasonable doubt") in a criminal trial.
Now I am remembering the days when I used to understand every video of Ted-Ed at once and it used to be the best explanation on the entire internet. :(
Really, really liked this video. I could tell you only went into a little detail on what is basically a piece of ubiquitious mathematical chicanery, but since I love that sort of stuff, thanks!
I did a statistic course, this is one of the first things they tell you to do otherwise you can just BS the p-value to fit or change other variables to prove your desired hypothesis.
My questions: 1. 3:42 Why father’s age can be used for “control for variation in baseline age across participants”? 2. 3:52 What’s the significance of “they also paused their experiment after every ten participants”? Why did they do that? What impact could it make? 3. 3:55 Why would they “continue when the p value was above .05 but stopped when it dipped below .05”? I thought we want the p value to be below .05. So shouldn’t they have done the opposite?
Let's say the data we are lookin at is how many candies a bunch of kids have. That data is: 2, 2, 3, 4, 10 The mean is the average of that data. (2 + 2 + 3 + 4 +10) / 5 = 4.2 candies The mode is the more frequent data value. The mode here is 2, because two kids have 2 candies. The median is the value which splits the data 50-50. Two kids have less than 3 candies, two kids have more than 3 candies, one has exactly 3 candies. The median is 3. I hope that clarifies things for you
This is clearly an issue that is more prevalent in certain fields, whereas in others that's just very basic knowledge. That's why clinical trials publish their statistical analysis plan ahead of time, and why you have to limit primary and secondary endpoints. As for early discovery/biology work, that's why you do multiple testing correction.
@@julieugolini4195 from what I have seen most people are equally smart, but what sets people apart is their interest in a particular subject or field, which is what determines how knowledgeable they are. Now regarding the video if you had to study statistics as a main subject or as a part of your undergraduate course in college then you would easily understand what the narrator was saying throughout the entire video.
@@gohansesshomaru8400 seems I'm awarding you a medal also 🥇💐 you sound really smart,, I admire people that are either born smart or work hard at education to get smart I unfortunately did not get to University (your college) due to some head Trauma but I have had to think about your reply and you have just cleared something in my thinking that my Neurosurgeon has been trying g to explain to me for years Thankyou so much Yes you are correct, I'm smart about life because I'm 63yrs old so have probably more wisdom than what I call "smart" I have struggled with some memory loss last couple of years and beating myself up when I get confused. So once again, thank you your reply has done a good thing Cheers Julie 🇦🇺😍
@@julieugolini4195 please don't be hard on yourself life is what it is, never meant to be perfect (as a 25 year old, my intention is not to lecture someone more than twice my age on life but that my words bring atleast some satisfaction to you). So glad to hear that my first comment meant something to you, I'm just a guy who wants to know more about this world as I slowly grow older. Lots of love from a small coral island in the Arabian sea(Minicoy if anyone cares), sincerely hope that your life goes better than you ever expected. XOXO, Shamsheer
That's called Family Wise error. When you keep doing T-Tests, because each time you do one, the chance of getting a type 1 error. That's where ANOVA'S come in. Analysis of Variance. So you can test Three or more Variables to find where the difference lies, as opposed to where the difference DOESN'T lie. Simply put, T-Tests are for 2 variables, ANOVA's are for Three or more 😁
Actually, it’s just where Bonferroni corrections come in, which account for the family-wise rate. If you have two variables, but one has multiple levels, you could still be doing t-tests to look at differences between those levels.
Publishing a research plan beforehand will not prevent the researchers from doing more experiments than planned and only releasing the favourable ones. However, publishing a research plan has a great advantage that even researches with negative results will get published, because it's another unrelated, but a very significant problem: currently publishing a negative research result is deemed to not be a significant contribution to science.
Omg from so many days, I was searching for a simple explanation for P value and here it is! TED-Ed is a saviour! Thank you for making this video and helping me to understand the concept in an easier manner.
I learned new stuff from this too, don't know what's up with the people who said that they find it confusing and hard to understand. I mean, they are talking simple/basic explainations for even the ones who haven't heard of this before could take grasp of it.
I think it would be interesting to discuss how while there is an easy solution to make experiments more reliable in design, that solution doesn’t work in reality because researchers are ENCOURAGED to P-hack based on the incentives their job provides. When negative results see much less publication, and a researchers job depends in part on getting papers published, they sometimes have to game the system to get a false positive result just so that they can keep their livelihood.
There's not really anything wrong with using the presence of low p-values to assert the degree of relatedness of studied variables; however, p-hacking has become something like 'lying through omission' in several fields, especially psychology. Selective blindness, confusing correlation with causation, and sometimes, even the deliberate obfuscation of facts has led to the Replication Crisis in psychology.
Maybe someone can help me. If the null hypothesis is that she can’t tell the difference. And if the chance of her randomly getting it right is .014. And that is what she gets, then wouldn’t that mean the null hypothesis holds? In that, she can’t tell the difference which is the null hypothesis. This video seems to be saying that just because the p-value is less than the constant .05 then she can tell the difference. But I feel like this isn’t how math works. Like all you have to do is increase the sample size and all of a sudden you have a p value smaller than the arbitrary constant .05. Simplified to: Suppose she can’t tell the difference, hence the null hypothesis. Null hypothesis Suppose she can’t tell the difference => p-value=.014 =>p-value < .05 => She can tell the difference Null hypothesis fails
In VERY simple words, it's like this: Null hypotheses usually state the results you intuitively expect and they don't go alone. They come with an alternative hypothesis, which can be the opposite result. So in our case: Null: She CAN'T tell the difference Alternative: She CAN tell the difference When you state the "more likely result", you pretty much give a 95% chance for it to happen, and the rest 5% goes to the alternative (hence the 0.05 limit). So, when she gets a p-value of 0.014 what that means is basically "she got it right even though I expected her not to". Therefore, there is an indication that this is result is probably not by chance and so you reject the null hypothesis. Hint: Be careful of the wording, you never "accept" a hypothesis (either null or alternative). You reject one and you're left with the other. It doesn't sound really concrete, I know, but that's how it basically works. Btw, by increasing the sample size, you don't necessarily get smaller p-values, rather with each increase you get a p-value closer to the true one. Whether this is above or below 0.05
The smaller a p-value, the less likely a concrete correlation between measured values, or the validity of a positive assertion is. Larger p-values mean that the null hypothesis is more likely to be correct. The 0.05 p-value measure is just a statistical standard, a broad convention followed across researches to assert the significance of a finding.
This explanation is plain wrong or they are trying to prove something by explaining it wrongly just like p-hacking. The video says 1:11 "if she cant distinguish the teas she'll still get the right answer 1 in 70 times by chance." *No* she wont get the right answer because, the chance of that hapening is 1 in 70 or 1.4% which is actually less than impossible(impossible is capped at 5% thus 0.05). But if she can distinguish the teas and she got it in one try then, like I said, it would be so impossible in fact it is less than impossible 0.014 < 0.05, that it would be a miracle it happened and we would be forced to accept the hypothesis the she has the miraclous power of predicting tea mixture and thus reject the null "she cant predict". Note that when I say impossible, I mean less than or equal to 0.05, that is the standard and that is how a hypothesis is accepted. p < .05 = null rejected, hypothesis accepted p > .05 = null cannot be rejected, but does not mean null is true
Thank you so much for throwing light on this with such entertaining visuals..loved every aspect of it...Ted-ed is one of the consistently best sources of learning for me for a long time & looking forward to even more..thank u folks 💓
You should have mentioned power. In the analysis one can make two types of errors - I and II. p-values are kind of a check for type I errors and power is a check for type II errors. But the power of a test usually is considerably harder to evaluate, so people just don't bother. In research reproducibility is one of the main conditions. A result must be reproducible. No pre-planning would prevent people from reporting only what they want to report. In the example from the video they could fake data of the 3d experiment using other two as a template.
1:532:42 - She's suspicious, she is. I hope I don't have nightmares because of her! As for the P-value, I was thinking 'Why not make some strong rules to abide by?' And yep, at 4:24 the pre-registration/rules had been considered to ensure the experiments didn't continue until the desired result was achieved.
Hello guys.. One suggestion Pls make different channels for different langaugaes... And post the same animation with that langauge.... It would help you to reach out more wide audience and help me in understanding too😊 Arabic - hindi/urdu - Brazilian - Spanish These 5 are the most spoken
it isnt, different studies have different level of significance, p-value is a way to show readers or other researchers to let them decide if the value is significant enough to reject the null hypothesis.
The fact that its lower than 0.05 means its significant, if its higher then it doesnt mean its insignificant, just that that we cannot reject the opposite hypothesis.
To anyone who is confused, I've tried to explain the content of the video a bit: The point of the video was that a lot of researchers keep trying different methods until they find something big, then they only report the significant results. This method has a huge problem: if you test something a lot, you get a lot of false positives. We call this method P-hacking (we'll get to why later). In statistics, the P value should be the chance that the whole research is wrong. As an example, a P value of 0.05 means there is a 5% chance that the research is wrong (and 95% chance it's correct). However, if the researcher uses the previously mentioned method "P-Hacking", then there is a much higher chance of a false positive. This means that the P value (a.k.a. probability of messing up) is much higher than reported, and nobody can prove otherwise. This is a big problem, because a lot of researches have done this without even realizing that they were making a mistake. If anyone is still confused, feel free to discuss it in the replies.
Torture the data long enough until it confesses! Thought provoking video! Music is really very interesting, somehow it made understanding the P concept easier.
Ted-Ed is such an amazing channel! They offer all these amazing and interesting educational videos that they work so hard on for free! I find this so much more interesting than school! Keep up the good work!
This shows that the traditional frequentist way of "rejecting or accepting hypothesis" is flawed. Instead we need Bayesian analysis and always present the result as a direct probability of the statement rather than using P-value. By doing this, we don't need "pre-registration" as the prior assumptions used in Bayesian analysis can be checked later.
I am a researcher, another methods to avoid issues related to p-values is to Abingdon them completely. Many researchers have gotten together and suggest that we, researcher, should move away from significance testing and rely on sound methodology, appropriate statistical testing and replication. This approach is called the “retiring statistical significance” approach,
The common p value of 0.05 would mean that you can reject your null hypothesis with a confidence of 95%. So you're giving a 5% chance to be wrong about your idea. That's about 1 BILLION times more than your little funny point here states.
Impressive video, TED Ed. Seriously, one of my college professors use your videos as additional study materials, in addition to some TED talks related to our topic. Thank you very much. Edit: grammar correction
In a sense, this is p-hacking: showing only one algorithm that produces a significant result (multiply by 0) and ignore everything else (like multiplying by any other value)
Unfortunately a lot of the people who do the peer review are unpaid and/or don't get any credit for it. The larger journals can do pretty good peer reviews for the papers they publish but they cost a ton of money to get published in; whereas the smaller or online journals can't do as much peer reviewing so their papers aren't taken as seriously. It is something that needs to be addressed in my opinion.
if all tea-first cups are identical, and all milk-first cups are identical, then there are only two options for each in the group. Since the second group is the complement of the first, there are only 2^4=16 different ways to sort the cups into two groups of four if the order doesn't matter. Unless I'm wrong, the combination formula "8 choose 4" = C(8,4) = 8!/4!/4! = 70 would only apply if there were 8 *different* cups to put in a group. The article on Wikipedia explains how it was calculated, but if tea-first are 1,2,3,4 and milk-first are 5,6,7,8... the combination formula assumes that 1256 in one group is counted separately from 1358, and thus double-counts many different combinations. It's not relevant for this video, but is this common example not actually 1%?
The tea will mix differently depending on which they put in first. If she didn't swish it around in her mouth like in the illustration she could taste the difference. If the milk is in first it will be more blended an even taste of creamy and bitter if put in after it will mix more slowly making it taste either more bitter or more creamy depending on how its drunk. Typically the milk would settle at the bottom but either way the taste will taste less even as would the temperatures
Loving the animation, but I'm a little sad that you didn't include the history behind this. For example, the tea animation was just set with some random character that looks rather male (but hey if it's a girl no problem), but you're referring them as "she" and "her", this lady was actually some mistress that claimed she didn't like her tea poured first, and rather the milk. Because it just tastes better, so they made a test to see if she could get it correct. She got every answer correct! By this to be random chance, I believe it was around a 1 in 70,000 chance. This chance they calculated was the first "P" value calculated to test the probability. And another instance of P hacking, was when a guy went to the market for a dead, gutted salmon. They put it in the MRI, they did a series of tests and they showed that the P value was proving that the salmon was alive and showing brain activity, showing that the P value can be manipulated to prove impossible results.
its really simple - its basically saying that as flawed human beings we may BS the results or search for an analysis that yields a p-value less than 0.05 in order to report a statistically significant finding. It may be a result of poor test structure, poor resources, poor hypothesis (i.e., not testable which in turn is not well defined), confirmation bias, ethical concerns, inexperience, poor training, poor oversight or review, lack of interest by community to confirm tests results from an independent peer in the community - all these could contribute to p-hacking. P-hacking is the intentional search for an experiment that produces a p-value less than 0.05 and report the result without focus on the scientific method/experimentation.
I'm gonna be replaying this lots before I understand it
The p-value is the likelihood that the result was due to random chance.
A p-value of 5% is considered statistically significant, but there's still a 5% chance it was only due to random chance and not a real effect.
If you do 40 studies on whether mobile phone masts cause cancer you expect to find one statistically significant study saying they do, one statistically significant study saying they reduce cancer risk and 38 studies saying they don't make a statistically significant difference. If you only publish the one study out of 40 saying they increase cancer risk you've now hacked the p-value by not publishing the 39 studies that didn't show the result you wanted. Taken together those 40 studies would have a p-value closer to 50%, but you only looked at the one with the p-value you wanted. If you also publish the study saying masts statistically significantly reduce cancer risk, you now have two conflicting outliers as your only results and people will be tempted to come up with some other factor to explain why one group had positive and another had negative health effects.
Same 😅
@@soylentgreenb thanks for this brief summary 👀🙌🏻
@@soylentgreenb thank you
@@soylentgreenb I'm gonna be reading this lots before I understand it
The P value animation is adorable
Also - engagement.
hey, great picture amigo =)
You mean it's *P* erfect? Haha
He's just vibing honestly
Only animation?! )))
There is a .014 chance that I just understood that video
Lol. Good one! XD
Well, in simple terms it's cherrypicking.
When you're doing an experiment, there are chances that the results don't represent the reality by pure chance. For example, if you wanted to know what is the male to female ratio in the population, you could go somewhere and count the number of male and female. In theory, that ratio should be close to 1 : 1. However, supposing you chose the perfect place, there is still a chance that the population is, say, majoritarily women for no specific reason other than luck.
Let's say you wanted to prove that there were twice as many men as women. You could go to a ton of places until one of them gives you the ratio you expect, and only report that result. That's the idea behind p-hacking.
In reality, to be efficient, you could do the experiment by gathering lots of different datapoints for a specific experiment. If one of these datapoints gives a significant unexpected result, only report that one.
@Richard Long so ... did you? or did you not?
@@EnteiFire4 I did kinda understand the entire p-hacking thing but I want to ask how to avoid or reduce it. Like if we collect data from different points do we just report the unusual data or report all the data.
The method that can "prove" almost everything: _H A C K I N G_
haha I like your name, is it a reference to something? =)
@@ScumfuckMcDoucheface Thanks! Yep, it's a line from an old comedy called "Bringing up Baby". It's a nice watch :D
not to be associated with hacking, the method that can IMprove almost anyone
Phacking
Ah yes, the truly elite and sophisticated scientific method; Phacking
2:00 Not quite: The P value is the probability you'd have gotten a result _at least as extreme_ as you did, _just due to chance,_ assuming the null hypothesis is true.
That still makes no sense
I think I can provide a fairly simple explanation to convey what this video tried to do.
Imagine you are trying to see if eating snow cones causes mind-reading. Your partner thinks of a number from 1-5, and you eat a different colored snowcone each experiment. Just based on probabilities you will guess it 1 out of 5 times on average, so, you eat 5 different colored snow cones before you guess the number right.
This is the data:
Orange snowcone - No
Red snowcone - No
Green snowcone - No
Blue snowcone - No
Yellow snowcone - Yes
Therefore, you publish the last data result showing that eating yellow snowcones leads to mind-reading. Of course, this is a simplification but it conveys the general idea that by increasing the data collected and ignoring certain results, you can prove something just by sheer probability. Had we repeated the yellow snow cone experiment 5 times, we would have seen it was luck but by neglecting the other data we were able to make a false conclusion.
So then why would researchers use it? (I’m disagreeing with you, just asking)
@@sammarrasheed3000 To reduce computation costs and time I believe?
Basically they're trying to prove what they believe through statistical information which is an hypothesis. right?
Don't eat yellow snow, mate...
@@sammarrasheed3000 Because they're eirher getting paid by a company or corporation to do so and come out with specific results that'll favor them, or because they have a personal bias that'll benefit themselves and/or whatever their cause might be.
"Science doing science on itself..."
Yo dawg, we heard you liked SCIENCE!
Excited Jesse Pinkman emerges.
Let's call it "scientology", oh wait.
"Metascience"?
Good ol' meme, haven't heard that in a long time.
I remember learning this in statistics class! Nice to see a video on this topic. Thank you Ted-Ed!
I learned Statistics for 2 semesters and this is still confusing.
Yep
Honestly I think unlike previous ted videos, this one is throwing some random words without detailed explanation (like null hypothesis, appeared here and there, and explained partially each times)
@@ericw2391 honestly,this was a total waste of time...99% people didn't learn a single thing and that is the whole point of watching ted ed video to learn something new....so they should be careful about topics which we can understand..
@@gohilravirajsinh5212 lol pre-requisite to this video: AP stat or higher education
phack statistics
This kind of method which "proves" almost anything is widely used by corporates especially in consumer products to justify their "product's" credibility from research.
I agree, but to avoid this issue, researchers who "pre-register" a plan for the experiment cannot manipulate the data and analysis. As a result, " This Method" would defend the corporates' results with validity and reliability concerning their "products."
Do you think this might apply to vaccination studies?
"four out of five experts surveyed recommend our product."
@@wliaputs it applies to antivaxxers.
@@wliaputs Definitely not.
They preregister their studies, and real-world performance matches the predicted performance.
I’ve been a ted Ed fan for years and this artist is definitely one of my faves. Thanks for all you guys do!
This was honestly not a very good explanation. Even as someone who somewhat (!) gets the concept of p-hacking I was pretty confused throughout most of the video and couldn't follow the logic.
For example I've watched the part at 1:25 and after several times to try to understand what they're trying to say and I think it's that if she sorts the cups correctly and the p-value is low then there is a good reason to believe that she was actually able to taste the difference. But something so simple is worded so poorly that I had to infer that from my own knowledge and reasoning and I feel like if this was the first video someone watched about this topic it wouldn't be very helpful.
Or maybe I'm just exceptionally slow, that's possible as well...
The p-value explanation is lacking, but the p-hacking explanation is ok IMO.
I second this! First TED-Ed video that failed to explain to me what they were trying to explain to me.
To be fair to them they are trying to explain a concept that many scientists misunderstand in a 5 min video.
Yes exactly. The basic idea is we want to see how likely something happened purely by chance versus because of some actual cause. Say your friend claimed they have a supernatural ability to predict a coin flip. After you skeptically tested them, you find that they got around 50 out of 100 correct. That's not really impressive enough for you to reject your belief that they have no special ability: since we expect anybody to get around 50% just by pure chance. If however you found that they got 99 out of 100 correct, you have more evidence to assume they might have some special ability since this is way better than pure chance. It would be a highly unlikely event to happen under the assumption that they have no ability.
I didn't get anything tbh
There is a slightly more subtle way in which this happens. Even when an experiment is planned in advance there is likely to be a vested interest in proving the hypothesis. So studies that do not do this are not finally published (studies that prove hypotheses are more cited for example) so when one looks for the number of studies that support a hypothesis (rather than the quality) it is easy to conclude that a certain hypothesis is true just because there are more documented cases.
Video recap: P-value cannot tell us if the reverse of null hypothesis is true when the null hypothesis is rejected. Instead, using p-value can only point in the right direction that one possibility is ruled out because of how (i.e., the terms) the null hypothesis is stated. Meaning, if the p-value rejected the null hypothesis, it just puts a “not” (i.e., alternative hypothesis) in front of the null hypothesis thus the reverse is still technically not true based on the phrasing of the null hypothesis. Thus, this result would only narrow down towards the reverse possibly being true. So, the pair of hypotheses statements (null and alternative) would need to be rephrased to prove if the reverse is true. Even when true, the tests need to be repeated for repeated “reverse-proven” results. Moreover, the irony here increased testing means increasing the chance of false negatives. So, there is no perfect p-value but still use the p-value to narrow down results.
That was phacking educational.
Torture the data till it confesses.
-what I learned
Hi, Ted Ed
Michael here
Selin here
Harry here
Mix here!
Chris here
I learned about this in stats! I'm so surprised that I can kinda understand this because the terms "null hypothesis" and "p-value" and the bell chart are familiar to me! :D
"Sometimes small true true different from the big true true"- jerry smith.
That's big true
Correction ~ Cloud Atlas Jerry😉
I struggled to understand this until now🤣
I feel like everyone is high watching this except for me: the stories are so clever, clear to understand and clean-cut animation, but videos like this blow my mind. I’ll never stop watching!
You don't have to be 'high' to be smart - it just takes a lot of time and effort. 🙂
The fact that P keeps tumbling into the experiment and ruining it is hilarious.
Fascinating stuff! This is analogous to optimization bias in machine learning, or even the idea of a standard of proof (e.g. "beyond a reasonable doubt") in a criminal trial.
Now I am remembering the days when I used to understand every video of Ted-Ed at once and it used to be the best explanation on the entire internet. :(
its not you, the explanation was terribly bad
Yes, it's true..m
This video is going straight to my statistics study playlist on TH-cam!
Really, really liked this video. I could tell you only went into a little detail on what is basically a piece of ubiquitious mathematical chicanery, but since I love that sort of stuff, thanks!
This video is extremely convoluted
I did a statistic course, this is one of the first things they tell you to do otherwise you can just BS the p-value to fit or change other variables to prove your desired hypothesis.
My questions:
1. 3:42 Why father’s age can be used for “control for variation in baseline age across participants”?
2. 3:52 What’s the significance of “they also paused their experiment after every ten participants”? Why did they do that? What impact could it make?
3. 3:55 Why would they “continue when the p value was above .05 but stopped when it dipped below .05”? I thought we want the p value to be below .05. So shouldn’t they have done the opposite?
I don't understand statistics like mean, mode and median.
Is that normal?
well if you understand concept of "normal" then you already understand mean/average
Let's say the data we are lookin at is how many candies a bunch of kids have. That data is: 2, 2, 3, 4, 10
The mean is the average of that data. (2 + 2 + 3 + 4 +10) / 5 = 4.2 candies
The mode is the more frequent data value. The mode here is 2, because two kids have 2 candies.
The median is the value which splits the data 50-50. Two kids have less than 3 candies, two kids have more than 3 candies, one has exactly 3 candies. The median is 3.
I hope that clarifies things for you
@@SKyrim190 Legend 🙏
@@SKyrim190 incredible good example
it's taught in the 6th or 7th grade, so I guess it depends on your age...
Very good explanation of P. Well done.
Very good subject. Relevant question at a time when science is the product of tasks with predefined goals and not of thoughts and ideas.
This is clearly an issue that is more prevalent in certain fields, whereas in others that's just very basic knowledge. That's why clinical trials publish their statistical analysis plan ahead of time, and why you have to limit primary and secondary endpoints. As for early discovery/biology work, that's why you do multiple testing correction.
I feel so smart! I knew exactly what they were talking about throughout the entire video! Someone hand me an award! 😂
I'm not so smart.....I only understood the first half then slowly got more confuuuusseedd 🤯🤯😭🤪
I'm awarding you a Gold Medal 🥇💐
@@julieugolini4195 from what I have seen most people are equally smart, but what sets people apart is their interest in a particular subject or field, which is what determines how knowledgeable they are. Now regarding the video if you had to study statistics as a main subject or as a part of your undergraduate course in college then you would easily understand what the narrator was saying throughout the entire video.
@@gohansesshomaru8400 seems I'm awarding you a medal also 🥇💐 you sound really smart,, I admire people that are either born smart or work hard at education to get smart
I unfortunately did not get to University (your college) due to some head Trauma but I have had to think about your reply and you have just cleared something in my thinking that my Neurosurgeon has been trying g to explain to me for years
Thankyou so much
Yes you are correct, I'm smart about life because I'm 63yrs old so have probably more wisdom than what I call "smart"
I have struggled with some memory loss last couple of years and beating myself up when I get confused.
So once again, thank you your reply has done a good thing
Cheers Julie 🇦🇺😍
@@julieugolini4195 please don't be hard on yourself life is what it is, never meant to be perfect (as a 25 year old, my intention is not to lecture someone more than twice my age on life but that my words bring atleast some satisfaction to you). So glad to hear that my first comment meant something to you, I'm just a guy who wants to know more about this world as I slowly grow older. Lots of love from a small coral island in the Arabian sea(Minicoy if anyone cares), sincerely hope that your life goes better than you ever expected.
XOXO, Shamsheer
🏅🏆
I'm so amazed and grateful that there is a TED-Ed video about the p-value. I think I understand it a liiittle better now ;)
That's called Family Wise error. When you keep doing T-Tests, because each time you do one, the chance of getting a type 1 error. That's where ANOVA'S come in. Analysis of Variance. So you can test Three or more Variables to find where the difference lies, as opposed to where the difference DOESN'T lie. Simply put, T-Tests are for 2 variables, ANOVA's are for Three or more 😁
Actually, it’s just where Bonferroni corrections come in, which account for the family-wise rate. If you have two variables, but one has multiple levels, you could still be doing t-tests to look at differences between those levels.
I didn't understood a thing!
FyreStryder and use Benjimini Hochberg for when you have thousands of tests.
Actually, Romello-Eisingen is a better one, which allows for the Heinrich rate. And finds the Schneider value of the Hoffman-Bianchi levels.
Publishing a research plan beforehand will not prevent the researchers from doing more experiments than planned and only releasing the favourable ones. However, publishing a research plan has a great advantage that even researches with negative results will get published, because it's another unrelated, but a very significant problem: currently publishing a negative research result is deemed to not be a significant contribution to science.
Ted-Ed can you do a History On Trial of Oliver Cromwell.
Oooooo, great idea.
Thank you for helping me be more intuitive thanks
Knowledge is power I am power I am energy we are 1
I just offered my family milk tea with breakfast. I told them the P value was only 0.015, but nobody wanted to drink it.
hahaha yeah I pee in my family's tea too!! hahaha wait, what?
The animation style is amazing.
Omg from so many days, I was searching for a simple explanation for P value and here it is! TED-Ed is a saviour! Thank you for making this video and helping me to understand the concept in an easier manner.
I learned new stuff from this too, don't know what's up with the people who said that they find it confusing
and hard to understand. I mean, they are talking simple/basic explainations for even the ones who haven't heard of this before could take grasp of it.
This is one of my favorite animations
No one:
Me reading the title: *PHACKING*
(F)acking
I was looking for thus comment
I remember learning the p value and the null and alternate hypothesis in high school but now that I truly understand it's use
I think it would be interesting to discuss how while there is an easy solution to make experiments more reliable in design, that solution doesn’t work in reality because researchers are ENCOURAGED to P-hack based on the incentives their job provides. When negative results see much less publication, and a researchers job depends in part on getting papers published, they sometimes have to game the system to get a false positive result just so that they can keep their livelihood.
Pre-registering and all would be great. What would be even greater is a link to the study.
Everyone's gangsta until the P-value starts waving to the reseracher....
I knew all those years of LSD would catch up with me =) haha yeeehawww!!
There's not really anything wrong with using the presence of low p-values to assert the degree of relatedness of studied variables; however, p-hacking has become something like 'lying through omission' in several fields, especially psychology. Selective blindness, confusing correlation with causation, and sometimes, even the deliberate obfuscation of facts has led to the Replication Crisis in psychology.
Maybe someone can help me. If the null hypothesis is that she can’t tell the difference. And if the chance of her randomly getting it right is .014. And that is what she gets, then wouldn’t that mean the null hypothesis holds? In that, she can’t tell the difference which is the null hypothesis.
This video seems to be saying that just because the p-value is less than the constant .05 then she can tell the difference. But I feel like this isn’t how math works. Like all you have to do is increase the sample size and all of a sudden you have a p value smaller than the arbitrary constant .05.
Simplified to:
Suppose she can’t tell the difference, hence the null hypothesis.
Null hypothesis Suppose she can’t tell the difference
=> p-value=.014
=>p-value < .05
=> She can tell the difference
Null hypothesis fails
It is very wordy and bad explained tbh.
In VERY simple words, it's like this:
Null hypotheses usually state the results you intuitively expect and they don't go alone. They come with an alternative hypothesis, which can be the opposite result. So in our case:
Null: She CAN'T tell the difference
Alternative: She CAN tell the difference
When you state the "more likely result", you pretty much give a 95% chance for it to happen, and the rest 5% goes to the alternative (hence the 0.05 limit). So, when she gets a p-value of 0.014 what that means is basically "she got it right even though I expected her not to". Therefore, there is an indication that this is result is probably not by chance and so you reject the null hypothesis.
Hint: Be careful of the wording, you never "accept" a hypothesis (either null or alternative). You reject one and you're left with the other. It doesn't sound really concrete, I know, but that's how it basically works.
Btw, by increasing the sample size, you don't necessarily get smaller p-values, rather with each increase you get a p-value closer to the true one. Whether this is above or below 0.05
The smaller a p-value, the less likely a concrete correlation between measured values, or the validity of a positive assertion is. Larger p-values mean that the null hypothesis is more likely to be correct. The 0.05 p-value measure is just a statistical standard, a broad convention followed across researches to assert the significance of a finding.
This explanation is plain wrong or they are trying to prove something by explaining it wrongly just like p-hacking.
The video says 1:11 "if she cant distinguish the teas she'll still get the right answer 1 in 70 times by chance." *No* she wont get the right answer because, the chance of that hapening is 1 in 70 or 1.4% which is actually less than impossible(impossible is capped at 5% thus 0.05). But if she can distinguish the teas and she got it in one try then, like I said, it would be so impossible in fact it is less than impossible 0.014 < 0.05, that it would be a miracle it happened and we would be forced to accept the hypothesis the she has the miraclous power of predicting tea mixture and thus
reject the null "she cant predict".
Note that when I say impossible, I mean less than or equal to 0.05, that is the standard and that is how a hypothesis is accepted.
p < .05 = null rejected, hypothesis accepted
p > .05 = null cannot be rejected, but does not mean null is true
The chalk writing was really cool !!
Thank you so much for throwing light on this with such entertaining visuals..loved every aspect of it...Ted-ed is one of the consistently best sources of learning for me for a long time & looking forward to even more..thank u folks 💓
Having said that I feel that a more detailed elaboration might help more especially for folks like me who are not exactly the brightest bulbs..
You should have mentioned power. In the analysis one can make two types of errors - I and II. p-values are kind of a check for type I errors and power is a check for type II errors. But the power of a test usually is considerably harder to evaluate, so people just don't bother. In research reproducibility is one of the main conditions. A result must be reproducible. No pre-planning would prevent people from reporting only what they want to report. In the example from the video they could fake data of the 3d experiment using other two as a template.
Putting milk in before the tea
Englishman: death it is then. Slow and painful or drawn out and agonising?
1:53 2:42 - She's suspicious, she is. I hope I don't have nightmares because of her!
As for the P-value, I was thinking 'Why not make some strong rules to abide by?' And yep, at 4:24 the pre-registration/rules had been considered to ensure the experiments didn't continue until the desired result was achieved.
I am not first, I am not last
but whenever I get your notification I click very fast
Your 18th just to share Somen fax
Some not somen
Gosh this animation is so rad 🤩
Ted Ed: The method that can “ prove ” almost anything
Me: Joe Rogan saying it. Bam, no need for this lesson.
If you take seriously that clown, I just 💀
@@sof1675 Aah Yes, Joe Rogan the clown, who also happens to be successful at almost everything he's tried.
Hello guys..
One suggestion
Pls make different channels for different langaugaes...
And post the same animation with that langauge....
It would help you to reach out more wide audience and help me in understanding too😊
Arabic - hindi/urdu - Brazilian - Spanish
These 5 are the most spoken
I don’t think that’s how the p-value works, the fact that 1/70 is lower than 0.05 is insignificant
it isnt, different studies have different level of significance, p-value is a way to show readers or other researchers to let them decide if the value is significant enough to reject the null hypothesis.
The fact that its lower than 0.05 means its significant, if its higher then it doesnt mean its insignificant, just that that we cannot reject the opposite hypothesis.
TED videos are so good. It's a shame some of them are so left
isn't this case can be categorized as selective bias by those scientists?
Thanks for the video.
To anyone who is confused, I've tried to explain the content of the video a bit:
The point of the video was that a lot of researchers keep trying different methods until they find something big, then they only report the significant results. This method has a huge problem: if you test something a lot, you get a lot of false positives. We call this method P-hacking (we'll get to why later).
In statistics, the P value should be the chance that the whole research is wrong. As an example, a P value of 0.05 means there is a 5% chance that the research is wrong (and 95% chance it's correct). However, if the researcher uses the previously mentioned method "P-Hacking", then there is a much higher chance of a false positive. This means that the P value (a.k.a. probability of messing up) is much higher than reported, and nobody can prove otherwise.
This is a big problem, because a lot of researches have done this without even realizing that they were making a mistake.
If anyone is still confused, feel free to discuss it in the replies.
Torture the data long enough until it confesses! Thought provoking video! Music is really very interesting, somehow it made understanding the P concept easier.
Speedruner:Does 6 hour long speedrun only to cheat in the last few minutes.
Speedrun Moderator: Listen here you little __
FREE MEME
You smell good when you sleep
A video about cronbachs alpha please!
Ted-Ed is such an amazing channel! They offer all these amazing and interesting educational videos that they work so hard on for free! I find this so much more interesting than school! Keep up the good work!
sometime yes :D
This shows that the traditional frequentist way of "rejecting or accepting hypothesis" is flawed.
Instead we need Bayesian analysis and always present the result as a direct probability of the statement rather than using P-value.
By doing this, we don't need "pre-registration" as the prior assumptions used in Bayesian analysis can be checked later.
Ah yes, I remember studying this in college.
I failed the paper twice.
I am a researcher, another methods to avoid issues related to p-values is to Abingdon them completely. Many researchers have gotten together and suggest that we, researcher, should move away from significance testing and rely on sound methodology, appropriate statistical testing and replication. This approach is called the “retiring statistical significance” approach,
Ah yes, now I understand the Dream scandal.
Good job
Test: “You have a .0000000003% chance of winning by guessing.”
Scientists: *“This simply won’t do.”*
🤣
The common p value of 0.05 would mean that you can reject your null hypothesis with a confidence of 95%. So you're giving a 5% chance to be wrong about your idea. That's about 1 BILLION times more than your little funny point here states.
I love how this is in the same style of animation as the Rasputin music video.
Impressive video, TED Ed. Seriously, one of my college professors use your videos as additional study materials, in addition to some TED talks related to our topic. Thank you very much.
Edit: grammar correction
First TED-Ed video that failed to explain to me what they were trying to explain to me.
For me : Not the 1st { commentary #fail } 😕🤦▶️
When I read the tiltle I thought they would simply say multiply both side of an equation by zero and "prove" RHS=LHS lol 😅😅
In a sense, this is p-hacking: showing only one algorithm that produces a significant result (multiply by 0) and ignore everything else (like multiplying by any other value)
🤣😂🤣😂
@@enacrt ya
yes. very straight forward.
This is another reason why we have peer review.
Unfortunately a lot of the people who do the peer review are unpaid and/or don't get any credit for it. The larger journals can do pretty good peer reviews for the papers they publish but they cost a ton of money to get published in; whereas the smaller or online journals can't do as much peer reviewing so their papers aren't taken as seriously. It is something that needs to be addressed in my opinion.
if all tea-first cups are identical, and all milk-first cups are identical, then there are only two options for each in the group. Since the second group is the complement of the first, there are only 2^4=16 different ways to sort the cups into two groups of four if the order doesn't matter.
Unless I'm wrong, the combination formula "8 choose 4" = C(8,4) = 8!/4!/4! = 70 would only apply if there were 8 *different* cups to put in a group.
The article on Wikipedia explains how it was calculated, but if tea-first are 1,2,3,4 and milk-first are 5,6,7,8... the combination formula assumes that 1256 in one group is counted separately from 1358, and thus double-counts many different combinations.
It's not relevant for this video, but is this common example not actually 1%?
The probability of me getting this data is 0 percent
The tea will mix differently depending on which they put in first. If she didn't swish it around in her mouth like in the illustration she could taste the difference. If the milk is in first it will be more blended an even taste of creamy and bitter if put in after it will mix more slowly making it taste either more bitter or more creamy depending on how its drunk. Typically the milk would settle at the bottom but either way the taste will taste less even as would the temperatures
The P value character is for sure a narcissist 😭
The "this video" link at the end needs to be in the description for those of us on Chromecasts as we cannot click on that link.
Loving the animation, but I'm a little sad that you didn't include the history behind this. For example, the tea animation was just set with some random character that looks rather male (but hey if it's a girl no problem), but you're referring them as "she" and "her", this lady was actually some mistress that claimed she didn't like her tea poured first, and rather the milk. Because it just tastes better, so they made a test to see if she could get it correct. She got every answer correct! By this to be random chance, I believe it was around a 1 in 70,000 chance. This chance they calculated was the first "P" value calculated to test the probability.
And another instance of P hacking, was when a guy went to the market for a dead, gutted salmon. They put it in the MRI, they did a series of tests and they showed that the P value was proving that the salmon was alive and showing brain activity, showing that the P value can be manipulated to prove impossible results.
I am almost 100% certain I did your state lecture at Ted Vancouver.
People tell me Im significant, must be p-hackers
i got very excited when i saw p, thinking ted ed was going to make a video on p vs np. but this is interesting too i guess
P-hacking is very useful when your science has an agenda. ;)
This needs to be widespreadly known...
So is this an equivalent to “cherry-picking “ or cooking the books?
Nice Info
Statistics mean simultaneously everything... and nothing.
Yes it means nothing in the future, but for now it determines our daily lives.
no.
I may be confused throughout the vid but its still fascinating
First!
Beat me to it by 10 secs
Beat me to it by 30 seconds.
@@4Nulla you really are!
Congrats for being first!
No matter how many times I hear this, I have to be reminded of it.
Flip a coin 5 times, note down the results. Congratulations, your result had only a 1/32 < 0.05 chance of occuring! Very significant.
This was more helpful than my AP statistics teacher
👍👍👍
its really simple - its basically saying that as flawed human beings we may BS the results or search for an analysis that yields a p-value less than 0.05 in order to report a statistically significant finding. It may be a result of poor test structure, poor resources, poor hypothesis (i.e., not testable which in turn is not well defined), confirmation bias, ethical concerns, inexperience, poor training, poor oversight or review, lack of interest by community to confirm tests results from an independent peer in the community - all these could contribute to p-hacking. P-hacking is the intentional search for an experiment that produces a p-value less than 0.05 and report the result without focus on the scientific method/experimentation.