This crash course is great. The level is fairly high (I believe that most people with a PhD could even learn a thing or two), and it's still comprehensible. Great job!
During my Ba and MA I certainly did feel the strong push to "just try this or that" so the p-values would be a little bit more acceptable. I never really got a straight and clear answer how that wasn't similar to cheating. But everyone seemed to think it was quite okay - That was a weird experience. Gaslighting, that's called right?
If you aren't publishing (and as a student, you probably weren't expected to publish) then p-hacking isn't a concern. If you're learning the "art" of statistics, you need to experiment with the way different analyses work.
Actually, there is a small misconception in this video (the table in the beginning). When we reject the alternative hypothesis that does not mean that the the null hypothesis is true. It simply means that we do not have statistical evidence to reject the null hypothesis - we cannot say with 100 % statistical certainty that Ho is true. Andy Fields writes in 'Discovering Statistics in SPSS': "If the p-value is greater than 0,05 you can decide to reject the alternative hypothesis but that is not the same as the null hypothesis being true" (Fields, 2018: 76).
All of that and you didn't mention the common terms "fishing" and "exploratory analyses", nor how such approaches can be used ethically to generate new hypotheses, or be methodologically accommodated, such as with screening and hold-out samples (cross-validation).
Hi! I really like your videos, I'll eventually go through all of them. Please make a video on multi-variate analysis. I' beginning to understand them but would really like for you to explain them (PCA, CVA)
I just want to say, when you calculate that with twenty tests all looking for p < 0.05, there is a 65% chance that we will get a false positive, you said that this "might be higher than you would expect". My statistically challenged brain sees it the other way, though: if each test has a 5% chance of giving a false positive and we run 20 tests, then we get a 100% chance of a false positive because 20 * 5% = 100%!! (Of course I understand why this is wrong, and you've explained it very well over the course of this series--I just wanted to point out what seemed to me to be the obvious mistake to make!)
You can think of it like "what's the chances that I'll get NO false positives in 20 tests of p = 0.05", which is ( 1 - 0.05 ) ^ 20 = 0.358 or 36%, so on the other hand, 1 - 0.358 = 0.642, or 64.2% chance of getting more than 1 false positive.
I was a bit confused about the Bonferroni correction, because surely, for a total p-value of 0.05 and 5 tests, the p-value per test should be p < 1 - 5th.root(0.95) But apparently, the simple division always leads to a smaller value than this calculation, making it basically close enough, and an even stricter limit. Although it surprises me that you didn't point this out, as that quotient comes kind of out of nowhere.
Yes it's a conservative approximation (that's close enough if you don't have a huge number of tests) that almost everyone uses without knowing that it is always (a little) too conservative (I guess that's why she doesn't mention it). The accurate correction (for independent tests) which you pointed out is called the Sidak correction :)
Yes, she also didn't talk about other solutions ie cross validation. Bringing up only Bonferroni (and also not talking about how conservative it is) and not talking about the issue more fully can be a bit misleading /:
I use the Bonferroni correction quite often. There are more than 20 thousand human genes in our DNA. Sometimes I need to look for genes which **significantly** change under a treatment. Without the Bonferroni correction, I would be doing over 20 thousand different p-values, which of course I would find something that appears "significant" with a p < 0.05.
WIth such a number of tests though maybe you could use another correction. Bonferonni is always a little too conservative, you could use the more acurrate Sidak correction. Or even Holm's or Hochberg's procedure...
how to get p-hacking out of "for profit" research? 1-make only "no profit" & massively perreviewed research significant enough for laws. 2-implement UBI and/or have a "national scientific research fund" that allows for scientists to follow through theier research within ethical conditions for themselfs and their results.
So, in order to believe any statistic we see, and even then the statistic might itself be a chance result, we need Ph.D.'s in statistical analysis, an army of scientist employees, and sophisticated lab and computer equipment to verify claims. Oh, and access to the journals said statistic was published in, each of which will cost a pretty penny. Not just CC, but in general, the media and experts do a fine job of outlining the problem but rarely give solutions and even when they do, the solutions are impractical or theoretical. The same media and experts, due to shenanigans like p-hacking, have broken the public's trust in what they have to offer. My solution: mandatory statistics classes beginning in Kindergarten. This will take forty years for any critical mass of trust to return to the societal influencers because in forty years, today's kindergarteners will have the power to influence and the old guard of prevaricating and unethical doyens will have died or be too old to dictate the direction of society in any meaningful way. Or, wait for the Matrix plugin seats. They're coming!
Michael Crichton was talking about this 20 years ago. He suggested that the government fund a return to basic reseach and that researchers make their data available so that others can replicate their studies. His senate testimony is available online.
Regards Adrian, I am planning to carry out an empirical study with p hacking being the subject at hand, could you please a case study that could be worked upon. Thanks in advance !!!!!!
How to get a tiny p-value every time? It's very easy conceptually. Increase your sample size. In fact, in areas of science where very large sample sizes are the norm, the p-values are almost always less than 0.05. Hmm.....so if with very large studies p-values are kind of pointless....does this mean that p-values are kinda dumb in general. Yes, and many, many people think that and write about it. So how did p-value type analyses become so popular? Easy. It provides a veneer of objectivity when scientific decision making is not totally objective. You can't have a machine spit out a yes or no, usually. Also, with moderate sample sizes, a low p-value indicates a relatively strong strength of effect. Hmm...so why not just look at strength of effect directly and lay everything out so that individuals can decide upon it? Well, that's a better approach. Cue first year math students, people that have taken one stats course, or a semi-competent stats teacher/researcher to call me crazy. Research this question about null hypothesis and significance testing and the controversy around it. There are hundreds of resources.
Yeah, the p-value is just a number you can use to argue your results. You should still be able to adapt your argument to your experiment/methods. Picking .05 as a cutoff (as it was said on other CC:Statistics videos) is just arbitrary and means nothing without context. The problem is that statistics isn't the most prestigous field of mathematics, but it is one of the most important one in ALL fields of science. So there are few really good Statisticians, that are good at teaching new generations.
You still need to have an effect (however small) to get a tiny p-value no matter how large the sample size. The way p-hacking is done is with lots of tests on lots of variables, not lots of samples.
I liked this video, but it was basically a drawn out way of saying that if you don't take everything into account or intentionally leave some things out you can get bad results.
I still don’t understand. What does p-value mean, and how do we choose the p-value? Does low p-value mean significant affect, and high p-value means no affect? Is it just cherry picking? Can someone please explain I’m so confused😭
But that information useless in that it does not offer any details about which variables (jelly beans) are significant or even how many! By adjusting your p-values you've CHANGED your null hypothesis. There are better ways of doing this!
It has been shown that p-values barely under 0.05 are statistically over represented. This was shown with a p-value < 0.05.
I'm dead 😂
Haha. That's funny.
This crash course is great. The level is fairly high (I believe that most people with a PhD could even learn a thing or two), and it's still comprehensible.
Great job!
**sighs** You explained this so much better than my biometry prof. Thank you!
During my Ba and MA I certainly did feel the strong push to "just try this or that" so the p-values would be a little bit more acceptable. I never really got a straight and clear answer how that wasn't similar to cheating. But everyone seemed to think it was quite okay - That was a weird experience.
Gaslighting, that's called right?
If you aren't publishing (and as a student, you probably weren't expected to publish) then p-hacking isn't a concern. If you're learning the "art" of statistics, you need to experiment with the way different analyses work.
Actually, there is a small misconception in this video (the table in the beginning). When we reject the alternative hypothesis that does not mean that the the null hypothesis is true. It simply means that we do not have statistical evidence to reject the null hypothesis - we cannot say with 100 % statistical certainty that Ho is true. Andy Fields writes in 'Discovering Statistics in SPSS': "If the p-value is greater than 0,05 you can decide to reject the alternative hypothesis but that is not the same as the null hypothesis being true" (Fields, 2018: 76).
Poor kid... He fell asleep studying and gets called an honest mistake :D
I think I loved the peas on the laptop. I'm confident with a p=0.04 that did not NOT love them.
@Rowan Brown Luk je kemi roa? Juk no ke lemapu ke hoa?
This is amazing! I dont understand anything you talked about . Still love it👌🏽
The bowl of peas sitting at that angle and on a laptop is very distressing!
All of that and you didn't mention the common terms "fishing" and "exploratory analyses", nor how such approaches can be used ethically to generate new hypotheses, or be methodologically accommodated, such as with screening and hold-out samples (cross-validation).
Hi! I really like your videos, I'll eventually go through all of them. Please make a video on multi-variate analysis. I' beginning to understand them but would really like for you to explain them (PCA, CVA)
Plot twist: those peas are actually green jellybeans.
I just want to say, when you calculate that with twenty tests all looking for p < 0.05, there is a 65% chance that we will get a false positive, you said that this "might be higher than you would expect". My statistically challenged brain sees it the other way, though: if each test has a 5% chance of giving a false positive and we run 20 tests, then we get a 100% chance of a false positive because 20 * 5% = 100%!!
(Of course I understand why this is wrong, and you've explained it very well over the course of this series--I just wanted to point out what seemed to me to be the obvious mistake to make!)
You can think of it like "what's the chances that I'll get NO false positives in 20 tests of p = 0.05", which is ( 1 - 0.05 ) ^ 20 = 0.358 or 36%, so on the other hand, 1 - 0.358 = 0.642, or 64.2% chance of getting more than 1 false positive.
Awesome video! Thank you!
This is such an important video
Someone needs to start a scientific journal that only publishes papers in which the null hypothesis is not rejected.
XauriEL Zwaan already done. Journal of negative results or something like that. But it’s not much cited...
I always love your props! DFTBAQ!
I was a bit confused about the Bonferroni correction, because surely, for a total p-value of 0.05 and 5 tests, the p-value per test should be
p < 1 - 5th.root(0.95)
But apparently, the simple division always leads to a smaller value than this calculation, making it basically close enough, and an even stricter limit. Although it surprises me that you didn't point this out, as that quotient comes kind of out of nowhere.
Yes it's a conservative approximation (that's close enough if you don't have a huge number of tests) that almost everyone uses without knowing that it is always (a little) too conservative (I guess that's why she doesn't mention it). The accurate correction (for independent tests) which you pointed out is called the Sidak correction :)
Yes, she also didn't talk about other solutions ie cross validation. Bringing up only Bonferroni (and also not talking about how conservative it is) and not talking about the issue more fully can be a bit misleading /:
I use the Bonferroni correction quite often. There are more than 20 thousand human genes in our DNA. Sometimes I need to look for genes which **significantly** change under a treatment. Without the Bonferroni correction, I would be doing over 20 thousand different p-values, which of course I would find something that appears "significant" with a p < 0.05.
I can see how one could easily come to the wrong conclusion if one is not careful under that circumstance.
WIth such a number of tests though maybe you could use another correction. Bonferonni is always a little too conservative, you could use the more acurrate Sidak correction. Or even Holm's or Hochberg's procedure...
@@nadjal Yeah totally. I use some R packages that let me choose from a variety of multiple testing procedures. Bonferroni is just one
P-Hacking was my nickname in highschool.
i had to pause the damn video to laugh like a goat when i saw the peas on the laptop XD
Excellent
Thank you for doing this!
how to get p-hacking out of "for profit" research?
1-make only "no profit" & massively perreviewed research significant enough for laws.
2-implement UBI and/or have a "national scientific research fund" that allows for scientists to follow through theier research within ethical conditions for themselfs and their results.
Thank you for the advice
can you guys do a crash course forensics??
So, in order to believe any statistic we see, and even then the statistic might itself be a chance result, we need Ph.D.'s in statistical analysis, an army of scientist employees, and sophisticated lab and computer equipment to verify claims. Oh, and access to the journals said statistic was published in, each of which will cost a pretty penny.
Not just CC, but in general, the media and experts do a fine job of outlining the problem but rarely give solutions and even when they do, the solutions are impractical or theoretical.
The same media and experts, due to shenanigans like p-hacking, have broken the public's trust in what they have to offer.
My solution: mandatory statistics classes beginning in Kindergarten. This will take forty years for any critical mass of trust to return to the societal influencers because in forty years, today's kindergarteners will have the power to influence and the old guard of prevaricating and unethical doyens will have died or be too old to dictate the direction of society in any meaningful way.
Or, wait for the Matrix plugin seats. They're coming!
Michael Crichton was talking about this 20 years ago. He suggested that the government fund a return to basic reseach and that researchers make their data available so that others can replicate their studies. His senate testimony is available online.
Chad and Stacy studio 💪💪💪💪
Regards Adrian, I am planning to carry out an empirical study with p hacking being the subject at hand, could you please a case study that could be worked upon.
Thanks in advance !!!!!!
Peas and laptop for P-hacking... XD
Is this the last episode? There hasn't been an upload of this series in 2 weeks...
Justyna Izabela
She says “I’ll see you next time”, which implies that there is either another episode or that she will present another show.
Phaking hell
How to get a tiny p-value every time? It's very easy conceptually. Increase your sample size. In fact, in areas of science where very large sample sizes are the norm, the p-values are almost always less than 0.05. Hmm.....so if with very large studies p-values are kind of pointless....does this mean that p-values are kinda dumb in general. Yes, and many, many people think that and write about it.
So how did p-value type analyses become so popular? Easy. It provides a veneer of objectivity when scientific decision making is not totally objective. You can't have a machine spit out a yes or no, usually. Also, with moderate sample sizes, a low p-value indicates a relatively strong strength of effect. Hmm...so why not just look at strength of effect directly and lay everything out so that individuals can decide upon it? Well, that's a better approach.
Cue first year math students, people that have taken one stats course, or a semi-competent stats teacher/researcher to call me crazy.
Research this question about null hypothesis and significance testing and the controversy around it. There are hundreds of resources.
Yeah, the p-value is just a number you can use to argue your results. You should still be able to adapt your argument to your experiment/methods. Picking .05 as a cutoff (as it was said on other CC:Statistics videos) is just arbitrary and means nothing without context. The problem is that statistics isn't the most prestigous field of mathematics, but it is one of the most important one in ALL fields of science. So there are few really good Statisticians, that are good at teaching new generations.
You still need to have an effect (however small) to get a tiny p-value no matter how large the sample size. The way p-hacking is done is with lots of tests on lots of variables, not lots of samples.
Awesome!
Is there a relationship between p-hacking and Simpson paradox?
Vicente Rivera Not really, they are different things.
So what does hacked :p mean?
Pee hacking: pissing in a bottle while stuck in traffic
Or, using someone else's urine to pass a drug screening test.
Oh
I liked this video, but it was basically a drawn out way of saying that if you don't take everything into account or intentionally leave some things out you can get bad results.
I still don’t understand. What does p-value mean, and how do we choose the p-value? Does low p-value mean significant affect, and high p-value means no affect? Is it just cherry picking? Can someone please explain I’m so confused😭
contrast two results out of every 24. (3/3+1/3) : (3/4+1/4)
"You feel bad about taking their million dollars."
You should never feel bad about taking Kevin O'Leary's money. p
Oramchi poki chutki he? Mera nor ke gor cun chu! Lakchi goy he?
I only found out this statistic video 2 days before my exam dammit
This puts the sigh in science and the lie in believe the scientists, they all agree.
0:21 Groan. 🙄
I will never eat a green jelly bean again.
Watch out people: the green jellybeans are really peas.
oh no, is it over? =o
Jaii Raph
She says “I’ll see you next time”, which implies that there is either another episode or that she will present another show.
i heart your faith. Let us pray. But, in like a mathy way & that. w/graphs.
Wait you mean buzzfeed aint legit?
But that information useless in that it does not offer any details about which variables (jelly beans) are significant or even how many! By adjusting your p-values you've CHANGED your null hypothesis. There are better ways of doing this!
She reminds me a bit of Terri Garr.
Seems like p-values are obsolete
"Obsolete" implies there is an alternative that is better; if you know of a better mathematical tool, you might be eligible for a lot of money. lol
So much power with math. Hehehehe
20
phacking hell
Are the peas placed on the laptop because the peas are hacking? (P-Hacking?)
Haha a bowl of P's
> significant
lel
Statistics are always a good tool to lie.
Math T_T
What about it?
Aspiring Cloud Expert succks
i lovecars Search "vihart" in the TH-cam search bar. It might change your mind.
Flaming Basketball Club why?
*thumbs up*
Im early :)
Is calling it "hacking" supposed to make it sound cool? Did we need another neologism for this old concept?
22nd comment!
Yello
First :D
Congrats