Questioning Psychology's Findings: A Real Crisis
ฝัง
- เผยแพร่เมื่อ 10 มิ.ย. 2024
- The replicability crisis in psychology refers to the fact that many psychological findings, especially those from experimental studies, cannot be reliably reproduced. Replicability and reproducibility are cornerstones of the scientific method and are necessary for establishing the validity and robustness of a finding. In this video, I review how the replicability crisis started and then worsened as researchers adopted questionable research practices (QPRs), including the use of small sample sizes, selective reporting of data, and the manipulation of outliers.
0:00 Introduction
0:58 Antecedents
7:53 False Positives
9:30 P-curve analysis
13:10 Small sample sizes
16:10 Selective reporting
16:44 Optional stopping
17:47 p-value rounding
18:23 File drawer effect
18:49 Post hoc storytelling (HARKing)
20:12 Manipulation of outliers
22:48 Science paper 2015
28:11 Conclusion
#reproducibility #psychology #science #psychologyfacts #openscience
✨ABOUT ME:
I received my Ph.D. from the University of Southern California in Los Angeles. I joined the School of Psychology at the University of Queensland in Australia in 2007, where my research and teaching are focused on social neuroscience.
✨OTHER VIDEOS OF MINE YOU MIGHT LIKE:
=Finding the Joy in Missing Out: • Finding the Joy in Mis...
=How to Use the Psychology of Habits to Make You Happier: • The Psychology of Habi...
=5 Apps Every Psychology Student Needs (and They're Free!): • 5 Apps Every Psycholog...
=Being a Professor in Australia: • What's it like to be a...
=Australian vs. American Universities: • Australian vs. America...
=My Goal for the Year was to FAIL 10 TIMES! • My goal for the year w...
✨ WHERE TO FOLLOW ME:
- PERSONAL WEBSITE: socialneuro.com
- INSTAGRAM: / socialneuro
----
✨ Music From EpidemicSound: share.epidemicsound.com/tjrxd3
Subscriber count: 906
![[풀영상] 2024 김연경 초청 세계여자배구 올스타전 "팀 스타 VS 팀 월드" [유료광고 포함] #김연경 #국가대표 #배구 #배구올스타전 #팀스타 #팀월드](http://i.ytimg.com/vi/SUKUgH1ltng/mqdefault.jpg)
Although I discuss some big problems that psychology (and other sciences) have been facing in this video, I do believe psychology has done many great things for society. Check out this video that I recently made: th-cam.com/video/ajXrTkWfqFk/w-d-xo.html
Bwaaahahahahahahahahahahaha ... a complete psuedo-science, in fact the word science has absolutely no place anywhere near the word psychology.
I called the editor of JPSP re Bem's studies. Specifically, JPSP refused to publish several papers failing to replicate Bem. I asked why and the editor simply said "we do not publish failed replications". Speaks volumes about the "science" of psychology.
Thanks for your two comments. I know your work well and possibly met you in some of my visits to UCSB back in the ‘90s. Those Bem and Ruggerio papers still make me shake my head in frustration! It’s timely you commented now because I’m in the midst of making a new video about Science Fraudsters.
@@socialneuro I was bewildered by the comments regarding Bem's work on precognition, because no justification was given for regarding them with suspicion -- it seemed to be a case of people with no knowledge of basic findings in parapsychology reacting in horror because significant results had been obtained. As a psychologist myself (long retired) I found his designs ingenious and his results convincing, especially as they confirmed one of the major findings that precognition is actually the most widely reported experience in survey studies -- in the UK, for example, around a quarter of the population will admit to having such experiences. The debate reminds me of Hansel's book about research on telepathy. He claimed to have demolished or at least thrown doubt upon the results of many experiments but had to admit that several had delivered strong and statistically significant results. His response to this was to claim that maybe statistical theory itself (not the particulars of these experiments) may be somehow flawed! He failed to observe that statistical theory has been used for decades in many other less contentious fields (such as medical research and agricultural studies) and is also the cornerstone of quantum theory, so if all these are based on wrong assumptions then we are all in serious trouble. Premonition, precognition, whatever you call it, is real and is extremely common. So are many other psi phenomena. What we need to do now is to find how they operate. Check out the SPR website for excellent surveys of the area. Don't let negative subjective reactions influence your judgment!
@@socialneuro the sad truth is that psychology not being a science IS A GOOD THING, but so many psychologists want the status that comes with STEM that they overlook this fact.
@@thispersonrighthere9024How is it a good thing? Where and how do psychologist get the knowledge to apply in treatment? Implying you're referring to a clinical setting.
I'm a non educated fan of psychology. No formal education just a big fan of the study of human behaviour. It has seemed to me for some time that Psychology seems to have a very ideological core to it. The science is just a veneer.
Any psychologists, psychiatrists and, god help us, clinicians in the psychological field (therapists, councilors etc) I have spoken to seem to have the craziest beliefs that aren't backed up by science at all.
I know they are not representative of all professionals but there seems to be a lot of them.
There seems to be an attraction from people who in another age might have been church councillors or witch doctors of some description. Really strange ways of thinking I have no idea how they got through university with this way of thinking unless the universities are promoting it
And of course the issue of sampling bias. No, psychology students are not really a representative sample of anything except psychology students.
Considering the last book I saw about psychological diagnostic claims free thinking and questioning authority is a mental illness told me everyone in the field is to be treated with suspicion and distrust.
Yep, there's no reason to trust them
I used to trust them. Whoops.
The phenomenon that people make themselves believe they are "criticall" and "free" "thinkers" when they mindlessly accept what a bunch of crackpots and/or grifters tell them to believe is indeed a mental illness that plagues society.
Science + business = this
Me bringing this issue up got my business life ruined. Glad to find this video. I don't see notifications so this is a one way message. Thank you for this video!
The replicability problem runs through all the sciences. Novel findings are rewarded more than replication of previous work, and little effort goes into disproving previous work. Science advances only by disproving false knowledge. Many theories are no longer falsifiable, which makes them by definition unscientific.
Amen.
Non-falsifiable. I heard that term for the first time from It's Okay to be Smart about why some things, like the existence of a God, can't be called scientific discussions, because science is about failing to disprove as much as it is about proving.
@@Moses_VII I doubt the ability of science to prove anything.
Plus the science department is now a revenue center. Of course the land grant college is supposed to support agriculture and industry but there's no longer resources or room to do plodding basic research. Yet the world still needs it.
Scientific progress by disproving false knowledge has very very little to do with showing that previous experiments cannot be replicated. What is meant by disproving false knowledge is falsifying theories by coming up with an experiment that contradicts the theory.
Before a previously accepted theory in physics is discarded, you can be absolutely sure that the experiments falsifying the theory have been thoroughly checked and replicated by others.
Thanks for covering this, very good video 🙏
I'm afraid, since the pandemic, I've lost all trust in scientific institutions and academia.
I understand
And yet your smartphone works.
@@ronald3836
Is this supposed to be an argument for why I should trust institutions?
Because the computers that track the movements, behaviours, political opinions, consumer trends, personal messages and psychology "work"..?
I usually ignore quippy idiots - but that is a truly moronic response to a wholly justified loss of trust in public health, "evidence based" medicine, and the branch of "science" which was fooled by the Sokal squared hoax papers.
I don't expect an answer, but I am always left wondering if TH-cam has censored my comments, as they have certainly hidden replies from me. But hey, as long as my phone works, we should all suck up the next spoonful of B.S. from the "experts".
You are a true, *true* clown.
@@ronald3836 Not without a thousand backdoors.
The entire publish or perish model is to blame in my opinion. The permanent reputation of the entire field is at stake now.
So, throw out Capitalism you say ? I'm in..
@@sgramstrup sadly not easily laid at the door of ‘capitalism’. More likely ‘dodgy ethics’
Capitalism demands and rewards bad ethics. I don't think that rainforest indians are narcissistic like our society. If it's not from capitalism, where would the pandemic of narcissism come from?
@@sgramstrup No, it's already highly socialist field. From communist-style solutions, you should consider just banning free speech so people would not dare to mention the problem.
@@sgramstrup State-funded academic research has nothing to do with capitalism.
7:00 retraction watch 15:00 number of participants needed 16:00 selective reporting 16:40 option stopping 19:00 post hoc story telling
This has inspired me to become a psychology educator (adjunct), instead of a researcher. I don’t want to be part of this crisis and trained in the methods that have created it.
Could you not create a better way of doing it and lead by example or would you be forced into bad practice?
This isn't just psychology and neuroscience. It's pretty much all fields of science except the "hardest". In particle physics, studying the standard model, it takes 5 or 6 sigma for a novel result to be generally accepted by the community. p < 0.05 is just 2 sigma. It's just easier in the soft sciences because while all electrons and photons are exactly alike, people are all different from each other, so there is no possible psychology (or biology/health/medicine) experiment that could get to 5 or 6 sigma confidence.
The main problem, however, isn't the p-hacking or even the fraudulent data manipulation, it's that there is no "product" other than a published paper. It's all funded by grants, and getting published determines future grants and career success overall, so every single incentive is to publish a paper, and the papers are going to say whatever it takes to get them published. If this were work that somehow benefitted some consumer or another, then those consumers could be the impartial judges as to whether the product quality is high. There are no such consumers - publishing benefits the publishers, who have no real incentive for quality control other than the potential embarrassment of being found out and losing reputation.
Don't be so fast to call physics innocent. There have been big scandals with entire labs publishing papers with fake data. They only got caught because some countries do see (some) value in having physics research capacity and after several years of the team in another country being unable to reproduce the results, it all unraveled.
You are correct though that p-hacking is a term I learned as just a fact of life in psychology research but pretty much a non issue with physics.
Measurement error though...
I'd be tempted to scrap everything we think we learned and start all over again from scratch!
Good idea.
Have fun re-inventing mining techniques.
Just a hunch: Have we dug ourselves into this by allowing academic roles to be personal “careers” rather than “vocational” outside of economic good?
We know that grants today are less sourced through government or community, and more through private corporate interest. The latter will want “deliverables” that can be traded, while the former is more incentivised by community values (not a good vs bad issue, just by incentive and sanctionability).
We also see that some findings in social and neurosciences point to a general effect of solely allowing pure reason and reductionist cognitivity in science. That mindset may occlude what earlier scientists in any field attributed to their own imaginative and creative abilities. I.e. that “new” ideas originated aided by non-scientific “sensing”, leading to insights that could (hence did) lead to actual scientific breakthroughs.
Up until ca 1980’s, many would use quite a poetic language in their publishings, even stating that language itself is a poor metaphor. Terms we use scientifically now, were at some point “created”. Many used words like “beauty” (Darwin) or “awe” (Feynman), without this taking away from their actual empirically sound work.
I amateurishly posit that we have lost something humanely valuable - holistic balance of mind, integrity of thought? - at this stage of modernity, based on the above. I can only guess that any relevant studies are hard to conduct and publish, for the very same reasons.
I hope I’m wrong, though…👍
Rugerrio worked as a post doc at my school. She gave a talk in our weekly brown bag. I found the results so improbable that her post-doc advisor (and co-author) had to beg me to let her continue. Sadly, I complied.
My own experience with the field of psychology is that it's more a guessing game then a science. New psychologist; completely new diagnosis which alway initialed new medication(of course). If your not crazy before you see a psychologist you will be after. By the way crazy=sick. The whole field is nothing but a hustle for doctors to make money. Reform the insurance industry and it would disappear. The only people that would truly understand that are patents and we're all crazy. catch 22
"We've had 100 years of Psychotherapy and the World's Getting Worse"- James Hillman......
That's because psychotherapy has nothing to do with prevention.
@@Username-nu8el You'd 'think' (operative word) that it might have pondered 'that'....rather than arrogantly 'thinking' (operative word) that it could cure 'that'.But it is a good way to get $150 for an hour of listening to someone's ramblant babble, when you are impotent at connecting a hammer with a nail or creating a nice sandwich for a lunchtime customer, Wot!?...but let's talk and watch Rome and Babylon burn...with anxiety.
Another brilliant video. I learned a lot of new studies/information to look up further. Thanks again for making and posting this.
Thanks so much!
Back in the 90's, I feel like psychology wasn't really taken as seriously precisely because it isn't a hard, replicable science. A plant always photosynthesizes the same way. Steel always responds the same way to a vacuum. Protons always behave consistently. It seems like humans are too nebulous to pin down without enormous numbers of participants and then you've broadened your findings such that they may or may not be applicable/helpful. Add sometimes arbitrary, sometimes critical, distinguishing factors like sex, ethnicity, health, background, etc. and results are further confounded.
What affects one person might not affect the other and since you can't "dissect" and present the internal workings of the human mind, you can't really ~prove~ anything past taking the researcher's word for it. Compound this heightened necessity for strictness, adherence to procedure, and total transparency and you've got a lot working against the field. At some point, we forgot all that and started treating it like a "hard" science... or at least classing its results as similarly predictive.
Plus, the mechanism creating/executing the research (the scientist's mind) is the same kind of mechanism as the one(s) being researched, like biting one's own teeth or avoiding peculiarity while doing something akin to observing a telescope using only another telescope. Fuckery abounds.
this comment is exactly why psychology isn't science.
Honestly I wish there was more of a financial incentive (from somewhere) to try and replicate as much popular research as possible. If science is going to get better it has to be as self-critical and open to falsification as possible.
Statistics like Economics is a pseudo intellectual process and of course Creative Accounting. "What results would you like sir"?
I do think that the danger of policy based on false positives is underplayed, and this is not limited to the field of psychology. If a false finding justifies an expansion of government power, then large numbers of people profit from this at the expense of the rest of us. As the number of people with a vested interest in acceptance of the false idea grows, it becomes ever more difficult to question the findings. An establishment forms, and it will seek to deny funding to any research which challenges the false view. Because more and more of the funding goes to those who profess the false view, the false appearance of a consensus grows. The field becomes corrupted.
Maybe I misunderstand the scientific method, but I always thought that no study should even be considered an established part of any science until it has been replicated. I find it strange that it is flatly stated that "null results are hard to publish", apparently simply because they are null results. Why aren't these results considered just as important to science as the successful replications?
Awesome! There should be a journal with only repeated experiments in it.
Excellent video. Thank you. Reminded me of all the mistakes I made in my undergrad polysci statistical research class 😅 😂
I’m so glad you liked it!
the entire clinical psychology field is a house of cards.
Thank you. Very interesting video
Glad you enjoyed it. Thanks for the feedback!
Piaget's experiments come to mind.
Not only do I think we are going to survive this, we are going to grow from this.
Only when the harm caused necessitates effective punishment. As long as that isn't true, faked data will continue to occur and the harm can be ignored
Not until money is taken out of the equation. In Vegas if you bring this crisis up to a professional they will submit you to a legal 2000 hold, which is just a 72hour hold that medicates everyone! Diagnosis and meds all done within a 72hr or less hold. this whole thing is bad and will never change.
A partial solution: publish good studies that fail to reject the null hypothesis.
Thanks for the video! Unfortunately botched science in this field has real world implications. I've observed quite often that psychotherapy failed to help people. I've learned that some methods claimed to be very effective by their inventors, and widely used, couldn't be replicated. A German scientist claimed that up to 50% of the therapies fail. There was at least one study claiming that not the method used was influential of the outcome of the therapy but the quality of the relationship between patient and therapist.
When the incentive is to be cited and to be cited means having a published paper and to be published means findings confirming the researcher's pet theory, it seems the incentive structure may be a little warped.
What do you think of the viability of a multidisciplinary journal that purely publishes papers based on experiments that fail to replicate previous findings? 'The Journal of Failed Scientific Experiments' has a nice ring to it
I like this idea very much. There are also many journals that review and accept papers BEFORE the data are collected.
Brilliant idea! Crowd source funding.
I've helped with analysis for a few papers in a different field. It is very easy to fall into these kinds of "statistical jiggery". Even if you just see a result that doesn't make sense or contradicts something you are trying to confirm, you are much more likely to look for something you did wrong so that you can run the analysis again. It is hard to have the kind of statistical rigor to do things properly. The way the publishing system works probably doesn't help, and I think the video author makes a good point that researchers are unlikely to question their previous publications or publish things that contradict their previous results.
We could probably move a step forward if journals were willing to publish negative results, as long as the hypothesis you were using made sense and your study design looked reasonable, even if they just restricted the page size of those papers.
I do not know why results do not replicate, there are many reasons for this. This year has been tough for me in terms of publishing. I lost two rounds at important Marketing Journals, because reviewers asked me to redo and replicate the studies but the studies did not replicate, leaving me no option but losing the rounds, this was hard after many years of work. Just yesterday, I ran a study following a well-established procedure from (JPSP), but the results were not significant. This is hard as it not only consumed my time and resources but also made meeting deadlines and submitting my paper challenging. Surviving in this academic environment feels daunting.
Is there a list of the papers on the 100 studies looked at ?
It makes no sense to not publish if p
I do not know if you have dug into this enough to understand how this has happened since making this video but thought I would provide some additional context in case you have not. The sciences have for a long time been infiltrated by destructive ideologies and their zealot's. The ideology that bears the bulk of the weight originates in marxism and the results of this infestation were foreseen long ago. This was a scripted long term plan of incrementalism. What is going on in the sciences is but one of the vectors of attack. I implore you to watch Robert Welch speech from 1958, in great detail he describes what has be transpiring for the last 65+ years well before it ever happened and he was no soothsayer he simply understood the plan and those bent on making it a reality. Later in the 80s a KGB agent named Yuri Bezmenov defected and in an interview confirmed what Robert Welch warned us of more than 30 years prior. Cognitive dissonance unfortunately will be a hurdle most will not be able to overcome when faced with such a reality.
/watch?v=xRe7pPgg6yA&t=2s
Looking forward to AI, AGI and mostly, ASI.
I find much of psychology to be pseudoscience. Neuroscience may be the astronomy to the astrology of psychology.
Eric Van "The Man"
While sound statistical methods and replication are important, I think the real crisis is theoretical. Psychology needs good theories. Science is an attempt to describe the world. Multiple replications using the most rigorous statistics isn't going to weed out bad theories when the subjects are all undergraduate students. A good theories tend to be useful. Theories that are potentially useful make it outside the lab and then replication happens in a much more meaningful way. How many theories of psychology make it out of the lab and are useful? I mean useful, not merely used in a no risk of disconfirmation Myers-Briggs sort of way but used in a way that would count as good test of theory.
Would opening of reviewers and marking parts of a paper as 'peer reviewed' a good approach? This way the reviewers would a look also take some responsibility!
Great idea! Some journals are doing a version of this. Check out Frontiers, for example.
The business of science has exposed it's fallacy
Isn't this a failure of peer review? Are the peer reviewers getting flack over their inability to actually review these fraudulent studies?
I think one huge issue in peer review is that there is no requirement/ "strong encouragement" for replication in studies that are not time-extensive (e.g. logitudinal studies).
So prior to 2009 did people think of psychology as serious science?
You know those people who always said school is useless...
Well, we get to the point that we have to agree.
Stanford is a joke and it can never be the only one who does it.
Know it from my own school back in the day already.
It happens everywhere...
I live in a western country ''the free west'' where the one in charge of mental healthcare is put there by the state..that is all you need to know about psychology.
Science dishonesty kills it self….
For me psychology is broke and full of biased people.
I've been working in countless households and from that experience I know that majority of children is abused by mother.
In the same time father is usually portrayed as monster by her emotional signaling to the child that he is dangerous and abusive and worthless as a person.
Entire field seems to be blind to that phenomenon and even support this behavior by relentlessly searching for excuses to accuse father of abusive behavior.
Why?
Psychological departments are full of women and men with daddy issue.
I witnessed the most egregious case of father alienation in home of professor of psychology/sexology.
That explains everything.
Of course those kind of people will take side of abusive, controlling mommy and support delusional believes of the child.
Craig Childress describes this perfectly in case of divorce by from my experience this is the case in all conflicted (covertly and overtly) marriages.
A lot of confirmation bias
Die Lösung ist doch einfach.
Publiziert wird nur, was repliziert wurde.
Dann ist der Spuk rasch zu Ende.
I believe that than company researches cannot publish at all. Which is a pity because there products have to work, so are often more sophisticated. I am EE, no psychologist.
"Anybody order fried sauerkraut!!??" Wow. A women being honest.
If we look at (at least) US court cases...we have the situation where a Prosecution and Defense psyhologist (for example) reach diametrically opposed conclusions on tbe SAME individual....you can only shake your head....How is the science so bad ? AND YET...we still use junk like this ....
More essential oils and yoga ?
Most of the money or land or home ownership or recreation vehicle to escape stupid city solve 90% of psychological
Sounds like this is all about finding ways of investigating yourselves....
Better be careful, you are challenging the high priests and sacraments...
Seems to me that competition is the cause of bad science ?
Why he mention the 2011 ESP study like, it was an error for publish that? ESP is real an there are a lot of experiments since decades ago. It is a taboo talking about ESP. The problem is a lack of theory about how works and why. Also replication sometimes does not work... and actually if you do not believe, will not work. There is some influence about consciousness.
its definitely real, ive had experiences with this that I kind of wish i didnt have, its extremely unsettling to experience this kind of stuff and not know what to make of it
I recently watched a video about the issues with that study that goes more in-depth. It's on the "Neuro Transmissions" channel.
Quacks, seeking Grant funds.
Psychology has never been a real science. The scientific method requires observing facts, coming up with a theory to explain the facts, then performing objective tests to check the validity of the theory. That last part is the problem for trying to make psychology a science. Feelings, emotions and attitudes are inherently subjective. Human behavior is complex, and performing scientific, randomized, double blind experiments on people's minds, even if it were possible, is certainly unethical. I'm not saying psychology is completely useless, but like dating, decorating your house, writing poetry, or arranging flowers, some things just aren't sciences.
Psychology is bunk “science”
What is opinion? How is it different from truth?
Population wide we share fewer and fewer opinions on anything. At the same time each of us certainly shares (and was born in) a relationship with truth. The evidence of this is the sincerity of any infant’s heart. Guile develops along the way, perhaps it’s a human foible. This precedent in truth is what makes room for normal people, excellent ones, weirdos and liars. No personality type can exist outside of the precedent of truth this way. Its what makes room for the very best and the most loathesome, otherwise they would not exist.
Where there is merit to “the Torah” or “the Bible” then truth is a synonym with life. To wit, the life in our bodies which makes it possible to operate a keyboard, that life is the same thing as truth, this would be the “Holy Ghost” per Christianity. That truth is the cornerstone of the human personality, per capita in any population of human beings. Probably this is the same for any population of animals also. Our physiology sets us apart.
Perhaps the free will of persons is where the rubber hits the road. This precedent in truth provides a place for liars to exist as it is truth which is being “lied” about.
Fact is measured with math and science whereas truth is measured with the cubit. Otherwise the sincerity of individual opinion has no basis whatsoever. Swedenborg writes on this fwiw.
That’s cause it’s not science - you are all guessing it’s basically economics without the money
You should transfer into a real scientific discipline. Psychology has the reputation of alchemy and astrology.
I am an astrologer. I can read your chart and reveal deep aspects of personality in minutes. Genesis 1:14.
@@brendabadih8855 "And God said, “Let there be lights in the vault of the sky to separate the day from the night, and let them serve as signs to mark sacred times, and days and years," me think you've got a reading problem- no pun intended.
crisis in psychology and he uses ER docs and nurses as a thumbnail
Somebody call a wambulance
The last few decades have been utter trash. 😂
Why is this a surprise to people now? I have been saying this for over 25 years and nobody listened. Psychology is junk. Let's throw it away.
Sounds like a crisis in empirico-positivist psychology, not in psychology as a whole.
Interesting, but why on earth would you not mention how the Science/Scientists environment affects their judgement ? We have Capitalism that ensures that every scientist needs to deliver - 'publish or perish' I think they call it. ALL fields are soaked in monetary incentives where only scientific truth should be, and almost all problems in science are directly related to that 'intruder'. Even most of the 'whoopsies' mentioned as poor science practices here, is caused by economic incentives. In fact, the incentives inherent in Capitalism is incompatible with the incentives of good science - yet no one dares take this one on. The state of self criticism in ALL sciences is at a low place..
"We have Capitalism that ensures that every scientist needs to deliver"
If we had capitalism, we would most likely already have most of academia eliminated by cheap, mostly online courses. What we have here is a socialism where someone started demanding results for taxpayers money.