Broken Science? The Replication Crisis

แชร์
ฝัง
  • เผยแพร่เมื่อ 3 ธ.ค. 2024

ความคิดเห็น • 77

  • @davelowe1977
    @davelowe1977 ปีที่แล้ว +29

    The first time I came across this problem was the first time I ever read a paper about a genuine experiment. While at school it was drummed into us to write up clearly what we had done and what we found. When I read my first genuine paper it was because I wanted to repeat it but with some minor variation applicable to my specific case. I fell at the first hurdle though because _I couldn't figure out from the text what the authors had actually done._ I have come to realise that the authors were trying to obfuscate their own work because it had holes in its methodology.

    • @businessreform
      @businessreform  ปีที่แล้ว +7

      Thank you for sharing that. You just gave me an idea for another video about why things are sometimes intentionally written to obfuscate. Early in my career I was doing failure analysis on an automotive electronic module that caught on fire, uh, I mean “experienced a thermal event, resulting in rapid oxidation and the release of carbon particulates.” My mentor at the time taught me how to write a “proper” report for the automotive industry so that if the report was ever read to a jury, it would be 100% accurate but completely unintelligible to the average person.

    • @katim2644
      @katim2644 ปีที่แล้ว +1

      How did you get your text to show as italics? I've tried but have failed.

    • @davelowe1977
      @davelowe1977 ปีที่แล้ว +2

      @@katim2644 put an underscore before and after the text you wish to italicise with no space between the underscore and the first or last letter. __in italics__ --> _in italics_

    • @davelowe1977
      @davelowe1977 ปีที่แล้ว +3

      @@businessreform These days I see this obfuscation frequently in poor theses. It's the perfect cue for a question during a viva because it gives the candidate an opportunity to demonstrate that the obfuscation was accidental and that they do genuinely understand the subject, or not.

    • @katim2644
      @katim2644 ปีที่แล้ว +1

      @@davelowe1977 Thank you so much. I'm a bit of a grammar nerd and the use of italics has a role in grammar like book titles and some types of quotes. You have bold too?

  • @drowningbanana3890
    @drowningbanana3890 ปีที่แล้ว +14

    A great overview of the issue. I find it incredibly disturbing that the basic 'repeat your experiment for a more accurate and reliable result' that I learned in school goes out the window when you get to research level scientific work. Sensationalism and a struggle for funding takes over, and in so many cases, the pursuit of the truth goes out the window.
    It's why I quit academic study and now focus on practical things that directly help people, by working in A&E

    • @businessreform
      @businessreform  ปีที่แล้ว +5

      As several people have commented on one of my other videos “follow the money”. Nobody gives grants to study “non-problems”

    • @peterjones6507
      @peterjones6507 ปีที่แล้ว

      Spot on.

  • @samuelsmith1969
    @samuelsmith1969 ปีที่แล้ว +2

    Whether the general public understands the difference between science and scientism will be the defining factor of the 21st century.

  • @thomasdowe5274
    @thomasdowe5274 ปีที่แล้ว +9

    I stopped reading Psychology Today and Scientific American before I was 15 years old...too many road-apple articles with logical inconsistencies...and obvious biases...

  • @dante7228
    @dante7228 ปีที่แล้ว +3

    I'm glad I found your channel!
    Subscribed immediately!

    • @businessreform
      @businessreform  ปีที่แล้ว

      Thank you for subscribing. I am re-energized from all my new subscribers and all the great comments people have been leaving. I have a steady stream of new content planned :)

  • @grannyninja3253
    @grannyninja3253 ปีที่แล้ว +3

    Thank you for exposing this. I was researching the Mandela Effect and Elizabeth Loftus research into false shared memories. She ended up being sued and it came out that it was a small group of mentally handicapped individuals!?? My first thought was where is the Peer Review!!! And for me added Validity to the Mandela Effect. Thank you again.

    • @businessreform
      @businessreform  ปีที่แล้ว

      You are welcome. Thanks for watching!

  • @glitchy_weasel
    @glitchy_weasel ปีที่แล้ว +5

    Very informative. Science is supposed to be verifiable, not worshipped. I definitely learned something new, and would love to see the public more aware of the inner workings of the scientific community. Keep these videos up, alright; I'm sure you will eventually reached a lot of people. :)

    • @businessreform
      @businessreform  ปีที่แล้ว +1

      Thank you!

    • @WatZ-In-Ur-Head
      @WatZ-In-Ur-Head ปีที่แล้ว +2

      That is 100% Truth: "Science is supposed to be verifiable, not worshipped."

  • @freespeechforever
    @freespeechforever ปีที่แล้ว +6

    A pleasure listening to your commentary, far too late now but I wish I'd had one teacher who could have explained things the way you do. Excellent video. As an after thought, Dr. Vernon Coleman, a UK gp turned author, a highly repected voice until his work was found to be far more accurate than was comfortable for people like Pfizer. Nowadays his wikipedia entry straight away labels him as a conspiracy theorist, however, even minimal research and my own experence indicates that he touiched a few nerves.

    • @businessreform
      @businessreform  ปีที่แล้ว +2

      I tell people every criminal case that has ever been tried began as a “conspiracy theory”. A prosecutor believed someone was guilty of something that the accused person(s) denied. The prosecutor had some degree of evidence to back up his / her beliefs but prior to the trial, nobody knew if the evidence was sufficient to convince 12 jurors beyond a reasonable doubt. It is also true that sometimes the evidence is flat out wrong (my upcoming videos on “bullet fingerprinting” and “bite mark analysis”). And it is true that sometimes the prosecutor was right but the evidence was too weak to convince the jury.

  • @tanned06
    @tanned06 ปีที่แล้ว +2

    This replication problem is inherent in science esp. in life sciences whose findings are very much circumstantial, depending on experimental setting and environment to some degree not easy to replicate. This variable perhaps becomes the "legitimate" cause for those daredevils to manipulate their data for the sake of cooking up exaggerated, if not extraordinary finds.

    • @businessreform
      @businessreform  ปีที่แล้ว

      My background is in electrical engineering and I know in the “hard sciences” or maybe “physical sciences” is the better term, it is easier to replicate results because it is easier to control variables. In contrast, any research relying on self reported data, surveys, etc. are going to be more difficult to replicate because of the human factor. I know how surveys can be subtly biased to “encourage” certain responses through the art of “priming”. In other words, if I asked you to think of a good experience you have had with a product my company sells, then I ask you a question about your overall satisfaction with the product, you will probably give the product a higher rating than if I had not “primed” you by getting you to think about a good experience.
      I completely agree this creates a lot of opportunities for people to “influence” or bias the data they collect. It is also why the majority of published research favors the sponsor of the research, a topic I will be covering within the next 2 weeks in my upcoming video about conflicts of interest.

  • @TheEarthCreature
    @TheEarthCreature ปีที่แล้ว +2

    Wow it's very interesting that you made this video today because I just watched the other video where you'd mentioned the replication crisis in the comments and wondered if you came out with the video yet.

    • @businessreform
      @businessreform  ปีที่แล้ว +3

      To be completely honest, a month ago I was really burning out having posted 48 videos over the course of 9 months and only having 212 subscribers. Most of my videos were getting 200 to 300 views over their first couple of months so I really slowed down on making new videos. However, last week something happened and my “Corrupted Science” video exploded in views, likes and comments and my subscriber BASE jumped from less than 300 to over 4,000 in just a couple of days.
      I am now re-invigorated and plan to be releasing 1-2 videos per week from now on.

    • @TheEarthCreature
      @TheEarthCreature ปีที่แล้ว +1

      @@businessreform Yes the youtube algorithm REQUIRES consistency for reach. Keep posting, we will keep watching, and it will keep getting better. Do not burn out sir. Your videos are very much needed and appreciated even by a few people. I both found and subscribed to you yesterday. It's not that people weren't interested. It's that you weren't reaching us yet because you didn't have enough content up. You're getting close to the "snowball" forming.

    • @businessreform
      @businessreform  ปีที่แล้ว +1

      Thank you for your support and I will be posting new content regularly from now on. Just this morning I thought of 2 new videos I want to make that were inspired by comments that were left on my Corrupted Science video. One from a positive comment and another from a couple of negative comments. I don’t mind my critics, sometimes they inspire great ideas 😊

    • @TheEarthCreature
      @TheEarthCreature ปีที่แล้ว

      @@businessreform Looking forward to it!

  • @jdgvee9313
    @jdgvee9313 ปีที่แล้ว +4

    Need to discuss/look at climate change psyence

    • @businessreform
      @businessreform  ปีที่แล้ว +1

      That is coming in the future but there are several videos I want to create leading up to that so I can put them in a playlist. The videos will focus on the LONG history that mankind had making apocalyptic predictions that don’t come true.

  • @lukejezeph7871
    @lukejezeph7871 ปีที่แล้ว +1

    Great videos Jon - keep them coming :)

  • @sparkyy0007
    @sparkyy0007 ปีที่แล้ว +1

    Stanley Plotkin deposition
    time stamp 2:15
    "The problems I've discussed are not limited to psychiatry, although they reach their most florid form there. Similar conflicts of interest and biases exist in virtually every field of medicine, particularly those that rely heavily on drugs or devices. It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine."

  • @noahvale2627
    @noahvale2627 ปีที่แล้ว +1

    Glad I subbed, another good video.

  • @wernermesserer4464
    @wernermesserer4464 ปีที่แล้ว +3

    Is there any individual incentive to confirm the work of others? He was a great scientist, he repeated 10 experiments of his peers??

    • @businessreform
      @businessreform  ปีที่แล้ว +1

      Great point. That is what the abstract quote at the end of the video was really hitting at - there are lots of incentives for doing original research but very few incentives for doing confirmation studies.

  • @KN-vz8dj
    @KN-vz8dj ปีที่แล้ว +2

    The reason for bad replication is partly due to "look elswhere effect" and that the true distributions of the parameters are really not known. I would take any single study result with a big grain of salt even in the case of a "5 sigma result". I can confess myself being tempted to call a result when one of my many ways to analyze the data yields a significant result while others don't (the other methods must then be faulty, right?). Replication is an immensly importrant part of the scientific process and being unable to replicate something is an unevitable result of that process.

    • @businessreform
      @businessreform  ปีที่แล้ว +1

      Excellent points. I follow several TH-camrs in the health / nutrition / supplement area who all share their comments about recently published studies on things like carnivore vs keto vs vegan diets or the longevity effects of NAC and NMN. The research is all over the place with many conflicting studies. In a couple of cases, where a researcher holds a patent on a particular supplement or they are co-founder of a company selling a certain supplement, I get VERY suspicious of their work. However, most of the time I think this is just how science works. One study finds ____ supplements extend the life of humanized mice by 20% but another study shows that same supplement doesn’t do anything for humans. Both can be true because humanized mice are not a perfect analogue for humans.

  • @memine3704
    @memine3704 ปีที่แล้ว +1

    Good video. I wish more people were aware of this problem, and how that is being used for no good some times.

  • @lemonstealinghorsdoeuvre
    @lemonstealinghorsdoeuvre ปีที่แล้ว

    I would say that's a statistically significant difference in both studies.

  • @wingman1e6
    @wingman1e6 ปีที่แล้ว

    Not to diminish your valid point regarding the replication crisis. A concern within the scientific community. But, the motivating premise was a loss of public confidence in science. I seriously doubt the American public is looking at peer-reviewed journals to influence their confidence. This loss of confidence is occurring at a time when many of our institutions are under constant attack. Science, education, law enforcement, judiciary, etc are all subject to loss in confidence.
    Yes, the scientific community needs to solve the replication crisis. Yes, the scientific needs to police itself to eliminate junk science. No, I don't think either of these will solve the public relations problem.

  • @katim2644
    @katim2644 ปีที่แล้ว

    I feel like the last bastion of truth is in the peer-reviewed scientific community. There is SO much information out there (so much of it misinformation) that those that don't feel the need to look too far to find "their" truth, are easily persuaded to accept theories and facts and ideas that have NO basis in reality, NONE! For instance, when people go on about Chiropractic (don't get me started) I ask them to read the history of it, who started it, how it started, and what the theories are. It is simple. The idea of cracking your neck SOUNDS like it might help someone; BUT, it has been scientifically shown, through peer-reviewed science, that it doesn't offer any more medical benefits than a good massage; and, Trixie down the street doesn't demand that I call her a doctor! All I ask is that someone read the history; and, if they are conscious AT ALL, they will easily see that it is BS. Most people think (because of good marketing) that chiropractors are doctors working in a different field. NOPE! They are a basically massage therapists. The good marketing has convinced way too many people (including effing King Charles III) that it an actual medical field. Once again NOPE! This just shows how easily a population can be persuaded of just about anything through misinformation. SO - It is essential that the scientific community stick to THE TRUTH! They must, or else what is the reason for going on in a world full of crazy! Thank you thank you thank you so much for how you teach us this information. Descriptive enough for a scientist; but simple enough for us, the people that MUST hear it!

  • @rosomak8244
    @rosomak8244 ปีที่แล้ว

    It basically fails on statistical significance already. However reproducibility shouldn't be the end all criteria. Results should be viewed as significant if it can be demonstrated that they are not just reproducible but also transferable. It should be possible to take the results and device a different testing setup in which the same effect should manifest as well.

  • @sethbrown1763
    @sethbrown1763 ปีที่แล้ว

    If I am understanding you correctly, this means that even published results cannot be trusted in many cases.
    Does this mean that we need a better way to do science? That results should only be published if they can be reproduced?
    How much money would that cost? Is it feasible?
    If we cannot trust scientific results, how do we evaluate truth?

  • @billweasley1382
    @billweasley1382 ปีที่แล้ว +1

    I'm not sure of your expertise, but I wonder if you could do a video on the "peer review" process? I've always been confused about what actually happens during a peer review. I'm guessing that the experiments aren't actually replicated, but what does happen? How are the "peers" picked? Do they get paid? Is there an incentive (explicit or otherwise) to either verify or invalidate a study?
    I've heard that scientists "love proving each other wrong". Is that true? Unless it was some major high profile theory, I would imagine that scientists would just do the least work possible to get the review done.
    I'm curious because usually when talking to someone about the validity of a scientific study, people will throw out the fact that something was peer reviewed to basically just end the discussion. They seem to believe that the peer review process guarantees that the study is valid, accurate, and truthful, and if you dispute a peer reviewed study then you are just "anti-science".

    • @businessreform
      @businessreform  ปีที่แล้ว +1

      I have never received compensation for doing peer reviews and I am not aware of any journals offering compensation to peer reviewers. The only “incentive” for an honest peer reviewer who is a professor is that it looks good on your promotion and tenure documents.
      Peer reviewers do not replicate the experiments. They trust that the original author is honest with the data they include in the article. The focus of the peer review is really on robustness of the research. Did the author use a good methodology, did they have a random or a biased sample? Was the sample large enough? Are they overstating their results (I.e. if your experiment only included 18-22 year old male college athletes but your conclusion involves all people of all ages, you are overstating your results).

  • @samdolbear4502
    @samdolbear4502 ปีที่แล้ว +2

    Cos we have a normal distribution for the human populations ability to solve problems maybe a job like science is only doable by a smaller % on the right tail end and only specific psychologies out of that set and that in modern society it gets filled with lots of people from other sections of the distribution too. Cos mass university. And you have no character and citizen training which makes for crowd trajectories/vectors heavy to the corruption direction.

    • @businessreform
      @businessreform  ปีที่แล้ว +2

      I can’t remember where I read this quote but it has always stuck with me, “Never attribute to malice that which can be explained by incompetence.” I will give the benefit of doubt that the majority of researchers want to do a good job but they are often working for organizations that are more focused on publications.

  • @markcampbell7577
    @markcampbell7577 ปีที่แล้ว

    The CDC lost vials of Polio virus during the Reagan Admin and were questioned about the losses of Polio virus and small pox virus during the Obama administration on CNN in 2010. It is known by older people and educated people that coronavirus is the cause of Poliomyelitis. Coronavirus vials were lost during the Reagan Admin and the Bush administration claiming they were developing cell lines for medicine and Trump administration delivered vaccines that were cryogenically frozen vaccines. Would you accept a vaccination to go to work??

    • @businessreform
      @businessreform  ปีที่แล้ว

      I was not aware of those instances of Polio vials going missing and I am not familiar with Poliomyelitis. My objection to the vaccine mandates was based on the amount of safety testing that was skipped to bring the drug to market so quickly and the completely irrational push to force EVERYONE to take it. For instance, Chicago ordered all police officers to take the “vaccine” and some officers who were eligible to retire decided to retire rather than get vaccinated. They then had their pensions threatened if they did not take the vaccine. That was a huge red flag to me that this was not just about “public health”. news.yahoo.com/unvaccinated-chicago-police-officers-resign-181700973.html

  • @aeroeng22
    @aeroeng22 ปีที่แล้ว

    the difference(78. 3%-77.8%) between the two test groups is 0.5%, not 0.05% (0.05% is obviously 0.0005).

  • @whatbringsmepeace
    @whatbringsmepeace ปีที่แล้ว

    No sound. Has this video been censored?

    • @businessreform
      @businessreform  ปีที่แล้ว

      Thanks for your message. I’m not sure what is going on. I just played the video on 2 of my devices (iPhone and Apple TV) and the audio seems fine on my devices. What were you trying to watch the video on?

    • @whatbringsmepeace
      @whatbringsmepeace ปีที่แล้ว

      @@businessreform my Samsung phone. Will try again.

  • @pappapaps
    @pappapaps ปีที่แล้ว

    So competition once more brought out the worst in folks.. when are we going to reconsider this competition society? We're in what future scholars call the 'socioeconomic dark ages of deadly competition', trust me I have a time machine.

  • @whitebeardskydaddy6756
    @whitebeardskydaddy6756 ปีที่แล้ว +2

    I knew all along, as I witnessed lunacy after lunacy, that you and many others were out there. In fact, I suspected that we are the vast majority all along. We should be getting to know one another. We may not agree on every point of every thing, but there is a line which separates logic from illogic, and I think we will probably very closely agree about where it falls. The cult of imbeciles played their hand a bit too soon, it seems, fortunately, believing that the bubbles in which they dwell are the nation.

    • @businessreform
      @businessreform  ปีที่แล้ว

      I completely agree! There are so many issues that I do not expect anyone to agree with me on everything. I have friends of other political parties (I consider myself a conservative libertarian but I do not have blind allegiance to any party or candidate). If you notice in my newer videos I have a Statue of Liberty over my right shoulder and a statue of Athena (Greek Goddess of Wisdom and War) over my left shoulder. But it seems the only thing in the background anyone notices is my salt lamp 😂
      My goal with this channel is not to support a party or candidate or to get everyone to agree with me. Instead I want to encourage people to think critically and I want to inform people about important issues that don’t get much coverage in the media. One of my early videos was about the amount of documented toxic chemicals released into the environment. I quite staggering figures from the EPA that I am guessing 99% of the public have never heard about. I know some of my content will appeal more to the political right and some of my content, particularly some of the environmental issues I plan to discuss, will appeal more to the political left (although I am not a climate alarmist, I am talking about pollution and especially micro plastics).
      Anyway, thank you for watching. By the way, I am planning to do my first ever interactive live stream sometime near the end of April. I hope you can join and participate!

  • @MS-od7je
    @MS-od7je ปีที่แล้ว

    From an article (? Where)
    Something I’ve said for decades.
    Psychiatric Diagnoses Are Scientifically Meaningless
    According to an explosive study in Psychiatric Research, making psychiatric diagnoses based solely on symptom clusters is scientifically meaningless and disingenuous
    Take this stunning fact, for instance: “In the DSM-5 there are 270 million combinations of symptoms that would meet the criteria for both PTSD and major depressive disorder, and when 5 other commonly made diagnoses are seen alongside these two, this figure rises to one quintillion symptom combinations-more than the number of stars in the Milky Way.”
    The researchers conclude that following a different approach may be more effective than remaining committed to what they called a “disingenuous categorical system.”
    THE PROBLEM WITH STANDARD MENTAL ILLNESS DIAGNOSES
    The landmark study mentioned above was led by University of Liverpool researchers who focused on a meticulous analysis of 5 chapters in the DSM-5:
    * anxiety disorders
    * depressive disorders
    * trauma-related disorders
    * bipolar disorder
    * schizophrenia
    Their main findings highlight many of the shortcomings of the current diagnostic paradigm:
    * There is a major overlap of symptoms among diagnoses.
    * Many diagnoses overlook the role of psychological trauma and head trauma.
    * The current approach rarely takes the individual in mind.
    This study’s deep dive into the numbers shows just how murky and inconsistent the diagnostic model is. For example, “There are almost 24,000 possible symptom combinations for panic disorder in DSM-5, compared with just one possible combination for social phobia.”
    Equally concerning is their finding that “two people could receive the same diagnosis without sharing any common symptoms.” And the sheer number of combinations of symptoms makes the ability to arrive at an accurate diagnosis nearly impossible.
    It is thought that 50% of what physicians are taught in school will, in the course of their careers, be proven false.

  • @williamlott7612
    @williamlott7612 ปีที่แล้ว

    So, a researcher should attempt to replicate his own research? Seems to me that would prevent a researcher from jumping the gun. Darlington, South Carolina

  • @gregeconomeier1476
    @gregeconomeier1476 ปีที่แล้ว +2

    Where does funding for replication studies come from? How do you replicate science that requires multi billion dollar equipment?

    • @businessreform
      @businessreform  ปีที่แล้ว +2

      Very legitimate question / observation. As the Reproducibility Project said in their abstract, their is much more incentive to discover than to confirm. Ideally, whoever does the original research would do it multiple times to ensure it was reproducible before publishing but there is very little incentive to do so. To your point, there is also very little incentive for anyone else to do what the reproducibility project has done, especially if it involves expensive equipment or consumables.

    • @gregeconomeier1476
      @gregeconomeier1476 ปีที่แล้ว +2

      @@businessreform So it sounds like the PI confirms his/her own work.

    • @businessreform
      @businessreform  ปีที่แล้ว +2

      Ideally that is how it would work. If the PI did multiple replications of their experiments to confirm the results. They could still be subject to an audit like the Center for Open Science but the audit would be less necessary if the original authors spent more effort confirming their results. And in fairness to the original researchers, I don't think many of them are "cheating" or falsifying results, I think they are just in a hurry and are budget limited so they don't want to spend the time and effort repeating the original work when they would rather publish that work and quickly move on to the next study.