Critiquing scientific studies: Pollution and Stock Markets
ฝัง
- เผยแพร่เมื่อ 5 ก.พ. 2025
- Sign up for the November live simplistics class that starts on Monday! simplistics.ne...
You can use discount code nov95_simplistics to get $95 off!
To take a self-guided simplistics course, visit: simplistics.ne...
To see the paper, visit pmc.ncbi.nlm.n...
Please do more of these! These critiques will help us all become better in our own work.
This is a phenomenal video!! Looking forward to part 2 :)
public peer review... very interesting. Should we have a platform for public video peer review? like a peerScience platform. At least reviewers would be paid by ads.
Interesting...I hadn't thought of that as "peer review" :)
That could also lead to bad reviews. When countries started to pay researchers a bonus for publishing papers a lot of fraudsters got involved.
I’m training to become a statistician and have learnt most of the subjects that were mentioned in this video in the first course on statistics that I ever took (chi square test, contingency tables, Mann Whitney U tests, Friedman tests, etc). But I’ve not used any of then since and consequently barely remember them (also ANOVA - definitely learnt it but hardly remember it). This video made me encouraged to review them. My academic path has been very mathematical and so mostly focused on understanding the proofs and ideas. As a statistician, though, these things need to be readily available to me, so I will definitely put more emphasis on memorization also going forward.
I'm doing an MSc programme on Data science. Your channel has opened my mind in ways I can't describe. Thank you 🙏
This is amazing, I'm learning a lot. Thank you for making this
Love this Dustin. It’s a great way of showing how you would approach it. Promoting your skills and services. ❤❤❤
This was a great analysis of an analysis! Hopefully you plan to do more. Also, I think your background music is awesome!
Thanks!
This is a criminally underrated video
This is so great. Thank you!
Also Idk, I had zero training in statistics during my Master's and ended up doing mixed models after having researched a bit on what should be done to answer my research question. I don't think that "this is what happens when people who are not trained in statistics are expected to do their own statistics". You can still do some reading and double checking and not miss out on obvious things. Even if you're not a statistician, or even if you're not so experienced.
True...though you're probably an outlier :)
😂@@QuantPsych
Great video! Please keep making more like this.
This was wonderfully cathartic.
Cathartic? Interesting.
@@QuantPsych I think he's using that word because everyone is so overwhelmed with information from "scientific sources" that it is "cathartic" to know that many of these people that are predicting doom don't actually know what the heck they're talking about.
@QuantPsych I reviewed many papers with similar concerns, albeit without data for replication. This was the first time I've seen someone do a video review, and greatly enjoyed it.
@shawnshahpari8681 I think he might be russian speaking and might have been watching the lectures of Andrei Raigorodskii who often uses the word "catharsis"
@ if the professor was speaking English, why would we assume this gentleman is Russian?
Wondeful! Please more of it!
I concur with others; please do more of these - this was great. My only point of feedback, which I'm not sure if I'm alone in, is I would suggest you keep your music in the intro and outtro, but it makes it hard to focus (for me personally) throughout the video with the background soundtrack lol
Having been studying statistics from this channel for a few months now, I'm going to read the paper and try to analyze the data based on what I've learned before I watch part 2 when that comes out. It'll be a nice test of analyzing data based on this GLM approach to stats. I'd invite others to do the same!
Very cool idea! Honestly though, I have no idea what I'M going to do yet :)
Excellent video. This is an eye-opener! Thank you.
I want to take the Introduction to Simplistics course you just announced.
Nice presentation. As an earth scientist I cannot image why the links with pollution would exist. This sounds very much like a study that the Irrational (Ariely) would “perform”.
Really good content!!!
Great video !! Looking forward to part 2 and other replication videos
Wow! Very nice video format!
Thank you!
I don’t understand how you haven’t already blown up with views over 100-200k
Overall good critiques of this paper. I would like to see more of this type of content.
Hi! I was wondering if there's a chance you would make a video about survival analysis ((event history analysis eg. first birth, parity progression))
Are you planning on making a DOE playlist anytime soon
Thank you Dustin for this great video! I was just asking myself, how come that this study got published in the first place even though it has so many errors and inconsistencies? I mean it was published in PLoS One and as far as I know it has a decent impact factor...
It took a lot of effort on my part to figure out what was going on, which is something volunteer reviewers may not be willing to do.
Hi Dr. Fife, regarding the different covariates in the mediation model, it’s my understanding you can have different covariates. There should be two different regression equations to make up a mediation model, one modeling the mediator as an outcome and the other modeling the final outcome that uses the mediator as one of the predictors. since there are two different models, each may have its own set of covariates. But, you’re right to be suspicious. The covariates need to be justified as necessary to deconfound the specific estimate. To do so, I favor using directed acyclical graphs a la Pearl. I haven’t heard you talk about DAGs on this channel, but I’d be interested in getting your opinion. They’re one of those things that psychologist seem to know nothing about, although the literature I’ve read makes DAGs seem like an excellent way to determine correct covariates to use. I worry that my reviewers (and coauthors) are going to have no idea what I’m talking about, when I use DAGs, but it seems like the correct thing to do.
A few comments...I'm not sure I agree with you about the covariates. (Again, I'm not an expert in this, so take my rebuttle with a grain of salt). The primary parameter of interest in a mediation model is c versus c'. If the model that contains c has a different set of covariates than the model c', the two parameters have very different meanings. The first (c) would represent the effect of the variable on the outcome, after controlling for the covariates. The second (c') would represent the effect of the variable on the outcome, after controlling for the mediator, BUT NOT THE COVARIATES. They're entirely different parameters.
As for DAGs, yes I agree that psychologists shy away from them. (I actually started working on a paper with another more senior psychologist and recommended we incorporate DAGs and he vehemently disagreed). They're certainly useful, but still limited. They still require the user to think very carefully about their theory. This is a good thing, but also means many will be hesitant to use them. Also, they don't do anything about estimation. When I first learned about them, I kept wondering, "okay...so what software do I use to fit my model?" As I understand it, that's the wrong way to think about it. They don't fit models...they only tell you which models would be useful in estimating causal effects. Not to mention they don't give any information about nonlinear and/or interaction effects. I'm not critical of them and I should use them more, but my focus tends to be more on the estimation side than the theoretical side.
@@QuantPsych Are you not using either of the popular causal frameworks (Pearl's or potential outcomes)? If not, what does your process look like for selecting valid controls?
@@QuantPsych Relevant discussion on structural causal models vs (Bayesian) probabilistic models: th-cam.com/video/X65uihBRB6U/w-d-xo.html
So cool! Do you one from sports psychology?
I'm happy to try one in the future.
If this becomes a regular series, please write one of my papers!
and the background music ??
Wow! Look at the variety of options involved in quantitative analysis... And could these lead to different conclusions? It makes me wonder why quantitative analyses in methodology books are often presented as more objective.
Hello! Can you explain how they violated the assumption of independence?
They measured the same people multiple times, but treated their scores as if they were independent.
Awesome video. Would it be of interest for you to do this with a study by Daron Acemoglu? I would be very interested to know what you think of the (statistical) skills of a recent Nobelist. e.g. the joint work with Robinso: Democracy Does Cause Growth, where they look at the effect of democratic intitutions. they also provide the data.
In college I had multiple professors who were adamant on having a labeled x-axis and legend, I never understood why
That's weird!
Both the x-axis and legend containing the same information? That sounds like a waste of attention.
Another thing which strikes me as a little suspect, but not "weird", is starting with variables that are proportions. I don't know anything about these specific proportions, but proportions are sometimes calculated on a flawed premise that dividing by some variable like population size will make numbers which depend on population size "comparable".
However, it can be shown that this is not always (or even usually) the case. It is a form of "linearity bias": the tendency to assume that relationships are linear. This is an attempt to render random variables conditionally independent, which is not a terrible idea, but simply dividing can be a poor methodology for that purpose.
One class of cases is covered by Lukacs's proportion-sum independence theorem which shows that only under special assumptions would a proportion of the form P = X / (X + Y) be independent from X + Y.
Proportions are sometimes calculated to achieve the same bounds (e.g. being on an interval [0.1]), and this sometimes gives the false impression that all effects of the denominator have been removed. This illusion is strengthened if one doesn't know the difference between a sample (which you can see) vs the population (which you cannot always see).
"putting on your filter" = welcome to corporate world, dr fife
I could have skimmed too quickly, but at a glance it appear that the authors used STATA rather than R. I couldn't say for sure, but some of the differences between the results from the R-based analysis compared to the paper could have been due to different algorithms or implementations.
@QuantPsych , I recommend making your analysis script available (if you have not already). This makes it easier to run your reproduction analysis.
❤
There are special cases where OLS could be used to predict proportions, but that does not appear to be the case here.
My professor
Thanks for posting this critique - surely all of this should have been picked up in the double blind review and the AE is just as accountable as the authors. As always a great explainer video. Please please please can we lose the elevator muzak in the background as a musician it's drawing my attention away from your (as always) excellent observations.
cool video!
Thanks!
Great video! Skip the background music🙏
I do know a fair bit about mediation analysis thanks to Hayes (2023). Yes, any covariates you include on the “C” path must also be included on the “A” and “B” paths as well. Had they used the PROCESS macro (available for SAS, SPSS, and R) it would have taken care of that for the authors. That alone reeks of (I call it cherry picking) p-hacking.
Obviously you’re an expert. The question is simple though: Did the quality of the analysis and any of the simplifications lead the researchers to an erroneous conclusion? That’s the acid test really?
That's ONE question to ask. Because my channel is a stats education channel, the better question to ask is "how did they go wrong with their analyses?" But, I will answer that question too, when I do my own analysis of the data.
@@QuantPsych Very much enjoying your work. It is fantastic to see people taking mathematics and science seriously and importantly sharing their knowledge.
This is excellent. The results in the paper seems like p-hacking to me. It also seems like the authors took one intro to regression course. At least one of the assumptions, their hint, is false. Stock prices that went up yesterday, don't tend to go up today. It seems to me you are being generous to the authors of the paper
I intend to be generous, especially since they voluntarily made their paper open access.
Pardon my ignorance but quite frankly I don't understand the "significance" of such, might I say pretty damn underpowered, experimental studies. The hypothesis itself sounds kind of bizarre, I mean, pollution makes people regret things more? What does that even mean? You can come up with a proxy for any tom dick and harry concept, doesn't mean it's a good one. Then that "regret" somehow significantly affects (not financial decisions in general), but specifically portfolio behaviour? Sounds like they shuffled a box of ideas and picked one XD. I don't know the literature backing such an association but I doubt there is any compelling evidence. No wonder they had to p hack their way out of this mess!
Precisely lol. The majority of titles in, say psychology, are simply of the form: ‘the effect of X on Y through Z’, where X Y Z could seem unreasonably arbitrary (or obvious), making the result practically insignificant. Not much to relish even if they get the stats right.
I don't even understand what they are studying. How did this get funded? Aren't there more important questions than does pollution affect stock market traders?
I'm not one to criticize the topic. There are people out there that are fascinated by frog feces and that's what they want to study. More power to them. But at least do the analysis correctly :)
@@QuantPsych Haha, yeah I agree with people being interested in anything. But usually you have to be somewhat convincing in your proposal to get funding. I doubt they paid these volunteers to be deliberately polluted with their own money.
@@QuantPsych 💯
@@InfiniteQuest86 Totally agree! The proposal itself has to make sense if it's going to make use of taxpayer money.
Wait, youre just telling now that the 'T' on my bar graphs arent the same as a dotted 'i'??? Next youll tell me my pinkie cramps at tea-time dont actually make me fancier
Sorry, not following what you're saying.
Y'know, the video is cool and all, but I've noticed it always looks like you just finished a workout
why, cuz I'm SWOL!!!!
Actually, i don't know where to start with my comment, and that's rare for me.
First, as a Non-Statistician, I could actually follow your "issues" with the paper.
Next, I encounter this hodge-podge stuff all the time. Let me give you a "light" example; I once had a boss who said he likes statistics because of all the cool squiggly symbols. 😳 Yes, what a mess!
The common denominator, in my experience, is that people want to sound bright and smart 🤓 so they do Stat Analysis and talk Stat Talk to LOSE others despite their being totally lost and totally untrained.
Another problem I encounter is that folks don't understand the structure of the data and variables they're using.
Yet another challenge is BLINDLY applying Statistical Models without knowing the underlying assumptions.
And last, but not least, untrained folks keep throwing axes until something sticks to support whatever they're trying to achieve, and they go with that regardless.
I'll close by saying that SPREADSHEETS can be seriously dangerous when performing Statistical Analysis because people use them to do MIRACULOUS STUFF that can't be traced! BAM!
Yes! I think the biggest problem is the last--there's no record of what went on! I had hoped the formulas were still intact in the spreadsheet, but they were not.
@QuantPsych To inject a bit of humor into a very serious problem, I'd say - better bring out your Crystal Ball🔮 ... or just go ahead and cry.
Spreadsheets are a horrible choice for data analysis if reliability and reproducibility are priorities.
@@galenseilis5971 You've said it better than I could ever say it. Additionally, I can tell that you're speaking from experience, and I'm going to guess that your words of wisdom come from your vast experience with Debugging or Reproducing Code or Results.
Thanks again for your wise comment. 👏
This is instructional because you reviewed a very bad paper. Obviously the peer review process failed. This is not simply bad statistics, it is bad science.