An easier way to do sample size calculations
ฝัง
- เผยแพร่เมื่อ 20 ก.ค. 2024
- You just got to know a little bit of code. The code shown in this video can be found at: github.com/very-normal/explained
Stay updated with the channel and some stuff I make!
👉 verynormal.substack.com
👉 very-normal.sellfy.store
I've been shocked at how broadly useful Monte Carlo approaches are in general.
I remember one problem I spent weeks figuring out the correct way to solve an issue, by that point it had been so convoluted to figure out, I decided writing a Monte Carlo simulation to verify I hadn't made a mistake would be smart.
The simulation got the exact same results out to three decimal points, and took about 10 minutes to write.
The other great thing about Monte Carlo simulations is they make all of your assumptions exceedingly clear, while equations tend to obfuscate your assumptions.
Right?? One time I was doing a technical interview for a company, and they gave me a probability question. I didn’t know how to do it but they also gave me a code editor, so I just ran a quick simulation and got the right answer from it
Yes. I found this method myself after being unsatisfied with traditional power analysis. What's nice is how flexible it is, and how it can be used to quantify and challenge assumptions you have about your data / population.
this video comes exactly at the right time for me as I'm trying to run a power analysis for maximum likelihood fitted sigmoid functions and I was really running out of ideas :))
Just one step away from using a Bayesian approach :-)
Yup, this is standard practice in particle physics. Eventually the technicality boils down to the modeling of the physical process being investigated, which may involve hundreds of gigabytes of equations. One of the reasons that this is necessary is that there can be signal-background interference in the particle physics processes. What also makes it extra worth it is that the same MC simulation will be used again during data analysis when the actual data collection reaches a checkpoint.
Hopefully, the day-to-day business applications do not often involves complex modeling, and formulae for rough estimations may still be the most economic, especially when the signal and background do not interfere significantly. However, when heavy machiniary, such as MC simulation based on complex model is built, its value can exceed mere advising on sample size. For example. After the statistical analysis with real-life data is done, if the business wants to improve its operation, the model and simulation can be adjusted to provide outlooks for the improvements being considered.
How are these videos so well made?!
Undergrad in math here, love this!
great video man. really enjoy brushing up on my skills via your channel.
This is similar to a method I use to show why we "fail to reject the null" instead of just rejecting it. If we change the criteria from the confidence interval not including the null to simply the p-value, then plot the returned sims as a histogram, we see when the null is actually true the p-value is simply a uniform random variable. The "falser" the null becomes, the more right tailed our p-value distribution becomes.
library("foreach")
sims = foreach(i=1:10000, combine = c) %do% {
groupA = rnorm(30, mean=0, sd=1)
groupB = rnorm(30, mean=0.125, sd=1)
test = t.test(groupA, groupB, conf.level = 0.95)
result = test$p.value
}
hist(unlist(sims), freq = FALSE)
That’s a great demo, I think I might use that for future TA office hours
I love your videos!!
When i was thinking about creating an statistical test, I thought about doing the same to find out how powerful my test could be!
I personally still prefer deriving the sample size needed for my estimators from concentration bounds given a certain level of control, which makes more intuitive sense to me. But I also like having other tools in my belt, so thank you for the video, great as usual 😀
This channel is so underrated🔥
Thank you for doing your part.
You make good videos. Keep it up!
Very helpful, thank you! In your code you should replace the magrittr pipe: %>% with the new native pipe in R: |>
Just a thought for future videos, so that no one gets hung up with an error that "%>% doesn't exist" if they don't load the tidyverse.
I just learned how to replace magrittrs pipe with the native pipe for the keyboard shortcut. Will do, thank you!
This video is a blessing.
Hey! I saw that simulations are used to estimate sample size for mixed models too, but it seemed a bit more complex. If you'd like to make a video on that, it would be super super useful :)
One plus to the mathematical formulae (which are not always equations but sometimes also inequations) is that they are computationally fast. A Monte Carlo simulation requires more electrical power than most formulae. The downside of the formulae is primarily that they can be very technical to obtain in the first place and they are only known to be valid under the assumptions they were derived. What's the electrical power cost of spending some length of time working on a formula? I don't know.
"you can't" - I cackled
can you do a video of renewal processes or renewal theory it's rare to find videos about it,i would really appreciate it.
Hi, what books do you recomend for begginer, intermediate and advance levels on stat?
Idk if you saw my reply from the other video but.. Could you possibly look into doing a video on set theory? I feel as if that is a foundation on making statistics more accessible as it is a whole different language from basic xyz variables.
might as well to do a high level overview of measures at that point
The hard part is that you don't really know the true effect, and it heavily affects the sample size you need to get the same confidence
Yeah, the best you can really do is check your sims against a range of realistic effect sizes, and it gets computationally expensive fast
StaTiSTiCS iS aLl liEs!!! Great vid
Awesome vid. Makes me wanna go monte carlo something.
But what's this about plugging variables into a function from some 95-star opensource library which magically gives us the right numbers? What's it do? Where's the code for it? Feels incomplete.
It’s a popular library for power and sample size calculations. The code for it is there, but I didn’t give it a lot of spotlight cuz it would mean I’d need to explain it.
It takes a bit to learn, but it gets the job done once you know how it works. It uses a different method for getting the sample size, which even I don’t entirely understand after reading their source code. Could be a future video topic
What do you use to create your videos? Manim? What video editing program?
I use Final Cut Pro for the editing, and manim to produce my plot and equation animations
@@very-normalthanks! I love your work, great stuff!
Amazing videos!!
Congratulations for the lessons and how easy is to understand you :)
I am thinking about sampling from one population and test the hypothesis like a bootstrap method also as a kind of sample size lesson.
Is this approach right?
Thank you very much!
Do you have some suggestions for testing differences between the variances instead of the mean following the same idea?
You could replace the t-test with the F-test to test the ratio of variances, and then alter the two variances in the data generation. I’m not sure about case for a difference of variances though, I’m not aware of a hypothesis test for that.
@@very-normal I will definitly give a look on that solution. Thank you very much for your reply and congrats for the nice job!
You should make a class on Udemy covering how to use statistics for different job titles. Maybe partner with ZeroToMastery? A lot of us have degrees but have to make a shift to a new field that needs more statistics. I'm biology but now I'm going towards Business Analyst and Project Management. I need help connecting the theory to the business world. Coding examples that use SQL, Python, and R is needed, too.
That’s always been a vague feeling I’ve had about my audience.
One of my long term goals is to ultimately make courses, but it feels very different from making these TH-cam videos. Thanks for the input, it definitely helps sculpt what I’d think about including in a course/s
Quick question here. With the MC approach, we need to know the difference we are looking for right?. In this scenario you had 0.5 as the difference to create the second sample and apply the test afterwards. Should we always try to have a specific difference in mind before running an experiment? Or how could I approach this issue if I'm not sure what difference I'm expecting. Thanks for the content!
Yes, you’re right, you’ll need to specify a treatment difference to do the simulation. The specific value of this difference will depend on your context. But since we don’t know what this difference is beforehand, it’s usually good practice to simulate a range of values that might span “minimally effective” to a large effect. For example, with a binary response, I might simulate new treatments that have a +10%, +20% and +30% increase in response rate over placebo. To get concrete values, you might have to read previous papers or consult a collaborator/expert on what are ranges they might consider
Are there distribution free methods for generating the data? If not, then I don't see the advantage of this approach over formulas. I guess this works even if you don't know the formila for the particular distributions you're working with.
I suppose you could do a bootstrap-type thing with a dataset you collect, but I’m not 100% about that. And yeah you’re right, from my own experience, they’re a boon for sample size calculations for complicated experimental designs where the distribution won’t be clear
Can your recommend some books for someone who knows nothing about statistics? Where would you start if you had to start over?
Yeah I can try to think of something. It would help me to understand your goals for learning statistics, can you tell me a bit more about them?
@@very-normal I come from a physics background, so basically anything that has to do with experimental science and data. Also, I know that probability is connected to statistics in some way but I’m not sure how to breach into that either. I am curious for the sake of quantum physics
Monte Carlo simulations feel like cheating or just dumb brute force, but I guess it's not if it works, which it does obviously.
That’s exactly how I felt when I first learned about it
why did you choose an alternative with a difference of 0.05?
For that example, my variance was one and I wanted an example that gave me relatively low power compared to 80%
Very gauss!
Can you do a video on the bootstrap? Unless you already have
I do have one on the theory, but you reminded me that I haven’t gone back to do the code demo for it! Future video!
@@very-normal awesome. Also, could you do a video on nonparametric regression. Just the idea behind it? Maybe compare it to the parametric regression case?
It dropped. Time to fire up obsidian and take some notes.
obsidian users RISE UP
I tried to run this code translated to julia to see how much faster
Let's just say that "waiting for it to finish" is a non issue
Ooh always looking for a tool to let me wait less
@@very-normal julia has some growing pains associated with it, but for operations like these its perfect with it's simple syntax and JIT compilation
@@jeffreychandler8418it's better to let him choose to tools he needed
Aren't we now going from an arbitrary sample size to an arbitrary mu_A - mu_B?
That’s also something that can varied in a Monte Carlo simulation. If you fix sample size and vary the true difference instead, you can plot the power function. But in this video, the true difference is fixed to 0.5, and the sample size is varied
i wish I were able to understand everything and to apply this kind of thing.. :S. I know statistics in a more broad way.
I feel dumb
Give it some time! Statistics is far from easy or intuitive. I didn’t understand a lot of it at first, even into my graduate studies. You’ll get it with time
I knew the answer from the top of my head without the use of a laptop does that mean I’m better than the half grads?
yes it does, im proud of you
@@very-normal t...thanks dad😭
1:50 - well man, in that case there should've been one person that has spoken.
You didn't had to come up with an answer you could've just said you knew but you needed to pull up a computer to calculate.
Maybe you could've simplified the method to quickly explain what had to be done and that would be a satisfying answer in the middle of a class.
missed opportunity for my past self
I'm sharing this.