Here's the link to the code: github.com/StatQuest/logistic_regression_demo/blob/master/logistic_regression_demo.R Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Hi Josh, Love your content. Has helped me to learn a lot & grow. You are doing an awesome work. Please continue to do so. Wanted to support you but unfortunately your Paypal link seems to be dysfunctional. Please update it.
Your videos never disappoint, Sir. I have gone through many of them and think you've earned the right to brand the phrase: "clearly explained" because your explanations are indeed very clear. I am building a better explanation of statistics thanks to you. I appreciate you and hope you continue to pass on the knowledge.
Thank you so much Josh for all these videos! I got Aplus for most of my stat courses quite a few years ago when I was doing my MSc of BIostat, but it took me quite some time to come up with a better understanding of a few concepts. You just summarized and presented these ideas and more in a few minutes! You are a genius and on top of that, you are so Kind to share all these work to everyone for free! With my limited vocabulary, all I can say is THANK YOU! It makes me feel the world is a beautiful place with beautiful mind and soul. I love your song “hello”, it reminds me of the day I met my daughter and brought happy tears to my eyes :)
You are an absolute life saver. My data science paper is due in two days and now I have my pretty log graph and I understand this better. DOUBLE BAM!!!!!
Josh, it’s Saturday morning here and I’m enjoying a cup of Bam! learning R from the best teacher on the planet. I’m so grateful and appreciative of your efforts to share your considerable talents with us!
Your simple English explanation of the meaning of "Intercept" in the output from 8:30 to 8:38 of this video was something I could not find after searching for 2 hours. Thank you!
What is great with your video is that even if I forgot my headphone I am able to follow the video in the computer room full of other students! Thank you so so so much !!!! From University of Bordeaux
Thanks you very much for all stuff. You have saved me to fail my exams. Amazing quality channel Unbelievable the low number of likes. Very appreciated channel, at least for me. Thanks again.
Both my husband and I learned so much from ur video. ( inspired by the top comment), whenever you come to Toronto let us know for a few free accommodation in our Asian restaurant/bubble tea surrounded neighborhoods (north York center)! Thx again! Xin
I was just here for the logistic regression but bam!! I would be watching all of your videos. As a ds learner using r, double bam!!!, your videos will surely help big time! Bambambam! 👌😅 Thank you. 🙂
good looking white background... graphs are beautiful... whatever you say, you write it on screen.... your sound and sound system, very good.. the way you explain things, CLEARLY EXPLAINS everything.. and loved that music part and BAM!!! and here, i have something to say about your work.. and that is VERY BIG BAM !!!... good luck.. keep growing..
Hoooray! We made it to the end of an exciting journey through logistic regression! Hope you have a nice day, and thank you for understanding the output for logistic regression in R, which really cant be understood thoroughly without watching all the logistic + odds videos!
Josh, joining all the folks here in thanking you! I have a question: around minute 9:05 you talk about odds of having being unhealthy for a female. How do we know that these are the odds of being unhealthy vs being healthy? I feel I am floating when it comes to intercept, reference categories, and baseline categories. Thanks a lot!
R orders factors ("healthy" vs "unhealthy") in alphabetical order. So that means "healthy" is first, and the default, and "unhealthy" is the difference from that. Likewise, "sexF" and "sexM" are ordered alphabetically, so "sexF" is the default value and "sexM" is the difference from that.
Thank you so much for this effort really appreciate We need a stat quest on three topics: 1-Chi-square test, 2- The Hosmer-Lemeshow goodness of fit test for logistic regression. And 3- Iteratively reweighted least squares (IRLS) by using Newton's method. If you don't mind :) of course. Can you tell us about the title of next video?!
The Chi-Square test is on the list. I've looked into the Hosmer-Lemeshow fit... Can you tell me what you think about the limitations? Specifically those mentioned in the wikipiedia article about it? en.wikipedia.org/wiki/Hosmer%E2%80%93Lemeshow_test#Limitations_and_alternatives And iteratively reweighted least squares is also on the list. However, up next are some basic statistics videos and then videos on lasso, ridge, and elastic-net regression.
the Hosmer-Lemeshow statistic was used to avoid problem in Pearson chi-squared statistic which was when observations being grouped by the values of the x variables, the Pearson chi-squared goodness of fit test cannot be readily applied if there are only one or a few observations for each possible value of an x variable, or for each possible combination of values of x variables. (A sample with a sufficiently large size is assumed. If a chi-squared test is conducted on a sample with a smaller size, then the chi-squared test will yield an inaccurate inference). So in the Hosmer-Lemeshow statistic, the observations are grouped by expected probability. But there is very little guidance on selecting the number of subgroups. The number of subgroups,g, is usually calculated using the formula g> P + 1. For example, if you had 12 covariates in your model, then g > 12. How much bigger than 12 g should be is essentially left up to you. Small values for g give the test less opportunity to find mis-specifications. Larger values mean that the number of items in each subgroup may be too small to find differences between observed and expected values. Sometimes changing g by very small amounts (e.g. by 1 or 2) can result in wild changes in p-values. As such, the selection for g is often confusing and arbitrary. Also, it doesn’t take overfitting into account and tends to have low power. For these reasons, the Hosmer-Lemeshow test is no longer recommended. Am I on right? Is it enough cues to no longer used of HL test? I have another question, ( Overfitting is happening when your sample size is too small. If you put enough predictor variables in your regression model, you will nearly always get a model that looks significant. While an overfitted model may fit the idiosyncrasies of your data extremely well, it won’t fit additional test samples or the overall population. The model’s p-values, R-Squared and regression coefficients can all be misleading. Basically, you’re asking too much from a small set of data.) If I have a small sample, is there any problem to use Maximum likelihood to fit model and McFadden's pseudo-R squared? Is there any rule to chose the number of sample for any regression? Sorry for the many of questions, it is my first year in biostatistics. :)
These are all great questions. You are correct about the HL test and you are correct about overfitting. There are, however, lots of tricks you can use to compensate for overfitting (lasso regression, ridge regression, elastic net regression etc.) One way to test to see if you have a model that is "overfit" is to use cross validation. As for a minimum number of samples for logistic regression - people often say "10 samples per level of each discrete variable". It's a general rule of thumb and it doesn't always apply. However, again you can use cross validation to verify if you have enough samples or not. Cross validation is a very practical tool!
Thanks for the video. Your video made it look like so simple. I request you to upload a video of how to get risk ratios in multiple logistic regression model.
Hi Josh, thanks for your videos they are very easy to understand. Really appreciate your efforts. I believe I speak for many, Because of you many people are able to understand with utmost clearity and you cover all the small details with super ease. Keep up the Nobel work. Cheers 👍 Would it be possible for you to put up a video on model evaluation i.e. determining cutoff and model performance. Thanks
Since the values or 0 and 1, it probably doesn't matter. However, to be safe, it's probably a good idea to make all categorical values, regardless of their values, factors.
At 11:30, the video states, "Since we are not estimating the variance from the data (and instead deriving it from the mean) it is possible that the variance is UNDERESTIMATED." Q. How can we say that we are UNDER-estimating the value of the variance? BTW, awesome vids, music man! ;)
@@statquest It's 1am but let me see if I got this straight... Due to the nature of discrete functions (like logistic functions) they do not always vary smoothly. With discrete functions it is possible to see variances (and their corresponding probabilities) differ from (in our case) the proposed logistic model. In other words, it is conceivable to have a leptokutic or platykurtic distribution. It is possible to see probabilities which differ from the expected probabilities due to the fact that the "real" model may be different and/or the samples may not be i.i.d. As it happens, the Bernoulli distribution tends toward the platykurtic. ...It's just the wrong dang model sometimes...
It's actually a little simpler than that. With binomial data (like logistic regression) we estimate the mean value = number of positive responses / total number of responses. Once we have the mean value estimated, we use that, and that alone, to calculate the variance. In other words, once we have calculated the mean, we do not need the data anymore to calculate the variance. This is in contrast to linear regression (or a lot of other things) where we estimate the mean with the data and then use the data again to calculate how it varies around the estimated mean. Thus, there is a possibility that with Logistic Regression (and other "generalized linear models") we did not correctly estimate the variance since the data were not involved in that calculation. If we over estimate the variance, that just makes the calculations more conservative and, generally speaking, that's not a problem. However, if we underestimate the variance, then that means we're more likely to say things are significantly different even if they are not, and that's no good. So the dispersion parameter takes care of that.
Great tutorials, I started with your PCA video and since then hooked onto other videos . Could I request you to do a video on various types of probability distributions when to use them.
Those are all in the works. I wish I could work 2 or 4 times faster than I can. I've wanted to cover the major probability distributions for over a year, but got sucked down a machine learning path and now feel spread pretty thin. However, these will happen eventually! :)
I like the way you presented the information in such a manner that is easily understood I have 2 questions 1. While doing the xtab, what do we need to do if we found that say it is either or both healthy and unhealthy under cp3 is 0 or very minimal (video clip at 6:04) 2. At 15:55 of the video clip , you mentioned about using cross validation to get a better idea of how well it might perform with new data. Do you have a separated video which is specifically for that topic ? Many thanks Williams
1) Unfortunately I don't understand what you're asking in this question. However, I think you are asking what do we do when one level from a categorical variable does not have strong preference for healthy or unhealthy or doesn't have much data to begin with. It really depends. You can just try it and see what happens, but you might also try removing the variable and see if that improves predictions. 2) I have a video on cross validation here: th-cam.com/video/fSytzGwwBVw/w-d-xo.html
@@statquest Thanks for your prompt reply let me clarify my 1st question at the video clip at 6:24, you mentioned that there are 4 patients represent level 1 under restecg category. My first question is, why only 4 can cause problem? is it because it is too mininal compares with others?(Level 0, and Level 2). How do I know exactly that it is causing the problem when I do the analysis? and if it does cause the problem. how to go about fixing it? just remove the Level 1 can solve the problem? Thanks for your help
@@williamstan1780 When you don't have much data supporting a specific category, then chance are it will have a lot of variance - in other words, further samples may be very different from the ones in the original dataset. You can test this with cross validation (use some of the data to fit the model, use the rest to see how well it performs). If things are no good, you can remove the variable, or try to lump categories together.
Hi Josh Firstly many thanks for your videos on this topic. I have noticed very odd and conflicting results between R and SPSS with regards to entering factors (with more than 1 level) into a logistic regression model. SPSS produces a simplified output containing an odds ratio with 95% CI and p-value, for each individual variable entered into a logistic regression model (rather than the factor levels, as displayed in R). In R - I have not found a good way to do this. I have used the logistic.display command as well as exp() to get odds ratios, but they do not provide an overall value like in SPSS (instead, listing these for each individual level within the factor). Do you have any idea why SPSS and R handle logistic regression differently like this? All I would like is a similar output to SPSS - where I get a single odds ratio, 95% CI and p-value for each individual factor variable entered.
Unfortunately I've never used SPSS so I'm not really familiar with the problem you are having. That said, perhaps this will help: stats.stackexchange.com/questions/543540/different-output-for-logistic-regression-between-r-and-spss-how-to-get-correct
at the end where you make the graph , you could have used the broom package and augment function to create the data frame to compute the fitted and actual values.
Awesome! Thank you so much! Please could you do a video about conditional logistic regression like clogit in R with result interpretation and how it works when using adjusted parameters.
I should have been more clear. Unless you tell R not to treat strings like factors, it does that by default. So, at that point "ca" is already a factor. The problem is that "?" is considered one of the levels of that factor. We can drop that level, or we can convert it to integers and then back to a factor. For some reason I did it the latter way. It would have been better to just drop the "?" level.
@StatQuest with Josh Starmer, After plotting the logistic regression graph in 16:47, how to do Confusion matrix? I searched Stack overflow and tried various Confusion Matrix codes but no luck, I get errors and couldn't make the matrix.
@@statquest I'm trying to uze a logistic regression model on a set of binary events. Each with a different probability of happening.. and I have no idea what I'm doing haha.. so I'm loading up on coffee and I'm going to start your videos soon
Hi, can we use a chi squared test first on (in this example), heart disease and sex then use a glm model on those 2 variables? I tried it with my data and the p values were different although both were significant. Afterwards I modelled all the other independent variables to check for confounding. I found some of them were significant along with the same variable in the chi test I did. In 13:47, since the output only says sexM, does that mean the P value 0.002503 only accounts for males?
The chi-square test and Wald test (used for logistic regression) are related are fundamentally different tests so it doesn't surprise me that you got different p-values, but because both tests are intended to work with the same data, it doesn't surprise me that they both were significant. As for "sexM" at 13:47, that means that both labels, "male" or "female", are useful for predicting heart disease. The reason it is called "sexM" is that "males" get a '1' for that variable, meaning that it is true that they are males, and females get '0', meaning that it is not true that they are males. For more details on how design matrices work, see: th-cam.com/video/CqLGvwi-5Pc/w-d-xo.html
Hi Josh, thank you very much for your teaching, I have really learned a lot over the past few days. May I ask a (stupid) question...At 8:45, you mention that you predict heart disease for a female patient. However, how can I tell whether it means health or unhealthy? I remember you set the hd variable as 2 level: health and unhealthy. How do I know which level we are predicting? In the github page, you tell us that: "## The intercept is the log(odds) a female will be unhealthy. This is because female is the first factor in "sex" (the factors are ordered, ## alphabetically by default,"female", "male")".... Does this alphabetical order also apply to "healthy" and "unhealthy"? I am not sure why the intercept is not predicting "healthy" female patient because "H" (i.e., healthy) comes before "U" (i.e., unhealthy). This part really confuses me. Thank you!
At 3:17 we code "healthy" as 0 and "unhealthy" as 1. Thus, the "base" or default is "Healthy" (since 0 comes before 1) and we are predicting the log(odds) that someone is different from the base.
Hi Josh, Firstly, forgive my ignorance. Can I refer to null and proposed LL results in 14:12? You mentioned that to pull from the log-likelihood, we need to divide the scores by -2. Can you please elaborate how do you get the -2? Additionally, in 14:46 do you use 1-pchisq to get the upper tail? If my assumption is wrong, can you please explain the 1-pchisq part? Thank you heaps
Your first question is answered in my video on Saturated Models and Deviance: th-cam.com/video/9T0wlKdew6I/w-d-xo.html For your second question, the answer is "you are correct!". :)
Is there a video explaining the 3 points on right of slide at 11:30? I watched all of the linear/logistic videos and my understanding is these regressions fit a line to predict data given new x. I don't understand what has this got to do with estimating mean/var. Is estimating mean/var part of the fitting process and important to creating the model?
14:37 is this the effect of all those variables on the response of just the significant ones? When building a linear equation out of this , we need to only include the variables with low p values on the RHS and can omit the others as they do not have as much impact right? 16:36 is it possible to get avoid getting differing number of rows for the number of fitted values vs response…it keeps appearing like that for me
Hi, I really like your videos, every topic is as clear as water after watching it. I've watched this one and also the three videos about logistic regression's details. If you want to go further in this topic, you could do a video explaining emmeans package for R. Many people, including me, would understand post hoc tests for glm using emmeans, if someone like you explained it. Thank you!
Great salute! If you can, please post a video on all machine learning models with a large dataset example implementation in r with clear intuition and mathematics statistics behind it. Thanks.
Hi Josh, Thanks for the really helpful video! Referring to the clip at 15:20. Whilst I know plotting a Predicted Y for a range of X values (say in a simple univariate logistic regression) we would expect to see that S shape. But for a multiple variable regression (as in yours) should the index of probabilities when ranked and plotted as you've done here always have to be in that S Logistic Shape? I am getting more of an exponential curve between 0 and 1, and can't tell if this means I have done something wrong/have something wrong with my model?
Hmm...I'm not sure. Your graph should definitely taper off as the predicted probabilities get closer to 1, but how visible this tapering is might depend on how many data points you plot.
Great video, thanks so much Josh! After the 4th minute you mention how to address the NA samples. Can you teach us the RANDOM FOREST method, if we don't want to get rid of our NA samples (e.g. in multivariate cases, where the rows include other useful info)? Thanks!
I cover the random forest method in this video: th-cam.com/video/6EXPYzbfLCE/w-d-xo.html (the theory is here: th-cam.com/video/sQ870aTKqiM/w-d-xo.html )
Hi, I love the way you explain all this things! I have a couple of questions. I observe that it's necessary to establish a code type for the predictors, if these are dichotomous, for example, they are assigned 1 and 0 (in the example male / female), so: - How should we proceed with polytomous predictors? - What results of the model should be reported in a scientific article? Thank you in advice and keep doing great content!
1) For all categorical data (with 2 or more classes), just make sure you are storing it in a factor. 2) That depends on the journal. I would look at other articles in that journal to figure it out.
All of your videos are great and fun to learn from! Could you please upload a tutorial on mediation analysis using STATA and R (using the mediation package)?
These videos are so amazing! Do you have a suggestion for a book that explains Logistic Regression to newbies? The videos are super awesome, but extra references may help too. Hopefully you will write your own book soon! Thanks!
I know this is probably 10 months too late, but the book “Introduction to Categorical Data Analysis” by Alan Agresti is a great book. Does a really good job explaining logistic regression and is pretty light on the math.
Hi Josh, thanks for this amazing tutorial. Would you be able to add something interactions between predictors and random effects? I am trying to run a mixed-model logistic regression and have three-way interactions but not entirely sure on how to deal with them. Thanks so much :)
Thanks for the detailed and super easy explanation, Josh. I'd like to ask you, shouldn't we check the below items before regression? 1. Would it be better if we use the WOE or log(odd) of a variable rather than raw variable (for example gender -> dummy). If I'm not wrong to use a dummy variable in the model is not a good choice; 2. Correlation between variables; 3. Factor transformation ; 4. PCA analysis. And as a result in order to calculate the probability from log(odds), we should use Sigmoid function? I mean transform log(odds) to the probability Thanks in advance!
If you want your model to be interpretable - in that you can look at the parameter value and make sense out of them - then removing correlated variables is a good idea and factor transformation and PCA can help with that. On the other hand, if you want to use your model to make the best predictions, then correlated variables are fine and can improve predictions. If you are interested in the details behind Logistic Regression, then check out these other StatQuests: General Overview: th-cam.com/video/yIYKR4sgzI8/w-d-xo.html Interpreting Coefficients: th-cam.com/video/vN5cNN2-HWE/w-d-xo.html Fitting the Model to Data with Maximum Likelihood: th-cam.com/video/BfKanl1aSG0/w-d-xo.html Calculating R-squared and its p-value: th-cam.com/video/xxFYro8QuXA/w-d-xo.html
Excellent video, very clear and easy to follow! Do you have any videos that show how to do best subsets and cross validation with logistic regression on R? I know you have a video that explains the concept of cross validation but I am looking for a video like this that goes through it step-by-step for logistic regression on R. Same thing for how to run all possible models (best subsets) using logistic regression on R. I have found one by another youtuber for linear regression but not for logistic.
Here's the link to the code: github.com/StatQuest/logistic_regression_demo/blob/master/logistic_regression_demo.R
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Hi Josh,
Love your content. Has helped me to learn a lot & grow. You are doing an awesome work. Please continue to do so.
Wanted to support you but unfortunately your Paypal link seems to be dysfunctional. Please update it.
Your videos never disappoint, Sir. I have gone through many of them and think you've earned the right to brand the phrase: "clearly explained" because your explanations are indeed very clear. I am building a better explanation of statistics thanks to you. I appreciate you and hope you continue to pass on the knowledge.
Wow, thanks!
I dont understand why you used both categorical for logistic regression??
7:00
This 89-year-old guy says BAM!! So clearly explained, indeed. DOUBLE-BAM!!!!
BAM!!! And thank you for your support!!!!
where have you been my whole thesis! thank you!!
Hooray! I'm glad to help! :)
I feel the same!! hah
Thank you so much Josh for all these videos! I got Aplus for most of my stat courses quite a few years ago when I was doing my MSc of BIostat, but it took me quite some time to come up with a better understanding of a few concepts. You just summarized and presented these ideas and more in a few minutes! You are a genius and on top of that, you are so Kind to share all these work to everyone for free! With my limited vocabulary, all I can say is THANK YOU! It makes me feel the world is a beautiful place with beautiful mind and soul. I love your song “hello”, it reminds me of the day I met my daughter and brought happy tears to my eyes :)
Thank you so much!!! I'm really glad you like my videos and my music. :)
I just wish one day all this information actually stays and sticks to my mind... thank you thought! Your videos are amazing!
Thanks for watching!
You are an absolute life saver. My data science paper is due in two days and now I have my pretty log graph and I understand this better. DOUBLE BAM!!!!!
Hooray!
so, how did it go today?
Josh, it’s Saturday morning here and I’m enjoying a cup of Bam! learning R from the best teacher on the planet. I’m so grateful and appreciative of your efforts to share your considerable talents with us!
Thank you very much! :)
You will surely be in my Thesis acknowledgments. Thank you for making our lives relatively easier but truly more ineligible. BAAAAAM!!
Thanks so much! :)
Your simple English explanation of the meaning of "Intercept" in the output from 8:30 to 8:38 of this video was something I could not find after searching for 2 hours. Thank you!
Awesome!!! Now that you have that concept down, a lot of other stuff in statistics should make more sense. (At least I hope!) :)
What is great with your video is that even if I forgot my headphone I am able to follow the video in the computer room full of other students! Thank you so so so much !!!! From University of Bordeaux
Solal Sténou Merci!! :)
Your videos are great! It's also so nice of you that you take the time reply to so many of the comments here !
Thank you!
Thanks you very much for all stuff. You have saved me to fail my exams. Amazing quality channel Unbelievable the low number of likes. Very appreciated channel, at least for me. Thanks again.
Wow, thanks!
Both my husband and I learned so much from ur video. ( inspired by the top comment), whenever you come to Toronto let us know for a few free accommodation in our Asian restaurant/bubble tea surrounded neighborhoods (north York center)!
Thx again!
Xin
Hooray!!! That would be awesome. I will dream of the day I can visit you in Toronto. :)
Thank you for saving my study. Not gonna lie, this video made me cry. I was about to drop out because of statistics, but this saved my project.
Hooray!
Its's 1:11 AM and what I am doing is DOUBLE BAM. Thank you for this awesome video. U are hero.
Thanks! :)
I was just here for the logistic regression but bam!! I would be watching all of your videos. As a ds learner using r, double bam!!!, your videos will surely help big time! Bambambam! 👌😅
Thank you. 🙂
Awesome! Thank you!
Thank you so much! I've a stat project to do in R with logistic Regression and this simplified the coding portion so much!
Hooray!
Doing a masters program on analytics and this video made more sense than all the lectures combined on logistic regression. thank you
Thanks!
I really enjoyed the clearly way to explain us this topic. So many thanks for the teaching!!!
Thank you very much!!!
Nice channel to land on! Happiest discovery of my 2020! Great job!
Thank you! :)
"one last shameless self promotion" got me 😂😂😂.....that's why I love your videos, u make learning stats fun
Hooray! Thank you! :)
Your videos cover everything in my course and I wish I found you sooner! So much detail and clear explaining in such little time
You sir deserve a promotion 👏 thanks for this incredibly helpful video
Thank you! :)
Thanks for also showing how to wrangle data and explore missing data in a simple helpful way ❤
My pleasure 😊
I dont understand why you used both categorical for logistic regression??
7:00
The outcome is dichotomous.
good looking white background...
graphs are beautiful...
whatever you say, you write it on screen....
your sound and sound system, very good..
the way you explain things, CLEARLY EXPLAINS everything..
and loved that music part and BAM!!!
and here, i have something to say about your work..
and that is VERY BIG BAM !!!... good luck.. keep growing..
Thank you very much! :)
You are an amazing teacher. God bless you!
Thank you! 😃
Great job bro.
Gratitude for your help. You also have where to stay if you come to Uganda (Africa).
Thank you very much!!! :)
I recommend all the videos by stat quest with Josh Starmer. Thank you for your good explanations.
Thank you very much! :)
Thank you so much for this video! I've been suffering with the coding for my project but this really helped. You're a star!
Thanks!
Hoooray! We made it to the end of an exciting journey through logistic regression! Hope you have a nice day, and thank you for understanding the output for logistic regression in R, which really cant be understood thoroughly without watching all the logistic + odds videos!
Yep, that is correct. That's why I made all those other videos first - the output is jam packed with stuff.
THANK YOU! somehow I couldn't find any websites explaining this
Glad you found it.
Josh, joining all the folks here in thanking you! I have a question: around minute 9:05 you talk about odds of having being unhealthy for a female. How do we know that these are the odds of being unhealthy vs being healthy? I feel I am floating when it comes to intercept, reference categories, and baseline categories. Thanks a lot!
R orders factors ("healthy" vs "unhealthy") in alphabetical order. So that means "healthy" is first, and the default, and "unhealthy" is the difference from that. Likewise, "sexF" and "sexM" are ordered alphabetically, so "sexF" is the default value and "sexM" is the difference from that.
Hi, Josh. I cannot thank you enough for these videos... Would also be good to have a similar video in Python..
Great suggestion!
@@statquest where's the video sir in python sir?
lets goooo StatQuest
bam!
It must be so much fun working with you! Thank you for this tutorial. =)
Thank you! :)
I am impressed, you are talented, thanks for your sharing your knowledge.
Thank you! :)
So helpful, thanks!
Whenever you come to Cyprus let me know for few free accomodations in our mountainous region, Marathasa!
Thx again!
Γ
Wow! That sounds awesome!!!
@@statquest
oh yes!
I owe you a lot - you saved me so many hours!
Γ
A small request, you have done a lot already, a big thank you for that. Is it possible to make a video on Logistic regression in Python ?
I'll keep that in mind.
@@statquest thank you so much
I won't forget you in the acknowledgments sir haha!!! Great job!
Thank you very much! :)
BAM_ spot on thanks for such video.. my journey with logis tic regression and r has started.
Awesome!!! :)
Mine too
Thank you so much for this effort really appreciate
We need a stat quest on three topics:
1-Chi-square test,
2- The Hosmer-Lemeshow goodness of fit test for logistic regression.
And 3- Iteratively reweighted least squares (IRLS) by using Newton's method.
If you don't mind :) of course.
Can you tell us about the title of next video?!
The Chi-Square test is on the list. I've looked into the Hosmer-Lemeshow fit... Can you tell me what you think about the limitations? Specifically those mentioned in the wikipiedia article about it? en.wikipedia.org/wiki/Hosmer%E2%80%93Lemeshow_test#Limitations_and_alternatives
And iteratively reweighted least squares is also on the list. However, up next are some basic statistics videos and then videos on lasso, ridge, and elastic-net regression.
the Hosmer-Lemeshow statistic was used to avoid problem in Pearson chi-squared statistic which was when observations being grouped by the values of the x variables, the Pearson chi-squared goodness of fit test cannot be readily applied if there are only one or a few observations for each possible value of an x variable, or for each possible combination of values of x variables.
(A sample with a sufficiently large size is assumed. If a chi-squared test is conducted on a sample with a smaller size, then the chi-squared test will yield an inaccurate inference).
So in the Hosmer-Lemeshow statistic, the observations are grouped by expected probability. But there is very little guidance on selecting the number of subgroups. The number of subgroups,g, is usually calculated using the formula g> P + 1. For example, if you had 12 covariates in your model, then g > 12. How much bigger than 12 g should be is essentially left up to you. Small values for g give the test less opportunity to find mis-specifications. Larger values mean that the number of items in each subgroup may be too small to find differences between observed and expected values. Sometimes changing g by very small amounts (e.g. by 1 or 2) can result in wild changes in p-values. As such, the selection for g is often confusing and arbitrary. Also, it doesn’t take overfitting into account and tends to have low power. For these reasons, the Hosmer-Lemeshow test is no longer recommended.
Am I on right? Is it enough cues to no longer used of HL test?
I have another question, ( Overfitting is happening when your sample size is too small. If you put enough predictor variables in your regression model, you will nearly always get a model that looks significant.
While an overfitted model may fit the idiosyncrasies of your data extremely well, it won’t fit additional test samples or the overall population. The model’s p-values, R-Squared and regression coefficients can all be misleading. Basically, you’re asking too much from a small set of data.)
If I have a small sample, is there any problem to use Maximum likelihood to fit model and McFadden's pseudo-R squared? Is there any rule to chose the number of sample for any regression?
Sorry for the many of questions, it is my first year in biostatistics. :)
These are all great questions. You are correct about the HL test and you are correct about overfitting. There are, however, lots of tricks you can use to compensate for overfitting (lasso regression, ridge regression, elastic net regression etc.)
One way to test to see if you have a model that is "overfit" is to use cross validation.
As for a minimum number of samples for logistic regression - people often say "10 samples per level of each discrete variable". It's a general rule of thumb and it doesn't always apply. However, again you can use cross validation to verify if you have enough samples or not. Cross validation is a very practical tool!
Thank you, Mr Josh, for answering me, I need to study more about Cross-validation.
Sorry l have more than one account 🙈🙊
Thanks for the video. Your video made it look like so simple. I request you to upload a video of how to get risk ratios in multiple logistic regression model.
I'll keep that in mind.
Thanks Josh - you are our saviour!
BAM! :)
@@statquest Triple Booyah BAM from my side!
incredibly brilliant tutorial!
Thanks! :)
Hi Josh, thanks for your videos they are very easy to understand. Really appreciate your efforts. I believe I speak for many,
Because of you many people are able to understand with utmost clearity and you cover all the small details with super ease. Keep up the Nobel work. Cheers 👍
Would it be possible for you to put up a video on model evaluation i.e. determining cutoff and model performance.
Thanks
Thank you! :)
amazing. thank you man!
Thanks!
You may like this video too:
Another great video about logistic regression in JMP
th-cam.com/video/9yN_yjGAJZE/w-d-xo.htmlsi=jUwEZUDobBudE8AE
Thanks for an excellent video. As usual.
Thanks again!
U are geneus...and ur teaching style too...hurray!!!! and Bamm!!!!
Wow, thank you!
me binge watching Josh's videos before midterm... anyone else? lmao
Good luck! :)
This is great stuff as I am just learning R; so pardon a very basic question: Why does "sex" need to be a factor vs number here?
Since the values or 0 and 1, it probably doesn't matter. However, to be safe, it's probably a good idea to make all categorical values, regardless of their values, factors.
This man is a legend
:)
At 11:30, the video states, "Since we are not estimating the variance from the data (and instead deriving it from the mean) it is possible that the variance is UNDERESTIMATED." Q. How can we say that we are UNDER-estimating the value of the variance? BTW, awesome vids, music man! ;)
That's a great question. Here's a (hopefully) useful discussion on the topic: newonlinecourses.science.psu.edu/stat504/node/162/
@@statquest
It's 1am but let me see if I got this straight...
Due to the nature of discrete functions (like logistic functions) they do not always vary smoothly. With discrete functions it is possible to see variances (and their corresponding probabilities) differ from (in our case) the proposed logistic model. In other words, it is conceivable to have a leptokutic or platykurtic distribution.
It is possible to see probabilities which differ from the expected probabilities due to the fact that the "real" model may be different and/or the samples may not be i.i.d. As it happens, the Bernoulli distribution tends toward the platykurtic.
...It's just the wrong dang model sometimes...
It's actually a little simpler than that. With binomial data (like logistic regression) we estimate the mean value = number of positive responses / total number of responses. Once we have the mean value estimated, we use that, and that alone, to calculate the variance. In other words, once we have calculated the mean, we do not need the data anymore to calculate the variance. This is in contrast to linear regression (or a lot of other things) where we estimate the mean with the data and then use the data again to calculate how it varies around the estimated mean. Thus, there is a possibility that with Logistic Regression (and other "generalized linear models") we did not correctly estimate the variance since the data were not involved in that calculation. If we over estimate the variance, that just makes the calculations more conservative and, generally speaking, that's not a problem. However, if we underestimate the variance, then that means we're more likely to say things are significantly different even if they are not, and that's no good. So the dispersion parameter takes care of that.
@@statquest Cheers,
Great tutorials, I started with your PCA video and since then hooked onto other videos . Could I request you to do a video on various types of probability distributions when to use them.
Those are all in the works. I wish I could work 2 or 4 times faster than I can. I've wanted to cover the major probability distributions for over a year, but got sucked down a machine learning path and now feel spread pretty thin. However, these will happen eventually! :)
StatQuest with Josh Starmer could you make a video on how to work 2 to 4 times faster? :-)
As soon as I figure that out, I'll make a video on it! ;)
BAM!!!
I like the way you presented the information in such a manner that is easily understood
I have 2 questions
1. While doing the xtab, what do we need to do if we found that say it is either or both healthy and unhealthy under cp3 is 0 or very minimal (video clip at 6:04)
2. At 15:55 of the video clip , you mentioned about using cross validation to get a better idea of how well it might perform with new data. Do you have a separated video which is specifically for that topic ?
Many thanks
Williams
1) Unfortunately I don't understand what you're asking in this question. However, I think you are asking what do we do when one level from a categorical variable does not have strong preference for healthy or unhealthy or doesn't have much data to begin with. It really depends. You can just try it and see what happens, but you might also try removing the variable and see if that improves predictions.
2) I have a video on cross validation here: th-cam.com/video/fSytzGwwBVw/w-d-xo.html
@@statquest
Thanks for your prompt reply
let me clarify my 1st question at the video clip at 6:24, you mentioned that there are 4 patients represent level 1 under restecg category.
My first question is, why only 4 can cause problem? is it because it is too mininal compares with others?(Level 0, and Level 2). How do I know exactly that it is causing the problem when I do the analysis? and if it does cause the problem. how to go about fixing it? just remove the Level 1 can solve the problem?
Thanks for your help
@@williamstan1780 When you don't have much data supporting a specific category, then chance are it will have a lot of variance - in other words, further samples may be very different from the ones in the original dataset. You can test this with cross validation (use some of the data to fit the model, use the rest to see how well it performs). If things are no good, you can remove the variable, or try to lump categories together.
@@statquest thanks Josh ..: appreciated
Sir, you are a savior
Thanks! :)
Your videos are awesome. Thank you very much.
Thank you! :)
YOU RAWK !! Awesome explains on ML concepts..
Thank you! :)
The more I watch your videos the more the wish I had a teacher like you in my school days..
Do we have a video on chi square test?
Not yet. :( But one day we will.
Hi Josh
Firstly many thanks for your videos on this topic. I have noticed very odd and conflicting results between R and SPSS with regards to entering factors (with more than 1 level) into a logistic regression model. SPSS produces a simplified output containing an odds ratio with 95% CI and p-value, for each individual variable entered into a logistic regression model (rather than the factor levels, as displayed in R).
In R - I have not found a good way to do this. I have used the logistic.display command as well as exp() to get odds ratios, but they do not provide an overall value like in SPSS (instead, listing these for each individual level within the factor).
Do you have any idea why SPSS and R handle logistic regression differently like this? All I would like is a similar output to SPSS - where I get a single odds ratio, 95% CI and p-value for each individual factor variable entered.
Unfortunately I've never used SPSS so I'm not really familiar with the problem you are having. That said, perhaps this will help: stats.stackexchange.com/questions/543540/different-output-for-logistic-regression-between-r-and-spss-how-to-get-correct
Hi!! You may like this video too:
Another great video about logistic regression in JMP
th-cam.com/video/9yN_yjGAJZE/w-d-xo.htmlsi=jUwEZUDobBudE8AE
kindly do logistic Regression in python from scratch.. Your way of teaching and explanation is amazing.. keep rockzz❤️
I'll keep that in mind.
You really are wonderful for explaining this in a way morons like me can understand, this is so incredibly helpful. Thank you so much!
at the end where you make the graph , you could have used the broom package and augment function to create the data frame to compute the fitted and actual values.
Cool.
Awesome! Thank you so much! Please could you do a video about conditional logistic regression like clogit in R with result interpretation and how it works when using adjusted parameters.
I'll keep that in mind.
At 3:04, why do you have to convert the column of Strings to an Integer before a factor? Why not just convert it straight to a factor?
I should have been more clear. Unless you tell R not to treat strings like factors, it does that by default. So, at that point "ca" is already a factor. The problem is that "?" is considered one of the levels of that factor. We can drop that level, or we can convert it to integers and then back to a factor. For some reason I did it the latter way. It would have been better to just drop the "?" level.
@@statquest thanks!
This has been extremely helpful. Thank you!
Thank you! :)
super dangg! Good explanation, bro!
Thank you! :)
@StatQuest with Josh Starmer, After plotting the logistic regression graph in 16:47, how to do Confusion matrix?
I searched Stack overflow and tried various Confusion Matrix codes but no luck, I get errors and couldn't make the matrix.
You have to first define a threshold for classification. The default is p > 0.5 gives you one classification and p
OK.. I haven't even watched the video yet but it looks like exactly what I need
I hope so! :)
@@statquest I'm trying to uze a logistic regression model on a set of binary events. Each with a different probability of happening.. and I have no idea what I'm doing haha.. so I'm loading up on coffee and I'm going to start your videos soon
@@Gypsy_Danger_TMC Good luck! :)
Hi, can we use a chi squared test first on (in this example), heart disease and sex then use a glm model on those 2 variables? I tried it with my data and the p values were different although both were significant. Afterwards I modelled all the other independent variables to check for confounding. I found some of them were significant along with the same variable in the chi test I did.
In 13:47, since the output only says sexM, does that mean the P value 0.002503 only accounts for males?
The chi-square test and Wald test (used for logistic regression) are related are fundamentally different tests so it doesn't surprise me that you got different p-values, but because both tests are intended to work with the same data, it doesn't surprise me that they both were significant.
As for "sexM" at 13:47, that means that both labels, "male" or "female", are useful for predicting heart disease. The reason it is called "sexM" is that "males" get a '1' for that variable, meaning that it is true that they are males, and females get '0', meaning that it is not true that they are males. For more details on how design matrices work, see: th-cam.com/video/CqLGvwi-5Pc/w-d-xo.html
@@statquest Thanks for the prompt reply! This vid saved my grade for my stats unit.
@@yutassmilehealsme6572 BAM!
you are just the best! Thanks for doing this!
Thank you! :)
This channel has helped me a lot understanding statistics! Could you please make a video explaining the linear mixed model too?
Yes! However, it might be a while before I get to it.
Hi Josh, thank you very much for your teaching, I have really learned a lot over the past few days.
May I ask a (stupid) question...At 8:45, you mention that you predict heart disease for a female patient. However, how can I tell whether it means health or unhealthy? I remember you set the hd variable as 2 level: health and unhealthy. How do I know which level we are predicting?
In the github page, you tell us that: "## The intercept is the log(odds) a female will be unhealthy. This is because female is the first factor in "sex" (the factors are ordered,
## alphabetically by default,"female", "male")".... Does this alphabetical order also apply to "healthy" and "unhealthy"? I am not sure why the intercept is not predicting "healthy" female patient because "H" (i.e., healthy) comes before "U" (i.e., unhealthy). This part really confuses me. Thank you!
At 3:17 we code "healthy" as 0 and "unhealthy" as 1. Thus, the "base" or default is "Healthy" (since 0 comes before 1) and we are predicting the log(odds) that someone is different from the base.
@@statquest Thank you for much for your help.....It is very clear!!!
Is it needed to turn all the variables into a factor before the regression analysis?
All of the categorical variables need to be converted to factors.
@@statquest Thank you very much. What do you classify as categorical?
@@afiapriscilla8276 Variables that represent discrete categories. Like "favorite color=Blue" or "Red"
Hi Josh,
Firstly, forgive my ignorance. Can I refer to null and proposed LL results in 14:12? You mentioned that to pull from the log-likelihood, we need to divide the scores by -2. Can you please elaborate how do you get the -2?
Additionally, in 14:46 do you use 1-pchisq to get the upper tail? If my assumption is wrong, can you please explain the 1-pchisq part?
Thank you heaps
Your first question is answered in my video on Saturated Models and Deviance: th-cam.com/video/9T0wlKdew6I/w-d-xo.html
For your second question, the answer is "you are correct!". :)
Is there a video explaining the 3 points on right of slide at 11:30? I watched all of the linear/logistic videos and my understanding is these regressions fit a line to predict data given new x. I don't understand what has this got to do with estimating mean/var. Is estimating mean/var part of the fitting process and important to creating the model?
Unfortunately this is not something that I talk about in other videos because it is rarely adjusted for logistic regression.
14:37 is this the effect of all those variables on the response of just the significant ones? When building a linear equation out of this , we need to only include the variables with low p values on the RHS and can omit the others as they do not have as much impact right?
16:36 is it possible to get avoid getting differing number of rows for the number of fitted values vs response…it keeps appearing like that for me
1) It's the effect of all of the variables.
2) I'm not sure I understand your second question. Are you asking about the call to ggplot()?
@statquest thank you so much. It's clear now.Is it possible for a model to have low pseudo R² but high accuracy?
@@tvvt005 I would find that odd.
@@statquest I see… thank you.., I’ll go try to find where I went wrong… 😓
Hi, I really like your videos, every topic is as clear as water after watching it. I've watched this one and also the three videos about logistic regression's details. If you want to go further in this topic, you could do a video explaining emmeans package for R. Many people, including me, would understand post hoc tests for glm using emmeans, if someone like you explained it. Thank you!
Thanks! :)
so incredibly helpful and well done. Thank you so much!!
Thank you! :)
Great salute! If you can, please post a video on all machine learning models with a large dataset example implementation in r with clear intuition and mathematics statistics behind it. Thanks.
Great video!!! Thank you so much!
Thanks!
Hi Josh,
Thanks for the really helpful video! Referring to the clip at 15:20. Whilst I know plotting a Predicted Y for a range of X values (say in a simple univariate logistic regression) we would expect to see that S shape. But for a multiple variable regression (as in yours) should the index of probabilities when ranked and plotted as you've done here always have to be in that S Logistic Shape?
I am getting more of an exponential curve between 0 and 1, and can't tell if this means I have done something wrong/have something wrong with my model?
Hmm...I'm not sure. Your graph should definitely taper off as the predicted probabilities get closer to 1, but how visible this tapering is might depend on how many data points you plot.
Great video, thanks so much Josh! After the 4th minute you mention how to address the NA samples. Can you teach us the RANDOM FOREST method, if we don't want to get rid of our NA samples (e.g. in multivariate cases, where the rows include other useful info)? Thanks!
I cover the random forest method in this video: th-cam.com/video/6EXPYzbfLCE/w-d-xo.html (the theory is here: th-cam.com/video/sQ870aTKqiM/w-d-xo.html )
You're the man! thanks for everything!
Thank you very much! :)
Hi, I love the way you explain all this things! I have a couple of questions. I observe that it's necessary to establish a code type for the predictors, if these are dichotomous, for example, they are assigned 1 and 0 (in the example male / female), so:
- How should we proceed with polytomous predictors?
- What results of the model should be reported in a scientific article?
Thank you in advice and keep doing great content!
1) For all categorical data (with 2 or more classes), just make sure you are storing it in a factor.
2) That depends on the journal. I would look at other articles in that journal to figure it out.
Great content and incredible value. Thank you so much
Thanks! :)
Can you explain to me why do we convert the column into a factor at 2:39?
Because we're using that variable as a factor instead of a numeric value.
All of your videos are great and fun to learn from! Could you please upload a tutorial on mediation analysis using STATA and R (using the mediation package)?
I'll keep that in mind.
These videos are so amazing!
Do you have a suggestion for a book that explains Logistic Regression to newbies? The videos are super awesome, but extra references may help too. Hopefully you will write your own book soon!
Thanks!
I know this is probably 10 months too late, but the book “Introduction to Categorical Data Analysis” by Alan Agresti is a great book. Does a really good job explaining logistic regression and is pretty light on the math.
Clear as water. Super BAM!!! Gracias por compartir
Josh you are amazing, thank you!
The last graph deserves a quadruple BAM!!!!🤣🤣🤣🤣
Yes!
Hi Josh, thanks for this amazing tutorial. Would you be able to add something interactions between predictors and random effects? I am trying to run a mixed-model logistic regression and have three-way interactions but not entirely sure on how to deal with them. Thanks so much :)
same!
Thanks for the detailed and super easy explanation, Josh. I'd like to ask you, shouldn't we check the below items before regression?
1. Would it be better if we use the WOE or log(odd) of a variable rather than raw variable (for example gender -> dummy). If I'm not wrong to use a dummy variable in the model is not a good choice;
2. Correlation between variables;
3. Factor transformation ;
4. PCA analysis.
And as a result in order to calculate the probability from log(odds), we should use Sigmoid function? I mean transform log(odds) to the probability
Thanks in advance!
If you want your model to be interpretable - in that you can look at the parameter value and make sense out of them - then removing correlated variables is a good idea and factor transformation and PCA can help with that. On the other hand, if you want to use your model to make the best predictions, then correlated variables are fine and can improve predictions.
If you are interested in the details behind Logistic Regression, then check out these other StatQuests:
General Overview: th-cam.com/video/yIYKR4sgzI8/w-d-xo.html
Interpreting Coefficients: th-cam.com/video/vN5cNN2-HWE/w-d-xo.html
Fitting the Model to Data with Maximum Likelihood: th-cam.com/video/BfKanl1aSG0/w-d-xo.html
Calculating R-squared and its p-value: th-cam.com/video/xxFYro8QuXA/w-d-xo.html
at 6:00, would there be an easy way to program a loop so that R could provide you all those xtabs with one line of code?
I'm sure there is. Maybe someone else will contribute the code.
Excellent video, very clear and easy to follow! Do you have any videos that show how to do best subsets and cross validation with logistic regression on R? I know you have a video that explains the concept of cross validation but I am looking for a video like this that goes through it step-by-step for logistic regression on R. Same thing for how to run all possible models (best subsets) using logistic regression on R. I have found one by another youtuber for linear regression but not for logistic.
Not yet. :(
@@statquest Wow thank you for the quick reply! That's alright, if you do make any videos like that, I'll be among the first to watch them! :)
This Video is very helpful. Do you also have a video about Multinomial Logistic Regression in R. Could be very helpful if you can post it.
I'm glad you like the video. I don't have one on multinomial logistic regression, so I'll put it on the to-do list.
@@statquest hello! did you ever make a video for this one? would love to check it out if you did, thanks so much for what you do!
@@joshuabudi4787 Not yet. :(
great
work done here
Thank you!
You are so good!! Thank you!
Thanks! :)
Excelent!!!! Thank you very much.
bam!