NOTE: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change. ALSO NOTE: You can support StatQuest by purchasing the Jupyter Notebook and Python code seen in this video here: statquest.gumroad.com/l/iulnea Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/
After eight years of employment after graduation, I got laid off in 2020. I went back to school to pursue my second master in Data Science. I was still confused after machine learning classes, but after I watched your videos which were the same topics as the ones in my classes, you led me into a totally different world. Same concepts were taught by you in much easier way. BAM!!!
I love this kind of webinar where you teach in real time and go through concrete examples. Just purchased the material package and can't wait to go through them with you. I hope you'll make more content like this in the future 😊(I love the short and sweet vids too but I learn by doing so this helps solidify all the theory stuff!)
Again great work Josh, thanks so much. I actually worked at UNC-Chapel Hill, but I discovered you after moving to another University. Hope will meet you one day to thank you in person for the amazing content you are creating.
Really amazing video, I've been in and around data science and ML for a while but this is the first time I feel like I've gone the full way from mathematical concept -> working program (using medium complexity ML methods) -> insight/ question answered.
I am not an expert but a small help for everyone here ^_^ , if you want to find the missing values very easily, you can type dataframe.isnull().sum() ; dataframe is the name of the object containing the data. And thank you Josh for the amazing webinar ♥
You already get a lot of love, but I have to add to it and tell you how great these are. No joke, I've had nights when I plan on watching some TV or some movies and I decide to check out some 'Quests instead!
I decided it was interesting to draw two different PCA versions: 1) of the training data - so we can see the classifier with respect to the data it was trained on and 2) of the testing data - so we can see the classifier with respect to the data it was tested with. So the code has both versions, however, one of them (the latter) is commented out. However, you can swap which line is commented out and draw the latter.
The most perfect guide for SVM in TH-cam. Will donate after I get my first job! Thank you so much. Btw, I have question, why don't you use PCA before doing the modelling part? Are PCA only been use for visualization?
@@statquest I see but, so far what I know it will reduce the accuracy, but will help to avoid multicollinearity. But because of we have done OneHotEncoder, multicollinearity will be not occur. Am I right?
@@joxa6119 Using PCA first would definitely reduce multicollinearity if that was something we thought we needed to deal with. Multicollinearity usually means that we have 2 or more highly correlated features (also called variables), and thus, they are somewhat redundant. One-hot-encoding will not change the fact that those variables are redundant.
Hi Josh, I have a question. in 32:30 ,we scale the X_test and X_train, but i think that they didnt scaled same way. Bc They are not in same sample and their standard dev and means are different from eachother. I tried with this tiny sets to check if i think correct, and looks like scaling process little wrong? xxx = [1, 4, 400, 10000, 100000] yyy = [1,4,400,10000,11] scale(xxx) scale(yyy) Can u check and write me, did i think wrong?
In a pinned comment I wrote: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
@@statquest I didnt realized, sorry. Thank you for the reply. I wanna thank you so much. There could be too much informative people on internet but you are the best. Thank you for having fun while teaching!!
Your videos are so informative as always. The way you explain the topics are on another level. But I see a Tabla(twin hand drums) behind you. Do you play that? I also loves to play Tabla. Double BAM!!!! :D
I used to play Tabla a lot. I spent a lot of time in Chennai when I was a kid because my dad taught at the IIT there. When I was there I took lessons on tabla and veena.
Your videos are amazing !!!! I am soo happy u clearly explain many of the the topics I need!! :) (p.s. do u receive requests? I would really love a StatQuest on AR,MA,ARIMA,SARIMA models)
So, although the publishing company is elsevier, they are not the ones who did the research. If you ever want to read a paper, you can send an email to the primary investigator (the last author of the paper) or any of the first authors really, and they will freely give you the article to read
I have a request. You explain brilliantly (also with your background info in other videos) how to create and optimize your SVM. Could you also make a video about how to actually use your svm in a target system? That would make sense I think. Because I think that this would necessitate saving the scaler during creation of the SVM and loading it at runtime. Regards.
Amazing work, I've watched all your videos and had an amazing time. You explained the concepts so simply even I could follow the steps and build it from zero to one! Thanks so much!. A question I had would be the suitability of an SVM to classify time-series data vs something like an LSTM. I'm trying to build a tool that forecasts agricultural commodities using the weather. And thought using an SVM would be a good idea. I plan to categories the commodity price into positive, neutral and negative sections and then have the weather data be the data. To predict the future stock price I thought I could use forecasted data and use the SVM to estimate a change in price. Do you think that would be suitable and do you have any tips for the development. My thinking against an LSTM would what I need to input and how that entire structure would even work. As I have to predict a series of weather data into commodity prices. Thanks again for all the help so far.
To be honest, I'm not very familiar with using SVM for time-series. I know people have used it with XGBoost well and of course have used it with neural networks as well.
Great tutorial Josh! You must truly have one of the highest thumbs up to thumbs down ratios on youtube. Just two questions. 1) Right now you are using standarscaler on all of your variables, including the ones you have encoded. What is your reasoning for this instead of just scaling the continous variables, or maybe it doesn't affect the result? 2) What are your thoughts on onehotencoding before vs after splitting the data? Obviously right now, when your doing get_dummies your are doing it before splitting the data. From what I have understood, whether to do it before or after splitting is a pretty heated topic and I have found several questions on stack exchange where half the people say do it before and the other half say that doing it before is absolutely wrong and that it instead should be done after. In this dataset it will have an effect, because using your random states will produce a train test that on some variables have fewer categories than the test data does, which would mean that those observations should be dropped if onehotenconding is done after splitting. If I instead used onehotencoding before splitting, they would not be dropped. Would love to hear your thoughts on that topic, because I have found no real consenus on what is the right approach. Thanks again Josh!
1) For support vector machines, I'm pretty sure it does not effect the result. However, I have not tried it both ways. 2) I think there is a fear that if you one-hot-encode before splitting the data, then there will be data leakage. With most transformations, this is a problem, but for one-hot-encoding this is not the case. If a value in one dataset does not occur in the other dataset, then the column representing that value will be full of zeros and not have an effect on classification. In fact, the preferred method for industrial pipelines is "ColumnTransformer()", which keeps track of the values during the initial one-hot-encoding and when a testing set has new values, it throws an error.
Hi Josh, Thank you very much for your lessons ! you explain very well unlike many teachers. I just have one doubt, when you scale(X_train) and scale(X_test) you're actually scaling the encoded 'categorical' variables. Thus the sparse encoded matrix of 0 and 1 encoded by the features ['SEX','MARRIAGE',....] will be scaled as well, is that correct ? Shouldn't be only the numerical features to get scaled ? Thanks a lot for your lessons
Thanks for the brilliant tutorial Josh! You are truly an inspiration. I just had two questions here :- 1) You applied a regularization technique here by finding the right value for C. What kind of regularization is this? L1, L2 or L1&L2? 2) Is it possible to apply L1, L2, and elastic net regularization on SVMs? If yes, how should I do it?
Josh - Thanks for the video and it is super helpful!! A couple of questions though: 1. Under "Transform the test dataset with the PCA...", should we use the code that you commented out - i.e. X_test_pca=pca.transform(X_test_scaled), instead of X_test_pca=pca.transform(X_train_scaled)? didn't get why we applied the PCA transformation on train dataset to derive testing data. 2. Noticed that 1,000 defaults and 1,000 non-defaults were selected to construct the training sample. Do the numbers of two classes have to be equal for SVM? If not, would this cause any bias as the ratio seems a lot different from the original data? Thank you!
1) Because the SVM was fit to the training data, I wanted to show how it "looked" relative to the training data. However, you can also "see" how the boundary applies to the testing data. It's up to you. 2) Typically it's a good idea to have "balanced" data - data with an equal number of both classes. However, this is not a requirement for SVM - and, whether or not you need it depends on how you want the SVM to perform. For more details, see: th-cam.com/video/iTxzRVLoTQ0/w-d-xo.html
Hello Josh Starmer, Can you explain more about some hyperparameter in resample? replace=False --> we will not change any data in original data (df_default) and if True mean original df_default will be changed? random_state --> help others can get the same result with you? So how many people can get same result to you? 42??? Thanks
1) Yes 2) We are setting the seed for the random generator to the number 42, this ensures that everyone will get the same results. In other words, the random number generator generates a sequence of random numbers based on a starting value. If we all set the starting value to the same number (in this case, 42) then we will all get the same sequence of random numbers.
Dear Josh, My understanding, n_components hyperparameter in PCA() is the number of dimensions that we want to reduce down to. Therefore, I make some confusion. 1. If we use PCA() with no reference any n_components, so what exactly is the number of components in this case? 2. In other tutorials, n_components can set in floating (0.0 to 1.0), it is not make sense if we understand as a dimension number. Thanks, have a nice week!
@@statquest Thank for your recommend video. I understanding in this way. when we use PCA() with no n_components hyperparameter, the program will calculate all PCs of data. n_components in this situation is equal to all dimensions of data (pca.explained_variance_ratio_.shape[0]) and when we use PCA(n_components=2) that we only take care 2 first PCs. Sorrry if this question make inconvenience from you. I am only want to sure that my understanding is correct.
Again great video , Thanks. just 1 question , hope you answer.. is there any thing like "model importance" in Rstudio ? i need those independent variable influence ..
@@statquest thank you so much.. but i meant in Python.. i am running svm and looking for that code in python.. i wanted to obtain variables importance after classification
Thank you very much.. In the radial basis function video, only hyperparameter gamma was involved.. regularization parameter C was not there in the radial kernel function.. Are we using different radial kernel function here or the same one which was shown in radial kernel video? Thanks again.. your videos are great help ..
We are using the same kernel - so the only kernel parameter that we are optimizing is gamma. However, most, if not all, machine learning implementations also include regularization in one form or another. So we'll be talking about that as well.
Hi i have a question, aren't we supposed to split the data even more, and then use the validation dataset for hyperparameter tuning, we can pass it to grid_search, e.g. grid_search(x_validation,y_validation) instead of using the training dataset again?
Thanks. Wondering if there are more than 3 classes as target, do we yes one hot encoding for that ? And if so how would the code work.. does y become a 3 column dataset. And do models take that as is.
If we are trying to predict more than 2 classes, we create one classifier per "one-to-one" comparison. For example, if we have 3 classes, a, b and c, we make 3 classifiers. One for a-to-b, one for a-to-c and one for b-to-c. For more details, see: scikit-learn.org/stable/modules/svm.html
Say, in the original data set, we had a ratio of 30:70 of defaulted to non-defaulted credit accounts. Is it obligatory to have a balanced down-sampled data frame before we proceed with the analysis?
Josh, this video has helped me out a lot in my studies, but I have a question. When we scale the data, we should also include the categorical variables? Shouldn't we just scale all the data excluding the categorical ones?
Because the categorical variables are one-hot-encoded, we can scale them. All of the 0s will stay the same and the 1s will all turn into another constant value. In other words, when one-hot-encoding, 1 is arbitrarily chosen to begin with, so it doesn't hurt to turn it into another arbitrary number.
Another great video, I wish I had found this channel years ago! I am assuming the way you have coded for the optimising of Parameters could be used as the basis code for other models like Random Forest and it will just be the parameters changing dependent on the model that is being optimised?
Sir, Your work is amazing and if you could help me with this as I am working on classification problem and I want the probability of all the target categorical output. So, how to do it?
One more question: when you're defining the param_grid, you have a comma after the last curly brackets. It actually works with or without that comma. I don't get why it isn't throwing an "error" in there, since that comma isn't supposed to be there. 🤔
Hi Josh! What if our dataset has 【continuous columns】 & 【"categorical number" columns】 at the same time, should we start with getting dummies first to convert our categorical columns to continuous columns AND Standardscaler the rest continuous columns in order to give the data 0 mean? Is there any correlation between "get_dummies" & "encoder" ? I really appreciate any answers you would share with US, cheers!
XGBoost works well with sparse data (data with lots of zeros), so it is probably a good idea to only one-hot-encode the categorical data. Do not standardize them as well.
Hi Josh, really great content, learning a lot. Out of curiosity when doing One Hot Encoding, is there a reason why you did not say drop-first=True to avoid Multi-collinearity?
If my df is 101, what value should i set on n_samples? is there a specific number? like in the video, you use n_samples=1000. one more thing is downsampling the same as splitting? i mean both are the same but different ways/methods or what?
Down sampling randomly selects a smaller subset of the data. In this case we downsample to balance the data. So, it really depends on your own data and goals.
Those who are facing error can update the code as follow. clf = SVC(random_state=0) clf.fit(X_train_scaled, y_train) predictions = clf.predict(X_test_scaled) cm = confusion_matrix(y_test, predictions, labels=clf.classes_) disp = ConfusionMatrixDisplay(confusion_matrix=cm,display_labels=clf.classes_) disp.plot()
NOTE: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
ALSO NOTE: You can support StatQuest by purchasing the Jupyter Notebook and Python code seen in this video here: statquest.gumroad.com/l/iulnea
Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/
By scaling do you mean data normalization?
@@Dani-hh3qd Normalization is a specific type of scaling.
After eight years of employment after graduation, I got laid off in 2020. I went back to school to pursue my second master in Data Science. I was still confused after machine learning classes, but after I watched your videos which were the same topics as the ones in my classes, you led me into a totally different world. Same concepts were taught by you in much easier way. BAM!!!
I'm glad my videos are helpful! :)
A 2nd master?
How much has the curriculum changed in the past 8 years?
I'm so happy to find out that saying BAM + DOUBLE BAM comes naturally to you (and was not just for the videos). Amazing walkthrough as usual, Josh!
Triple bam! :)
I will definitely donate to this channel as soon as I got a job! Thanks.
Thank you very much! :)
Really appreciate for your slow speaking speed ,which makes it possible for not a English speaker ,like me ,a Chinese,to learn.
Thank you!
455 likes and 0 dislikes.... that's a double BAM!
Thanks!
i will be the first one
I love this kind of webinar where you teach in real time and go through concrete examples. Just purchased the material package and can't wait to go through them with you. I hope you'll make more content like this in the future 😊(I love the short and sweet vids too but I learn by doing so this helps solidify all the theory stuff!)
Thank you, and thank you for your support!
man your way of teaching is much better than the master degree teachers.
Thank you!
Again great work Josh, thanks so much. I actually worked at UNC-Chapel Hill, but I discovered you after moving to another University. Hope will meet you one day to thank you in person for the amazing content you are creating.
Wow! Thank you very much! :)
You're a pretty amazing nerd, I love it. This is an amazing tutorial.
Thanks! 😃
Hi from Argentina.
Great video! It really was from start to finish, it covers every step with dedication.
Thanks for sharing your knowledge!
Thank you very much! :)
Really amazing video, I've been in and around data science and ML for a while but this is the first time I feel like I've gone the full way from mathematical concept -> working program (using medium complexity ML methods) -> insight/ question answered.
Glad you enjoyed it!
Thank you so much for making this amazing code-walkthrough for SVM. Looking forward for more code walkthroughs like this.
You're very welcome!
This is amazing! Am in love with your approach of handling these stuff. Very clear and concise.
Thank you! :)
I purchased the notebook and I also watched the whole ad so you can make more money.
Thank you so much for your support! It means a lot to me. BAM! :)
Josh, you are Phenomenal! Love and Respect from Madras !
Respect from kerala too
@@anjalivijay9577 adhaan💥💪
Hooray!!! Thanks! :)
@@statquest 🤩🤩🤩🤩
@@statquest BAAAAM
I am not an expert but a small help for everyone here ^_^ , if you want to find the missing values very easily, you can type
dataframe.isnull().sum() ; dataframe is the name of the object containing the data.
And thank you Josh for the amazing webinar ♥
Nice tip!
This guy doesn't deserve the dislike button to be in his videos...what a clear explanation!!!
Awesome! Thank you very much! :)
you video deserves to be translated into more languages so people don't speak English can also learn from your amazing content
Thanks! :)
Really detailed and nice lesson! I liked how detailed the explanations were, It is definitely DOUBLE BAM worthy!
Thank you.
Glad you enjoyed it!
You already get a lot of love, but I have to add to it and tell you how great these are. No joke, I've had nights when I plan on watching some TV or some movies and I decide to check out some 'Quests instead!
BAM! Thank you very much! :)
Triple BAM!! Guess What?? You are the best teacher I've ever come across. My life is saved. Good to know you play Tabla too.
Thank you very much!!! :)
Thank you Josh, this taught me a good lesson on both PCA and SVM. Great work!
Bam! :)
Haha the double bam at 31:22had me dying lol. Great content! And love your channel!
Thank you so much! :)
i always love those musical intros
Bam!
Precise and to the point. Luv this and I am def going to extend my support to you
Thank you! :)
When Josh said 'OH NO!!', I was waiting for the line 'Terminology Alert!!!'.
:)
I've reread the "hitchhikers guide to galaxy" again (first time I read I was 12) and now it makes a lot more sense why the random state is 42 :)))
Yes!
Josh, you are wonderful! Thanks a million form Italy!
Thank you very much!!!
You are amazing. Keep posting. Best wishes from India.
Thank you very much! :)
Your video is so awesome. Everything related to SVM in one video, BAM.
Glad you liked it!
I learned a lot from your channel. I am a big fan of you. Looking forward for your Deep learning and NLP tutorial with python
Awesome, thank you!
This video came out the same week I decided to learn this. Get out of my head!
BAM! :)
I feel a bit starstruck finally seeing your face... :p Love your videos as always!
😊 thank you
svm are kinda my favourite thing in ML. very simple and mathematically concise yet highly usable.
Nice!
You Sir are an outstanding educator.
Thank you!
Good to see no haters for the saintly man.
Thanks!
Hope to listen to the Tabla's behind you at the start of your training one day.
Maybe one day!
@@statquest Amen to that ✌️
You are awesome. I hope you do something on NLP (tf idf, word2vec, etc.), for some reason your style was made for my brain
Thanks! :)
Josh u r an inspiration in teaching...Plz keep it up
Thank you! :)
Can you explain why you used 'x_test_pca =pca.transform( x_train_scaled) when you wanted to transform test data with PCA?
I decided it was interesting to draw two different PCA versions: 1) of the training data - so we can see the classifier with respect to the data it was trained on and 2) of the testing data - so we can see the classifier with respect to the data it was tested with. So the code has both versions, however, one of them (the latter) is commented out. However, you can swap which line is commented out and draw the latter.
@@statquest Thank you so much
Wow. Thanks Josh . Your videos are always a go to resource
Thanks! :)
The most perfect guide for SVM in TH-cam. Will donate after I get my first job! Thank you so much.
Btw, I have question, why don't you use PCA before doing the modelling part? Are PCA only been use for visualization?
In this case, we only use PCA for visualization.
@@statquest I see but, so far what I know it will reduce the accuracy, but will help to avoid multicollinearity. But because of we have done OneHotEncoder, multicollinearity will be not occur. Am I right?
@@joxa6119 Using PCA first would definitely reduce multicollinearity if that was something we thought we needed to deal with. Multicollinearity usually means that we have 2 or more highly correlated features (also called variables), and thus, they are somewhat redundant. One-hot-encoding will not change the fact that those variables are redundant.
I love StatQuest. please continue to make video with python =)
Thank you! :)
Thanks! I really like the way you explain things: calm and simple :)
Thank you! :)
what a lovable smart man, thanks for the great work!
Thank you!
Thanks a bunc. Helping me a lot getting started with my SVM. Regards
Happy to help!
I always ser your videos! Please continue this series of videos and surely I will purchase a notebook soon.
Thank you very much! :)
Hi Josh,
I have a question. in 32:30 ,we scale the X_test and X_train, but i think that they didnt scaled same way. Bc They are not in same sample and their standard dev and means are different from eachother.
I tried with this tiny sets to check if i think correct, and looks like scaling process little wrong?
xxx = [1, 4, 400, 10000, 100000]
yyy = [1,4,400,10000,11]
scale(xxx)
scale(yyy)
Can u check and write me, did i think wrong?
In a pinned comment I wrote: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
@@statquest I didnt realized, sorry. Thank you for the reply.
I wanna thank you so much. There could be too much informative people on internet but you are the best. Thank you for having fun while teaching!!
Your videos are so informative as always. The way you explain the topics are on another level. But I see a Tabla(twin hand drums) behind you. Do you play that?
I also loves to play Tabla. Double BAM!!!! :D
I used to play Tabla a lot. I spent a lot of time in Chennai when I was a kid because my dad taught at the IIT there. When I was there I took lessons on tabla and veena.
Your videos are amazing !!!! I am soo happy u clearly explain many of the the topics I need!! :)
(p.s. do u receive requests? I would really love a StatQuest on AR,MA,ARIMA,SARIMA models)
I'll keep those topics in mind.
You are just awesome. I just love your videos as they are really amazing. Stay safe .
Thank you! You too!
It helped a lot! Thank You on shared time and knowladge.
Thank you! :)
33:46 how is your svc show many details like the value of C, degree and etc? when i run my code like yours, it only showed SVC(random_state=42)....
I have no idea.
The man behind the voice
:)
This is really great. Thank You Sir for this great effort!!
Glad you liked it!
So, although the publishing company is elsevier, they are not the ones who did the research. If you ever want to read a paper, you can send an email to the primary investigator (the last author of the paper) or any of the first authors really, and they will freely give you the article to read
That's a great idea! :)
This is amazing. Thank you, Josh!
Thank you!
Thank you very much for the video!
I have a question, in SVM should the variables only be numeric or does it also support text?
Thank you!
Only numeric
Hooray! :)
929 likes and 0 dislikes ... that's a triple BAM !!
Hooray! :)
Love python! Been using R much lately! Would love to have some of R videos
Yes, I'm going to cover all of these topics (and more) in R. For example, R does a much better job with Random Forests than Python.
StatQuest with Josh Starmer I totally agree! Expect videos to come~
I have a request. You explain brilliantly (also with your background info in other videos) how to create and optimize your SVM.
Could you also make a video about how to actually use your svm in a target system? That would make sense I think.
Because I think that this would necessitate saving the scaler during creation of the SVM and loading it at runtime. Regards.
Good idea!
Hello Josh, Do you have any lecture about support vector data description (SVDD) as well. Actually, your way of describing problems is amazing.
Not yet! :(
Aahhhh....Something that I was stuck with...thanks a lot❣
Hooray! :)
Amazing work, I've watched all your videos and had an amazing time. You explained the concepts so simply even I could follow the steps and build it from zero to one! Thanks so much!.
A question I had would be the suitability of an SVM to classify time-series data vs something like an LSTM. I'm trying to build a tool that forecasts agricultural commodities using the weather. And thought using an SVM would be a good idea. I plan to categories the commodity price into positive, neutral and negative sections and then have the weather data be the data. To predict the future stock price I thought I could use forecasted data and use the SVM to estimate a change in price. Do you think that would be suitable and do you have any tips for the development.
My thinking against an LSTM would what I need to input and how that entire structure would even work. As I have to predict a series of weather data into commodity prices.
Thanks again for all the help so far.
To be honest, I'm not very familiar with using SVM for time-series. I know people have used it with XGBoost well and of course have used it with neural networks as well.
Great tutorial Josh! You must truly have one of the highest thumbs up to thumbs down ratios on youtube. Just two questions.
1) Right now you are using standarscaler on all of your variables, including the ones you have encoded. What is your reasoning for this instead of just scaling the continous variables, or maybe it doesn't affect the result?
2) What are your thoughts on onehotencoding before vs after splitting the data? Obviously right now, when your doing get_dummies your are doing it before splitting the data. From what I have understood, whether to do it before or after splitting is a pretty heated topic and I have found several questions on stack exchange where half the people say do it before and the other half say that doing it before is absolutely wrong and that it instead should be done after. In this dataset it will have an effect, because using your random states will produce a train test that on some variables have fewer categories than the test data does, which would mean that those observations should be dropped if onehotenconding is done after splitting. If I instead used onehotencoding before splitting, they would not be dropped. Would love to hear your thoughts on that topic, because I have found no real consenus on what is the right approach.
Thanks again Josh!
1) For support vector machines, I'm pretty sure it does not effect the result. However, I have not tried it both ways.
2) I think there is a fear that if you one-hot-encode before splitting the data, then there will be data leakage. With most transformations, this is a problem, but for one-hot-encoding this is not the case. If a value in one dataset does not occur in the other dataset, then the column representing that value will be full of zeros and not have an effect on classification. In fact, the preferred method for industrial pipelines is "ColumnTransformer()", which keeps track of the values during the initial one-hot-encoding and when a testing set has new values, it throws an error.
@@statquest Thanks for your insights Josh! Really appreciate it
@@statquest is it the same for K-means cluster analysis also ?
@@causticmonster Presumably if you use ColumnTransformer().
Thank you so much, it was a wonderful video!!!
Glad you enjoyed it!
Awesome teaching! Very interesting lectures.
Thank you! :)
Hi Josh,
Thank you very much for your lessons ! you explain very well unlike many teachers. I just have one doubt, when you scale(X_train) and scale(X_test) you're actually scaling the encoded 'categorical' variables. Thus the sparse encoded matrix of 0 and 1 encoded by the features ['SEX','MARRIAGE',....] will be scaled as well, is that correct ? Shouldn't be only the numerical features to get scaled ? Thanks a lot for your lessons
It doesn't really matter if you scale binary variables or not: stats.stackexchange.com/questions/59392/should-you-ever-standardise-binary-variables
@@statquest thanks for the reply! BAM
Thanks for the brilliant tutorial Josh! You are truly an inspiration.
I just had two questions here :-
1) You applied a regularization technique here by finding the right value for C. What kind of regularization is this? L1, L2 or L1&L2?
2) Is it possible to apply L1, L2, and elastic net regularization on SVMs? If yes, how should I do it?
C controls L2 penalty. I think that might be the only regularization you can use with scikit-learn svm.
@@statquest Yes I read the documentation of scikit-learn svm and the only other penalty allowed is L1.
Josh - Thanks for the video and it is super helpful!! A couple of questions though:
1. Under "Transform the test dataset with the PCA...", should we use the code that you commented out - i.e. X_test_pca=pca.transform(X_test_scaled), instead of X_test_pca=pca.transform(X_train_scaled)? didn't get why we applied the PCA transformation on train dataset to derive testing data.
2. Noticed that 1,000 defaults and 1,000 non-defaults were selected to construct the training sample. Do the numbers of two classes have to be equal for SVM? If not, would this cause any bias as the ratio seems a lot different from the original data?
Thank you!
1) Because the SVM was fit to the training data, I wanted to show how it "looked" relative to the training data. However, you can also "see" how the boundary applies to the testing data. It's up to you.
2) Typically it's a good idea to have "balanced" data - data with an equal number of both classes. However, this is not a requirement for SVM - and, whether or not you need it depends on how you want the SVM to perform. For more details, see: th-cam.com/video/iTxzRVLoTQ0/w-d-xo.html
Hello Josh Starmer,
Can you explain more about some hyperparameter in resample?
replace=False --> we will not change any data in original data (df_default) and if True mean original df_default will be changed?
random_state --> help others can get the same result with you? So how many people can get same result to you? 42???
Thanks
1) Yes 2) We are setting the seed for the random generator to the number 42, this ensures that everyone will get the same results. In other words, the random number generator generates a sequence of random numbers based on a starting value. If we all set the starting value to the same number (in this case, 42) then we will all get the same sequence of random numbers.
Very approachable!
Thanks!
One final question (I swear!): At the final code segment, you type
X_test_pca = pca.transform(X_train_scaled)
Isn't that supposed to be X_test_scaled?
Hmm....I'm actually on vacation right now and can't dig through this code. Can you re-ask this question in a few weeks?
Dear Josh,
My understanding, n_components hyperparameter in PCA() is the number of dimensions that we want to reduce down to. Therefore, I make some confusion.
1. If we use PCA() with no reference any n_components, so what exactly is the number of components in this case?
2. In other tutorials, n_components can set in floating (0.0 to 1.0), it is not make sense if we understand as a dimension number.
Thanks, have a nice week!
The number of components is explained here: th-cam.com/video/oRvgq966yZg/w-d-xo.html
@@statquest Thank for your recommend video. I understanding in this way.
when we use PCA() with no n_components hyperparameter, the program will calculate all PCs of data. n_components in this situation is equal to all dimensions of data (pca.explained_variance_ratio_.shape[0])
and when we use PCA(n_components=2) that we only take care 2 first PCs.
Sorrry if this question make inconvenience from you. I am only want to sure that my understanding is correct.
@@tanphan3970 Yes, that is correct.
Again great video , Thanks.
just 1 question , hope you answer..
is there any thing like "model importance" in Rstudio ?
i need those independent variable influence ..
I'm sure there is. See: cran.r-project.org/web/packages/shapr/vignettes/understanding_shapr.html
@@statquest thank you so much.. but i meant in Python..
i am running svm and looking for that code in python.. i wanted to obtain variables importance after classification
@@samanvafadar7719 See: shap.readthedocs.io/en/latest/
Hey Josh, could you make a video explaining the softmax function? Thanks!
Noted!
How do we set values for C and gamma especially the penalty Parameter C .. is it only by Hit and trial?
Pretty much. You just test a bunch with values with cross validation and see which is best.
Thank you very much.. In the radial basis function video, only hyperparameter gamma was involved.. regularization parameter C was not there in the radial kernel function.. Are we using different radial kernel function here or the same one which was shown in radial kernel video? Thanks again.. your videos are great help ..
We are using the same kernel - so the only kernel parameter that we are optimizing is gamma. However, most, if not all, machine learning implementations also include regularization in one form or another. So we'll be talking about that as well.
Hi i have a question, aren't we supposed to split the data even more, and then use the validation dataset for hyperparameter tuning, we can pass it to grid_search, e.g. grid_search(x_validation,y_validation) instead of using the training dataset again?
You can definitely do that.
Thanks. Wondering if there are more than 3 classes as target, do we yes one hot encoding for that ? And if so how would the code work.. does y become a 3 column dataset. And do models take that as is.
If we are trying to predict more than 2 classes, we create one classifier per "one-to-one" comparison. For example, if we have 3 classes, a, b and c, we make 3 classifiers. One for a-to-b, one for a-to-c and one for b-to-c. For more details, see: scikit-learn.org/stable/modules/svm.html
@@statquest Thank you very much for your reply. It seems for the Target column we don't need to do one-hot encoding. The model takes care of that?
@@rizwanmuhammad6468 I'm not sure.
Say, in the original data set, we had a ratio of 30:70 of defaulted to non-defaulted credit accounts. Is it obligatory to have a balanced down-sampled data frame before we proceed with the analysis?
It's not obligatory.
Josh, this video has helped me out a lot in my studies, but I have a question. When we scale the data, we should also include the categorical variables? Shouldn't we just scale all the data excluding the categorical ones?
Because the categorical variables are one-hot-encoded, we can scale them. All of the 0s will stay the same and the 1s will all turn into another constant value. In other words, when one-hot-encoding, 1 is arbitrarily chosen to begin with, so it doesn't hurt to turn it into another arbitrary number.
@@statquest Got it Josh, thanks for responding
Thanks a lot Josh!!
Thanks! :)
Another great video, I wish I had found this channel years ago!
I am assuming the way you have coded for the optimising of Parameters could be used as the basis code for other models like Random Forest and it will just be the parameters changing dependent on the model that is being optimised?
Yes. However, the scikit-learn implementation of random forests is terrible...
that tabla behind you tho!😵
I used to play and took lessons when I lived in Chennai.
How it started:
df
How it is going:
df_23_without_missingdata_scaled_with_magic_powers
Bam! :)
Awesome as always!!! :)
Thank you! And thank you for your support!
Are you supposed to scale the one-hot encoded variables as well?
I think it can go either way.
Sir, Your work is amazing and if you could help me with this as I am working on classification problem and I want the probability of all the target categorical output. So, how to do it?
scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
@@statquest Thank you sir.
One more question: when you're defining the param_grid, you have a comma after the last curly brackets. It actually works with or without that comma. I don't get why it isn't throwing an "error" in there, since that comma isn't supposed to be there. 🤔
Python is sometimes a mystery to me.... :)
gr8 experience, looking for ANN.
Thanks!
Hi Josh!
What if our dataset has 【continuous columns】 & 【"categorical number" columns】 at the same time, should we start with getting dummies first to convert our categorical columns to continuous columns AND Standardscaler the rest continuous columns in order to give the data 0 mean? Is there any correlation between "get_dummies" & "encoder" ?
I really appreciate any answers you would share with US, cheers!
XGBoost works well with sparse data (data with lots of zeros), so it is probably a good idea to only one-hot-encode the categorical data. Do not standardize them as well.
Thank you Josh!!!
My pleasure!!
Thank you for great tutorial!!!
Thanks!
Hi Josh, really great content, learning a lot.
Out of curiosity when doing One Hot Encoding, is there a reason why you did not say drop-first=True to avoid Multi-collinearity?
Yes, this is different from a linear model.
why do you use 1:1 resampling instead of stratified resampling? The dataset contains 3.5 no_default:1 default. Does this affect SVM results?
What time point, minutes and seconds, are you asking about?
If my df is 101, what value should i set on n_samples? is there a specific number? like in the video, you use n_samples=1000. one more thing is downsampling the same as splitting? i mean both are the same but different ways/methods or what?
Down sampling randomly selects a smaller subset of the data. In this case we downsample to balance the data. So, it really depends on your own data and goals.
Those who are facing error can update the code as follow.
clf = SVC(random_state=0)
clf.fit(X_train_scaled, y_train)
predictions = clf.predict(X_test_scaled)
cm = confusion_matrix(y_test, predictions, labels=clf.classes_)
disp = ConfusionMatrixDisplay(confusion_matrix=cm,display_labels=clf.classes_)
disp.plot()
Yep. The notebook has been updated.
can SVM be used for forecasting timeseries ?
I've never done that personally, but a quick google search says that other people have done it successfully.
@@statquest what i mean by my question is it intuitive to do it ? like what could be the intuition behind it ?