Support Vector Machines in Python from Start to Finish.
ฝัง
- เผยแพร่เมื่อ 12 มิ.ย. 2024
- NOTE: You can support StatQuest by purchasing the Jupyter Notebook and Python code seen in this video here: statquest.gumroad.com/l/iulnea
This webinar was recorded 20200609 at 11:00am (New York Time)
NOTE: This StatQuest assumes that you are already familiar with:
Support Vector Machines: • Support Vector Machine...
The Radial Basis Function: • Support Vector Machine...
Regularization: • Regularization Part 1:...
Cross Validation: • Machine Learning Funda...
Confusion Matrices: • Machine Learning Funda...
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
If you'd like to support StatQuest, please consider...
Buying my book, The StatQuest Illustrated Guide to Machine Learning:
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
TH-cam Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
0:00 Awesome song and introduction
4:16 Import Modules
6:36 Import Data
11:27 Missing Data Part 1: Identifying
16:57 Missing Data Part 2: Dealing with it
21:04 Downsampling the data
24:35 Format Data Part 1: X and y
26:35 Format Data Part 2: One-Hot Encoding
31:25 Format Data Part 3: Centering and Scaling
32:45 Build a Preliminary SVM
34:55 Optimize Parameters with Cross Validation (GridSearchCV)
37:58 Build and Draw Final SVM
#StatQuest #ML #SVM
NOTE: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
ALSO NOTE: You can support StatQuest by purchasing the Jupyter Notebook and Python code seen in this video here: statquest.gumroad.com/l/iulnea
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
By scaling do you mean data normalization?
@@Dani-hh3qd Normalization is a specific type of scaling.
After eight years of employment after graduation, I got laid off in 2020. I went back to school to pursue my second master in Data Science. I was still confused after machine learning classes, but after I watched your videos which were the same topics as the ones in my classes, you led me into a totally different world. Same concepts were taught by you in much easier way. BAM!!!
I'm glad my videos are helpful! :)
A 2nd master?
How much has the curriculum changed in the past 8 years?
I will definitely donate to this channel as soon as I got a job! Thanks.
Thank you very much! :)
I'm so happy to find out that saying BAM + DOUBLE BAM comes naturally to you (and was not just for the videos). Amazing walkthrough as usual, Josh!
Triple bam! :)
455 likes and 0 dislikes.... that's a double BAM!
Thanks!
i will be the first one
Really appreciate for your slow speaking speed ,which makes it possible for not a English speaker ,like me ,a Chinese,to learn.
Thank you!
I love this kind of webinar where you teach in real time and go through concrete examples. Just purchased the material package and can't wait to go through them with you. I hope you'll make more content like this in the future 😊(I love the short and sweet vids too but I learn by doing so this helps solidify all the theory stuff!)
Thank you, and thank you for your support!
This is amazing! Am in love with your approach of handling these stuff. Very clear and concise.
Thank you! :)
Thank you so much for making this amazing code-walkthrough for SVM. Looking forward for more code walkthroughs like this.
You're very welcome!
Hi from Argentina.
Great video! It really was from start to finish, it covers every step with dedication.
Thanks for sharing your knowledge!
Thank you very much! :)
You're a pretty amazing nerd, I love it. This is an amazing tutorial.
Thanks! 😃
Really detailed and nice lesson! I liked how detailed the explanations were, It is definitely DOUBLE BAM worthy!
Thank you.
Glad you enjoyed it!
This guy doesn't deserve the dislike button to be in his videos...what a clear explanation!!!
Awesome! Thank you very much! :)
Really amazing video, I've been in and around data science and ML for a while but this is the first time I feel like I've gone the full way from mathematical concept -> working program (using medium complexity ML methods) -> insight/ question answered.
Glad you enjoyed it!
Again great work Josh, thanks so much. I actually worked at UNC-Chapel Hill, but I discovered you after moving to another University. Hope will meet you one day to thank you in person for the amazing content you are creating.
Wow! Thank you very much! :)
Triple BAM!! Guess What?? You are the best teacher I've ever come across. My life is saved. Good to know you play Tabla too.
Thank you very much!!! :)
Thank you Josh, this taught me a good lesson on both PCA and SVM. Great work!
Bam! :)
Precise and to the point. Luv this and I am def going to extend my support to you
Thank you! :)
You Sir are an outstanding educator.
Thank you!
Thanks! I really like the way you explain things: calm and simple :)
Thank you! :)
Josh, you are wonderful! Thanks a million form Italy!
Thank you very much!!!
Josh, you are Phenomenal! Love and Respect from Madras !
Respect from kerala too
@@anjalivijay9577 adhaan💥💪
Hooray!!! Thanks! :)
@@statquest 🤩🤩🤩🤩
@@statquest BAAAAM
Wow. Thanks Josh . Your videos are always a go to resource
Thanks! :)
Your video is so awesome. Everything related to SVM in one video, BAM.
Glad you liked it!
I learned a lot from your channel. I am a big fan of you. Looking forward for your Deep learning and NLP tutorial with python
Awesome, thank you!
You are amazing. Keep posting. Best wishes from India.
Thank you very much! :)
I always ser your videos! Please continue this series of videos and surely I will purchase a notebook soon.
Thank you very much! :)
Thanks a bunc. Helping me a lot getting started with my SVM. Regards
Happy to help!
Your dedication is unreal, you replied to all the comments. Wow!
p.s. thanks for the video
bam!
This is really great. Thank You Sir for this great effort!!
Glad you liked it!
Josh u r an inspiration in teaching...Plz keep it up
Thank you! :)
You are just awesome. I just love your videos as they are really amazing. Stay safe .
Thank you! You too!
I am not an expert but a small help for everyone here ^_^ , if you want to find the missing values very easily, you can type
dataframe.isnull().sum() ; dataframe is the name of the object containing the data.
And thank you Josh for the amazing webinar ♥
Nice tip!
This is amazing. Thank you, Josh!
Thank you!
It helped a lot! Thank You on shared time and knowladge.
Thank you! :)
This video came out the same week I decided to learn this. Get out of my head!
BAM! :)
You already get a lot of love, but I have to add to it and tell you how great these are. No joke, I've had nights when I plan on watching some TV or some movies and I decide to check out some 'Quests instead!
BAM! Thank you very much! :)
Awesome teaching! Very interesting lectures.
Thank you! :)
You are awesome. I hope you do something on NLP (tf idf, word2vec, etc.), for some reason your style was made for my brain
Thanks! :)
what a lovable smart man, thanks for the great work!
Thank you!
i always love those musical intros
Bam!
you video deserves to be translated into more languages so people don't speak English can also learn from your amazing content
Thanks! :)
Haha the double bam at 31:22had me dying lol. Great content! And love your channel!
Thank you so much! :)
Thank you so much, it was a wonderful video!!!
Glad you enjoyed it!
svm are kinda my favourite thing in ML. very simple and mathematically concise yet highly usable.
Nice!
I love StatQuest. please continue to make video with python =)
Thank you! :)
Your videos are amazing !!!! I am soo happy u clearly explain many of the the topics I need!! :)
(p.s. do u receive requests? I would really love a StatQuest on AR,MA,ARIMA,SARIMA models)
I'll keep those topics in mind.
I purchased the notebook and I also watched the whole ad so you can make more money.
Thank you so much for your support! It means a lot to me. BAM! :)
Aahhhh....Something that I was stuck with...thanks a lot❣
Hooray! :)
Thank you for great tutorial!!!
Thanks!
Josh - Thanks for the video and it is super helpful!! A couple of questions though:
1. Under "Transform the test dataset with the PCA...", should we use the code that you commented out - i.e. X_test_pca=pca.transform(X_test_scaled), instead of X_test_pca=pca.transform(X_train_scaled)? didn't get why we applied the PCA transformation on train dataset to derive testing data.
2. Noticed that 1,000 defaults and 1,000 non-defaults were selected to construct the training sample. Do the numbers of two classes have to be equal for SVM? If not, would this cause any bias as the ratio seems a lot different from the original data?
Thank you!
1) Because the SVM was fit to the training data, I wanted to show how it "looked" relative to the training data. However, you can also "see" how the boundary applies to the testing data. It's up to you.
2) Typically it's a good idea to have "balanced" data - data with an equal number of both classes. However, this is not a requirement for SVM - and, whether or not you need it depends on how you want the SVM to perform. For more details, see: th-cam.com/video/iTxzRVLoTQ0/w-d-xo.html
I've reread the "hitchhikers guide to galaxy" again (first time I read I was 12) and now it makes a lot more sense why the random state is 42 :)))
Yes!
Very approachable!
Thanks!
Good to see no haters for the saintly man.
Thanks!
I feel a bit starstruck finally seeing your face... :p Love your videos as always!
😊 thank you
The most perfect guide for SVM in TH-cam. Will donate after I get my first job! Thank you so much.
Btw, I have question, why don't you use PCA before doing the modelling part? Are PCA only been use for visualization?
In this case, we only use PCA for visualization.
@@statquest I see but, so far what I know it will reduce the accuracy, but will help to avoid multicollinearity. But because of we have done OneHotEncoder, multicollinearity will be not occur. Am I right?
@@joxa6119 Using PCA first would definitely reduce multicollinearity if that was something we thought we needed to deal with. Multicollinearity usually means that we have 2 or more highly correlated features (also called variables), and thus, they are somewhat redundant. One-hot-encoding will not change the fact that those variables are redundant.
Hope to listen to the Tabla's behind you at the start of your training one day.
Maybe one day!
@@statquest Amen to that ✌️
Hello Josh, Do you have any lecture about support vector data description (SVDD) as well. Actually, your way of describing problems is amazing.
Not yet! :(
Great tutorial! Thank you!
Glad you enjoyed it!
Another great video, I wish I had found this channel years ago!
I am assuming the way you have coded for the optimising of Parameters could be used as the basis code for other models like Random Forest and it will just be the parameters changing dependent on the model that is being optimised?
Yes. However, the scikit-learn implementation of random forests is terrible...
Thank you Josh!!!
My pleasure!!
Your videos are so informative as always. The way you explain the topics are on another level. But I see a Tabla(twin hand drums) behind you. Do you play that?
I also loves to play Tabla. Double BAM!!!! :D
I used to play Tabla a lot. I spent a lot of time in Chennai when I was a kid because my dad taught at the IIT there. When I was there I took lessons on tabla and veena.
Thanks a lot Josh!!
Thanks! :)
Very useful ! Thank you very much !
Thanks!
Hi Josh, really great content, learning a lot.
Out of curiosity when doing One Hot Encoding, is there a reason why you did not say drop-first=True to avoid Multi-collinearity?
Yes, this is different from a linear model.
Very helpful! thanks :)
:)
Amazing content! How do I know when you have a webinar planned? and where do you stream it? Thanks!!
If you subscribe, you can find out about webinars.
@@statquest excellent, will do!
Awesome as always!!! :)
Thank you! And thank you for your support!
Great tutorial! In fact, all your videos are great. I have just on question: When looking for the best value for C, the algorithm went for the upper limit. Shouldn't we try again with higher values as suggestions? I haven't tried myself so I really don't know what would happen.
Yes, we should probably try higher values.
Hi Josh,
Thank you very much for your lessons ! you explain very well unlike many teachers. I just have one doubt, when you scale(X_train) and scale(X_test) you're actually scaling the encoded 'categorical' variables. Thus the sparse encoded matrix of 0 and 1 encoded by the features ['SEX','MARRIAGE',....] will be scaled as well, is that correct ? Shouldn't be only the numerical features to get scaled ? Thanks a lot for your lessons
It doesn't really matter if you scale binary variables or not: stats.stackexchange.com/questions/59392/should-you-ever-standardise-binary-variables
@@statquest thanks for the reply! BAM
Thank you very much for the video!
I have a question, in SVM should the variables only be numeric or does it also support text?
Thank you!
Only numeric
Hooray! :)
THANK YOU!!!!!!!!!!
Thanks!
Thank you very much.. In the radial basis function video, only hyperparameter gamma was involved.. regularization parameter C was not there in the radial kernel function.. Are we using different radial kernel function here or the same one which was shown in radial kernel video? Thanks again.. your videos are great help ..
We are using the same kernel - so the only kernel parameter that we are optimizing is gamma. However, most, if not all, machine learning implementations also include regularization in one form or another. So we'll be talking about that as well.
929 likes and 0 dislikes ... that's a triple BAM !!
Hooray! :)
When Josh said 'OH NO!!', I was waiting for the line 'Terminology Alert!!!'.
:)
I have a request. You explain brilliantly (also with your background info in other videos) how to create and optimize your SVM.
Could you also make a video about how to actually use your svm in a target system? That would make sense I think.
Because I think that this would necessitate saving the scaler during creation of the SVM and loading it at runtime. Regards.
Good idea!
Hey Josh, could you make a video explaining the softmax function? Thanks!
Noted!
Can you explain why you used 'x_test_pca =pca.transform( x_train_scaled) when you wanted to transform test data with PCA?
I decided it was interesting to draw two different PCA versions: 1) of the training data - so we can see the classifier with respect to the data it was trained on and 2) of the testing data - so we can see the classifier with respect to the data it was tested with. So the code has both versions, however, one of them (the latter) is commented out. However, you can swap which line is commented out and draw the latter.
@@statquest Thank you so much
Hey Josh, thanks for the video.
One question: you drop the ID column right from the start. In real life, once you made sure your model is valid and accurate, you would actually need to match those IDs to the probabilities of default. How would you do that? Put the ID in a list before dropping and then adding the list as a column to the predict proba?
We don't really need to re-add the IDs for the training data. However, when we get new data, we can just keep track of the ID by hand.
gr8 experience, looking for ANN.
Thanks!
Love python! Been using R much lately! Would love to have some of R videos
Yes, I'm going to cover all of these topics (and more) in R. For example, R does a much better job with Random Forests than Python.
StatQuest with Josh Starmer I totally agree! Expect videos to come~
Thanks for the brilliant tutorial Josh! You are truly an inspiration.
I just had two questions here :-
1) You applied a regularization technique here by finding the right value for C. What kind of regularization is this? L1, L2 or L1&L2?
2) Is it possible to apply L1, L2, and elastic net regularization on SVMs? If yes, how should I do it?
C controls L2 penalty. I think that might be the only regularization you can use with scikit-learn svm.
@@statquest Yes I read the documentation of scikit-learn svm and the only other penalty allowed is L1.
You are a god.
:)
One final question (I swear!): At the final code segment, you type
X_test_pca = pca.transform(X_train_scaled)
Isn't that supposed to be X_test_scaled?
Hmm....I'm actually on vacation right now and can't dig through this code. Can you re-ask this question in a few weeks?
Hi Josh!
What if our dataset has 【continuous columns】 & 【"categorical number" columns】 at the same time, should we start with getting dummies first to convert our categorical columns to continuous columns AND Standardscaler the rest continuous columns in order to give the data 0 mean? Is there any correlation between "get_dummies" & "encoder" ?
I really appreciate any answers you would share with US, cheers!
XGBoost works well with sparse data (data with lots of zeros), so it is probably a good idea to only one-hot-encode the categorical data. Do not standardize them as well.
One more question: when you're defining the param_grid, you have a comma after the last curly brackets. It actually works with or without that comma. I don't get why it isn't throwing an "error" in there, since that comma isn't supposed to be there. 🤔
Python is sometimes a mystery to me.... :)
Hello Josh, from a statistical perspective, how do you deal with "weird data"? As an illustration, for this dataset,
EDUCATION, Category
1 = graduate school
2 = university
3 = high school
4 = others
However, df['EDUCATION'].unique()
array([2, 1, 3, 5, 4, 6, 0], dtype=int64)
How do you deal with "5 and 6"? They are not in the category. Do you treat them as "missing values'?
Also, how about some data values which are out of range? They are definitely wrong.
I would treat them as missing data.
Hello Josh Starmer,
Can you explain more about some hyperparameter in resample?
replace=False --> we will not change any data in original data (df_default) and if True mean original df_default will be changed?
random_state --> help others can get the same result with you? So how many people can get same result to you? 42???
Thanks
1) Yes 2) We are setting the seed for the random generator to the number 42, this ensures that everyone will get the same results. In other words, the random number generator generates a sequence of random numbers based on a starting value. If we all set the starting value to the same number (in this case, 42) then we will all get the same sequence of random numbers.
Great lecture ;) but anyway I have one question - is it correct to standardize X_train and X_test separately? I mean, shouldn't the standardization parameters be the same for both datasets?
In the current approach, the data are not comparable, as if they were from a completely different world. Am I correct?
In a pinned comment, I wrote "At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change."
@@statquest Thank for answer :)
So, although the publishing company is elsevier, they are not the ones who did the research. If you ever want to read a paper, you can send an email to the primary investigator (the last author of the paper) or any of the first authors really, and they will freely give you the article to read
That's a great idea! :)
Dear Josh,
I fell not logic when you set the C hyperparameter in 2-time(when apply to X_train_scaled and pca_train_scaled) you define the param_grid. The first, C= 1000 is not in your list, the second C = 1000 is adding and it is becoming the best parameter in grid-search.
Any ideal in this step?
Thanks and have a nice week!
It might have been originally but I forgot to add it back in.
Again great video , Thanks.
just 1 question , hope you answer..
is there any thing like "model importance" in Rstudio ?
i need those independent variable influence ..
I'm sure there is. See: cran.r-project.org/web/packages/shapr/vignettes/understanding_shapr.html
@@statquest thank you so much.. but i meant in Python..
i am running svm and looking for that code in python.. i wanted to obtain variables importance after classification
@@samanvafadar7719 See: shap.readthedocs.io/en/latest/
Hi i have a question, aren't we supposed to split the data even more, and then use the validation dataset for hyperparameter tuning, we can pass it to grid_search, e.g. grid_search(x_validation,y_validation) instead of using the training dataset again?
You can definitely do that.
Sir, Your work is amazing and if you could help me with this as I am working on classification problem and I want the probability of all the target categorical output. So, how to do it?
scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
@@statquest Thank you sir.
Josh, this video has helped me out a lot in my studies, but I have a question. When we scale the data, we should also include the categorical variables? Shouldn't we just scale all the data excluding the categorical ones?
Because the categorical variables are one-hot-encoded, we can scale them. All of the 0s will stay the same and the 1s will all turn into another constant value. In other words, when one-hot-encoding, 1 is arbitrarily chosen to begin with, so it doesn't hurt to turn it into another arbitrary number.
@@statquest Got it Josh, thanks for responding
The man behind the voice
:)
Say, in the original data set, we had a ratio of 30:70 of defaulted to non-defaulted credit accounts. Is it obligatory to have a balanced down-sampled data frame before we proceed with the analysis?
It's not obligatory.
Hi Josh,
I have a question. in 32:30 ,we scale the X_test and X_train, but i think that they didnt scaled same way. Bc They are not in same sample and their standard dev and means are different from eachother.
I tried with this tiny sets to check if i think correct, and looks like scaling process little wrong?
xxx = [1, 4, 400, 10000, 100000]
yyy = [1,4,400,10000,11]
scale(xxx)
scale(yyy)
Can u check and write me, did i think wrong?
In a pinned comment I wrote: At 31:25 we should use the mean and standard deviation from the training dataset to center and scale the testing data. The updated jupyter notebook reflects this change.
@@statquest I didnt realized, sorry. Thank you for the reply.
I wanna thank you so much. There could be too much informative people on internet but you are the best. Thank you for having fun while teaching!!
that tabla behind you tho!😵
I used to play and took lessons when I lived in Chennai.
Great tutorial Josh! You must truly have one of the highest thumbs up to thumbs down ratios on youtube. Just two questions.
1) Right now you are using standarscaler on all of your variables, including the ones you have encoded. What is your reasoning for this instead of just scaling the continous variables, or maybe it doesn't affect the result?
2) What are your thoughts on onehotencoding before vs after splitting the data? Obviously right now, when your doing get_dummies your are doing it before splitting the data. From what I have understood, whether to do it before or after splitting is a pretty heated topic and I have found several questions on stack exchange where half the people say do it before and the other half say that doing it before is absolutely wrong and that it instead should be done after. In this dataset it will have an effect, because using your random states will produce a train test that on some variables have fewer categories than the test data does, which would mean that those observations should be dropped if onehotenconding is done after splitting. If I instead used onehotencoding before splitting, they would not be dropped. Would love to hear your thoughts on that topic, because I have found no real consenus on what is the right approach.
Thanks again Josh!
1) For support vector machines, I'm pretty sure it does not effect the result. However, I have not tried it both ways.
2) I think there is a fear that if you one-hot-encode before splitting the data, then there will be data leakage. With most transformations, this is a problem, but for one-hot-encoding this is not the case. If a value in one dataset does not occur in the other dataset, then the column representing that value will be full of zeros and not have an effect on classification. In fact, the preferred method for industrial pipelines is "ColumnTransformer()", which keeps track of the values during the initial one-hot-encoding and when a testing set has new values, it throws an error.
@@statquest Thanks for your insights Josh! Really appreciate it
@@statquest is it the same for K-means cluster analysis also ?
@@causticmonster Presumably if you use ColumnTransformer().