Correction: 3:23 The array should only have wt through wt5, ko1 through ko5. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Thank you, I was mentioning 3:23. Your videos are great. I am a medical doctor from Turkey and currently, I am planning a career change to data science and I have been watching your videos to get prepared for a data scientist position. Could you create a few videos regarding data science interviews if it is relevant for your channel content? Best Regards, Göktuğ Aşcı, MD.
@@keerthik3791 Unfortunately the random forest implementations for Python are really bad and they don't have all of the features. If you're going to use a random forest, I would highly recommend that you do it in R instead.
"Note: We use samples as columns in this example because... but there is no requirement to do so." "Alternatively, we could have used..." "One last note about scaling with sklearn vs scale() in R" This is some of the gold that sets StatQuest apart. Thank you! ❤
The fact that you said bam when the plot showed what we wanted really shows that even if you are a pro python programmer, you still feel happy when you code correct, relatableeee
One of the best videos ever made on this topic. This channel has helped me a lot in understanding machine learning in greater detail. Keep up the good work !!
Simply loving StatQuest. Concise, clear and fun videos. One point I noted while watching this video is that the latest version of sklearn PCA() will center the data for you, but not scale it. So if you just need centering for doing pca, you don't need to worry about preprocessing.
I learn so much better in Python for some reason, I think it's because it's more interactive and you can play around with the data! Good one. Stattttquueeeeeest.
Awesome. Please create more videos about how to implement the machine learning as well as data science concepts explained here into Python. That would be super helpful for us, in particular beginners.
Thanks for the tutorial! One thing I don't understand is why the PC1 can separate the wt and ko samples. Their gene expression values are generated in a same way.
Hello Josh, Thank you for the amazing video! Quick question, at 9:18 how can I adapt "index=[*wt, *ko] for an excel input? Lets say that we have the same variables (Genes vs wt/ko) but in an excel file. How can I add these labels to the final plot (9:47)? Thank you again!!
I'm not sure I understand your question. You can export your data from excel and import it into python (or R or whatever). Or are you asking about something else?
Amazing video! I initially watched the video explaining PCA and i was mind-blown, thank you so much! I was hoping to ask if anyone on the comment section or even StatQuest if possible, would know how to implement PCA in a multivariate timeseries dataset and also "examine the loading scores" in such a dataset. Thanks in advance! :) P.S - extremely clueless on anything coding or ML, but Ive got to use PCA (and other dimensionality reduction methods) on my timeseries dataset. so would greatly appreciate any direction on how to proceed.
Queation please... 09:50 wt and ko samples are both created with the same random function Poisson (10, 1000). Why are wt samples (and ko samples) more correlated??
Because rd.randrange(10, 1000) returns a random number between 10 and 1000. Once we get that random value, we use it to generate 5 values for the wt samples using a poisson distribution. Then we select another random value between 10 and 1000 and use it to generate 5 values for the ko using a different (because the random value is different) poisson distribution.
We select a random number between 10 and 1000 to be the mean of a poisson distribution. That's just the average value, and there can be larger and smaller values.
@@statquest excellent going - really. difficult to know what's up and down in data science, and so i'm happy your videos cover subjects from mathematical concepts to code implementation. excellent spirit and explanations, again. (sorry about the superlative avalanche - in the vast ocean that's the net, it's difficult finding authoritative sources covering subjects well ) bests from Germany/Denmark ;)
Very concise, I will surely be coming back to this video, however I would like to know why PCA is able to group these two categories (wt and ko), when it's shown they are generated from the same random method. If all indexes were generated at the same time, I would get it, but as they are generated index by index, I seem not to be able to grasp it.
The trick is at 3:48. For each group, wt and ko, we select a different parameter for the poisson distribution and generate 5 measurements from each of those two different distributions. One set is for wt and the other set is for ko.
@@statquest I think my confusion comes from the fact that these will make the two groups different from one another (all w's different from ko's), but I wouldn't predict them to be similar within the group (wt1 is close in vertical to wt2, and to wt3...,), thus I tend to believe PCA should tell them apart, but not in exactly two groups (wt's vs ko's), I would predict more like two clouds instead of two "vertical line of points" in the 2-D.
@@3stepsahead704 Remember how PCA actually works, it finds the axis that has the most variation (which is between WT and KO) and focuses on that. And then find the secondary differences (among the WT and KO). However, because the differences between WT and KO are big, the scale on the x-axis will be much bigger than the scale on the y-axis. Thus, the samples will appear to be in a vertical line rather than spaced apart like you might guess they should be. In short, check the scales of the axes, they will explain the difference between what you think you see and what you expect.
Hi Joshua, thanks for that. really helpful. i'm quite new to python myself, and i'm trying to compile a PCA across a range of macro-economic factors (inflation,gdp,fx, policy rate etc.,), now in all that you've done above where is the display of the PCA i.e: the newly uncorrelated data set, is it the loading scores you printed? or the wt, and ko variables you plotted? Thanks
PCA does not have predictors and targets. All variables are just...variables. For more details about PCA, see: th-cam.com/video/FgakZw6K1QQ/w-d-xo.html
Hi, You are a lifesaver. I am trying to do PCA analysis on my own data but since every demo video either use the databases and you created your own data. I am missing some crucial steps, especially in defining index when i am doing it with my data. Will it be too much to ask few more videos on machine learning where you use the excel sheet data from your laptop.
@@statquest Thank you very much for the clarification! I googled it, and seems that it's two different things, but sometimes they can be used interchangeably or be the same thing.
@@Cat_Sterling Yes, I guess it depends on how you want to use them and whether you divide by 'n' or 'n-1', but, at least on a conceptual level, they are the same.
It depends on the field you are in. I used to work in Genetics and this is the format they used. So it's always worth checking to make sure you have the data correctly oriented.
3 ปีที่แล้ว
I don't know why in the PCA graph you plot the "features", in some other videos, they plot all the data point and visualize the data in the new subspace... And I don't know what are the meaning of the x-axis in the same plot, what does -10 mean in the PC1-89.9%? thanks
Great tutorial, sorry if my question may be ammature, but how did they differentiate WT and KO apart in the final PCA, I thought the data set was randomly generated?
BAM!!! I understood what u said. I show my gratitude. But I have a query. I am confused with my dataset regarding which to consider a row and which as columns My dataset is regarding Phase measurement units (PMU) used in electrical grid or sort of the distribution lines we see around. One single PMU measures 21 electrical parameters for a timestamp. We use around Four PMU each measuring the 21 parameters at different locations at the same time continuously over a period of time. How can I arrange the above data for Performing PCU sir?
Sir those two case you mentioned that PCU would work is what I am also interested in calculating apart from the combination of all of the PMUs time stamp. Can u mention how to arrange the data (Rows and columns) for both of the mentioned viable cases? Thanking you so much!!You are really awesome sir
First I want to thank you for all the awesome videos on PCA. I wanted to experiment with the demo code you published but I'm having problems in the data generation. The asterisk method used to stack wt and ko series is not working.
Firstly, very good video. Secondly I am running the code in some spectra I have and I get good PCA plot but the loading scores seems to be wrong. Have you any idea why?
Question please. This line trasforms the original data to a 10x10 array: pca_data = pca.transform(scaled_data) The video says: it generates the coordinates for PCA graph based on loading score and scaled data. Apart from the coordinates in graph, what do the values actually represent? How should I interpret them - Is it the amount of variance of sample values attributed to each PC? The distance of each sample on PC line to the origin? What is the unit?
Oops!! I make a mistake a deleted your follow up comment. Sorry about that. However, my response is "Yes, the PCA graph is a graph that uses PCs as the axes."
Hey Josh, how do I find out which feature in the original dataset is to be removed(the one that least affects the variance im assuming)? I know we use PCA for the same, but I just can't understand how we select the unimportant feature from the original dataset using PCA.
what happens if we give the n_components=d, d being the no of dimensions? Does PCA denoise the data because there won't be any reduction in dimensions?
I don't think it will. However, there is still value because you can draw the scree plot and see how many PCs are really useful (it might only be a few, or it could be all of them).
Hi Josh!, this is a very excellent video that helped me a lot!!! I have a question, what if PC3 PC4 is also essential? Do I need to draw 2 2-D graphs, or what do I need to do?
If you want to draw the PCs and the data, then you'll have to draw multiple graphs. Or you can use the projections from the first 4 PCs and input to a dimension reduction algorithm like t-SNE: th-cam.com/video/NEaUSP4YerM/w-d-xo.html
Superb video... How does PCA perform if it has one-hot encoded data(transformed from categorical data) and discrete data. I suppose Z-scaling cant be done for such data types.
As PCA can be used to identify principal components which cluster a sample away from some clutter, can i then use these principal components as an input to a neural network to help better identify the sample? Or is this something which happens as matter of course in a neural network (ie the neural net doesnt need any help from PCA)? (i'm new to data science, and just discovering PCA, and where i can go now with all these amazing techniques, to help identify samples.)
PCA is often used to "denoise" data (by removing PCs that account for only a small amount of variation). before inputting it into all kinds of algorithms, neural networks included.
@@statquest Many thanks, i see now, PCA is a good tool to begin to investigate the usefulness, or otherwise, of metrics, before moving on to more sophisticated algorithms of data science. Your TH-cam videos are about the best there are for an introduction into data science. Thanks, i look forward to move of your videos!
Thanks for the "easy" to follow tutorial. I am trying to do a PCA for my RNAseq data but when I run scaled_tmp = StandardScaler().fit_transform(tmp.T) I get an error message: 'could not convert string to float: 'lcl|NC_000913.3_cds_NP_414542.1_1'. The lcl... is my target gene ID and I cannot edit it since i will need it later on to identify speficific genes. Please how do I solve this error message?
It looks like one of the columns in your matrix is some sort of identifier instead of sequencing data. In the video, when we create the data, we move identifiers to be row names or column names (see: 3:17). Other than the row and column names, the matrix that we do math on can only contain numbers because... how do we do math with identifiers?
@@statquest Thanks I was able to rectify the issue. I did not indicate "my gene_id" column as my index when loading data. After setting the index column it now works well.
Hi Josh Thank you for the video. It was a great tutorial. Just one question. What you called in the python code as loading_score, isn't in fact component score? It was score for each record (gene). Please correct me if I am wrong but isn't loading score the correlation between original fields (wt1, wt2 ...etc) and components? Thank you
What is the optimum number of components in a PCA analysis? I can have have say 3 or 20, i just dont know the relative importance of each component, so i need to run an analysis. However, is there some maximum number of components, beyond which a code will crash, or take too long to run? Many thanks, N
@@statquest Hi Josh, thanks for your reply. OK so i guess you take as many components as you can, and the PCA then tells you the dimensionality of a reduced form of your data? Ie you might take 1000 components, only to be told that PCA can reduce the dimensionality to say 4, just for argument sake? You then continue your work with just these main components.
@@neilanthony7596 As shown in the video, you can look at the scree plot and see how many PCs account for most of the variation in the data. So, you might start out with 1000 dimensions, and see that you only need 4 PCs to account for 99% of the variation. So you can use those 4 PCs, and only those, to continue with your work.
You don't have to scale the data, but it is highly recommended. For more details why scaling is important, see this StatQuest: th-cam.com/video/oRvgq966yZg/w-d-xo.html
@@statquest I saw that video. I want to use PCA to determine weights of variable in my index. I have 2 doubts: 1) Shall I take square of load factor and multiply by variation of the principal component to get weight for that variable? 2) what is the role of negative and positive sign if I am using PCA method to derive weights. Shall I give negative sign load factors, negative weights in my index?
@@manannawal8466 1) The loading scores are weights to begin with and determine how much of each variable is combined to make the principal component. In other words, you should need to transform the loading scores. 2) The positive and negative signs are just relative to the other variables and how they contribute to the slope of the principal component. For example, if you had two genes, 1 and 2, then loading scores 5 and -2 (for genes 1 and 2 respectively) would give us the exact same slope as -5 and 2. Since the positive and negative signs are somewhat arbitrary, people often ignore them and instead concentrate on the magnitude of the loading scores as a way to determine variable importance.
Please could someone tell me which are the variables\dimensions we want to reduce and which are the observations\samples? im a little confused, especially that i found 10 principal components in the scree plot the genes are the variables/features , and the wt, ko are the samples right?
The genes are features and the types of mice are the samples. Did you run the code that I wrote, or did you write your own? You can download my code for free.
@@statquest thank you very much, you know due to the reverse of order of variables and samples, i got a little confused , plus the scree plot you showed in the video had 10 principal components which is equal to the number of samples, but perhaps the scree plot was just for visualizations and there is more than 10 principal components, however i didn't run the code I have another small question please, what if the samples are far less than the variables, let's say it's an image and the variables are the pixels with 4000 pixels in total for each image, and the samples are just 200 , in that case i would not get more than 200 principal components right? or in other words, only 200 PC or less would be useful to me right?
hai, could help me to this question please? Xnp = np.asarray(X.todense()) # Run a principal component analysis on Xnp # How much of the variance can be explained # by the first 10, 50, 100, 200, and 500 principal components?
Thank you Joshua for this wonderful explaination. Thanks a lot. I am using your code for generating a scree plot in the same way and I obtain this error: bar() missing 1 required positional argument: 'left'
Your channel has helped me immeasurably :) I just had one question here, and that is how precisely to go from the data sample array you start with, to the scaled data by hand? I tried but didn't get the correct answer? I did watch the PCA Explained video as well, but just didnt get the same result here and wonder if you could clarify exactly how it gets from one to the other... should be: scaled_data = (data['wt1'][i] - np.mean['wt1])/ np.std(data['wt1']) ... for each datapoint i and each column right? this isnt real code im just making a point that its z = x-u / s :)
Correction:
3:23 The array should only have wt through wt5, ko1 through ko5.
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Thank you, I was mentioning 3:23. Your videos are great.
I am a medical doctor from Turkey and currently, I am planning a career change to data science and I have been watching your videos to get prepared for a data scientist position. Could you create a few videos regarding data science interviews if it is relevant for your channel content? Best Regards, Göktuğ Aşcı, MD.
@@GoktugAsc123 I'll keep that in mind.
@@keerthik3791 Unfortunately the random forest implementations for Python are really bad and they don't have all of the features. If you're going to use a random forest, I would highly recommend that you do it in R instead.
@@statquest Thankyou for the suggestion. I am good at Python, MATLAB. Can I do random forest in MATLAB? Or is learning R necessary here?
@@keerthik2168 I have no idea. I've never tried to do random forests in Matlab.
Dude you deserve a humanitarian award.
Thanks! :)
he is a good human in my eyes
@@joshuamcguire4832 bam!
@@rezab314 super bammm!!!
Not only the best PCA demonstration but also THE BEST introduction to Python. Hats off to you man!!
Thank you! :)
"Note: We use samples as columns in this example because... but there is no requirement to do so."
"Alternatively, we could have used..."
"One last note about scaling with sklearn vs scale() in R"
This is some of the gold that sets StatQuest apart. Thank you! ❤
Thank you! :)
I have been dabbling in data science for a while now, and only now learned that pandas stand for "panel data" xd
This channel never ceases to amaze
:)
Whenever I search for some machine learning based explanation, I add 'by statquest' in it ^_^. Keep up the great work :')
Thank you very much!
@@statquest It's True I do the same thing ..thank you for your hard work
The fact that you said bam when the plot showed what we wanted really shows that even if you are a pro python programmer, you still feel happy when you code correct, relatableeee
bam! :)
Finally! You explain in the language I understand much better than English haha Thanks !!!
:)
but you are watching a tutorial \(-_-)/
YOU ARE SAVING MY DEGREE I LOVE YOU SO MUCH I CANT EVEN BELIEVE THIS IS THE SAME MATERIAL IM LEARNING IN MY MACHINE LEARNING CLASS RIGHT NOW.
Happy to help!
Python. Now you're speaking my language :)
want me to take out my python?
@@HK-sw3vi ...weirdo
One of the best videos ever made on this topic. This channel has helped me a lot in understanding machine learning in greater detail. Keep up the good work !!
Thank you!
Simply loving StatQuest. Concise, clear and fun videos. One point I noted while watching this video is that the latest version of sklearn PCA() will center the data for you, but not scale it. So if you just need centering for doing pca, you don't need to worry about preprocessing.
Thanks for the update!
I learn so much better in Python for some reason, I think it's because it's more interactive and you can play around with the data! Good one. Stattttquueeeeeest.
Thanks! There should be a lot more Python videos and learning material out soon.
@@statquest looking forward to it :).
You've got the right formula for simple explanations. Teach me dawg
Thank you! :)
This channel is the best TH-cam channel that I discovered. Thank you, sir!
Thanks!
Awesome. Please create more videos about how to implement the machine learning as well as data science concepts explained here into Python. That would be super helpful for us, in particular beginners.
Thanks, will do!
Thanks for the tutorial! One thing I don't understand is why the PC1 can separate the wt and ko samples. Their gene expression values are generated in a same way.
Just stating I have the same question 2 years later.
I am watching the 1st minute and I'm already super excited. Thanks!!
Hooray!!!!!! :)
6:31 using scikit PCA
8:35 plotting scree plot
10:37 loading scores for each principal component
Thanks for the time point! I'll add those to the description to divide the video into chapters.
I push the like button even before I play the video. Because Josh never fails to amaze me.
bam!
The only good step by step explanation I found on the web. Thank you so much!
Hooray!!! Thank you so much! :)
Wish there were more statquest coding in python videos, they are the best! Much prefer to regular content although that is still really high quality
Noted.
Hi Josh... Simply incredible all StatQuest videos... Triple Bam!!!
Thank you! :)
MAKE MORE PYTHON CONTENT PLEASE I LOVE IT
I'm working on it. :)
I like the way you plot the ratio of each PC~~
It is really easy to read!
BAM~~~~~~~~~~
Thank you!
You are one the best teacher that i've ever found. Thank you very much!
Thank you! :)
Always can find a new and detailed explanation of steps from your videos! Thank you!
Thank you! :)
Wow Josh.. Thanks for that unpacking concept. I never knew that my whole life...
You bet!
It's awesome to have the explanation based on python code. Thanks a lot!
No problem. I'm doing a lot more python coding these days, so hopefully I'll more of these "in python" videos.
Wow, your explanation is so clearly!!
Thank you! 😃
Hello Josh, Thank you for the amazing video! Quick question, at 9:18 how can I adapt "index=[*wt, *ko] for an excel input? Lets say that we have the same variables (Genes vs wt/ko) but in an excel file. How can I add these labels to the final plot (9:47)? Thank you again!!
I'm not sure I understand your question. You can export your data from excel and import it into python (or R or whatever). Or are you asking about something else?
Really appreciate this and would love to see more concepts implemented in python.
Thanks!
Man, u r a gem. I will pay for the knowledge later after my graduation bro. lol
Wow! Thank you! :)
Another Great StatsQuest in the books!
Thank you Josh. Such practice is important and valuable!! And you really also taught some Python tricks that I don’t know.
Thank you! :)
You are the best!!!! It would be great if you could make a video on speculative decoding using medusa and quantization of neural networks in general
@statquest
I'll keep that in mind! :)
Amazing video! I initially watched the video explaining PCA and i was mind-blown, thank you so much! I was hoping to ask if anyone on the comment section or even StatQuest if possible, would know how to implement PCA in a multivariate timeseries dataset and also "examine the loading scores" in such a dataset. Thanks in advance! :)
P.S - extremely clueless on anything coding or ML, but Ive got to use PCA (and other dimensionality reduction methods) on my timeseries dataset. so would greatly appreciate any direction on how to proceed.
See: stats.stackexchange.com/questions/158281/can-pca-be-applied-for-time-series-data
Woww! That was absolutely awesome!!! Thank you so much!
Glad you liked it!
That's a cool one. The fact that observations are columns makes it so confusing though. I'm really used to the tidy data notation
Noted
Excellent work!!! 👏👏
Thanks a lot!
Queation please...
09:50 wt and ko samples are both created with the same random function Poisson (10, 1000). Why are wt samples (and ko samples) more correlated??
Because rd.randrange(10, 1000) returns a random number between 10 and 1000. Once we get that random value, we use it to generate 5 values for the wt samples using a poisson distribution. Then we select another random value between 10 and 1000 and use it to generate 5 values for the ko using a different (because the random value is different) poisson distribution.
Hi Josh. The best PCA explanation. Thanks a lot :-) May GOD bless you 😊
Thank you! :)
Yes, May god bless you 100 times. May the troubles of today’s world not reach your doorstep. You’re a great person.
As always a great presentation and the python code just give the extra bite...
Thanks!
i really like your clear explanation. please do some videos about deep learning and NLP.
I'm working on them.
@@statquest yeah! I am waiting for that
4:46 Why the gene4,ko1 has a value over 1000 if the command says "get a random value between 0 and 1000?
Thanks for the value !!
We select a random number between 10 and 1000 to be the mean of a poisson distribution. That's just the average value, and there can be larger and smaller values.
@@statquest oh! i see!! thank you so much, I still learning about this
This was so clear, thanks! Finally I can do PCA in python, BAM 😊 You DA BEST!
Thanks!
What a playlist, I simply loved it 😘
Thank you!
very much enjoy your explanation style.
many thanks for the great videos!
Thanks!
@@statquest excellent going - really.
difficult to know what's up and down in data science, and so i'm happy your videos cover subjects from mathematical concepts to code implementation.
excellent spirit and explanations, again.
(sorry about the superlative avalanche - in the vast ocean that's the net, it's difficult finding authoritative sources covering subjects well )
bests from Germany/Denmark ;)
@@miskaknapek BAM! :)
This was a reallly good explanation using Python
Very concise, I will surely be coming back to this video, however I would like to know why PCA is able to group these two categories (wt and ko), when it's shown they are generated from the same random method. If all indexes were generated at the same time, I would get it, but as they are generated index by index, I seem not to be able to grasp it.
The trick is at 3:48. For each group, wt and ko, we select a different parameter for the poisson distribution and generate 5 measurements from each of those two different distributions. One set is for wt and the other set is for ko.
@@statquest I think my confusion comes from the fact that these will make the two groups different from one another (all w's different from ko's), but I wouldn't predict them to be similar within the group (wt1 is close in vertical to wt2, and to wt3...,), thus I tend to believe PCA should tell them apart, but not in exactly two groups (wt's vs ko's), I would predict more like two clouds instead of two "vertical line of points" in the 2-D.
@@3stepsahead704 Remember how PCA actually works, it finds the axis that has the most variation (which is between WT and KO) and focuses on that. And then find the secondary differences (among the WT and KO). However, because the differences between WT and KO are big, the scale on the x-axis will be much bigger than the scale on the y-axis. Thus, the samples will appear to be in a vertical line rather than spaced apart like you might guess they should be. In short, check the scales of the axes, they will explain the difference between what you think you see and what you expect.
@@statquest Thank you very much for taking the time to explain this. I now get it!
Hi Joshua, thanks for that. really helpful. i'm quite new to python myself, and i'm trying to compile a PCA across a range of macro-economic factors (inflation,gdp,fx, policy rate etc.,), now in all that you've done above where is the display of the PCA i.e: the newly uncorrelated data set, is it the loading scores you printed? or the wt, and ko variables you plotted? Thanks
Your videos are great! Thanks
Thanks!
This video is really awesome! I am just confused on one thing, what are your predictors and what is your target?
PCA does not have predictors and targets. All variables are just...variables. For more details about PCA, see: th-cam.com/video/FgakZw6K1QQ/w-d-xo.html
fantastic, like always.
I wonder how Poisson distribution caused each wt samples and ko samples to be correlated with each other?
Because we generated the data, I selected different lambda values for the wt from the ko samples.
Hi, You are a lifesaver. I am trying to do PCA analysis on my own data but since every demo video either use the databases and you created your own data. I am missing some crucial steps, especially in defining index when i am doing it with my data. Will it be too much to ask few more videos on machine learning where you use the excel sheet data from your laptop.
I am a newbie in data science and programming. I am a Molecular Biologist who would love to learn machine learning.
I'll keep that in mind for a future video.
Thank you!!! When we are speaking about variation in PCA, is that the same as variance?
Yep.
@@statquest Thank you very much for the clarification! I googled it, and seems that it's two different things, but sometimes they can be used interchangeably or be the same thing.
@@Cat_Sterling Yes, I guess it depends on how you want to use them and whether you divide by 'n' or 'n-1', but, at least on a conceptual level, they are the same.
@@statquest Thank you so much again! Really appreciate your reply! Your channel helped me so much!!!
at 5:10 why do we scale our data?
I explain why we scale the data in this video: th-cam.com/video/oRvgq966yZg/w-d-xo.html
dear instructor, will you release a python version of your ml course. supper fan here!
One day I will.
@@statquest hope that day comes quick. stay well.
generally, in ML, we use 'columns' as 'features(variables)' and ''rows' as 'examples', but in the video, it is inverse. but is is not a big deal.
It depends on the field you are in. I used to work in Genetics and this is the format they used. So it's always worth checking to make sure you have the data correctly oriented.
I don't know why in the PCA graph you plot the "features", in some other videos, they plot all the data point and visualize the data in the new subspace... And I don't know what are the meaning of the x-axis in the same plot, what does -10 mean in the PC1-89.9%? thanks
I don't plot the features, I plot the subjects. For details, see: th-cam.com/video/FgakZw6K1QQ/w-d-xo.html
great video! thanks for these!!! have you done a redundancy analysis and dbRDA plot video? thank you for contributing to our education
I haven't done that yet.
@@statquest let us know if you ever do! It would be a double bam from me. It just clicks the way you explain! Thank you again for your content!!!
Great tutorial, sorry if my question may be ammature, but how did they differentiate WT and KO apart in the final PCA, I thought the data set was randomly generated?
Early on we gave the rows and columns names and kept track of them.
Is loading score eigenvalues? Wish to see a more linear algebra method of explaining pca!
For more details on how PCA works, see: th-cam.com/video/FgakZw6K1QQ/w-d-xo.html
BAM!!! I understood what u said. I show my gratitude. But I have a query.
I am confused with my dataset regarding which to consider a row and which as columns
My dataset is regarding Phase measurement units (PMU) used in electrical grid or sort of the distribution lines we see around.
One single PMU measures 21 electrical parameters for a timestamp.
We use around Four PMU each measuring the 21 parameters at different locations at the same time continuously over a period of time.
How can I arrange the above data for Performing PCU sir?
Sir those two case you mentioned that PCU would work is what I am also interested in calculating apart from the combination of all of the PMUs time stamp.
Can u mention how to arrange the data (Rows and columns) for both of the mentioned viable cases?
Thanking you so much!!You are really awesome sir
is centering included in sklearn's pca model and that's why there is no extra step to center?
I believe so.
First I want to thank you for all the awesome videos on PCA. I wanted to experiment with the demo code you published but I'm having problems in the data generation. The asterisk method used to stack wt and ko series is not working.
Which version of Python are you using? The code was written for Python 3 and I'm not sure the asterisk method works in Python 2.x.
Oh, you are right, I was accidentally using Python 2. Now it works well in Python 3
Hooray! I'm glad you got it working :)
If u end up using the PCA data...would not cause data lakeage in our predictive model since scaling should be done after train test split?
If you're using for machine learning, presumably you can come up with a standard scaling and centering procedure.
Python ε> now we are talking!
:)
Firstly, very good video. Secondly I am running the code in some spectra I have and I get good PCA plot but the loading scores seems to be wrong. Have you any idea why?
No idea.
Question please. This line trasforms the original data to a 10x10 array: pca_data = pca.transform(scaled_data)
The video says: it generates the coordinates for PCA graph based on loading score and scaled data.
Apart from the coordinates in graph, what do the values actually represent? How should I interpret them - Is it the amount of variance of sample values attributed to each PC? The distance of each sample on PC line to the origin? What is the unit?
The coordinates do not have units. And, as far as I know, they are just coordinates.
Oops!! I make a mistake a deleted your follow up comment. Sorry about that. However, my response is "Yes, the PCA graph is a graph that uses PCs as the axes."
@@statquest No problem. Thanks for confirming.
Hey Josh, how do I find out which feature in the original dataset is to be removed(the one that least affects the variance im assuming)?
I know we use PCA for the same, but I just can't understand how we select the unimportant feature from the original dataset using PCA.
You can set a threshold for the loading scores. All features with loading scores below that threshold can be discarded.
@@statquest okay, thanks a lot!
Hi , Great video ..thanks. one question as there are 100 genes (features) wont there be 100 PCs ?
In theory, the answer is "yes", but in practice the answer is "no". To learn more about why, see: th-cam.com/video/oRvgq966yZg/w-d-xo.html
@@statquest Thanks 😊
Amazing! this is so important, thanks a lot.
Thanks! :)
what happens if we give the n_components=d, d being the no of dimensions? Does PCA denoise the data because there won't be any reduction in dimensions?
I don't think it will. However, there is still value because you can draw the scree plot and see how many PCs are really useful (it might only be a few, or it could be all of them).
Cool video thanks! How can ı generate numbers basen on poisson dist. that wont change when ı rerun the code for dataframe?
np.random.seed(0) for more details, see: stackoverflow.com/questions/21494489/what-does-numpy-random-seed0-do
Hi, by any change, you have video about theory of PLS and how to implement it in Machine Learning?
I'll keep it in mind.
Hello, thanks for the videos. I think you explain great. I have a question. How can we make rotations with this package?
You can multiply data by the loading values.
@@statquest Thanks for the response. That will allow me to make rotations such as varimax, quartimax, etc? :(
Thanks for the great video! :)
:)
Hi Josh!, this is a very excellent video that helped me a lot!!!
I have a question, what if PC3 PC4 is also essential? Do I need to draw 2 2-D graphs, or what do I need to do?
If you want to draw the PCs and the data, then you'll have to draw multiple graphs. Or you can use the projections from the first 4 PCs and input to a dimension reduction algorithm like t-SNE: th-cam.com/video/NEaUSP4YerM/w-d-xo.html
dude i'm trying to do isotonic regression with toydataset, but the error show x is not a 1D ARRAY, can i use PCA to turn it into 1D?
Maybe!
What will be the negative values in the loading scores indicates?
Loading scores are explained here: th-cam.com/video/FgakZw6K1QQ/w-d-xo.html
@@statquest Thank you.
Superb video... How does PCA perform if it has one-hot encoded data(transformed from categorical data) and discrete data. I suppose Z-scaling cant be done for such data types.
Scaling to be done only for numeric features
Categorical features to be used before PCA after one-hot encoding....
Incredible French accent “Poisson distribution” , I saw it three times 😆
:)
Thank you very much for this tutorial. Please can you explain how to get correlation matrix
With numpy, you use corrcoef().
@@statquest Thank you very much
Thank you! This video helped a lot with what I'm trying to do.
Awesome!
As PCA can be used to identify principal components which cluster a sample away from some clutter, can i then use these principal components as an input to a neural network to help better identify the sample? Or is this something which happens as matter of course in a neural network (ie the neural net doesnt need any help from PCA)? (i'm new to data science, and just discovering PCA, and where i can go now with all these amazing techniques, to help identify samples.)
PCA is often used to "denoise" data (by removing PCs that account for only a small amount of variation). before inputting it into all kinds of algorithms, neural networks included.
@@statquest Many thanks, i see now, PCA is a good tool to begin to investigate the usefulness, or otherwise, of metrics, before moving on to more sophisticated algorithms of data science. Your TH-cam videos are about the best there are for an introduction into data science. Thanks, i look forward to move of your videos!
Thanks for the "easy" to follow tutorial. I am trying to do a PCA for my RNAseq data but when I run scaled_tmp = StandardScaler().fit_transform(tmp.T) I get an error message: 'could not convert string to float: 'lcl|NC_000913.3_cds_NP_414542.1_1'. The lcl... is my target gene ID and I cannot edit it since i will need it later on to identify speficific genes. Please how do I solve this error message?
It looks like one of the columns in your matrix is some sort of identifier instead of sequencing data. In the video, when we create the data, we move identifiers to be row names or column names (see: 3:17). Other than the row and column names, the matrix that we do math on can only contain numbers because... how do we do math with identifiers?
@@statquest Thanks I was able to rectify the issue. I did not indicate "my gene_id" column as my index when loading data. After setting the index column it now works well.
@@godsperson5571 Hooray!
Hi Josh
Thank you for the video. It was a great tutorial. Just one question. What you called in the python code as loading_score, isn't in fact component score? It was score for each record (gene). Please correct me if I am wrong but isn't loading score the correlation between original fields (wt1, wt2 ...etc) and components?
Thank you
Is scalling to be done for both test and train dataset?
Yes.
What's about the source of dataset??
The dataset is created within the code.
@@statquest thanks for your quick reply
What is the optimum number of components in a PCA analysis? I can have have say 3 or 20, i just dont know the relative importance of each component, so i need to run an analysis. However, is there some maximum number of components, beyond which a code will crash, or take too long to run? Many thanks, N
I've done PCA with 1000's of components before.
@@statquest Hi Josh, thanks for your reply. OK so i guess you take as many components as you can, and the PCA then tells you the dimensionality of a reduced form of your data? Ie you might take 1000 components, only to be told that PCA can reduce the dimensionality to say 4, just for argument sake? You then continue your work with just these main components.
@@neilanthony7596 As shown in the video, you can look at the scree plot and see how many PCs account for most of the variation in the data. So, you might start out with 1000 dimensions, and see that you only need 4 PCs to account for 99% of the variation. So you can use those 4 PCs, and only those, to continue with your work.
thanks for the great videos, it's fantastic stuff!
is it necessary to scale the data ? becuse sometimes a variable might have a std near 0 wich generate NaNs
You don't have to scale the data, but it is highly recommended. For more details why scaling is important, see this StatQuest: th-cam.com/video/oRvgq966yZg/w-d-xo.html
how do we interpret positive and negative load factors of features in terms of separating the sample?
I show some examples of this in my main PCA video: th-cam.com/video/FgakZw6K1QQ/w-d-xo.html
@@statquest I saw that video. I want to use PCA to determine weights of variable in my index. I have 2 doubts:
1) Shall I take square of load factor and multiply by variation of the principal component to get weight for that variable?
2) what is the role of negative and positive sign if I am using PCA method to derive weights. Shall I give negative sign load factors, negative weights in my index?
@@manannawal8466 1) The loading scores are weights to begin with and determine how much of each variable is combined to make the principal component. In other words, you should need to transform the loading scores.
2) The positive and negative signs are just relative to the other variables and how they contribute to the slope of the principal component. For example, if you had two genes, 1 and 2, then loading scores 5 and -2 (for genes 1 and 2 respectively) would give us the exact same slope as -5 and 2.
Since the positive and negative signs are somewhat arbitrary, people often ignore them and instead concentrate on the magnitude of the loading scores as a way to determine variable importance.
@@statquest thank you :)
Please could someone tell me which are the variables\dimensions we want to reduce and which are the observations\samples? im a little confused, especially that i found 10 principal components in the scree plot
the genes are the variables/features , and the wt, ko are the samples right?
The genes are features and the types of mice are the samples. Did you run the code that I wrote, or did you write your own? You can download my code for free.
@@statquest thank you very much, you know due to the reverse of order of variables and samples, i got a little confused , plus the scree plot you showed in the video had 10 principal components which is equal to the number of samples, but perhaps the scree plot was just for visualizations and there is more than 10 principal components, however i didn't run the code
I have another small question please, what if the samples are far less than the variables, let's say it's an image and the variables are the pixels with 4000 pixels in total for each image, and the samples are just 200 , in that case i would not get more than 200 principal components right? or in other words, only 200 PC or less would be useful to me right?
@@timokimo61 I answer that question in this video: th-cam.com/video/oRvgq966yZg/w-d-xo.html
@@statquest i have seen the video before , yea.
But a simple answer here would help me alot 😄
How do we import all those libraries???
Do we have to download anythiny extra ???
It probably depends on what Python you have. I believe I used Anaconda which comes with all of these libraries.
hai,
could help me to this question please?
Xnp = np.asarray(X.todense())
# Run a principal component analysis on Xnp
# How much of the variance can be explained
# by the first 10, 50, 100, 200, and 500 principal components?
If I had more time I could help, but today is super busy. Maybe someone else wants to help.
Thank you Joshua for this wonderful explaination. Thanks a lot.
I am using your code for generating a scree plot in the same way and I obtain this error: bar() missing 1 required positional argument: 'left'
Yes, I was using the original code given. I am using Python3, could that be the issue?
How can I select the final components to apply them to new data?
You can use the loading scores. For details, see: th-cam.com/video/_UVHneBUBW0/w-d-xo.html
@@statquest Thanks! Were can I find the loading scores in python?
@@dafran500 For details on how to do PCA in python, see: th-cam.com/video/Lsue2gEM9D0/w-d-xo.html
Please post some intuitions on sparse deconvolution and compressive sensing..Would love to understand your approach..❤️
Your channel has helped me immeasurably :) I just had one question here, and that is how precisely to go from the data sample array you start with, to the scaled data by hand? I tried but didn't get the correct answer? I did watch the PCA Explained video as well, but just didnt get the same result here and wonder if you could clarify exactly how it gets from one to the other... should be: scaled_data = (data['wt1'][i] - np.mean['wt1])/ np.std(data['wt1']) ... for each datapoint i and each column right? this isnt real code im just making a point that its z = x-u / s :)
It depends on how the data are oriented. Sometimes it's in columns, sometimes rows. So check to make sure your data is in columns.
@@statquest for the test code you supplied, so columns, am I using the correct method?
@@RachelDance It sounds like it.