Quantile-Quantile Plots (QQ plots), Clearly Explained!!!
ฝัง
- เผยแพร่เมื่อ 12 พ.ย. 2017
- Quantile-Quantile (QQ) plots are used to determine if data can be approximated by a statistical distribution. For example, you might collect some data and wonder if it is normally distributed. A QQ plot will help you answer that question. You can also use QQ plots to compare to different datasets that you collected to determine if their distributions are comparable. This video shows you how to do both things.
NOTE: The data in this video are measures of gene expression. If "gene expression" doesn't mean anything to you, just imagine that the data represents how tall a bunch of people are, or how much they weigh. Then consider the y-axis to be the height or weight of the people, and the x-axis just represents all of the data you collected on a single day. In this case, all of the data were collected on the same day, so they form a single column.
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
If you'd like to support StatQuest, please consider...
Buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
TH-cam Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
Corrections:
4:35 The Uniform Distribution has one extra quantile
5:30 I should have said that Quartiles divide the data into 4 parts.
#statquest #quantile #qqplot
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
that intro alone, made me forget my hate for statistics and instantly fall in love with it
Hooray!
Haven't seen the video yet, but that intro earned you a subscription
It made me think that the whole video was going to be a song lol. Very interesting nonethless
I'am not suscribed for the plots, but for the music!
That intro hit me hard xD
Thanks sooooooo much! This is the only video I found explained the details of generating QQ plot and also make the concept so clear and easy to understand!
Thank you very much! :)
I was waiting for the "BAAM" all video long, got just a couple of great "HOORAY!".
Thank you for the awesome channel Josh!
You made me laugh. :)
🤔🤔🤔🤔🤔 well I thought that q-q plot was difficult but thanks to you I got it now. thanks and keep it up!!!
I really appreciate from your very easy way explanation.
I faced with so difficult and rough terminologies that I could not even understand the meaning of them.
Couldn’t have asked for more clear explanation, thanks!
Glad to help!
Thanks for all the videos! Great music BTW. Also I'm looking forward to rockin' my new SQ hoodie!
TRIPLE BAM! Thank you for your support!
You had my like at the beginning with the jingle. Thanks for explaining this so well!!
Glad you liked it!
Very well explained, thanks so much
Awesome video. Explained so clearly. Really helped me a lot!
Hooray! :)
Your videos are cool and concise , thank you .
It's so clear! Thanks a lot for your video.
Triple BAMM! Serious man your channel is pure art. Thanks
Thank you!
Great explanation, have a nice day :)
Can you do a video on normality tests like shapiro wilk and anderson darling? If not anytime soon, can you share link to some good materials?
Best intro song, it can be used as a 'mnemonic' for what QQ plots are used for =)
Thank you for the video! It was short and easy to understand :)
Thanks! :)
very nicely explained. it was a tricky concept until this video! thanks!
Hooray! I'm glad the video helped. :)
Awesomely explained! Good job!
Thank you! :)
Really excellent presentation, Josh. ⭐️
Thank you! :)
I just noticed when you said "please subscribe" at the end of the video, the subscribe button lit up:)
bam! :)
thanks , awesone , this is useful in measuring gene expression effect
You made it very clear man !!! Great doing
Glad to hear that!
Your videos are awesome, thank you so much!
Glad you like them!
wow, now I can clearly understand it ! thanks alot !
Hooray!!! :)
so clear, so good , so nuce thank you , Josh
Thanks!
phenomenal explanation and really cool intro music man!
Thanks!
How helpful! Thanks a lot for your amazing videos
Thanks!
Legendary explanation! Fantastic!
Thank you!
very clear with great examples!
Thanks!
such an awesome video, thanks!
Glad you liked it!
Hoorray!
Thx for the video :)
BAM! :)
Not gonna lie, stats is my super weak spot. You've helped me a lot in my Data Models course and interpreting my results. +1
Happy to help!
@@statquest Thank you! I'm just kicking myself for not taking more stats courses at this point!
@@jameswhitaker4357 My stats courses were all pretty terrible, so you never really know what you're going to get. I had to teach myself statistics, and these videos are how I taught myself.
@@statquest That's what I'm going through right now! I've been using your videos and a "Intro to Statistical Learning with Applications in R" textbook which has helped a lot. I think when I saw terms like "heteroscedasticity" or the crazy formulas I would get scared and put off the studying, until I took a course that required knowing it LOL. And luckily most of these statistical tests and concepts are now pretty easy to perform in programming. Cheers!
@@jameswhitaker4357 I actually wrote a little about heteroscedasticity. Maybe I should record it.
This was super helpful!
Thank you! :)
i love statquest!
Hooray! :)
Nice video. Explained everything in just under 7 mins. Awesome.
😄👍👍
bam!
@@statquest bam indeed. 😁
Thanks for the great explanation as always! So QQ is just a way to plot and visualize the similarity of two distributions? Are there any other scenarios when these can be used? Thanks!!
QQ is mostly used to check tail conditions. Density plots and cumulative plots are the best way to check distribution symmetry.
Explained like a pro.
Tripple BAM!!!
Thank you! :)
Very well explained !
thank you!
Great explanation, thanks!
Thank you! :)
Thanks a lot for the wonderful explanation!
Thank you!
is there a statistical test we can do to determine how far away the dots are allowed to deviate, rather than just eyeballing it? Or is eyeballing good enough? I.e. a stat test that could say 'the chance of these 2 distributions being the same is less than X%
The "K-S Test" is what you want. However, it is very strict and tends to reject the null too easily. It's one of the few statistical tests where a large p-value (suggesting no difference) is more convincing than a small one. en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
@@statquest How were the lines drawn? Least Squares? Maybe doing R^2 calculations can provide an idea?
Still trying to grasp my statistics a bit better :( :)
It helps a lot. Thanks!
Glad it helped!
Thank you, Josh.
Thanks! :)
Sweet, its not that difficult to grasp anymore! Thanks
This is very very cool, more likely to learn on TH-cam than in a classroom. Grazie
Glad it was helpful!
Thanks for this! Finally I understand
Hooray!
Helped a lot! Thank you :D
Hooray! :)
QBUS?
Thanks for the videos, if you're still looking for ideas how about k-l divergence?
Awesome video - thank you SO much for saving my sanity.
Thanks!
Best intro by far so far
Hooray!!!! :)
great video. a video on the intuition on why q-q plot works might be interesting.
I'll keep that in mind.
It's a party time with Josh Starmer and his quantiles! 😆🤘 Party on, Wayne!
:)
This video vas quite helpful
And again, thank you for another amazing video ! A little question : most of the points have to fit in the straight line for the data to be considered as normally distributed and at 4:15 you said it is not the case. Althought the intersection points are really close to the line, it does not matter, most of the point have to be strictly ON the line, right ? The fact that other intersection points are close or far from the line does not give any relevant information ?
I'm not sure I understand your question. For more details on how to interpret QQ-plots, see: stats.stackexchange.com/questions/101274/how-to-interpret-a-qq-plot
@@statquest ok thank you !
Best Explanation ever!!! 🎉🎉🎉
Thanks!
Thank you for your help! Greetings from Brazil.
Muito obrigado! :)
How beautiful and simple is that explaination 🥳
Thank you!
You are really Awesome!!!
Thanks! :)
Helpful and easy to undderstand.
Thanks!
Thanks for saving my life
Hooray! :)
You simply saved my life
Bam!
Understand it now - thank you!
Hooray! :)
Thanks for this wonderful explanation! I'm curious if we can tell anything about the slope in a QQ plot? Does the slope always equal 1 when the data follows a certain distribution?
As long as the x and y-axes are normalized to quantiles, then the slope should be 1 if the data follows a certain distribution. However, usually the x and y-axes are in the original units, and this means that the actual slope isn't super important. What is important is that the points form a straight line.
@@statquest Thanks!!
what a nice lecture!
Thanks! :)
What's the approach for determining which distribution has the best fit for the data? Would the r-squared of the data against the straight line be a suitable measure for how well the distribution describes the data?
This is a good question, and, to be honest, I'm not sure what the answer is. I like your idea, but it may oversimplify the problem. i.e. you could get a high R^squared value, but still have some real obvious problems if you looked at it visually.
du hast zerfetzt bro, danke
Bitte!!!
@@statquest quick question to 2:04 in your video:
if we have 15 data points and we divide the dataset into 15 quantiles, then shouldn't the smallest quantile be 0.06666 so around 0.07? because in your video you are saying that it is 0.7, which would mean, that 70% of all data is covered by just one datapoint. Thank you for your video :)
Hello, the video was amazing and I was able to get an idea of QQ plots. I do have a question though. How do we draw the normal distribution and uniform distribution? Is it just random?
The normal and uniform distributions are well defined by equations. So we just plug numbers into them to get the values out.
I could have better grades if i had faculties like you...thank you Josh!!
Thanks!
Thanks! It helps.
Wow!!! Thank you very much for your support! BAM! :)
really cool video
Thanks!
great video
Thanks!
precious help, thank you
Thanks!
Interesting!
Hi, I understand how the quantile points are plotted wrt observed vs theoretical distributions, what I don't understand is what determines the slope of the straight line. While this is fairly intutitive for a normal distribution, for say the Weibull distribution I am unclear how the slope of the striaght line is used to determine whether the observed vs theoretical quantiles are a good fit for a given distribution. Any ideas?
I came here for the same question and left with no answer LOL
Thank you! Awesome explanation
Thank you! :)
i have a doubt...why to use this method,instead just plot the points and see if it forms a bell curve....correct me
@@bharathkumar5870 I'm not sure I understand your question. Are you asking, "why don't we just create a histogram with the data and see if the histogram looks like a normal distribution"? If so, histograms can be very tricky in terms of selecting the correct bin size. In contrast, with a q-q plot we don't have to worry about optimizing a bin size or anything else.
@@statquest thank you sir ..u cleared my doubt. Different bins give different distributions😀
Hi Josh, Amazing Video there :) Just want to understand the intuition behind the working of QQ plots ? Is it the fact that quantiles of every normal distribution are just scaled up values of a standard normal distribution and that is why we expect a straight line ?
Pretty much
@@statquest Thanks for the response ;) Would really appreciate if you could make something on the same or share some content that could explain the intuition behind QQ Plots.
THANK YOU!!!!!!
Thanks!
I have another question: does shape of a QQ plot also has some information? Like difference in the beginning or at the end, or overall shift of the values.
Thank you!
Thank you!!!
bam!
you are the best!
Thanks! :)
You are great!
Thanks! :)
Thanks man!
No problem!
Could you do a video on Lorenz curves please
can we change the color in qqplot , if we can then how
How would you tell skewness from looking at the QQ plot? For example if some of the data points fell below the straight line (representative of a normal distribution) does that indicate positive/negative skew? Cheers :)
Ah, is it correct to say that, roughly, if at the beginning the majority of points were consistently/mostly above the line since the beginning then you'd say the data is positively skewed, and conversely, if most points laid below the line since initially then the data is negatively skewed?
Here's a link that explains how to interpret qq-plots: stats.stackexchange.com/questions/101274/how-to-interpret-a-qq-plot
Best intro 🔥
bam!
Sir I have been facing problem in ggplot2 package in R programming now a days
Could you please help?
Do I need to matter the exact size size or probability when dividing the contribution? Or just need to only make sure the sizes are equal?
Just equal sizes.
Nice lecture, but how do u identify boundary conditions, like - 1.5, for normal distribution?
This is explained at the very start of the video at 0:38. We have 15 data points, so our data have 15 quantiles. We then divide the normal distribution into 15 quantiles. Each quantile should have an equal probability - thus, with the normal distribution, the quantiles on the edge are relatively far apart, to compensate for the relatively low probability of observing a value out there. In the middle of the curve, the quantiles are close together since there is a higher probability of observing a value there. Since each quantile has to have the same probability, then there is only one way to configure the 15 lines that we draw. If that last part doesn't make sense, then just imagine we only had one quantile - so we needed to divide the normal distribution into two equal parts. Where would we put the line? Well, there's no choice involved here because there is only one location for that line - right in the middle. Similarly, when we have to divide the normal distribution into 15 equal parts, there isn't a choice about where to put the lines, there is only one option.
Little confused by the scales for the axis of a q-q plot (I think I'm overthinking the fitting of the straight line part)
What if my sample contains values strictly between 100 - 200 (normally distributed) and I want to check this distribution against the normal distribution. My interpretation of the video is that a q-q plot should return values that fit perfectly on y = x.
However because of the scale of my data, surely this isn't true. Rather is it that we can say that they will fit some line y = x + C(arbitrary constant)? Not even sure if I'm right about this haha...
Is this the latter the correct interpretation? And in a hands on context we run a regression on a Q-Q plot to confirm this sort of thing?
You are correct that the line will not always by y = x, but depends on the values for the different distributions. So y = ax + c is a more "general" line that the data should be on - the important thing is that they data are on a line, any line, if so, then you can use quantile normalization (or z-scale normalization, depending on the distribution) to compare the two samples.
For example, I could compare two normal distributions with very different means, one with mean = 4 and the other with mean = 400 (here's the R code):
data1
Thank you! And as others have said, you're truly a phenomenal teacher!
Hi Josh,I wanted to know that how did you choose the 4 data points from the original set by observing the another set of datpoints containing only 4 data points.
Can you tell me what part of the video (time and seconds) you are talking about?
Yessssss. Makes sense.
Hooray! :)
What a great song :)
:)
Tysm for this
You're welcome!
Hi Josh, can we use percentiles in place of quantiles to plot QQ plot ? If so, in case of percentiles we can only have upto hundred percentile no matter how big our data is then how to have a definitive answer whether or not the 2 datasets have similar distributions as mention in the video at 6:30 ?
The terms "quantiles" and "percentiles" are often used interchangeably, and in this case you can swap out quantiles for percentiles. And you can have as many percentiles as you want - however, the largest percentile is always 100. For example, you could have the 0.5 percentile, or the 1.23 percentile.
@@statquest Thanks Josh.
thanks for the explanation, can you clarify this please?: if we have 15 quantiles, then I thought you should plot 14 red lines in the normal distribution and the 15th line should reside in +infinite. and a little question: is the straight line generated by linear regression?
Plotting a line at infinity would be hard to do and you can fit the line with regression.
Hi!!! Great video!!!! It was very helpful to understand Q-Q Plots!!!! But just one question, how do you calculate the quantiles for your dataset?? I mean, the first observation of your dataset is 0.6, but I don't understand why, since the first observation leaves 0 observations on one of its sides. Should the quantile be 0? In the video where you explain how to calculate quantiles, you explained that the quantile for each observation is calculated dividing the number of observations that this value leaves below between the total number of observations... So, for the first point... 0/15 = 0. Why 0.6??
I think I see the confusion here. The x and y-axes on the QQ-plot (on the right side) are labeled "Normal Quantiles" and "Data Quantiles". This is a little misleading - what we are plotting are the values at each quantile, not the quantile name itself. So if the first quantile is called "quantile 0", but it represents -1.5 in the normal distribution and 0.6 in the data, then we draw a dot at -1.5, 0.6 to represent the first quantile. Does that make sense?
@@statquest Perfectly. I understood this after watching some more videos. I would suggest you to clarify this if you make a new version!! As I already told you, congratulations for your videos and of course, your quick reply!! You explain really well, and the videos are perfect (easy to follow and to understand). I'm doing my Ph.D and it is really helpful people like you. Thanks a lot.
@@statquest Thank you very much for all your videos! They help me a lot. Just a follow-up question on this: how can we decide where to start as smallest quantile value in the theoretical distribution? Like you mentioned, "quantile 0" value in the sample distribution is 0.6, but how can it represent -1.5 in the normal distribution? My confusion is normal distribution doesn't technically have "quantile 0" value because it's infinity on the both tails.
@@Fan-fb4tz On the left side the first quantile is defined for the first point of 15 data points, meaning that 1/15 of the data is equal to or less than that point. Thus, we find the corresponding point on the normal curve such that 1/15th of the area under the curve is to the left of it.
@@statquest strictly speaking, the 15 lines(15 data points) divide the whole data into 16 equal groups or parts，So corresponding to normal distribution should be divided into 16 bins so that every bin has the same probability of 1/16 ,right?