MINI-LESSON 1: Breaking down intuitively the concept of standard deviation. Why pple don't get it.
ฝัง
- เผยแพร่เมื่อ 20 ก.ย. 2024
- SIMPLIFIED TUTORIAL, 1 (in the series where we break down concepts intuitively):
Before we talk about correlation, let's discuss standard deviation, its analog in dimension 1. People don't get while using it as a metric for deviation!
This is simplified of course (n, not n-1); sorry nitpickers..
More technical: my book at researchers.on...
Also we wrote a paper (Goldstein & Taleb, 2007) showing PhD students in stat-math-finance and fund managers got the point wrong (mismatch between intuition and computation). Yet it is elementary.
This is why the internet is the greatest gift to humanity. If only most people used it for knowledge we as humans could grow so much.
Well said my friend
Blah blah you didnt understand a thing
You’re right. However, many use the internet as a technical surrogate to memorizing, thus becoming digital illiterates as they almost completely depend on sourcing “their” knowledge from the internet, instead of using their own brain as a “database”.
I use it to follow cardi b? You dont !?
@@petergianakopoulos4926 ;-)
"let's talk about STDs" nassim talking about them tail risks
Watch out for them black swans, they'll getcha
Incredible pun
Artem "The GOAT" Lobov.
Here's my summary of what I think he said:
1. Correlation is the relationship between two variables and any dependence they have on each other.
2. Before correlation is explored, just looking at one variable, the tools we have in stats are messed up.
3. Start with Standard Deviation (STD). Look at one variable in one dimension.
4. Most people, even 'experts' think STD is “how much something moves on average every day.” Even if they can write down on paper the 'correct' definition of STD, they explain something else in words.
5. The formula for STD requires you to sum the SQUARE of the differences between each observation and the mean of all observations. Then divide by the number of observations and then take the square root.
6. Because you squared the deviations, you will give much larger weight to large observations and vice versa.
7. Despite the formula, most people think of STD as Mean Absolute Deviation
8. Ratio between STD and MAD corresponds to fat tails.
9. For Gaussian Distributions, STD/MAD = sqrt(pi/2) = 1.25
10. Why use STD? For Gaussian Distributions, it makes some sense because you can use interesting things like 68% of observations fall within 1 STD of mean. It has some probability properties that are useful.
11. Notion of SQUARES in STD is what has messed up stats quite a bit
12. Consider: 1,000,000 observations of some phenomenon. All are zero except one observation is 1,000,000. STD = 1,000. MAD = 2. SAD/MAD = 500. This is because you SQUARED the deviations but only the large one was changed because the others were zero.
13. STD is not intuitive. With a mean of 0, take negative STDs and positive STDs and sum them. They are not additive.
14. So, before getting into correlation and covariance, you have a problem because they use STD and it's not what most people think.
15. In correlation, beta fails in a non-linear environment.
END OF PARAPHRASE
To me, the point is that you should understand how standard deviation is calculated (and thus its limitations) before using it in other calculations.
Great summary,
was a little confused when he did the example so went and worked it out, the mean is 1. so a million is once again the only one that changes but I think this is needed to get the same results. If mean was 0 MAD would be million/million =1 versus (|0-1|*999999 + 9999999)/million ~=2
kindly note that there is a mistake in your observation 1. Corelation is the measure of "linear" relationship between two variables. It cannot capture any non-linear relationship between them.
Thanks
Thank you for this, which I've just read prior to watching the video. I now know that I will not BE watching the video!
@@AnnoulaXeni you can still watch..it will be fun
"That notion of squares is what really has fucked up statistics quite a bit."
- Nassim Taleb
then why is the board he is writing on such a big square?
There is a 9 missing
This is why I love this guy. It doesn't get more unpretentious than this. Stand in front of a blackboard with bad lighting and take 10 mins of your time. No high level production. Nothing. I see the spirit of the old internet here.
Please, please, please continue uploading this content more regularly. I came across your videos tonight and this is exactly the kind of content a college student needs.
Second that. 🙏
not just college students.
Do you really need to thank us for listening? to you? WE should thank you...
The size of the contribution you made to the trading world is unmeasurable.
NNT the man who is a natural philosopher (one of the best of our times), mathematician, trader, and a true hustler!
Btw those of you that don't know NNT makes a killing in selloffs and even during big rallies, most people don't get it but NNT makes money even on big rallies because he always exploits inconsistencies.
Best way to spend 10min. So much power and clarity of delivery. Profound learning.
Absolutely FANTASTIC video, love the crude, natural form of expression, continue this. I wish my stats classes were this interesting.
Nasim please make more videos like this. You clarify so much in such brief and concise snippets
I've always tought that there is something familiar in Taleb, something ancient, like if his person itself carried a whole generation of intelectuals, kind of an archetype of the thinker of the era
bless his heart!
Excellent video. I am not a mathematician or statistician, but I certainly held the misunderstanding of thinking that STD is MAD.Would be exciting to see a follow-up video on examples of where this misunderstanding has caused people to make bad decisions.
Nassim is allergic to not having a sweater over his shoulders
Having sweater over shoulders (during spring) while writing on the board makes u more stoic. Hehe
Ty for the giggle bro
Bag for me
@@Senecamarcus It is sign of flâneurism :)
Cold shoulders run in his family
Very helpful, and perfect timing for me as I had just finished reading chapter 3 of Statistical Consequences of Far Tails today! One small point, I actually think when you ask a layperson about standard deviation, I don't think they're actually thinking of MAD, I think they are more thinking of a time series looking at the average daily change, like the average of x(n)-x(n-1) across n observations.
He is basing his opinion on what people think standard deviation means from experiments he has run asking people with another scholar.
Is it possible to read the book with basic math knowledge (e.g. college degree)?
NNT's {0,0,0, ..., 10^6} example is the quintessential black-swan -- nothing happens, until BOOM! Fat tails everybody ... pay attention.
great video! that example with the million 0s and 1 one million was excellent at demonstrating how the sigma/MAD coef works.
It was my understanding that the reason we use standard deviation (using the square) is a leftover from a time before iterative computation (computers).
absolute value function is not continuous, so using it vs. squaring takes away the ability to do a lot of calculus tricks that where useful for closed form solutions (and when everything is "gaussian")
Amazing as always. Never run for the bus Mr Taleb
YES!! Good comment!!
Hi Nassim! Could you please write a small 100 page book with what everyone should now about statistics/probably?? In similar style to this lecture. I'd buy it in a second :)
It's a good idea. Otherwise, to understand "Statistical Consequences of Fat Tails" you must first learn standard statistics to have the basic vocabulary, but then "undo" that learning.
@Asindu Cool, I'll look into it! Thanks!
Yeah this would be so useful in the world. Would be grateful if he did this.
@Asindu thanks!
Condensing it down that far necessarily makes some of it wrong or so void of nuance that is practically isn't the same concept anymore. That's why people walk out of stats 101 classes with more misconceptions than before they took the class.
That's gonna change my whole perspective about association in medical research papers, especially the ones with fat tails. I'm gonna use MAD instead of STD in my current research and see for myself the change.
holy shit, that's EXACTLY how Ive been thinking of standard deviations... I always suspected I was wrong and just figured I don't really understand statistics or probability (probably true). I bought taleb's book after this incredible explanation that really hit home with me
Always go to mathematics to understand statistical concepts, any metaphors are probably wrong.
That said, the square in the std formula does have some justification in linear algebra. The standard deviation formula with the square is the Euclidian distance between a sample vector and its mean vector, thereby directly showing how "far away" a sample is from a theoretical sample that contains no variability. In the same scenario, MAD would be the L1 distance between the two vectors, which I would personally find much less intuitive. So I don't find it very fair to rag on std for not being intuitive.
To answer the questions, why do we use standard deviation? We are not supposed to use it all the time, but it does produce a simple foundation for proof of the Gauss Markov theorem, as it makes the math really simple. So perhaps it is a good starting point for the intuition required to understand how to then break the rules of the simple linear model. It's really a teaching tool in that sense.
Profesor Taleb you transmit a genuine fun and love for learning about probabilities and statistics.
Excellent accompaniment to my morning cuppa. Excited for part 2 on beta.
In 10 seconds he solved why the replenishment algorithm we created at my work doesn't work 100% of the time.
I'm not sure why anyone would look into standard deviation without first trying to understand the nature of the distribution. Without knowing distribution I'd stick to median and range.
Because most policy makers in medicine, politics, and economics take a single regression course and don't understand anything about actual statistics let alone distributions.
For most, there is only one type of distribution lol i mean they have no concept that there could be another
@@jrab227 You're probably right. I'm trained as an engineer, and not as a statistician, but even to me it seems intuitive to plot data frequencies and look at the distribution before crunching numbers. Only some specific things in nature follow a normal distribution, and there are specific underlying reasons for it.
STD also differs from mean-absolute if normal distributed
Yes, as a chemist, I was told always to do the relevant plots (including histograms etc) before performing any statistics! Never assume or even perform a test to see whether the data follows a particular distribution unless it makes sense!
Least square minimize the entropy of residuals. Thus if you want to earn money from daily movements you will overestimate the most common opportunities while underestimating the rare big ones if you use standard deviation. You try to make every sample residual as typical as possible at the least cost. If the distribution is gaussian the typical values are close to the mean but the aim is the same.
I wish I watched your videos during college. Unfortunately I graduated today :(
Use it in your life time
I don’t understand a word this man is saying. I guess the search continues for me to find true understanding of probabilities.
what a goldmine of a channel
I am so happy to find this course! I am currently reading The Black Swan book (about the fat tails) and I am eager to understand more.
That but in mentally:intuitively reducing std to mad is a freakin eye opener
Taleb seems like a genuinely nice person
He is. Just not to twitter idiots
I think Sam Harris would disagree.
He is very very kind, but Intolerant of idiots acting like they know what they don’t know and leading people astray.
He is very nice to ordinary people but not so to the iyi
@@justinflannery8671 Perhaps that is Sam's fault
Mr. Taleb
Thank you for your great service, bringing to light the problems of probability and statistics.
As someone who has a great interest in the underlying ideas but little to no experience in the formal expressions of them, I'm having a hard time following the reasoning fully. This is not a critique of your lecturing, but rather a question in disguise on where I should turn my gaze to learn the things nessecary to grasp these concepts.
Are there any introductory books in maths, statistics and probability you recommend?
Thanks again, blessings to you.
Adam
This one was simple yet the most eye opening lecture for me so far.
I came here to learn about Sexually Transmitted Diseases (STD) - but now I'm hooked on statistics
You are a gift to the world. It never crossed my mind to find you on TH-cam, so happy that it somehow appeared on my home page.... Love your writings, love your views! And yes, many "scientists" have no fucking clue!
Nassim, get a tripod so the camera is up to your eyes line. Love your videos. That will improve a lot.
I was wondering that just today. Engineers also use STD without knowing why or when it is valid.
Proportional to AC power for electrical engineers
Engineers rarely get STDs
@@randpaul9863 😂😂😂😂😂😂😂😂
@@randpaul9863 unless you are Randy Paul!
@@randpaul9863 damn I got the worst of both worlds then
Thank you Nasim for passing along your knowledge.
Let's go for a lecture from the Man himself.
Not entirely sure, but around 03:24 when talking about weights it sounds like "observation" is used twice in place of "deviation". Like "bigger observations get bigger weight", which I either misheard or it was misspoken.
I think he misspoke - I'd imagine he already had his example in mind
I always thought STD was BS. I asked all the time why MAD wasn't the default in most settings, but my teachers just said "STD makes sense mathematically".
@Hunters Not yet chad enough for Gentoo, I afraid
Hey Nassim! Thank you for your contribution to humanity by correcting all the faulty beliefs that we have been exposed to in this area. I am just curious what you think of Bayesian statistics. I would guess that it would be more to your liking because it sort of allows for intuition and emperical data to enter the equation so to speak. What is your take on Bayesian statistics?
"presently in the republic of Italy" Hilarious. what a flex!
This channel is grossly undersubscribed
Thank you for the video, sir. Please keep making more.
i'm glad he has that sweater on his back like that.
I COULD NOT BELIEVE MY EYES FOR A WHILE. N N TALEB and MOOCS.
I listened to your book's audio on FOOLED BY RANDOMNESS as a crypto trader/investor
I did not think of Standard Deviations that way. It was always a measure of variance in any given data set, rather than average variance per data item. What I did not get was why it was not applicable to fat tails.
Excellent explanation to improve intuitive understanding of standard deviation.
Gaussian distribution is the maximum entropy distribution if there is a fixed mean and variance. If there is not a fixed mean and variance, other distributions maximize entropy
Just to push back a little: the point of using something like STD for normally distributed data is that we shouldn't want a measure of deviation that's sensitive to small deviations near the mean, but one that is very sensitive to large deviations, or those far from the mean. In other words, if a few points very near the mean shift over by some small amount, I don't want my measure of variability to change wildly. But if I have some very aberrant data point in one data set that's absent in another, I want to end up with two very different STDs for those two data sets, which reflects this difference. That is to ask: when data points shift from sample to sample, do we want to treat the same size shifts as equivalent, irrespective of where the shifts occur in the distribution, or do we want to prioritize shifts in aberrant data points, far from the mean?
Coming from Twitter Nassim....!! Loots of thanks...!
He wears his sweater as a cape because he's come to save probability.
Thanks for the class, Taleb!
Looking great, maestro!
We square the difference to the mean to get rid of negative values.
9:59
If it fails in a non linear environment, what’s the solution? Transformations sounds like a cop out
Amazing insights ... We have taught not to interpret these definitions and results are catastrophical
Great info. The graphs of abs(x) and x^2 are approximate to each other in the neighborhood of x = 0, and they become quite different from each other as you move away from the said neighborhood.
Abs(x) vs sqrt(x**2)
There's a misstatement at 3:20 oh, he says you're going to give much greater weight too large observations and a smaller way too small observations. His statement would be correct if he said a much larger weight too large deviations oh, that is differences from the mean. Something that is much smaller than average will also carry a lot of weight
after this video, I found out one crucial thing; I NEED TO STUDY MORE MATH AHHHHHH
Thank you. Good course format, short and visualizing. It was helpful.
Bravissimo Taleb, come sempre! Antifragile e' il mio libro preferito!
Mr. Taleb looks like he’s sticking like glue to his gym routine. I bet you he’s hitting new records on the press which he finds “exhilarating”.
Thank you very much mr. Taleb.
Std dev is often expressed as the 68%, 95% etc rules. But that's only valid for Gaussian. So maybe we should just have a quantile based measure?
Thanks, Taleb. These are much appreciated.
it is also easier to use squares than absolute value when computing regressions (working with the first derivative of f(x)^2 is easier than working with the derivative of abs(f(x))).
I feel vindicated after so many years since my last statistics class when the professor gave a poor explanation for why we use STD instead of mean absolute deviation. Thank you!
Excellent
I have never seen a sweater worn this way.
Nassim, thank you for this video!
Very excited for this series!
In business school, I was fascinated by statistics. Though didn't manage to pursue it further.
Now I have a little more time. Are there any good beginner resources to learn stats from NNT school of thought?
Love the sweater. Keeping it OG.
YES! I WANT MORE OF THESE! THANK YOU.
So there's a parametric exponent/root that is 2 for STD and 1 for MAD. That parameter can be chosen at will, and there is no natural value that automatically establishes scientific validity. Indeed, there are infinite choices that are wrong, at most 1 value that is optimally valid, and possibly (hint: nonlinear cases) no valid value and *no standard*. "Standard" is a misnomer, because all that is standard is "a conventional value that makes the mathematical abstractions easier to manipulate." This is what an engineer would call a "magic constant". It is not "standard" in the sense of "something most likely to be correct" or "what fits our intuitive notion". Thinking that "standard" means "good" is a fallacy. Yet entire industries and much of public policy hinge and operate on this fallacy. It is a pillar of the cargo cultism and magical thinking that passes for science these days.
Let's call it "magic deviation" or "cartoon deviation", because it corresponds to nothing real, just an intellectual cartooning of reality. The IYIs love these cartoons, because it shows off how great their artistry is, and people enjoy cartoons.
And Taleb is just getting started, with this nonlinear criticism. He hasn't spoken the more serious fallacy of probability distributions of real phenomena (not) being continuous, which is a far worse ignorance than assuming linearity when it comes to sifting truth from the numbers. And the incompleteness theorems destroy any potential proof of statistical validity. The best you can do is be anti-fragile.
Thanks for the content. I was afraid I’d have to recalculate all the statistically significant results of my research. Thanks to the normal distribution.
Nassim, the few who think in this world, we need you. Please take care of your weight, you don't breathe right, when you speak. You are making very shallow inhales, cheating yourself out of Oxygen. I bet you don't sleep well either or have Apnea. I will be happy to help, you can drop 50# in no time, while eating well and not at ell doing heavy exercise. It's all in HOW you eat. I got rid of numerous issues - Diabetes Type II, Peripheral Neuropathy, Sleep disorders, mental issues, memory, performance, improved eye sight just to name the few, after I dropped 50#. My entire recommendation will be less than 1/2 page, simple, highly effective. You will live to 120. I want nothing in return, just an honor to be of service to a great man.
They should call it RMS deviation. That would clear up so much confusion. What is the definition of 'standard'? Root mean square deviation more precisely describes what is happening and therefore is much more intuitive to visualize.
Only content worth watching on yt tbf
So as an complete layman, what should I read / listen to to really get the grip of statistic and finally understand what he is talking about :D
Let me know what you find useful. I'd also like to know. :)
I'm a little confused. How would you define standard deviation, then?
According to Wikipedia: the standard deviation is a measure of the amount of variation or dispersion of a set of values.
Would you say that this is incorrect?
Thanks!
Please more videos like this
Why isn't SQRT(xn-x_)² = ABS(xn-x_)? (as said in 4:00). For SQRT(X²) = X (or -X) = ABS(X).
Oh, the problem isn't with the squaring and absolute values. In his example the problem is that in the former, 1/N is sq.rooted, while in the latter, it isn't.
Thank you for your explanation!
Blessed Holy Week and Pascha, Mr. Taleb
Thank you Maestro! Also noticed quite a bit of weight loss!
Std dev is just a numerical measurement on how spread apart numbers are to the mean. Do the khan academy intuition exercises and it really demontrates it with an interactive exercise
No
@@nntalebproba what does it measure then?
@@pebre79😂
2:33 - "The Economist of course, you can always look if you want a wrong definition". LOL!
I don't understand this statement at all, the definitions of basic metrics in econometrics are usually taken directly from statistics.
@@ReTr093 He meant the Economist magazine publication. It is basically left-wing propaganda. Their staff don't understand much of anything, preferring to misrepresent the news to fit their agenda. No real factual reporting, no proper statistical analysis, that stuff just flies straight over their head. Politically corrupt to the core.
@@gmshadowtraders Left wing? Ha!
@@danlewis92 I missed the part where they were for Trump policies. lol nice try. Honestly though, I haven't read that garbage in years.
@@gmshadowtraders Quelle surprise. I watched - endured - the 2020 RNC, and it was notably absent of policy, besides 'school choice'. A party entirely devoid of ideas, opting instead for cult of personality.
I wish he did a video on how to get a healthy tan - like he has. I don't have anything useful to do with all this knowledge of standard deviations now.
Be from Lebanon
Eat 3 or more carrots for 3 days and you'll have a nice golden skintone.
Vacation on the beaches of Palermo, “which is currently in the republic of Italy”.
definitely needed this
"Why the fuck do we use SD?" I died
Thanks again for these explanations !
Thank you Dr Taleb. Could you please explain if the ratio of Standard Deviation to Mean Absolute Deviation would be useful in identifying extreme events in asset pricing?
still I why STD is calculated by squaring the deviation from mean ......if squaring is for weight why square only why not cubing and higher order so on .................. why covariance is calculated Multiplying
deviation from mean of two variables....... is there any anything to do with the joint variation of two variables of direct proportionality with proportional constant as 1............. kindly explain............
Great stuff. Just became a sub. Thanks for sharing
I've always thought that standard deviation wasn't useful as useful as mean deviation in most cases where you're describing things in ordinary terms. Standard deviation is useful when comparing probability density functions (PDFs), viz. the Gaussian PDF, which has only the 1st & 2nd moments (mean & variance) non-zero, vs. others, e.g., like the logistic PDF.
I just checked NNT's calculation @ 6:00 using MATLAB; it checks out (as I'd expect, considering NNT's rep.). Here's the code I used:
x = [zeros(1, 999999) 1e6]; % don't forget the semicolon, if you want to avoid pain
std(x)
mad(x)