Clear, to the point, with real life examples. I've been learning pandas and I decided to do a recap on math. The fact that you provided examples in pandas is the happiest coincidence I've come across this week. Thank you!
Thank you Sir, for making such kind of beginners friendly videos. I really enjoyed and learned a lot. Please make make more such kind of videos so that we can understand easily. ❤️
Your way of teaching is incredible, I love your videos. Whenever anyone ask me from where you learn all this then, I share link of ur channel to my juniors.
Hi Dhaval, Greetings! Thanks a lot for creating wonderful videos for everyone and these videos are invaluable. I was watching your Mean, Median, Mode video and found one issue with the example at 2 mins and 47 secs where you explained the median of even values by mistake you didn't sort the values because the values you took for a median were (7000+8000)/2 = 7500 but instead of this it should be (7000+7500)/2 = 7250.
In the median example at minute 2:40 , shouldn't we order the values first before guessing about which value is the median? shouldn't the values be like that: 4,000 < 5,000 < 6,000 < 7,000 < 7,500 < 8,000 < 8,000 < 10 million so, the median would be the average of 7,000 and 7,500 which is 7,250
Thank you very much for detailed and nice explanation. Have a question, Do we need to remove outlier all time? What if the salary range is constant not like unusual high salary(Elan musk as per mentioned use case)?
Hi sir, thanks a lot for your extraordinary teaching, I have learned lot and did my homework by following your machine learning tutorial. Sir, Can you do for a video about Generative Adversarial Network (GAN) for regression prediction?
Hi Dhaval, Greetings! Thanks a lot for creating wonderful videos for everyone and these videos are invaluable. I was watching your Mean, Median, Mode video and found one issue with the example at 7 mins and 17 secs where you explained the 25th percentile of the dataset. Here the 25th percentile is 1.75 which is less than 2 data points. As per my understanding, it should be between the first two data points (4750) or approximately the second data point which is 5000. Please let me know if my understanding is incorrect. I really appreciate any help you can provide.
Hi! in time = 2:44 for the median you take Tao and Prem, but they must be first sorted and Prem it is not counted in the median, but Sofia do. So m=(Tao + Sofia)/2?
Sir Your quartile calculation seems to be wrong. The formula for the rank of 25th percentile is 25/100*(7+1) which is 2. This is universally accepted. It means the value should be 5000 only. I really don’t know how pandas is also doing the same mistake
Sir , how is the median of the data points 7500 , since the median has to be the average of Tao's and Sofia's income so it will be (7000+7500)/2 = 7250 right.. So I meant after arranging in ascending order
percentile_95=df.price.quantile(0.95) sir i do by this approach, it give 350 something if i increase value of quantile outlierss comes max gap that's why i remove all values upon this condition
am I just scatter brain, or did you not include the link to video where you explain how to use iqr to remove outliers? I only see a link to a playlist, but none of them seem to be on that particular topic? EDIT: okay, seems you explained it later in this video, but it really sounded like you had a link for us...
Again Great Video Sir. I have a silly doubt. As you said we cant take average to fill null value if outlier have very large value like Elon musk(10 million$) and now we are going to take Median to fill na values.but nan values itself present at the middle of datapoints .So how we gonna calculate median if nan value is present at those points. median=(nan+nan)/2 ?
I love maths even though I flunked this subject in college, I have 5plus years of dev experience want to move to Data science pls share some beginners project
when we can use the median??? when we have an outlier use case 1: (descriptive analysis التحليل الوصفي for cars) median ---> odd it is the middle median --> even is adding both numbers / 2 (their average) use case 2: Handling missing values 4:27 summary 4:52 Outlier: is a datapoint is very different than the rest of the data points How to remove the outlier? percentile النسبة المئوية 50th percentile of this dataset is 7000 (the median) = 50% of my data has value < 7000 and 50% of my data has value > 7000 100th percentile? what is the data for which 1oo % of that data is < or = that value? A\ 10M 0th percentile? obviously is 4000 what is the 25th percentile? ❌❌median / 2 ??❌❌ 🟩🟩🟩🟩total values of data = 7 25% of 7 ??? 0.25 * 7 = 1.75 -- but this is not a (Fractional Number or Decimal) so approximately will be 2 data sets or two values of data = 7 so what we will do is we pass the first two values and after will be the 25th of the data set which is the middle data between that and the data after (the data before + the data after) / 2 = (5000+6000)/2 = 5500 So what does the 75th percentile mean what is the 75% of the 7 data 7*0.75 =5.25 (7500+8000)/2= 7750 The range between the 25th and the 75th percentile: IQR: Interquartile Range normal range is the range between the min and max but the IQR
One very basic question - Should the outlier removal be applied on labels (values to be predicted) as well if outliers exist on such data labels as well ?
Nice video. I would like to suggest a change. 100th percentile doesn't exist, only 99th. In your example, Musk would have to be earning higher than himself to be the 100th percentile.
i am a jr. data analyst with less than a year experience if i apply for jobs is it expected of be to be able to code advance python funcions? cause now i feel that i am just able to understand code by debugging it but if i try to write similar code i am not able to but i k what function does what and if i a problem statement is given i will be able to identify what thing we should be doing to achieve the result but i am unable to implement it. please give your opinions on this. cause coming from non-it i am always havinng a sens eof insecurity that i dont know python enough.
I'm near about 50 . I have completed MCA from IGNOU and Digital marketing from NIIT imperia. I worked as a software developer and now im a digital marketer. If I want to change my career in data science after learning this field, can i get a job in data science field?
Hi , I have a dataset where 3 columns are independent categorical features and 5 dependent features that are 10th ,25th, 50th ,75th , 90th percentile of annual wage. How can I get values (annual wage ,which is missing) from the 5 percentile columns ?
Consider my data points: 100 100 100 100 here the 50th percentile which is 100 is kinda misleading right? because 2 more 100 values are present in the right side of median. SO.. 100% of the data values are equals to 50th percentile. Can you please explain where I am confused??
For suppose the data is like this 4,4,6,7,40,100,110,120,1300...in this case taking median doesn't make sense right ....same for mean outlier 1300 involved...and for mode also 4,4 just repeating 4 for 2 times doesn't make sense right... What to do in this case please any one answer me ...could we find solution from this video..
Taking mode of 4 is perfectly ok because you are looking for a value that is most frequently occurring and 4 is that value. It really depends on what problem you are trying to solve here. Can you suggest what type of dataset this is? You just made up the values and are generally curious about such distribution?
@@codebasics I just take it as an example...but just for repeating 4 for 2 times blindly we can't take 4 for filling the missing value right because it is far less than other higher values
Hi, I'm a bit confused with the solution of the exercise. To me, the outlier is not simply removed by percentile, we should exclude the line with 365 availability and 0 reviews + 0 availability and 0 reviews because those lists are just "ghost" lists that no one actually rent them or just the data is not accurate. If we go further down, we should probably clean the data by review date also, I see some of them are with 2011 date, but if we are analyzing the average of this/recent year then there should be a cut off of the latest year we can use. Please let me know your thoughts. Thanks.
Totally agreed with your thoughts here. Percentile is just one of the ways, using common sense simple logic is totally a legit way of treating outliers
@@codebasics Thank you for replying to me so quickly, so if I apply what I said in the post first and then apply percentile, is that going to be right, or let's say with better accuracy? Also, how do we measure the accuracy? should the mean be close to the 50% percentile? how do we know our analysis is good or bad? Thank you so much!
Subscribing to this channel proved to be lot more helpful than enrolling into college for graduation
You are the best teacher and have the best content on data analysis. NO need to go any channel.
I am happy this was helpful to you.
Mean, median, mode and percentile are also known as 'Measures of Central Tendency'.
Yeah bro I listened them from Khan Academy
Thank you so much Sir you're a good teacher and you're different from others because of the practice you demonstrate
Clear, to the point, with real life examples. I've been learning pandas and I decided to do a recap on math. The fact that you provided examples in pandas is the happiest coincidence I've come across this week. Thank you!
2:45 When you added an extra value you did not sort them in ascending order (7000,7500,8000) instead of (7000,8000,7500).
As a. Beginner, I should say this is the best.
I have no words to say, really awesome series!
Thank you Sir, for making such kind of beginners friendly videos. I really enjoyed and learned a lot. Please make make more such kind of videos so that we can understand easily. ❤️
Tell me Your insta id bro...plz
Your content and examples are great😃.
Please don't stop making such easily explained content.
The best video to understand the concept of removing outliers
Sr please make one video for freshers on real life data science project, your teaching skills are so simple everyone can understand very easily
This series are masterpiece. Thank you.
Yeah . So true ...*uck education system
Before exploring the Codebasics channel. I never had an interest in Math & Stat. Thanks, Bro. Love & Respect from Pakistan
I am happy this was helpful to you.
you earned a sub man!!! what an explanation
Your way of teaching is incredible, I love your videos. Whenever anyone ask me from where you learn all this then, I share link of ur channel to my juniors.
Thanks for sharing! I am happy this was helpful to you.
Hi Dhaval, Greetings!
Thanks a lot for creating wonderful videos for everyone and these videos are invaluable. I was watching your Mean, Median, Mode video and found one issue with the example at 2 mins and 47 secs where you explained the median of even values by mistake you didn't sort the values because the values you took for a median were (7000+8000)/2 = 7500 but instead of this it should be (7000+7500)/2 = 7250.
Ultimate Explanation🎉 Got a good idea on using mean and medain
In the median example at minute 2:40 , shouldn't we order the values first before guessing about which value is the median?
shouldn't the values be like that: 4,000 < 5,000 < 6,000 < 7,000 < 7,500 < 8,000 < 8,000 < 10 million
so, the median would be the average of 7,000 and 7,500 which is 7,250
Sir your way of teaching is very awesome
furthermore to learn, this was even a enjoyable video, thanks a lot sir.
Sir upload real life data science project 👍😁
On TH-cam search for "codebasics data science project", you will find my videos please watch it
Thank you very much for detailed and nice explanation.
Have a question, Do we need to remove outlier all time? What if the salary range is constant not like unusual high salary(Elan musk as per mentioned use case)?
Beautiful explanation
Glad it was helpful!
Hi sir, thanks a lot for your extraordinary teaching, I have learned lot and did my homework by following your machine learning tutorial. Sir, Can you do for a video about Generative Adversarial Network (GAN) for regression prediction?
Exactly what I wanted a mentor 👍🏻❤️🙂.
Hi Dhaval, Greetings!
Thanks a lot for creating wonderful videos for everyone and these videos are invaluable. I was watching your Mean, Median, Mode video and found one issue with the example at 7 mins and 17 secs where you explained the 25th percentile of the dataset. Here the 25th percentile is 1.75 which is less than 2 data points. As per my understanding, it should be between the first two data points (4750) or approximately the second data point which is 5000. Please let me know if my understanding is incorrect. I really appreciate any help you can provide.
Only one word loved your explination
Really great no raatta
Very informative video.
Hi! in time = 2:44 for the median you take Tao and Prem, but they must be first sorted and Prem it is not counted in the median, but Sofia do. So m=(Tao + Sofia)/2?
Sir
Your quartile calculation seems to be wrong. The formula for the rank of 25th percentile is 25/100*(7+1) which is 2. This is universally accepted. It means the value should be 5000 only. I really don’t know how pandas is also doing the same mistake
One of the best tutorial ❤️🔥
Glad it was helpful!
Awesome learning 🆗😎👍
Glad you enjoyed it
While calculating the median( when data values are even) we need to sort data values in ascending order.
Thanks for making this video its very helpful
Sir , how is the median of the data points 7500 , since the median has to be the average of Tao's and Sofia's income so it will be (7000+7500)/2 = 7250 right.. So I meant after arranging in ascending order
percentile_95=df.price.quantile(0.95) sir i do by this approach, it give 350 something if i increase value of quantile outlierss comes max gap that's why i remove all values upon this condition
2:43 there should be sorted values and median will be equals to (7000+7500)/2
in the example at 3:00 u havent sort data in ascending order for median
Example of Mode is lit 😀
great sir 🥰
Thank you that was very informative content.
am I just scatter brain, or did you not include the link to video where you explain how to use iqr to remove outliers? I only see a link to a playlist, but none of them seem to be on that particular topic?
EDIT: okay, seems you explained it later in this video, but it really sounded like you had a link for us...
mimosvera, you are right I forgot to include a link but I just added it now. Please check video description
Again Great Video Sir. I have a silly doubt. As you said we cant take average to fill null value if outlier have very large value like Elon musk(10 million$) and now we are going to take Median to fill na values.but nan values itself present at the middle of datapoints .So how we gonna calculate median if nan value is present at those points. median=(nan+nan)/2 ?
maybe you can take the median of non-null values and fill up
I think for taking median of dataset first we have to rearrange data to ascending order that will shift position of Nan value!!
@@shutterup24-7 yes thats the first step
Sir apney last maen outlier ko remove kaisey kia ?
do you have any full course on data analysis?
At 3:02 adding prem to the dataset is disturbing the ascending sorting order. So the median should really be 7000+7500 / 2 = 7250.
Thanks for such a nice tutorial
Glad it was helpful!
I love maths even though I flunked this subject in college, I have 5plus years of dev experience want to move to Data science pls share some beginners project
Im hoping that by end of covering all playlist ill become master at data science and the following know of ML,DL,LLM
links to softwares used?
why is using median better than leaving musk out and getting the average of the rest? is it compulsory for all data to be used?
Sandeep Jain sir GFG samjhne wale haath uthao😅😅
when we can use the median???
when we have an outlier
use case 1: (descriptive analysis التحليل الوصفي for cars)
median ---> odd it is the middle
median --> even is adding both numbers / 2 (their average)
use case 2: Handling missing values
4:27 summary
4:52 Outlier: is a datapoint is very different than the rest of the data points
How to remove the outlier?
percentile النسبة المئوية
50th percentile of this dataset is 7000 (the median) = 50% of my data has value < 7000 and 50% of my data has value > 7000
100th percentile?
what is the data for which 1oo % of that data is < or = that value?
A\ 10M
0th percentile? obviously is 4000
what is the 25th percentile?
❌❌median / 2 ??❌❌
🟩🟩🟩🟩total values of data = 7
25% of 7 ???
0.25 * 7 = 1.75 -- but this is not a (Fractional Number or Decimal) so approximately will be 2 data sets or two values of data = 7
so what we will do is we pass the first two values and after will be the 25th of the data set which is the middle data between that and the data after
(the data before + the data after) / 2 = (5000+6000)/2 = 5500
So what does the 75th percentile mean what is the 75% of the 7 data
7*0.75 =5.25
(7500+8000)/2= 7750
The range between the 25th and the 75th percentile: IQR: Interquartile Range
normal range is the range between the min and max but the IQR
14:33 what if you have thousands of data points and you don't know the 100 pctile ?
Why don't we fill missing values with mode?
Mode is the one that appears most but why we use mean and median most of the time?
so clear, thx.
Glad it helped!
16:04 df.income.iloc[3] =Nan will work too
Hi Sir,can we use multiple median for multiple NaN data like you did in sofia;s case?
So, How to identify there is an outlier in the dataset? please calrify
Is the amount of statistics required for data science and data analytics the same?
One very basic question - Should the outlier removal be applied on labels (values to be predicted) as well if outliers exist on such data labels as well ?
No
Sir, could you please add the assignment link?
Nice video. I would like to suggest a change. 100th percentile doesn't exist, only 99th. In your example, Musk would have to be earning higher than himself to be the 100th percentile.
i am a jr. data analyst with less than a year experience if i apply for jobs is it expected of be to be able to code advance python funcions? cause now i feel that i am just able to understand code by debugging it but if i try to write similar code i am not able to but i k what function does what and if i a problem statement is given i will be able to identify what thing we should be doing to achieve the result but i am unable to implement it. please give your opinions on this. cause coming from non-it i am always havinng a sens eof insecurity that i dont know python enough.
Sir I want to know which language is very important? Should we start with Java or python
Sir can u explain the steps to become a data analyst and skills required for that
On TH-cam search for "codebasics learn data analyst skills", you will find my videos please watch it
@@codebasics tq sir
Sir what if the data is missing from or below 25% ,75% then how to find The Average.please reply
Is careerera a good institute to join as a beginner.
Im final yr ECE student.
Sir there could have been possibility that sofia's income would really high then median will not work well?
What is the difference between 0.99 and 0.999 quantile range as in exercise 0.999 is used?
I'm near about 50 . I have completed MCA from IGNOU and Digital marketing from NIIT imperia. I worked as a software developer and now im a digital marketer. If I want to change my career in data science after learning this field, can i get a job in data science field?
Hi , I have a dataset where 3 columns are independent categorical features and 5 dependent features that are 10th ,25th, 50th ,75th , 90th percentile of annual wage. How can I get values (annual wage ,which is missing) from the 5 percentile columns ?
nice
what is the difference between Linear Quantile and Midpoint quantile ??
why cannt we use trimmed mean?
What is the difference between average and mean?
Should we first learn pandas then attempt exercises?
Sir, I need your suggestion. Can you help me ?
Power Bi KO Course Kaha Cha Hola?
🙏🏻
outliers kaha hai?
How can i code in Jupyter, just like you.
Consider my data points: 100 100 100 100
here the 50th percentile which is 100 is kinda misleading right? because 2 more 100 values are present in the right side of median. SO.. 100% of the data values are equals to 50th percentile. Can you please explain where I am confused??
I am not sure still, you can double-check with someone else too .In your case, you should consider mode as your measure and ignore mean or median.
Sir tamari sathe contact kai rite kari saku?
It is good if you should have taught why not median and mode in some cases
how do you know that your data has oulier
For suppose the data is like this
4,4,6,7,40,100,110,120,1300...in this case taking median doesn't make sense right ....same for mean outlier 1300 involved...and for mode also 4,4 just repeating 4 for 2 times doesn't make sense right... What to do in this case please any one answer me ...could we find solution from this video..
Taking mode of 4 is perfectly ok because you are looking for a value that is most frequently occurring and 4 is that value. It really depends on what problem you are trying to solve here. Can you suggest what type of dataset this is? You just made up the values and are generally curious about such distribution?
@@codebasics I just take it as an example...but just for repeating 4 for 2 times blindly we can't take 4 for filling the missing value right because it is far less than other higher values
why 0.999 in the exercise ?
in case of even n.of data point you have not sorted them so median is wrong
Legend
2:44 median will be 725
The funniest part is if Elon Musk lives in our town😂😂
Ha ha.. yes he is my neighbor ☺️🧐
removing elon from twitter as an outlier is also great
Hi, I'm a bit confused with the solution of the exercise. To me, the outlier is not simply removed by percentile, we should exclude the line with 365 availability and 0 reviews + 0 availability and 0 reviews because those lists are just "ghost" lists that no one actually rent them or just the data is not accurate. If we go further down, we should probably clean the data by review date also, I see some of them are with 2011 date, but if we are analyzing the average of this/recent year then there should be a cut off of the latest year we can use. Please let me know your thoughts. Thanks.
Totally agreed with your thoughts here. Percentile is just one of the ways, using common sense simple logic is totally a legit way of treating outliers
@@codebasics Thank you for replying to me so quickly, so if I apply what I said in the post first and then apply percentile, is that going to be right, or let's say with better accuracy? Also, how do we measure the accuracy? should the mean be close to the 50% percentile? how do we know our analysis is good or bad? Thank you so much!
The video is great but i didnt like the exercise because there is more in it than it has been covered in the video
Sorry sir , you miss one part in the video first we have to sort the nos. ( When the count of no is even (while finding median )
1:55
This is legend . Go to hell teachers and education system...
I didn't get percentile in first glance