In pandas version 0.21 (released October 2017), they added 'isna' and 'notna' as aliases for 'isnull' and 'notnull'. Learn more in my latest video, "5 new changes in pandas you need to know about": th-cam.com/video/te5JrSCW-LY/w-d-xo.html
Even in the final of 2019 your material form 2016 still gives incredible help. I have certainty the DataSchool will keep been a success and helping people. Excellent job Kevin Markham. Thanks.
@@Taranggpt6 hi, that's because there is already a category named "VARIOUS" in the dataset, so the new filled in data gets added up to the existing count of "VARIOUS".
can we get a video on how to handle missing values for data time related datasets. may be sensor values or any sensitive values. multiple varieties of handling missing value would be very useful.
I like his way of teaching, he doesn't assume that the audience knows by default. He breaks down the explanation piece by piece, it is a great learning experience, concise and clear stated lectures as always! Thanks!
6 years has gone released this video and i m watching it now and it still made me fall in love with the series ... beautifully explained every concept in detail.
I have been watching Kevin Videos, needless to say he is an Awesome Instructor. His explanation in all of his videos is Conceptual, In-depth and breaking down any complex topic into the easiest way. Thanks Kevin for your great Work!!! It would be great if you could make videos on visualization using Matplotlib & Seaborn.
I am really loving your videos. Explored your channel just 2 days back!! Earlier I had no idea about pandas but after watching your video, I feel that I will be able to work on my assignment. Great Work! Thank you!
Thanks for all your well-made videos! I got to know you and your classes from Datacamp. As a beginner in the ML field, please allow me to ask a silly question. So if we have categorical features with missing values, do we need to handle missing values first then do categorical feature transformation using encoders? Or the order doesn't matter? Thanks!
Great question! Previous to scikit-learn 0.24, missing values need to be handled first if you are going to one-hot encode them. Starting in 0.24, OneHotEncoder can handle missing values itself. Hope that helps!
Thanks for your videos. most of the python online course i took... i just couldn't get into. Something about your cadence, data sets, and or approach just clicks with me. Thanks for the content.
Thanks so much for making this video. You spoke slowly, clearly and very concise. Other videos I have to rewind and watch over, but i dont have to do that here. Looking forward to watching other.
Awesome, simple and straight to point with code, what i have been looking for weeks. Thank you so much. Do you have any video where you have used NSL KDD or KDD 99 data set to demonstrate data pre-processing as this is driving me naught.
Could you please make a video on how to handling missing values in multiple sheets in pandas? Or any recommendation source that I can read about it? Thanks in advance
Fantastic explanation , however at the end would be good to mention that there are more ways to fill with value_counts , eg. With the mean of all other values etc and not just merging null column with any other column. Cheers!
i rarely leave youtube comment but thank you!! if it werent for your video i wouldn't understand how to do my assignment at all, you did a great job at explaining!
Really helpful. This means if one needs to figure out number of rows with 1 or more Null values, the code should look like dataframe[dataframe.isnull().sum(axis=1) > 0].
Hey, first of all thanks a lot for your videos! One question regarding the fillna() method you use. At the end of the video, when you check the NAs in Shape Reported it said that there were 2644 NaN. However, when you use the fillna() method, it appears that there are 2977 VARIOUS. I dont understand why there are more VARIOUS than NaN? Thanks in advance
Kevin Thank you very much for taking so much time to teaching us about pandas and much more. I have a litte question. Why when you ran this statement ufo['Shape Reported'].value_counts(dropna = False) the output showed NaN it was 2644 rows. but when you ran the fillna ufo['Shape Reported'].fillna(value='VARIOUS',inplace=True) the Various showed 2977. It shouldn't show 2644 ? Thanks for any comments
Great question! I think that the Shape Reported column already had the value "VARIOUS" for some rows, and then I set that value for an additional 2644 rows.
Excellent! Still have one doubt: how do I creat a third column (dummy variable) based on others two columns (dummy variables), considering that they have missing values. I don´t want to lose information, in other words, I want to consider the pair (NaN, 1) or (0, NaN) as 1 or 0.
Great video! I learned a lot! I just wished you talked about non discrete values as well. I'm having some trouble to replace missing numerical data and I don't want to replace them with zero because that would make my dataset biased. My goal was to replace that missing data with the mean of the data that I have.
Thanks a lot for your valuable video, I have a question: how can I drop the nans of all the nan rows except the rows who has values of one or two columns?
Do you have posted any video on how to replace NaN values for a column by deriving it from other columns like using KNN or any other imputation technique
Doubt: Sir , If I want to assign Na to a value suppose 5.Means where ever 5 is present in a DataFrame it will be replaced by Na.then how I have to proceed???? Thanks
Thanks! Regarding fillna, it is limited to filling missing values with defined values or back-fill/forward-fill methods, but it doesn't offer any nearest neighbor approaches. You can read more here: pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.fillna.html
Hi there! Congrats for your videos you're super clear! I was wandering: what if I want to substitute all the NaN values of the Shape Reported series with values chosen from the normalized distribution obtained from ufo['Shape Reported'].value_counts(normalize=True)? How can I ask pandas to evoke values from the frequency distribution of a series? Is it possible to do it directly? Thanks a lot! You're doing super good!
Hi Kevin , What do you do for missing values if all your columns/features are categorical. I understand for numerical we can take mean or most frequently occurring value etc but for categorical features where we just have 'words' as values( may be converted using LE/OHE or get_dummies) what are our options and what in your opinion works best?
For a categorical feature, you can fill missing values with the most frequently occurring category, or you can treat missing values as their own category. There's no universally "best" strategy - it is a judgment call depending on the nature of the problem and your knowledge of the dataset. Hope that helps!
Dear Sir , Small question here , i want to replace "..." in specific column name "Energy supply" . What i am doing is en1['Energy Supply']=en1['Energy Supply'].str.replace("...", "NaN") what this does is it disturbs all other values that are correct into NaN My objective here is to replace "..." to NaN
I would not advise using the text "NaN" to denote missing values. Rather, you should be setting those values to "nan" from the NumPy library. An example is shown in this video: th-cam.com/video/4R4WsDJ-KVc/w-d-xo.html Does that help?
I actually have question, I have a dataframe grouped by month and country. Some of those countries don't have a value for a certain month which is causing anomalies in the visualization. I want to generate a record for the month and the country with zero if no record is found, how can I achieve that? Thanks in advance
Dude it's just an awesome video, forgive me for saying this turning playback speed 1.25 is felts more normal hahah .Love ya, appreciate for your effort about teaching piece by piece !!!!!
Hi, is there an acceptable cut off point for dropping an entire column if it has missing values for a large % of rows? For example if 90% of rows displayed NAN for City, should you consider dropping this col?
There's no rule of thumb... rather, it depends entirely on what you are planning to do with the data. Maybe having 10% of City values present is plenty. Or maybe that column is only useful if it's 100% complete. Or maybe you only want to keep the rows in which City is present. This is where having a deep understanding of your data and the problem you are trying to solve is critical!
hey , how can we replace a NaN value with the previous value in a database like on ufos (shapes ) instead of various you place maybe rectangle shape if it was before the NaN value
In pandas version 0.21 (released October 2017), they added 'isna' and 'notna' as aliases for 'isnull' and 'notnull'. Learn more in my latest video, "5 new changes in pandas you need to know about": th-cam.com/video/te5JrSCW-LY/w-d-xo.html
Even in the final of 2019 your material form 2016 still gives incredible help.
I have certainty the DataSchool will keep been a success and helping people.
Excellent job Kevin Markham. Thanks.
Why after replacing na with *various* the count is different .
Coubts of various must be equal to na values earlier which was 2644
@@Taranggpt6 hi, that's because there is already a category named "VARIOUS" in the dataset, so the new filled in data gets added up to the existing count of "VARIOUS".
can we get a video on how to handle missing values for data time related datasets. may be sensor values or any sensitive values. multiple varieties of handling missing value would be very useful.
@@bragattemas I must say even in 2021 it is still completely up to date
I like his way of teaching, he doesn't assume that the audience knows by default. He breaks down the explanation piece by piece, it is a great learning experience, concise and clear stated lectures as always! Thanks!
You're very welcome! Thanks for your kind words!
@@dataschool first time watch,,,,those positves comments are true,,,,,thanks a lot
I never leave this place unsatisfied or without answers, total treasure.
Thank you so much!
Now i am in love with Pandas just by seeing a couple of your videos, Shukran Jazeelan !
That's awesome! Thanks for sharing!
6 years has gone released this video and i m watching it now and it still made me fall in love with the series ... beautifully explained every concept in detail.
Thank you so much! 🙏
The most amazing python tutorial I've watched so far. Fell in love with python.
Awesome- you are gifted.... -- your explanation and content are clean and effective.
Thanks very much for your kind words!
This is actually the most clearly explained video on DataFrames that I have ever come across. Glad I found you. Thank you so much.
Glad it was helpful!
Great videos covering the basics. I enjoy how the additional values within the functions are covered, i.e. axis, etc.
Glad it was helpful!
I have been watching Kevin Videos, needless to say he is an Awesome Instructor. His explanation in all of his videos is Conceptual, In-depth and breaking down any complex topic into the easiest way.
Thanks Kevin for your great Work!!!
It would be great if you could make videos on visualization using Matplotlib & Seaborn.
Thanks for your kind words, and for your suggestions! :)
I am really loving your videos. Explored your channel just 2 days back!! Earlier I had no idea about pandas but after watching your video, I feel that I will be able to work on my assignment. Great Work! Thank you!
You are doing an excellent job. You are called to do this for sure. Cheers
Wow, thank you so much for your comment! I really appreciate it.
I cannot tell you how much you have helped me, with all sorts of problems! You have the clearest way of explaining things, thank you so much!
You're so very welcome, thanks for your kind words! 🙏
You saved my life twice today, your videos are great and the way you explain is really good. Thank you!
Thank you!
This was awesome. Clear, concise, incredibly easy to follow. Your explanations (and bonus) were exactly what I was looking for.
Excellent! I'm glad the video was helpful to you!
Your way of teaching makes learning Data Analysis very interesting to me. I really appreciate and wish you success.
Thank you!
Thanks for all your well-made videos! I got to know you and your classes from Datacamp. As a beginner in the ML field, please allow me to ask a silly question. So if we have categorical features with missing values, do we need to handle missing values first then do categorical feature transformation using encoders? Or the order doesn't matter? Thanks!
Great question! Previous to scikit-learn 0.24, missing values need to be handled first if you are going to one-hot encode them. Starting in 0.24, OneHotEncoder can handle missing values itself. Hope that helps!
Thanks for your videos. most of the python online course i took... i just couldn't get into. Something about your cadence, data sets, and or approach just clicks with me. Thanks for the content.
That's awesome! Thanks so much for sharing!
Thanks so much for making this video. You spoke slowly, clearly and very concise. Other videos I have to rewind and watch over, but i dont have to do that here. Looking forward to watching other.
That's awesome to hear! Thanks for watching my videos 👍
Great videos. I love how all the CSVs are available online.
Thanks! 😄
u are superb.i took a paid course but they were not able to make me explain these things as u explained me in such a easy way.thnx a lot.
You are very welcome! Thanks so much for your kind comment!
Awesome, simple and straight to point with code, what i have been looking for weeks. Thank you so much. Do you have any video where you have used NSL KDD or KDD 99 data set to demonstrate data pre-processing as this is driving me naught.
I'm sorry, I don't have a video like that... good luck!
Good video. Learnt a lot in short and crisp way
Thanks!
great video series, I always fall back here whenever I'm stuck..Thanx for making them so informative...cheers
Thanks very much for your kind words!
The best pandas tutorial ever. Hands down.
Wow! Thank you so much for your kind comment!
Really clear and amazing tutorial
Glad it was helpful!
Awesome as usual !
Thanks!
Could you please make a video on how to handling missing values in multiple sheets in pandas? Or any recommendation source that I can read about it?
Thanks in advance
Thanks for your suggestion!
Exceptional would be a single word to describe your tutorial. Looking forward to binging on your videos lol. Thank you for such clear explanation.
Thank you! 🙏
Fantastic explanation , however at the end would be good to mention that there are more ways to fill with value_counts , eg. With the mean of all other values etc and not just merging null column with any other column. Cheers!
Thanks!
Thanks a lot, your course is really helpful and very detailed. You are a great teacher!
Thank you!
Excellent video Data School, very helpful, your explanations are clear and objective. Thank you !
i rarely leave youtube comment but thank you!! if it werent for your video i wouldn't understand how to do my assignment at all, you did a great job at explaining!
This video is quite helpful and easy to understand. Thanks a lot!
You're welcome!
The content you shared is Gold!!
Thanks!
This yung gentleman is simply amazing.
Thank you! I'm actually 40 years old now 😊
Great explanation. This was a huge help. Thanks so much!
You're very welcome!
Thank you , You made learning pandas a cake walk.
Awesome, that's great to hear!
Really helpful. This means if one needs to figure out number of rows with 1 or more Null values, the code should look like dataframe[dataframe.isnull().sum(axis=1) > 0].
you are a life saver man...... i was fucked up with errors for only 2 missing values in a row of 1000 data
Glad to hear I could be of help!
Thank you for simple and detailed explanation including the use of features.
You're very welcome!
Thank you so much. It is always clearer to listen to you!!
You are so welcome!
and by the way i love the way you teach , its just perfect
Thank you!
Great video and explanation as always!
Thanks!
Great video btw. Just a quick question. I am trying to build a benchmark, would it be okay to make the data standardized before creating it or?
Amazing, clear, precise and I got it working as well :)
Great to hear!
Explanation techniques is great........want to thank you for sharing your knowledge......Grt videos
Truly amazing videos. Can you do a series on Matplotlib and Seaborn
Thanks for your kind words and suggestion!
This is super helpful, thank you!!!!!
Glad it was helpful!
Hey, first of all thanks a lot for your videos!
One question regarding the fillna() method you use. At the end of the video, when you check the NAs in Shape Reported it said that there were 2644 NaN. However, when you use the fillna() method, it appears that there are 2977 VARIOUS. I dont understand why there are more VARIOUS than NaN?
Thanks in advance
Okay nvm, there was already a group called various with 333 observations
Kevin Thank you very much for taking so much time to teaching us about pandas and much more. I have a litte question. Why when you ran this statement
ufo['Shape Reported'].value_counts(dropna = False) the output showed NaN it was 2644 rows.
but when you ran the fillna ufo['Shape Reported'].fillna(value='VARIOUS',inplace=True) the Various showed 2977. It shouldn't show 2644 ?
Thanks for any comments
Great question! I think that the Shape Reported column already had the value "VARIOUS" for some rows, and then I set that value for an additional 2644 rows.
Excellent! Still have one doubt: how do I creat a third column (dummy variable) based on others two columns (dummy variables), considering that they have missing values. I don´t want to lose information, in other words, I want to consider the pair (NaN, 1) or (0, NaN) as 1 or 0.
Thank you for the clear and quick explanation. Very helpful !!
You're welcome!
very nice tutorial, your style of teaching is awesome like an amazing opera singer
What a compliment, thanks! :)
Great video! I learned a lot! I just wished you talked about non discrete values as well. I'm having some trouble to replace missing numerical data and I don't want to replace them with zero because that would make my dataset biased. My goal was to replace that missing data with the mean of the data that I have.
The only problem is that I don't know how to do that (yet).
Glad you liked the videos! You can do something like this: df.fillna(df.mean())
Sir what you told in the video is applicable only for the numbers and what should be done for the string values?
If you like, you can impute missing string values with the most common values using scikit-learn's SimpleImputer.
ROHIT SINGHAL Hi programmer🔌🤩 pleaaaase see my channel🌹
What a beautiful video and such great explanation. Beautiful. Keep it up
Thank you! 🙏
You are good. Your explanation really made it simple.
Thanks!
great tutor...great way of making us understand.... so easy and intuitive
Thanks!
Thank you so much.Very useful. I have no wordings to appreciate you. I liked your way of teaching very much. You became my ideal in teaching.
You're very welcome! Thanks so much for your kind words!
Great video, how do I get the codes? Does any know or can help? Thanks
See here: nbviewer.jupyter.org/github/justmarkham/pandas-videos/blob/master/pandas.ipynb Hope that helps!
Hi Kevin!
How can I fill na based on a condition? Say I want to fill NA for all missing cities, but only if the color is red.
Great question! ufo.loc[(ufo.City.isnull()) & (ufo['Colors Reported']=='RED'), 'City'] = 'New value'
Man thx btw😁
Thanks a lot for your valuable video, I have a question: how can I drop the nans of all the nan rows except the rows who has values of one or two columns?
all the content in the video are presented clear!!! thanks very much!! we love you!!
And we love you! 😉
Fantastic video Sir.Your work is really commendable.It would be great if you can make a video about imputing the missing values in Python.
Thanks for your suggestion, and your kind comments!
Thank you! Great instructions!
You're welcome!
Do you have posted any video on how to replace NaN values for a column by deriving it from other columns like using KNN or any other imputation technique
No, sorry!
In the last part of the video, why the number of "VARIOUS" made by fillna doesn't match the previous NA number?
Thank you so much! you are amazing as always ! I really appreciate it ! Please don't stop making these videos !
Thank you!
Super. I understood everything. Nice explanation
Thanks!
Superb video! Thanks a lot it helps alot !
Glad it helped!
Doubt: Sir , If I want to assign Na to a value suppose 5.Means where ever 5 is present in a DataFrame it will be replaced by Na.then how I have to proceed????
Thanks
df.column_name.replace(5, np.nan, inplace = True)
check to make sure values are replaced with df.info()
Nice!
Very useful and easily explained.
Thanks!
Great video, would like it twice if I could. Is there perhaps a way to fillna with a user defined function, or a nearest neighbour (that is not NaN)?
Thanks! Regarding fillna, it is limited to filling missing values with defined values or back-fill/forward-fill methods, but it doesn't offer any nearest neighbor approaches. You can read more here: pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.fillna.html
hello and tnx for your greats tutorials, have a question
is there any way to handle exceptions in pandas?
It's no different than exception handling in every other area of Python, as far as I know. Hope that helps!
thank you so much the explanation is very clear
Glad it was helpful!
What inspires a down vote on any of these videos?? Always great content!
Thanks Paul! :)
As always, your videos are very helpful!
Thanks very much for your kind words!
Hi there! Congrats for your videos you're super clear!
I was wandering: what if I want to substitute all the NaN values of the Shape Reported series with values chosen from the normalized distribution obtained from ufo['Shape Reported'].value_counts(normalize=True)?
How can I ask pandas to evoke values from the frequency distribution of a series? Is it possible to do it directly?
Thanks a lot! You're doing super good!
Great question! I'm not sure if there's a simple way to do that in pandas.
Thanks for your kind words!
Hi Kevin ,
What do you do for missing values if all your columns/features are categorical. I understand for numerical we can take mean or most frequently occurring value etc but for categorical features where we just have 'words' as values( may be converted using LE/OHE or get_dummies) what are our options and what in your opinion works best?
For a categorical feature, you can fill missing values with the most frequently occurring category, or you can treat missing values as their own category. There's no universally "best" strategy - it is a judgment call depending on the nature of the problem and your knowledge of the dataset. Hope that helps!
You are simply awesome :) .thank you for making such wonderful videos
That's so nice of you to say - thank you!
You told how to handle NaN values but if there are some other values such as "Not Provided" then what to do?
How to ignore them?
Excellent answer! 👏
Dear Sir , Small question here , i want to replace "..." in specific column name "Energy supply" . What i am doing is
en1['Energy Supply']=en1['Energy Supply'].str.replace("...", "NaN")
what this does is it disturbs all other values that are correct into NaN
My objective here is to replace "..." to NaN
I would not advise using the text "NaN" to denote missing values. Rather, you should be setting those values to "nan" from the NumPy library. An example is shown in this video: th-cam.com/video/4R4WsDJ-KVc/w-d-xo.html
Does that help?
Thank you!!!! All the good stuff, all in the same place...love it!
Great! :)
What about displaying the rows where columns 'A' and 'B' both of them have any missing values?
I watched your first video, you are legend!
Ha! Thanks :)
That's exactly what I was looking for!
Great to hear!
I actually have question, I have a dataframe grouped by month and country. Some of those countries don't have a value for a certain month which is causing anomalies in the visualization. I want to generate a record for the month and the country with zero if no record is found, how can I achieve that?
Thanks in advance
thanks for Nice lecture we love ur sir
You're welcome!
Dude it's just an awesome video, forgive me for saying this turning playback speed 1.25 is felts more normal hahah .Love ya, appreciate for your effort about teaching piece by piece !!!!!
Thank you!
Amazing video, thanks!
Hi, how you can handle the ValueError: arrays must all be same length ? when df.transpone() is not an option?
I think it would be great if you can make lecture about handling missing missing values for machine learning.
Thanks for your suggestion!
Hi, is there an acceptable cut off point for dropping an entire column if it has missing values for a large % of rows?
For example if 90% of rows displayed NAN for City, should you consider dropping this col?
There's no rule of thumb... rather, it depends entirely on what you are planning to do with the data. Maybe having 10% of City values present is plenty. Or maybe that column is only useful if it's 100% complete. Or maybe you only want to keep the rows in which City is present. This is where having a deep understanding of your data and the problem you are trying to solve is critical!
Hi Data School,
Hope you are doing very well,
My Question : Can we highlight NaN value in result data?
Waiting of your appreciate result.
Thanks
There's no simple way that I'm aware of.
Hi let's say I accidentally changed the value like the one I line 19 where NAN is change to Various can I reverse the change?
No, changes made through assignment (or inplace operations) are permanent!
This is great video,I just have one question how do I fill missing 25 city values with the city with most ufo sightings of the same state?
I'm not sure how to do this off-hand. Good luck!
Sir , you are simply awesome !! Could you please tell me from where could I get a complete Data Science Course taught by you ??
Thanks! I don't have quite what you are looking for, but all of my tutorials and courses are here: www.dataschool.io/start/
@@dataschool thank you so much ☺️☺️☺️☺️
hey , how can we replace a NaN value with the previous value in a database like on ufos (shapes ) instead of various you place maybe rectangle shape if it was before the NaN value
I thought we said that no values were missing from the City or Shape reported columns. Why do we see rows dropped at @11:07?
Values are missing from both the City and Shape Reported columns.