Correcting Skewed Data with Scipy and Numpy
ฝัง
- เผยแพร่เมื่อ 29 ส.ค. 2024
- Skewed data can adversely affect your analysis and machine learning models. In this video, I demonstrate five methods for cleaning skewed data using the NumPy and SciPy modules. The methods include taking the square root, cube root, fourth root, log, and Yeo-Johnson transform. I also showcase the effectiveness of each method by summarizing the skewness of the data after each transformation with a bar plot.
First time I have seen your videos. This is genuinely a very good video. Very well explained and clear. I am subscribing.
The music wasn’t off putting either!
Thank you so much!!! I really appreciate it. If there’s anything you’d like to see just let me know!
@@CJP3Sent you an invite on LinkedIn!
the on-screen text is a great addition, Dr. P!
🙏🏽, I’ll incorporate more in upcoming videos! Thanks for the feedback!
Thank you for the video, subscribed! TH-cam needs more quality content like this.
Bro this is data science ASMR 🤤
Hahaha I didn’t mean for it to be but glad you enjoyed it (I hope) 😂
Amazing video
I like it's structure: motivation, overview with examples, practical advices
Thanks!
Thanks for the feedback! I’ll do more of this style!
Outstanding explanation, professor
Thank you so much!
So what about if we were to standardize using z-scoring? It seems like that would get largely the same impact, wouldn't it?
Amazing video! I was creating a function for measuring the same you forgot to name log1p Wich is log of (x+1) really useful for right skewed data with values less than 1
This is interesting. If one corrects the original skewed data, via doing these kinds of transformations, in the context of linear regression or multiple linear regression, will that not change the interpretation of the original data. Curious to know.
Perhaps, but that change may be for the better. I’d say it’s worth considering these transformation if you know you have skewed data. Many models especially linear models assume normally distributed variables.
I usually build models with and without significant preprocessing and feature scaling/engineering.
Thank you! That was helpful!
So we basically can make the root of any power? Is there a drawbag for exploiting it , like keep increasing the n value for feature to the power of 1/n?
Hi Danny! Context definitely matters. For analytical chemistry 1/n scaling is usually ok. a few downsides are that it makes the models less sensitive to potential outliers. Also its not suitable for certain distributions. Lastly, because 1/n scaling is non-linear, it can make data interpretation more difficult.
Skewing doesn’t necessarily matter if you’re using XGBoost, correct? For classification or regression, that is
Exactly! Skewed data doesn’t impact all model frameworks.
SUBSCRIBED! What should one do before? Or, what's the correct order? - treating outliers, impute missing values, correct symmetry? Thanks Dr. P!
You’re not going to like the answer 😂… it depends a lot on the application. It’s first best to be aware they exist and then evaluate their impact on your outcome. For example if you’re trying to determine outlier samples - then outlier msmts wouldn’t be so bad.. maybe. Or missing values could be useful depending on the application so instead of imputing maybe you engineer a new feature.
Don’t unsubscribe after my answer! 😂 🤣
What if my data contains a lot of useful '0' values?
Howdy! Can you explain more about the 0’s?
@@CJP3 it is a expenditure data where the budget column contains a lot of '0' (not null) values.
please dont add background music
Thanks for the feedback. Most of the newer coding tutorials don’t have background music. Have a great day!
I like it. Don’t listen to this hater!
Bro you explain a concept, but go you need the music!! It’s distracting
I 💯 understand, they newer videos don’t have the music and the audio has a better EQ :)