Missing Data? No Problem!
ฝัง
- เผยแพร่เมื่อ 7 เม.ย. 2023
- 5 Ways Data Scientists deal with Missing Values.
Check out my other videos:
Data Pipelines: Polars vs PySpark vs Pandas: • The BEST library for b...
Polars for Data Science: • Polars: The Next Big P...
Speed up Pandas Dataframes: • This INCREDIBLE trick ...
Avoid These Pandas Mistakes: • 25 Nooby Pandas Coding...
Links to my stuff:
* TH-cam: youtube.com/@robmulla?sub_con...
* Discord: / discord
* Twitch: / medallionstallion_
* Twitter: / rob_mulla
* Kaggle: www.kaggle.com/robikscube - วิทยาศาสตร์และเทคโนโลยี
![🔴 [TRỰC TIẾP] U20 VIỆT NAM VS U20 THÁI LAN | Cúp VTV9 - Bình Điền 2024 | JET STUDIO](http://i.ytimg.com/vi/tdmZlw-XJJI/mqdefault.jpg)
This is data fabrication which (at least in physics) is academic malpractice and can lead to your data seeming more precise than it is
this is usually used to make the most out of a given data set that is then used to make a predictive model which in and of itself is not 100% accurate anyways. it is not an academically perfect solution but a workable one, that is needed in data science regularly.
@@101McAvoy101 ah this makes sense I guess so long as the graph isn't being passed off as actual measurements thanks for clarifying
@mapo499 your comment just show you don't know shxt about data science. Quite shocked coming from someone with a stem physics background
@@user-vn4jw3ch8w why so hostile? he just didnt know and he isnt wrong about what he said about his domain of experience either😮
@mapo499 your statement is broadly accurate, but there are very few specific instances where what you state will ever be true. The reason being, is that to be sued for malpractice, the evidence of your malpractice would need to show a clear and obvious deviation that would require outlier data points. There are also very few niche topics that would be subject of malpractice, and most of them relate to health and safety, legal and finance professions. Its use in professional practice must be significantly flawed or misapplied to rise to the level of malpractice.
Didn’t know about the built in interpolate, back fill, and forward fill functions. Thanks!
“We make it up!”
If you want to use that to train a neural net or a supported vector machine, you may want to add a new column indicating that the original value was missing ...
Please never drop samples with missing values. This will create a bias in your analysis and will break your pipeline on new data. Mean/Mode Imputation is a great and fast method to handle the problem. If you have non-linear predictive models MM is a great imputation method that doesn't affect the performance.
Thank you. This guy gets it.
Actually, it's fine. Just reduces your sample size. Why would it break anything?
What do you mean with "predictive Model mm?
@@Blacky95Geh predictive models, Mean/Mode
@tomaszubiri4588 mmhm thanks - I just can't wrap my head around the purpose of using mode here. "Mean / rolling mean" I get. Personally I like splining better, as it uses a cubic function which in turn will work better when generating values for missing temperature peaks.
Excellent. Can't remember the last time I learned so many things in such a short time. 👍
Glad you found it helpful!
Awesome! More shorts please!
Glad you like it!
Love me a lil casual data manipulation
so informative, clear, presice and easy to understand
These shorts are awesome; thank you for sharing 😀.
This is awesome! I had no idea you could interpolate like that. Thanks Rob!
Tks Rob! Great content!
Thanks for watching!
Great content. Keep it up man!
Thanks!
Great video! Wish i knew this when i was working with time series data at a past job... those NaNs gave me a couple of headaches for sure 😂
I definitely should read the f..ing pandas manual 😅
Or just watch my shorts! 😊
Please don't use NAN and null interchangeably it will probably confuse people. They are different concepts that have different implications. Null generally means there was an unreported error or there's a serious bug in the program.
NAN generally means either there was nothing collected, or the value calculated/measured is not expressible as a floating point number. This can suggest a flaw in the math performed on floating point numbers.
Null only is possible with numbers that are stored as integers of some kind and NAN is only possible on specifically floating point numbers. The closest thing to null that python has is `None` but it's not identical
While I certainly agree with your main point that null and NaN are completely different concepts that shouldn't be conflated, a lot of the details you gave are incorrect or misleading.
Null and integers don't really have anything to do with each other at all. Sometimes, null can be indicated with a "sentinel value" for integers, but this is generally considered bad practice because it is very error prone.
Instead, in a robust system, null is indicated outside the actual value of something. You can sort of think of this as wrapping your type SomeType in another type NullableSomwType, where NullableSomeType stores a boolean variable indicating whether it is null, and also an instance of SomeType which is only valid to read from if the boolean has a specific value. Rust's Option type is a good example of how this works.
Null does not necessarily mean there is a bug. It can be a completely sensible return value for some functions. For example, if you have a set of survey responses, where some questions could be skipped if the respondent elected not to answer, the only sensible thing to return for what that respondent entered would be null by some name (you might call it None or NotAnswered, but it's serving the same purpose in this context).
NaN is part of IEEE 754, the standard that defines how floating point numbers work. The actual value of a float can be NaN, without needing to wrap the fundamental value type in something like Option. Typically, NaN is returned for floating point operations that have no sensible value, such as inf - inf, 0÷0, sqrt(-1), etc. NaN actually does usually indicate that something has gone wrong, because it means that there *was* a value, but at some point, something was done to it that was invalid.
Thank you sir for help doing time series analysis
Actually the best data science channel. This tips encourage you how to choose a solution for missing info
I love the interpolate method
why is having gaps in time data a bad thing if we drop all NaN values, i dont see why this would cause problems. can anyone pls explain?
With a temperature graph, for example, deleting time slices will affect the overall average temp. Interpolating the missing temps is fine to do in time-series datasets. Not perfect but neither are the prediction models.
Side note: blindly fill is NOT acceptable. It is best to estimate the missing values based on existing information. Interpolation is a good one.
Absolutely perfect.
There is a sixth way, tell the data engineer to build a better pipeline
The true approach:
1. Determine why the data was lost: defective measuring equipment, data corruption, etc.
No point in data if the measurement is bad.
2. Think thoroughly about the topic/domain of ressearch.
For instance, the temperature of what - air in a specific city? Then you could probably find the missing values online by the forecast.
An experiment? Probably, you'd have to repeat it anyways.
3. If you can't find other sources, only then you should choose these methods. And still think about what is measured and why the data went missing.
Great video sir! Which one do you think the best solution for time seriea data?
the last one is very useful!!
Doing awesome job.... Thanx a lot....
Thanks for watching 🙏
Becareful in many situations you must write your interpolation algorithm if you have to fill more than one interval in the data set. And remember in 2023 you can easily generate meaningful synthetic data to fill gap in a dataset.
I'll use this some day
Great video! Is there a way of doing multiple imputation methods in python? Is that what interpolate is doing?
Can you mention the scenarios in which interpolate is used in real-life projects?
In developing a predictive model on sensor data that has missing data.
Useful for the pog champs sleep dataset 😅
Could be! But trust cross validation.
in my study field (thermal engineering), we dont really have any data scientists or even python gurus, just some script kiddies who can barely copy/understand code given by ai, in a master's thesis there was a subject using ai to determine the best config for the required humidity of a room, after collecting data, the student and the supervisor didnt really preprocess data and did the unforgivable: 0ing all the NaNs, the problem is that when they injected the data into an ANN, it gave the worst results ever, and both of them didnt know why so they ended up dropping the idea of using ai coz 'humans cant be replaced yet'
Awesome content
which method is best... or which method should be used in which case... it always confuses me.. pls answer..
nice content btw... keep hustling!
Amazing content as usual.
How do we use interpolation to fill missing values in panel data?
Please, can you reply my comments?🙏🙏🙏
So useful 🎉
Very useful information. Thank 🙏 u
Glad you found it helpful.
Your channel is totally different. You bring to light things that people don't even know exist in many libraries. Big ups to you.
Thanks for telling me that. That’s exactly what I’m trying to do. Thanks!
thank you veryuch😊
Is there a difference with making your own system to handle a task or using a function that is inherent in the language?
In which situation should we use interpolate?
Lets say if i have 15% missing values in a column? Also which one is better interpolate or KNNImputer ?
I would just use interpolation to be honest. I work with sensor data and they rarely jump. With looking through generic data streams and daily record in general, it usually becomes easy to simply interpolate the data. It is easy, versatile and works for waaaay more thing than just missing data in a graph.
How do we account for Greenland? Serious question.
Hey! What if we're working with hourly data and there's a full month missing? (PS: thanks for your content)
Ouch. That’s a lot of missing data. It depends on how many years you have and if you feel comfortable imputing values from previous years. Also, what are you using the data for- model building or descriptive analysis? Lots of context needed to answer your question.
Awesome
Skip > . You just fill up data when there's a recognized pattern
Awesome interpolate.
Thx
when He says interpolate, does He reffer to regression?
Missing data? Make it up! :D
How are missing time values worse than fabricating new data points?
interpolate ... use loess for temperature kinda data
How to do the interpolation in R?
The data is taken every hour, so interpolation will be fine. But personally, i would just drop the NaNs.
why not using interpolate all the time, wouldn’t it solve every case you mentioned?
6. They just go home.
Does ffill and bfill only working with linear variables?
What do you mean by linear. It works for all data types I believe.
@@robmulla Such the situation like a linear regression data or some task like stock prediction need ARMA model and timestamp,sir.
why is there a need to fill missing data?
Man I just do fillna(“it’s too hot”)
You mean this whole time I've been taking the mean of ffill and bfill, but I could have just used interpolate!!??
What about knn imputation?
This gets the job done but obviously it’s inaccurate as the missing value could’ve possibly been a zero, or could’ve possibly been orders of magnitude, higher and valid. We will never know because the data is missing.
thats the difference between a „data scientist“, a computer scientist, and a real data scientist, a statistician. you cant just fill missing data with sourrounding data, no matter how well you interpolate it. If its missing randomly just delete it and if not your fucked. Because if its missing because its a certain value and you still analyse the data there wont be a valid result
Pythond devs be like. So we import pandas to do all pur work for us.
Why do you think it's that simple? Because it is not.
or just u know... remove it from the data and plot the rest?
This helps a lot in a real business situation
I have so much to learn yet just can’t :/
🎉
Or you could leave the data as is and dont make it seem better than it is. why are values missing in the first place?
Don't fabricate data. If no data was available. Leave the empty spots that's relevant and says something different than if you fabricate a trend
In what context. In machine learning imputation is common and sometimes required depending on the algorithm.
@@robmulla If you have missing data, don't use an algorithm that can't deal with missing data, simple as that.
isn't NaN not a number and not null?
Whats the interpolate code to get the average of the last and forward data? Or is the default method basically it?
You can select from different interpolation methods. The default is linear.
Well, technically, if you were to use JS, then... You know... NaN would be a number, not null... And null, ugh... Would be an object...
Alright this confuses me, why are we screwing with the data again?
Rob, quick question. If continues rows have missing values. Should we use Ffill or Interpolate.
I think interpolate would be better. What are your thoughts
It depends on the data and what you are trying to do. If the values are continuous then interpolation can help
So when we use the best?
It depends on the data and what you are trying to do.
@@robmulla Can you explain each one?
is this in pandas?
"Five Ways to Cook Your Data". Highly NOT recommended.
Oh and never believe math from someone who uses 'x' both as a variable and a multiplication sign.
Is it only me who can't see gaps in the plot?
Yea. They are kind of hard to see I agree.
Dont have gaps. Ez
You need to understand the NULL values before you do anything with them. Are they TRULY outliers or is there a specific reason they're there? In some cases there's a real reason they're present and can drastically change your analysis or how you end up modelling something. I'm experiencing this right now with a project at work, coincidentally.
Let me just insert some missing values.... Look! climate change!
imputation methods
Always interpolate. Got it!
Just learn 17th century maths ->interpolation. Otherwise crazy skills to call this function
Awsome, but NaN is not NULL
This is malpractice
I hate this video and represents everything wrong with people dumbing down data science. If you aren’t able to figure out this stuff by simply looking at the documentation for fillna, then you are not a data scientist. Any data scientist knows what interpolation means, you don’t need to flash the equation on the screen to make it seem complicated
newdf = dropna() problem solved 😂😂
Sometimes, yes.
@@robmulla thanks for your short bite size lecture that stick insanely well :)
this doesn't feel very scientific!
Ah yes, how to bastardise science.
How to not understand data science.