Yeah. Except I'm no 'rookie' I'm a seasoned veteran of what was called back in the day, a "database programmer.' We carved our code into stone with flint arrowheads. Now, things are very different. No more stone carving. We've got "Full stacks" now. So it took me about oh... couple of weeks to get any data on my console. Still can't get the "Full Monty", maybe because 3.8 GHz Pentium and 8 Gb isn't enough desktop power? Hard to believe, but hey, it's 2023. Time to face Big Data. By the way, I studied computational biology recently and the amount of DNA sequences is known as "totally insane." Back to thanking Mr. Mulla. Can't thank you enough for getting the important imports imported in the correct order not to confuse the wonderful "Full Stack" too much!
I am working exactly on my programming assignment of the date manipulation and it is very confusing. With the PyArrow datatype support, it is a lot easier from now. Thank you for highlighting the significance to us. You are a saint.
Rob, you are making the best videos. I am always watching them and learn new stuff. I learned python and pandas myself, and your videos helped me improve them several times fold. thank you!
ah yess the much needed Pandas improvement! this is what I like in a competitive environment..it sharpens everyone involved. I agree with what has been said in the comment section: now we can use Pandas to handle large dataset properly. especially that 20x speed improvement when reading data using Pyarrow backend is so good!
I agree. I'm excited to see how it takes off once officially released. The only problem is people won't be able to ask chatGPT how to properly use pandas 2.0 since chatGPT only goes up to 2021 😆
In a few projects I’ve worked on recently, I found that switching from Pandas to pure Numpy speeded up execution time from about 30 minutes to less than a second on my hardware with about 50m rows of data.
I was about to switch to Polars but now with Pandas Arrow dtypes, I need to do more research. Speed is one thing but lazy processing and memory usage are very important.
That’s a good question that I should experiment with. The article says the only thing it actually needs to move is metadata, so I’m guessing it doesn’t take very long at all.
My very first thought was: yeah, with pyarrow as a backend, everything single software that relies on numpy has to be rewritten. But at the ends it doesn't looks like a big problem or a problem at all.
Rob, do you have any limitations of Arrow to share? Until recently, for example, if I'm not mistaken, Arrow didn't seem to handle Pandas' multi indexes.
That’s a good question! I’m sure we will know a better answer once people start using it more. I forgot about the multindex limitation! I need to see how that works in pandas 2.0
When you imported Polars, I thought you were going to do a versus on speed between Polars and Pandas with the new backend. But still very informative video, thanks so much!
When you swap between pandas and polars, is python creating a copy of the dataframe into memory or is it just referencing the already allocated block of memory with the pyarrow backend?
Great question. It's a little complicated and for things like metadata there is a copy made, but my understanding is that the underlying data is not copied. The article I've linked in the description goes into a lot more detail about it and I'd suggest checking that out.
Just one thing I don't understand. If boolaens are stored on single bits (really a great idea), how comes they can have 3 states (True, False, None). Wouldn't that require 2 bits ?
Can you make a video giving techniques to update pandas (or polars) columns using vectoring instead of .apply() for more complicated custom function calls? eg. I want to take a str column, split it on a delimiter, do work on each section and then combine them back and return it as the new value for the same column
Thanks very much Rob! Your content is off the charts good, a true outlier, and a great outlier! To get my grad degree in stats, I had to take a PhD-level course in statistical consulting. The prof for that course actually wrote a whole book solely on the topic of outliers, outliers can be extremely important. To me, data science is a subject of stats, similar to the way experimental design or survey sampling are subjects of statistics. I think it is unfortunate that data science is not widely seen as belonging to “statistics” since it makes it harder (at least for me) to use data science tools when teaching stats, as the expectation is to use an old school approach to descriptive and inferential statistics in the classroom. I think your content has the clarity and insight to convince teachers that data science methods MUST be brought into the standard curriculum, e.g. courses like Introductory Business Statistics. Thanks again for posting truly remarkable content!
Thats a good idea. I'll try to think about how I could do that. There might be some settings for limiting the CPU use but also I know h2o does some benchmarks across different libraries.
Hey, your DS videos are awesome! I was using ChatGPT to learn more about Apache Arrow, Polars, etc from all your videos. Can you do a coding livestream or video about bioinformatics data like biological sequence data or drug development data? Thank you!
One day, I learned that Python was slow for large data, so I started to study Julia who's reputation is "the fastest". One programmer I talked with said to me "No Python is fast!" And maybe new Pandas helps for speed matter. How should I take it from your point of view?
do you think this can have any "consequences" to the develop of numpy? I use pandas for data management but also use numpy and scipy and is very handy the way you can work with all of them till now. don't know if arrow can make the job so easy. also, can this make things more difficult for plotting stuff with matplotlib and seaborn, i.e.? p.d. thanks for the video and also for the previous 1h live stream in-ter-oper-a-bi-li-ty :D
Haha. I pronounced interoperability in this video 😂. I don’t think numpy is going anywhere. When working with non-tabular types of data numpy is strong. Also libraries like PyTorch and Tensorflow allow you to convert between tensors and numpy and those are very popular.
Hey Rob, Great stuff! .. I moved my data transformation app to PD2.0 and I have a problem with data conversion. When Im converting data from int[pyarrow] to string using astype(str) im loosing pyarrow and pandas convert it to object :/. I cant find the tool to do it properly. All best!
With a library as established as pandas, I don’t think they want to implement breaking changes, so having it as an option, at least in early releases I think is the right choice.
It's a very good Video and this Library a good improvement, however ^^ nobody stops you from using a single int and store 32 different boolean states into it with bit shifting operations, but using a Interpreted Script Language means from the beginning "I trade resources for convenience". The underlaying Engine of Python is C and in C there is no actual boolean datatype, the smallest unit is the exact smallest unit the Operating System can provide which are 1 byte / 8 bits which the Operating itself only can provide to you if it reserves a whole virtual memory page for you. I just think, to nitpick about 7 wasted bits as a Python Programmer is a little bit awkward. The rest was great, thanks for the Video
Wow! I'm still struggling in pronouncing "Interoperability" and it came out so fluid out of your mouth. Bet you are to release a video comparing pandas 2.0 vs Polars. Is Numpy still in pandas 2.0 due to backward compatibility?
Wait... did I actually say "Interoperability" correctly? I filmed myself saying it a bunch of times and just keep the one that sounds the best :D To answer your question yes, pandas has backwards compatability, as I show in the video by default it still uses the numpy backend.
I think eventually it will become adopted by most users, but pandas doesn’t want to make major changes. It also might be some time before it’s stable enough to trust in production code.
Good catch! I think the bool values use a single bit but every column has a null array associated with it. But I also don’t know that for sure. But that would mean that a nullable bool would me more than one bit.
Hmm, Arrows seems great, but I had no idea something as fundamental as Pandas and NumPy was so janky, with really really sketchy quirks. Now I'm wondering what the speedup would be if you tried these operations with a system that wasn't using at least some scripting language interpreter calls for raw data shuffling? Like just a normal struct.
@@robmullahey Rob actually haven't updated my yt using one year older version of TH-cam I always get 5 sec ads and i can also download videos in 1080p idk how that's possible but working for me even sometime i am able to see private videos those youtubers also always asked me how i am able to play their private videos. Conclusion:- Google is stupid
Thanks for the feedback. Unfortunately the TH-cam algorithm favors videos with higher click through rates. I’m experimenting with more enticing titles although I agree with you I wish I didn’t have to.
I'm not sure what'd i'd be without you and this YT channel Rob. Thanks for being and amazing teacher to a rookie like myself
lots of respect and love
Thanks for the feedback!
Yeah. Except I'm no 'rookie' I'm a seasoned veteran of what was called back in the day, a "database programmer.' We carved our code into stone with flint arrowheads. Now, things are very different. No more stone carving. We've got "Full stacks" now. So it took me about oh... couple of weeks to get any data on my console. Still can't get the "Full Monty", maybe because 3.8 GHz Pentium and 8 Gb isn't enough desktop power? Hard to believe, but hey, it's 2023. Time to face Big Data. By the way, I studied computational biology recently and the amount of DNA sequences is known as "totally insane."
Back to thanking Mr. Mulla. Can't thank you enough for getting the important imports imported in the correct order not to confuse the wonderful "Full Stack" too much!
I am working exactly on my programming assignment of the date manipulation and it is very confusing. With the PyArrow datatype support, it is a lot easier from now. Thank you for highlighting the significance to us. You are a saint.
Thanks Rob! Couple months seeing your videos and your content is on the spot ! 🎉
Hey! I apprecaite the feedback Marcos. I'll try to keep it up.
Your channel is pure gold.
Thanks. I appreciate that!
Great video Rob! I think Pandas will continue to be very useful in the data science community.
Glad you liked the video. I agree, I don't think it's going anywhere anytime soon and 2.0 is a good move for them to adapt to newer backends.
I agree, they used the right approach. Don't break legacy code, while allowing people to refactor it easily when necessary
Epic video Rob!
I use pandas everyday and glad to hear it’s getting faster!
Rob, you are making the best videos. I am always watching them and learn new stuff. I learned python and pandas myself, and your videos helped me improve them several times fold. thank you!
Thanks for sharing the update! Very well articulated
Appreciate the feedback!
ah yess the much needed Pandas improvement!
this is what I like in a competitive environment..it sharpens everyone involved.
I agree with what has been said in the comment section: now we can use Pandas to handle large dataset properly.
especially that 20x speed improvement when reading data using Pyarrow backend is so good!
I agree. I'm excited to see how it takes off once officially released. The only problem is people won't be able to ask chatGPT how to properly use pandas 2.0 since chatGPT only goes up to 2021 😆
In a few projects I’ve worked on recently, I found that switching from Pandas to pure Numpy speeded up execution time from about 30 minutes to less than a second on my hardware with about 50m rows of data.
@@robmulla chatgpt 4 incoming
thank you so much for such a kind information.
Glad you found it helpful!
I was about to switch to Polars but now with Pandas Arrow dtypes, I need to do more research. Speed is one thing but lazy processing and memory usage are very important.
I would be interested to know how much time you lose when converting from pandas to polars and back to pandas?
That’s a good question that I should experiment with. The article says the only thing it actually needs to move is metadata, so I’m guessing it doesn’t take very long at all.
As a non coder I thought this was a joke
Depends on the temperature
@@dansplain2393 lmao
Thanks for this update!
Thanks for watching!
Awesome! Thanks for the update.
Anytime Chuck.
Good job pronouncing Interoperability!
Thank you Rob! Great explained!
Thanks for watching. 🙏
My very first thought was: yeah, with pyarrow as a backend, everything single software that relies on numpy has to be rewritten. But at the ends it doesn't looks like a big problem or a problem at all.
Yeah, cool thing indeed.
Thank you Rob! Your videos are always useful with a such nice flow!
Thanks, Rob. Great video!
Thanks for the comment!
Dope video! Would be interesting to see how it compares to polars. Also, what software do you use to record your face in that circle?
I particularly liked the conversion between the Pandas with Arrow backend and polars.. maybe you should make a separate video on it
Rob, do you have any limitations of Arrow to share? Until recently, for example, if I'm not mistaken, Arrow didn't seem to handle Pandas' multi indexes.
That’s a good question! I’m sure we will know a better answer once people start using it more. I forgot about the multindex limitation! I need to see how that works in pandas 2.0
When you imported Polars, I thought you were going to do a versus on speed between Polars and Pandas with the new backend. But still very informative video, thanks so much!
Maybe I can do that in the next video. I did test it out in my live stream video from yesterday. For speed generally: Polars > Pandas 2.0 > Pandas 1.5
@@robmulla Seeing the same here, Polars is still much faster than Pd 2.0.
very informative, thanks
Thanks for watching!
When you swap between pandas and polars, is python creating a copy of the dataframe into memory or is it just referencing the already allocated block of memory with the pyarrow backend?
Great question. It's a little complicated and for things like metadata there is a copy made, but my understanding is that the underlying data is not copied. The article I've linked in the description goes into a lot more detail about it and I'd suggest checking that out.
@@robmulla Awesome! Thank you so much! I've been avoiding pandas for large datasets, but this looks like it will make me a more dedicated user!
@@Erosis Exactly! I think that's the idea.
Hi Rob, thank you for the content. I'd be interested in seeing a video on using the old np.where() and np.select() in the new pandas 2.0.
great video, tks Rob
Just one thing I don't understand. If boolaens are stored on single bits (really a great idea), how comes they can have 3 states (True, False, None). Wouldn't that require 2 bits ?
Aint nothing stored as one bit in python
Can you make a video giving techniques to update pandas (or polars) columns using vectoring instead of .apply() for more complicated custom function calls?
eg. I want to take a str column, split it on a delimiter, do work on each section and then combine them back and return it as the new value for the same column
I actually have a short about this topic - using the str methods on string columns in pandas: th-cam.com/users/shorts5eYTaYHzoEE
What's the issue, you just return the new value in the applied function. You will be needing to use python functions to split and join though.
This int converting into float has been a long time PITA for me. This upgrade will be much welcome.
Thanks very much Rob! Your content is off the charts good, a true outlier, and a great outlier! To get my grad degree in stats, I had to take a PhD-level course in statistical consulting. The prof for that course actually wrote a whole book solely on the topic of outliers, outliers can be extremely important. To me, data science is a subject of stats, similar to the way experimental design or survey sampling are subjects of statistics. I think it is unfortunate that data science is not widely seen as belonging to “statistics” since it makes it harder (at least for me) to use data science tools when teaching stats, as the expectation is to use an old school approach to descriptive and inferential statistics in the classroom. I think your content has the clarity and insight to convince teachers that data science methods MUST be brought into the standard curriculum, e.g. courses like Introductory Business Statistics. Thanks again for posting truly remarkable content!
I would like to see more about aggregate performance. How it scales in CPU cores if it is still single-core.All for varying sizes of dfs
Thats a good idea. I'll try to think about how I could do that. There might be some settings for limiting the CPU use but also I know h2o does some benchmarks across different libraries.
Hey, your DS videos are awesome! I was using ChatGPT to learn more about Apache Arrow, Polars, etc from all your videos. Can you do a coding livestream or video about bioinformatics data like biological sequence data or drug development data? Thank you!
Pandas is life.
Even if Polars is cool.
Yes it is! 😆
Awesome! Are there any drawbacks to using the pyarrow backend?
Thanks Ken! Backwards compatibility and the fact that the integration is still very new and might be buggy are two that come to mind.
One day, I learned that Python was slow for large data, so I started to study Julia who's reputation is "the fastest". One programmer I talked with said to me "No Python is fast!" And maybe new Pandas helps for speed matter. How should I take it from your point of view?
I think it’s mostly about choosing the correct tool for the job. But this in general is moving pandas towards being a faster library.
do you think this can have any "consequences" to the develop of numpy?
I use pandas for data management but also use numpy and scipy and is very handy the way you can work with all of them till now. don't know if arrow can make the job so easy.
also, can this make things more difficult for plotting stuff with matplotlib and seaborn, i.e.?
p.d. thanks for the video and also for the previous 1h live stream in-ter-oper-a-bi-li-ty :D
Haha. I pronounced interoperability in this video 😂. I don’t think numpy is going anywhere. When working with non-tabular types of data numpy is strong. Also libraries like PyTorch and Tensorflow allow you to convert between tensors and numpy and those are very popular.
Hey Rob, Great stuff! .. I moved my data transformation app to PD2.0 and I have a problem with data conversion. When Im converting data from int[pyarrow] to string using astype(str) im loosing pyarrow and pandas convert it to object :/. I cant find the tool to do it properly. All best!
Now that the beta is out is there any estimation about release ?
Btw thx for the video !
I only know what has been publicly said by the core dev team. Hope to know soon
Is the PyArrow backend not gonna be the default? it would be annoying to have to specify the dtype for every table I have
With a library as established as pandas, I don’t think they want to implement breaking changes, so having it as an option, at least in early releases I think is the right choice.
Do you have any affiliations to Polars?
Nope. I’m just me.
Hey Rob!
That text editor looks awesome. Which one is it?
When should I be using pandas 2.0 vs Polars?
Check out my video on data pipelines comparing them and pyspark!
Can you make a tutorial on how to use the new upgraded MediaPipe object detection in python with live stream footage?
Great idea. I can try! Too many good ideas and not enough time.
@@robmulla Nice to hear! If you make this work, you'll make my whole month. I've been struggling with this one since the upgrade came out..
pandas 2.0 with pyarrow backend VS pyspark dataframes?
I get unknown engine: pyarrow but I have pandas 2.0 and Pyarrow 10.0.1. Am I missing something ?
It's a very good Video and this Library a good improvement, however ^^ nobody stops you from using a single int and store 32 different boolean states into it with bit shifting operations, but using a Interpreted Script Language means from the beginning "I trade resources for convenience". The underlaying Engine of Python is C and in C there is no actual boolean datatype, the smallest unit is the exact smallest unit the Operating System can provide which are 1 byte / 8 bits which the Operating itself only can provide to you if it reserves a whole virtual memory page for you.
I just think, to nitpick about 7 wasted bits as a Python Programmer is a little bit awkward. The rest was great, thanks for the Video
Wow! I'm still struggling in pronouncing "Interoperability" and it came out so fluid out of your mouth. Bet you are to release a video comparing pandas 2.0 vs Polars. Is Numpy still in pandas 2.0 due to backward compatibility?
Wait... did I actually say "Interoperability" correctly? I filmed myself saying it a bunch of times and just keep the one that sounds the best :D
To answer your question yes, pandas has backwards compatability, as I show in the video by default it still uses the numpy backend.
and what about loading big dataframes? it will improve?
Wow. First time seeing these data types. When would you not use pyarrow and just use default python or something else?
I think eventually it will become adopted by most users, but pandas doesn’t want to make major changes. It also might be some time before it’s stable enough to trust in production code.
Wait, how can Arrow store booleans as one bit each AND allow None values?
Good catch! I think the bool values use a single bit but every column has a null array associated with it. But I also don’t know that for sure. But that would mean that a nullable bool would me more than one bit.
Hmm, Arrows seems great, but I had no idea something as fundamental as Pandas and NumPy was so janky, with really really sketchy quirks. Now I'm wondering what the speedup would be if you tried these operations with a system that wasn't using at least some scripting language interpreter calls for raw data shuffling? Like just a normal struct.
Hey rob this video is not showing not even got a notification re-upload it, only way I got access to this video by going to your playlist.
Hey! You aren't supposed to see this yet :D - I have it as unlisted and am suprised you could even find it!
@@robmullahey Rob actually haven't updated my yt using one year older version of TH-cam I always get 5 sec ads and i can also download videos in 1080p idk how that's possible but working for me even sometime i am able to see private videos those youtubers also always asked me how i am able to play their private videos.
Conclusion:- Google is stupid
This sorts two big issues, non-nullable ints, and the object dtype both can be frustrating.
First time watcher feedback:
Great video! Clickbait-y title, could at least have a more positive spin :)
Thanks for the feedback. Unfortunately the TH-cam algorithm favors videos with higher click through rates. I’m experimenting with more enticing titles although I agree with you I wish I didn’t have to.
what is purpose of use_nullable_dtype=True
I wonder how much of my code this will break. Upgrading sqlalchemy to v2 was like sticking a grenade in my computer.
Its supposed to be backwards compatible. You won't know for sure until you try.
Hi Rob my second comment I'm glad)
Nice! The first commenter was able to see it before I even made the video public :D so lets call it a tie.
Id be living in the dark if it wasnt for this video 🤣
pd.options.mode.dtype_backend = 'pyarrow'
This option is dead now in pandas 2.0.3.
It must have only been a part of the release candidate
do you think 90% of data analyst will be eliminated due to GPT4.0 and further integration of machine learning in MS 365?
Absolutely not.
I think with all this talk about pandas and polars people these days often neglect to even consider grizzlies.
I can't BARE it! 😄
in·tr·aa·puh·ruh·bi·luh·tee. Remember that Rob, 😉
Oh no. Did I say it wrong 😑
@@robmulla Not at all. You nailed it 👍
lol. funny stuff.
Apache Arrow is so much better.
Oh yea? In what way? Do you mean how it’s integrated into pandas 2.0?
@@robmulla yessir 💯
Background music is slightly annoying. Your normal voice is peasant, so no need to supplement it.