As always, awesome video...a real eye opener on most efficient file formats. I have only used pickle as compression, but will now investigate feather and parquet. Thanks for putting this together for all of us.
Very good video :). One note: pickle files can be compressed. If you compress them, they become much smaller but reading and writing becomes slower. Overall parquet und feather are still much better.
One really cool feature of .read_parquet() is that it passes through additional parameters for whichever backend you're using. For example the filters parameter in pyarrow allows you to filter data at read, potentially making it even faster: df = pd.read_parquet("myfile.parquet", filters=[('col_name', '
Whoa. That is really cool. I didn't realize you could do that. I've used athena which allows you to query parquet files using standard SQL and it's really nice.
I saw people mentioned feather on Kaggle sometimes, but had no clue what they were talking about. Finally, I got answers to many questions in my mind. Thank you!
This was the first video from the channel that randomly appeared in my feed. I clicked, I watched - I liked and subscribed :D. This video plant a seed into my mind, some others inspired me to try. So few days later I got running playground environment in the docker. I'm not data scientist but tips and tricks from your videos could be useful for any developer. I used to code before to check some datasets, but with pandas and jupiter notebook it way more faster. Thank You for sharing your experience !
Wow, I really appreciate this feedback. Glad you found it helpful and got some code working yourself. Share with friends and keep an eye out for new videos dropping soon!
This video greatly helped me. I didn't know so many ways to dump a DataFrame. I then did a further test, and found the compression option plays a big role: df.to_pickle(FILE_NAME, compression='xz') -> 288M df.to_pickle(FILE_NAME, compression='bz2') -> 322M df.to_pickle(FILE_NAME, compression='gzip') -> 346M df.to_pickle(FILE_NAME, compression='zip') -> 348M df.to_pickle(FILE_NAME, compression='infer') -> 679M # default compression df.to_parquet(FILE_NAME, compression='brotli') -> 334M df.to_parquet(FILE_NAME, compression='gzip') -> 355M df.to_parquet(FILE_NAME, compression='snappy') -> 423M # default compression df.to_feather(FILE_NAME) -> 500M
@@robmulla Actually if you check the docs parquet files are snappy compressed by default. You have to explicitly say `compression=None` to not compress it. Snappy is the default because it adds very little time to read/write with modest compression and low CPU usage while still maintaining the very nice columnar properties (as you showed in the video). It is also the default for Spark. Other compressions like gzip get it smaller but at a much more significant cost to speed. I'm not sure this is still the case but in the past they also broke some of the nice properties because it is compressing the entire object.
I looked this up, and it's a pretty cool format, I kinda guessed that it could be a column-based storage strategy when you said that we can efficiently get only select columns, but after I looked it up and found it to be true, it felt very exciting. Anyways, hats off to Google's engineers for thinking out of the box on this, the number of things we can do just by storing data as column-lines rather than row-lines is a lot. Of course, the trade-off is that it's very expensive to modify column-wise data, so this is more useful for static datasets that require multi-dim analysis
Rob, You're a natural communicator (or you worked really hard at acquiring that skill) - most effective. I follow you on twitch and I'm currently going through your youtube content to come up to speed. Thanks for sharing your time and experience. Have you thought about aggregating your content into a book as a companion to your content - something like "Data Analysis Using Python/Pandas - No BS, Just Good Stuff" ?
Hey. Thanks for the kind words. I’ve never considered myself a naturally good communicator and it’s a skill I’m still working in but I appreciate your positive feedback. The book idea is great, maybe sometime in the future….
This is good to know. I`m going into web development now, so I usually use JSON format for serialization... I`m still new to python so I didn`t know about parquet and feather. Thank you!
Parquet really saved me ) Around one year data, each day is appr 2GB (csv format). Parquet is both compact and fast. But have to use filtering and load only necessary columns “on demand”.
On the first pass when you timeit the csv writing you time both the writing to csv and generating the dataset. So you are likely having biased results as you only time the writing with other format. (Sure it does not change the final message, just want to point it out) Also with timeit, you can use the -o flag of timeit to output the result to a variable, and this can help you to for example make a plot of the times.
Good point about timing the dataframe generation. It should be negligable but fair to note. Also great tip on using -o. I didn't know about that! It looks like from the docs it writes the entire stdout, so it would need to be parsed. ipython.readthedocs.io/en/stable/interactive/magics.html#magic-timeit Still a handy tip. Thanks!
Thanks, great comp. One thing about Parquet - it has some limitations in what chars column names can take, I spent quite some time renaming col names 1 year ago - perhaps that has fallen away by now.
Informative video! I've heard about feather and pickle, but never used them. I think I should give feather and parquet a try! I'd like to get some materials on machine learning and data science that are not introductory - something for middle and senior engineers :)
I’m working on a little project and I have a csv file that’s 15GB. If I get what you’re telling me, I could turn it into a parquet file and save tons of memory space and time?
Thanks for the great benchmark. In R / Python hybrid environment I sometimes use `csv.gz` or `tsv.gz` to address the size issue with CSV but retain the ability to quickly pipe these through line based processors. It would be interesting to see how gzipped flat files perform. I do agree that parquet/feather is a better way to go for many reasons, they are superior especially from the data engineering point of view.
In addition to everything, parquet is the native file format to spark and can fully support spark‘s lazy computing (spark will only ever read the columns and rows that are needed for the desired output). If you ever prep really big data for spark, parquet is the way to go.
Another awesome video. It has become my favorite channel. Only regret is that I found it too late. Small correction. It should be 0.3s 0.08s for parquet files. You mistakenly wrote 0.3ms and 0.08ms while converting. Thanks.
Experiment add the compression "Brotli" at the file create. The file size reduce considerably and the read is more fast a lot. Example: to save file: from pyarrow import csv, parquet parse_options = csv.ParseOptions(delimiter=delimiter) data_arrow = csv.read_csv(temp_file, parse_options=parse_options, read_options=csv.ReadOptions(autogenerate_column_names=autogenerate_column_names, encoding=encoding)) parquet.write_table(data_arrow, parquet_file + '.brotli', compression='BROTLI') to read file: pd.read_parquet(file, engine='pyarrow')
the parquet really help me , i was super confuse deal with such a big data , i try any way to reduce the size but never got the best accuracy and fast respons but this one really cahnge it
Hello! Very interesting! Thank you! Can you please tell me is any limitation for a DF to save in parquet in terms of number of columns? Excel allow around 16-17k columns to save! Thank you for the answer!
Great video - It would have been good to at least mention the downsides to pickle and also the built in compatibility with zip files. Haven't come across feather before, will try it out
@@robmulla - Agreed, parquet has some serious benefits You know it also supports a compression option? Use it with gzip to see your parquet file get even smaller (and you only need to use it on write)
Interesting video!.. thanks.. i tried to compress several csv into one parquet, but I had several problems with datetime ISO 8601 with time zone, I just couldn't change the format after all my eforts, I had to continue using csv as it didn't have problems transforming it into to_datetime, any suggestions for compressing files without having problems with datetime?, thanks!
Oh yes! I've had this same problem and it can be really annoying. Have you made sure you updated apache arrow to the latest version? stackoverflow.com/questions/58854466/best-way-to-save-pandas-dataframe-to-parquet-with-date-type
I think you use something like this in the markdown: [section title](#section-title) check this link: stackoverflow.com/questions/28080066/how-to-reference-a-ipython-notebook-cell-in-markdown
JSON has a lot of great use cases, and that is probably one of them. This video if more focused on large datasets to be processed in bulk. JSON is great for data that is less structured.
I don't know much about ORC file formats but I'll look into it. I don't think pandas has a built in function for saving to it. It looks liked pandas does have a `read_orc` method though.
I didnt know about just reading select columns in polars, wanted to see how much bigger coverting a 320mb parquet file to csv and json, csv was over 5 times and json 17.5
Hey, just want to mention that when you wrote the pickle file to disk, you did so with no compression. While other formats have compression by default.
What about saving and reading massive data from PostgreSQL? I have some troubles loading csv to DB using psql. Them changed to pgloader, so much fast and without errors, but still super slow. Like 30min for 1.7mi rows.
stumbled on to this awesome video and absolutely loved it. Just out of curiosity - what tool are you using for making Jupyter notebook with themes especially dark theme?
Hey Rob, this was a really nice video! Can you please make a tutorial where you try to write this data to a database? Maybe sqlite or postgres? And explain bottlenecks? (Optional: with or without using an ORM).
Great comparing, thanks, not sure if feather/pickle files i'm creating from Julia's script use some compression - none that i'm specifying out of the box .. but happens that the pickle files always end up being 1/2 the size smaller than the feather ones. (havent compared those 2 to a parquet made file)
Hi all ! I'm a newbe, some difficulties using %time, seems not a standard python command, but only for some specific python release. Did I do something wrong ? Thanks !
First post! That’s my husband he knows about data…
He knows a lot of good stuff about data 😁. His the first non-introductory Python TH-camr I have found so far 🎉
aww this is cute
Guess he's really in a "pickle" now.
Awww now you guys need a The DataCouple channel if you both do data science! Love your content
Nice work Mr. ROB
As always, awesome video...a real eye opener on most efficient file formats. I have only used pickle as compression, but will now investigate feather and parquet. Thanks for putting this together for all of us.
Glad it was helpful! I use parquet all the time now and will never go back.
You are my new favorite TH-camr, Sir. I'm learning more from you than anyone else, by a country mile!
Very good video :). One note: pickle files can be compressed. If you compress them, they become much smaller but reading and writing becomes slower. Overall parquet und feather are still much better.
Good point! There are many ways to save/compress that I probably didn't cover. Thanks for watching the video.
Man, I thought this video is a clickbait, but it was awesome. Thank you!
One really cool feature of .read_parquet() is that it passes through additional parameters for whichever backend you're using. For example the filters parameter in pyarrow allows you to filter data at read, potentially making it even faster:
df = pd.read_parquet("myfile.parquet", filters=[('col_name', '
Whoa. That is really cool. I didn't realize you could do that. I've used athena which allows you to query parquet files using standard SQL and it's really nice.
Athena is amazing when backed with parquet files, I've used it in order to be able to read through 600M+ records that were in those parquets easily
That's the real use case for parquet. Feather doesn't have this.
I saw people mentioned feather on Kaggle sometimes, but had no clue what they were talking about. Finally, I got answers to many questions in my mind. Thank you!
Yes. Feather and parquet formats are awesome for when you want to quickly read and write data to disk. Glad the video helped you learn!
Exactly what I needed to know, and to the point. Thanks.
As Einstein said, 'Everything should be as simple as possible, and no simpler!'
That’s a great quote. Glad you found this helpful.
as someone moving into datascience this is such a great explainer! thank you
This was the first video from the channel that randomly appeared in my feed. I clicked, I watched - I liked and subscribed :D. This video plant a seed into my mind, some others inspired me to try. So few days later I got running playground environment in the docker. I'm not data scientist but tips and tricks from your videos could be useful for any developer. I used to code before to check some datasets, but with pandas and jupiter notebook it way more faster. Thank You for sharing your experience !
Wow, I really appreciate this feedback. Glad you found it helpful and got some code working yourself. Share with friends and keep an eye out for new videos dropping soon!
Hi Rob. I'm from Argentina, you are the best!!!
Amazing! Got one new member. Thanks, Rob! 😉
Glad you liked it. Thanks for commenting!
This video greatly helped me. I didn't know so many ways to dump a DataFrame. I then did a further test, and found the compression option plays a big role:
df.to_pickle(FILE_NAME, compression='xz') -> 288M
df.to_pickle(FILE_NAME, compression='bz2') -> 322M
df.to_pickle(FILE_NAME, compression='gzip') -> 346M
df.to_pickle(FILE_NAME, compression='zip') -> 348M
df.to_pickle(FILE_NAME, compression='infer') -> 679M # default compression
df.to_parquet(FILE_NAME, compression='brotli') -> 334M
df.to_parquet(FILE_NAME, compression='gzip') -> 355M
df.to_parquet(FILE_NAME, compression='snappy') -> 423M # default compression
df.to_feather(FILE_NAME) -> 500M
Nice findings! Thanks for sharing. Funny that compressing parquet still works. I didn't know that.
@@robmulla Actually if you check the docs parquet files are snappy compressed by default. You have to explicitly say `compression=None` to not compress it.
Snappy is the default because it adds very little time to read/write with modest compression and low CPU usage while still maintaining the very nice columnar properties (as you showed in the video). It is also the default for Spark.
Other compressions like gzip get it smaller but at a much more significant cost to speed. I'm not sure this is still the case but in the past they also broke some of the nice properties because it is compressing the entire object.
I looked this up, and it's a pretty cool format, I kinda guessed that it could be a column-based storage strategy when you said that we can efficiently get only select columns, but after I looked it up and found it to be true, it felt very exciting.
Anyways, hats off to Google's engineers for thinking out of the box on this, the number of things we can do just by storing data as column-lines rather than row-lines is a lot. Of course, the trade-off is that it's very expensive to modify column-wise data, so this is more useful for static datasets that require multi-dim analysis
Very clear, very structured, and the details are intuitive to understand!
Thanks a lot! Loved the video and helped me with what I need it ❣️ Keep going with these videos. They really worthy 🔥
Very clear and insightful explanation, thanks Rob, keep it up!
Thanks Gustavo. I’ll try my best.
A major design objective of feather is to be able to be read by R. If you are doing pandas-type data science stuff, this is a significant advantage.
Great point. The R package called "arrow" can read in both parquet and feather files.
Hey Guy, nice job. Congratulations! Thanks for video.
Thanks for watching Humberto.
Rob, you did it again...keep'em coming, good job!
Thanks!
These file saving methods are really useful 😊
You explain very clearly. Thank you.
I really love it man, thank you. You saved a life
Thanks! Maybe not saved a life, but saved a few minutes of compute time!
Excellent as usual Rob...very very useful indeed
Thank you sir!
Rob, You're a natural communicator (or you worked really hard at acquiring that skill) - most effective. I follow you on twitch and I'm currently going through your youtube content to come up to speed. Thanks for sharing your time and experience. Have you thought about aggregating your content into a book as a companion to your content - something like "Data Analysis Using Python/Pandas - No BS, Just Good Stuff" ?
Hey. Thanks for the kind words. I’ve never considered myself a naturally good communicator and it’s a skill I’m still working in but I appreciate your positive feedback. The book idea is great, maybe sometime in the future….
This is good to know. I`m going into web development now, so I usually use JSON format for serialization... I`m still new to python so I didn`t know about parquet and feather. Thank you!
Glad you found it helpful. Share it with anyone else you think would benefit!
Excellent tutorial Rob. Subscribed!
Thanks so much for the feedback. Thanks for subscribing!
Thanks Rob, awesome information! Learning a lot from your channel. Keep it up!
Isn’t learning fun?! Thanks for watching.
thanks rob, its help me a lot for beginner like me to realize there is weakness in csv format 😉
Thanks for sharing; I think I'll start using feature and parquet for some of my data needs.
Very good stuff. The essence of information.
Glad you liked it!
@@robmulla I saw few more videos, insta sub. Thank you. Glad to find you.
Very engaging and clear. Thanks!
Thanks for watching. 🙌
learnt something new today. Thank you Rob for this useful & informative video.
Learn something new every day and before long you will be teaching others!
I've learned a great deal with this video. Thank you!
Thanks so much for the feedback. Glad you learned from it!
interesting to learn the existence of parquet and feather files. nothing beats csv for portability and ease of use
Yea, for small/medium files CSV gets the job done.
Parquet really saved me )
Around one year data, each day is appr 2GB (csv format). Parquet is both compact and fast.
But have to use filtering and load only necessary columns “on demand”.
Great video!! Small things matter the most. Thanks
Absolutely! Thanks.
Great stuff! Thanks for sharing.
Glad you enjoyed it!
@@robmulla 👍
Very informative video! Subscribed :)
Glad it helped! 🙏
On the first pass when you timeit the csv writing you time both the writing to csv and generating the dataset. So you are likely having biased results as you only time the writing with other format. (Sure it does not change the final message, just want to point it out)
Also with timeit, you can use the -o flag of timeit to output the result to a variable, and this can help you to for example make a plot of the times.
Good point about timing the dataframe generation. It should be negligable but fair to note. Also great tip on using -o. I didn't know about that! It looks like from the docs it writes the entire stdout, so it would need to be parsed. ipython.readthedocs.io/en/stable/interactive/magics.html#magic-timeit Still a handy tip. Thanks!
Good tips on speeding up large file read and write
Glad you liked it! Thanks for the feedback.
great comparison. What about HDF5 format? Is it in anyway better?
super clear and useful! Subscribed
Awesome, thank you!
Thanks a lot, just brought down my database backup size to MBs.
Glad it helped. That’s a huge improvement!
Great summary of data types. Thanks
Thanks for the feedback! Glad you found it helpful.
This blew my mind, duuude
Happy to hear that! Share with others so their minds can be blown too!
Great! Thank you for this very helpful video.
Glad it was helpful!
Thanks, great comp. One thing about Parquet - it has some limitations in what chars column names can take, I spent quite some time renaming col names 1 year ago - perhaps that has fallen away by now.
Good point! I've noticed this too. Definately a limitation that makes it sometimes unusable. Thanks for watching!
Huge thanks for sharing 🍀
Glad you liked it? Thanks for the comment.
This content is really awesome
Appreciate that!
Nice video. I'm going to rewrite the storage on the parquet
You should! Parquet is awesome.
Lol this video changed my life :D Thank you so much.
Informative video! I've heard about feather and pickle, but never used them. I think I should give feather and parquet a try!
I'd like to get some materials on machine learning and data science that are not introductory - something for middle and senior engineers :)
Glad you found it useful. I’ll try to make some more ML videos in the near future.
Really useful video - thanks.
I was just searching for some Pandas videos for some light upskilling on the weekend, so this was a great find.
Glad I could help! Check out my other videos on pandas too if you liked this one.
12:28 "When your data set gets very large." - Me working with 800GB json files: :)
Good video regardless, i might give them a test sometime.
Haha. It’s all relative. When your data can’t fit in local ram you need to start using things like spark.
I’m working on a little project and I have a csv file that’s 15GB. If I get what you’re telling me, I could turn it into a parquet file and save tons of memory space and time?
I really hope you make a video about Data Cleaning in Python soon. Thanks a lot for all your awesome tutorials
I'll try my best. Thanks for the feedback!
Thanks for the great benchmark. In R / Python hybrid environment I sometimes use `csv.gz` or `tsv.gz` to address the size issue with CSV but retain the ability to quickly pipe these through line based processors. It would be interesting to see how gzipped flat files perform. I do agree that parquet/feather is a better way to go for many reasons, they are superior especially from the data engineering point of view.
I do the same with gzipped CSV files. Good idea about making a comparison. I’ll add it to the list of potential future videos.
Thank u very much for sharing such useful skills! 😉Subscribed!
Anytime! Glad you liked it.
In addition to everything, parquet is the native file format to spark and can fully support spark‘s lazy computing (spark will only ever read the columns and rows that are needed for the desired output). If you ever prep really big data for spark, parquet is the way to go.
That’s a great point. Same with polars!
@@robmulla Need to have a closer look at polars then! 🙂
Awesome informations ! Thankyou for this.
Glad you liked it!
Amazing.
Congrats for the video
Glad you like the video. Thanks for watching.
Another awesome video. It has become my favorite channel. Only regret is that I found it too late.
Small correction. It should be 0.3s 0.08s for parquet files. You mistakenly wrote 0.3ms and 0.08ms while converting.
Thanks.
Apprecate that you are finding my videos helpful. Good catch on finding that typo!
i was going to comment that, but decided to check first, least should have caught that. Good video.
Experiment add the compression "Brotli" at the file create. The file size reduce considerably and the read is more fast a lot.
Example:
to save file:
from pyarrow import csv, parquet
parse_options = csv.ParseOptions(delimiter=delimiter)
data_arrow = csv.read_csv(temp_file, parse_options=parse_options, read_options=csv.ReadOptions(autogenerate_column_names=autogenerate_column_names, encoding=encoding))
parquet.write_table(data_arrow, parquet_file + '.brotli', compression='BROTLI')
to read file: pd.read_parquet(file, engine='pyarrow')
Oh. Very cool I need to check that out.
the parquet really help me , i was super confuse deal with such a big data , i try any way to reduce the size but never got the best accuracy and fast respons but this one really cahnge it
Fantastic video
Fantastic comment. 😎
Hey this was very useful to me thank you for sharing!!
So glad you found it useful.
Hello! Very interesting! Thank you! Can you please tell me is any limitation for a DF to save in parquet in terms of number of columns? Excel allow around 16-17k columns to save! Thank you for the answer!
This is excellent, thank you man
Glad it helped!
Was very useful, thanks much
Thanks! Glad you learned something new.
Amazing content!
Is there a way you could make those dataframes editable where user could change the value?
There are ways but not really what they are intended for. More so bulk data analysis
Great video - It would have been good to at least mention the downsides to pickle and also the built in compatibility with zip files. Haven't come across feather before, will try it out
Great point! I did forget to mention that pandas will auto-unzip. I still like parquet the best.
@@robmulla - Agreed, parquet has some serious benefits
You know it also supports a compression option? Use it with gzip to see your parquet file get even smaller (and you only need to use it on write)
Interesting video!.. thanks.. i tried to compress several csv into one parquet, but I had several problems with datetime ISO 8601 with time zone, I just couldn't change the format after all my eforts, I had to continue using csv as it didn't have problems transforming it into to_datetime, any suggestions for compressing files without having problems with datetime?, thanks!
Oh yes! I've had this same problem and it can be really annoying. Have you made sure you updated apache arrow to the latest version? stackoverflow.com/questions/58854466/best-way-to-save-pandas-dataframe-to-parquet-with-date-type
It's useful for me, thanks a lot!
Happy to hear that!
Can i do this for building text recognition? how to save and read the data if it's image?
Not sure what you mean. That's possible but not really related to this. There are other ways of saving text and images that would be better.
Rob, could you explain how to add contents with internal links to paragraphs of the jupyter notebook?
I think you use something like this in the markdown: [section title](#section-title) check this link: stackoverflow.com/questions/28080066/how-to-reference-a-ipython-notebook-cell-in-markdown
@@robmulla clear explanation, thank you!
Very good and informative video
So nice of you. Thanks for the feedback.
3:34 another example: if the data is going to be read in via JavaScript (like on a website) then wouldn't JSON be the best option?
JSON has a lot of great use cases, and that is probably one of them. This video if more focused on large datasets to be processed in bulk. JSON is great for data that is less structured.
Great comparison, thanks
When I import csv files always column names cause problems during query how to handle it do I change column names
Great video. Thanks
You are welcome!
the pickle format seems to be significantly faster (10x) than parquet in the final 5mil row test
Great video and content.
How about ORC file format, it is also widely used to store the data.
I don't know much about ORC file formats but I'll look into it. I don't think pandas has a built in function for saving to it. It looks liked pandas does have a `read_orc` method though.
what shortcut are you using to switch block from code to markdown?
Y for code and M for markdown. You should check out my video on setting up jupyter where I go into detail about the shortcuts I often use.
If we use df to parquet. Can we see the data?
You can’t open the data in something like excel. But you can open it with pandas.
Hi Rob! I love our channel. It is very helpfull. I would like to ask you a question: is HDF5 any better than all the options you showed in the video?
Good question. I didn't cover it because I thought it's an older, lesser used format.
@@robmulla so the answer is no?
@@leonjbr The answer is - I don't know but probably not. 😁
@@robmulla ok thanks.
I don't know about "better" but HDF5 is a very popular data format in science.
I didnt know about just reading select columns in polars, wanted to see how much bigger coverting a 320mb parquet file to csv and json, csv was over 5 times and json 17.5
awesome ! thank you for this tutorial
You're very welcome! Share with a friend.
what was the purpose of generating a new dataset per test, couldnt you run the save load functions seperately to the dataset generation?
Hey, just want to mention that when you wrote the pickle file to disk, you did so with no compression. While other formats have compression by default.
Good point. I guess it would be slower and smaller if compressed.
you can compress in parquet too with the compression parameter =D, at is even smaller
What about saving and reading massive data from PostgreSQL? I have some troubles loading csv to DB using psql. Them changed to pgloader, so much fast and without errors, but still super slow. Like 30min for 1.7mi rows.
When we create a parquet dataset, can we dummycode the columns?
super awesome tricks, thank you
Glad you like them! Thanks for watching.
stumbled on to this awesome video and absolutely loved it. Just out of curiosity - what tool are you using for making Jupyter notebook with themes especially dark theme?
Glad you enjoyed the video. I have a different video that covers my jupyter setup including theme: th-cam.com/video/5pf0_bpNbkw/w-d-xo.html
Hey Rob, this was a really nice video! Can you please make a tutorial where you try to write this data to a database? Maybe sqlite or postgres? And explain bottlenecks? (Optional: with or without using an ORM).
I was actually working on just this type of video and even looking at stuff like duckdb where you can write SQL on parquet files.
Very nice explanation. Can you compare Dask and PySpark ?
Great comparing, thanks, not sure if feather/pickle files i'm creating from Julia's script use some compression - none that i'm specifying out of the box .. but happens that the pickle files always end up being 1/2 the size smaller than the feather ones.
(havent compared those 2 to a parquet made file)
Hi all ! I'm a newbe, some difficulties using %time, seems not a standard python command, but only for some specific python release. Did I do something wrong ? Thanks !
It only works in jupyter. Otherwise check out the timeit package.