You went to a very dangerous territory here... Using pickle to store state of objects in long term has a lot more caveats than just "don't load random stuff from the internet'. 1. Pickle is Python only format. You can't read it from anything else or look at stored data. Even within python there are several versions of pickle and you can't load files stored in newest pickle format on older versions of python. (Last change happened at python 3.8) 2. More importantly, pickle depends deeply on the structure of your code, modules and classes. You can only safely unpickle objects that were stored with the same code structure. Programs tends to change over time. Storage formats change less often and it's easier to consider what to do with old data. Add new attribute to your class or rename one. If you attempt to load pickle file stored in previous version, an object will be restored with attributes as they were in a previous version and they probably won't match the changes you made. Rename the class or put it into different module - and loading will fail. 3. There is a reason why saving into json takes a lot of repeating code: you actually need to say explicitly, what are you planning to store and what name should be attached to it. Your classes could contain some data that is deduced on the fly and there is no use in storing and loading them. And without additional code pickle would just store everything from object (well, technically not everything, there are some exceptions by default). You could specify what to pickle and what not to pickle, but then we return to the same state as before which is - we need to be explicit about what to store and how. 4. There are proper ways to stay with normal interchangeable formats and not writing too much boilerplate. Projects like pydantic, dataclass-json, dataclass_wizard, dataclass_factory and others allows you to specify mapping between most of formats (like json, yaml, or some dictionary loaded from sql database) and your classes. 5. Main purpose of pickle remains to be short term storage of objects so they could be transferred between parts of the system that you maintain, for example, for interprocess communication or sending tasks to worker farm run by celery. And pickle was never intended for long term storage of the system state like you are implying. To summarise: don't use pickle this way if that's not a one-time task and you are planning to use what you have written two month later, there are better solutions.
I'm sure that would make for a much more interesting video than what I explained in 10 minutes. Thanks for the informative comment! I will look into creating a video that goes far more in depth.
I also see a potential vulnerability, wouldn't you be able to just rewrite this pickle file from another python application and embed some malicious code. Even more of a reason to stay away from it for any long term storage.
@@md.redwanhossain8822 i think the main purpose of pickle is for easy and small project. People with massive code will eventuelly use json or other formats.
" with open('banana.json', 'w') as file: json.dump(fruit.__dict__, file) " & " with open('banana.json', 'r', encoding='utf-8') as file: fruit = Fruit(**(json.load(file)))" Does this not achieve the same thing but for json? rather than a need to manually create the data you could dump the object as a dictionary?
I now always have my EarPods in for monitoring to avoid unpleasant surprises like this. I was editing the video without headphones so I didn't even notice that humming until I put my headphones on a couple of days later 🥲
Yeah unfortunately that happened for 2 videos, I learned it was because my iPhone was connected to my adapter and for some reason that created this noise, it will be gone in a couple videos.
TH-cam suggested the video at very right time. I was trying to save a pandas dataframe in database and I think this will do the job. Whole dataframe can be saved in single cell in a pickle I guess
from food.buns import * from food import fried_patty from food import salad from food.sauces import mayo,mustard,ketchup from food import onion,tomatoes friedpattyc=food.fried_patty.config(beef=True,ground=True) # NOTE: its a joke
Why the hell it's called "piclkle?" Why the hell web server is called "flask?" Why the hell xml library is called "beautifulsoup?" Why the hell people should taky Python seriously, if python's naming conventions are so infantile? I love python, but this is just what I don't get.
Personally, I think you should sway away from that mentality, I would relate Python to being a social language that everyone can understand. If you want to really get serious, then I recommend just using C or a lower level language. The comedy that comes with Python is what makes Python, Python 😉
@@Indently I know, I am using multiple languages, I just don't get this "forced cringe." Fortunately this awesome language pushed old baddies like pascal out of Universities (yes, in 2013 as I was here, they learned Pascal as a main language 😞 )
@@IndentlyIndeed, even Python is arguably an odd name for a computer language. Who would name their premier software masterpiece after a TV comedy? Guido apparently has an offbeat sense of humor, and surely his followers could be excused for continuing in a similar vein.
You went to a very dangerous territory here...
Using pickle to store state of objects in long term has a lot more caveats than just "don't load random stuff from the internet'.
1. Pickle is Python only format. You can't read it from anything else or look at stored data. Even within python there are several versions of pickle and you can't load files stored in newest pickle format on older versions of python. (Last change happened at python 3.8)
2. More importantly, pickle depends deeply on the structure of your code, modules and classes. You can only safely unpickle objects that were stored with the same code structure. Programs tends to change over time. Storage formats change less often and it's easier to consider what to do with old data. Add new attribute to your class or rename one. If you attempt to load pickle file stored in previous version, an object will be restored with attributes as they were in a previous version and they probably won't match the changes you made. Rename the class or put it into different module - and loading will fail.
3. There is a reason why saving into json takes a lot of repeating code: you actually need to say explicitly, what are you planning to store and what name should be attached to it. Your classes could contain some data that is deduced on the fly and there is no use in storing and loading them. And without additional code pickle would just store everything from object (well, technically not everything, there are some exceptions by default). You could specify what to pickle and what not to pickle, but then we return to the same state as before which is - we need to be explicit about what to store and how.
4. There are proper ways to stay with normal interchangeable formats and not writing too much boilerplate. Projects like pydantic, dataclass-json, dataclass_wizard, dataclass_factory and others allows you to specify mapping between most of formats (like json, yaml, or some dictionary loaded from sql database) and your classes.
5. Main purpose of pickle remains to be short term storage of objects so they could be transferred between parts of the system that you maintain, for example, for interprocess communication or sending tasks to worker farm run by celery. And pickle was never intended for long term storage of the system state like you are implying.
To summarise: don't use pickle this way if that's not a one-time task and you are planning to use what you have written two month later, there are better solutions.
I'm sure that would make for a much more interesting video than what I explained in 10 minutes. Thanks for the informative comment! I will look into creating a video that goes far more in depth.
I also see a potential vulnerability, wouldn't you be able to just rewrite this pickle file from another python application and embed some malicious code. Even more of a reason to stay away from it for any long term storage.
@@TheGoldenPro tbh if you have write access you could also just overwrite the program with malicious code
@@TheGoldenPro just use hashing
@@md.redwanhossain8822 i think the main purpose of pickle is for easy and small project. People with massive code will eventuelly use json or other formats.
Cool video, thanks for the disclaimer at the end, these types of videos rarely mention the downsides of certain methods
" with open('banana.json', 'w') as file:
json.dump(fruit.__dict__, file) "
&
" with open('banana.json', 'r', encoding='utf-8') as file:
fruit = Fruit(**(json.load(file)))"
Does this not achieve the same thing but for json? rather than a need to manually create the data you could dump the object as a dictionary?
I used this today on a Project. Thanks!!
This is pretty awesome; I will definitely use this in my future projects! Thanks for the tutorial 😄😄
Thank you so much. Your teaching is very concise and I was able to understand it super easily.
Lovely quick and easy lesson. Love it. Very instructive too !
Are we suppose to import pickle before our coding?
This is a good one, Didn't know the real usage of pickle before, Tnx
I will know the usefulness of pickling a model for example a machine learning model you have trained.
For shorter, more concise and safer code, look into dataclasses and the "dacite" package :)
I'll check it out! Thanks for sharing!
'data.bin' is a typical file name for the pickle module. Pickle creates binary files.
doing my CS MSc using pickle is a big no no, assignment gradings are penalised for doing so. I can see the appeal...
thanks! really useful for me as a Python beginner. Subscribed!
I will pickle today, thanks
What's the difference between object creation like the one shown in the video and `fruit = Fruit('Banana', 100)`.
pickling isnt safe so it should only be used on cases were you trust the data, like cache or something
I mentioned that in the video
Very useful Info, Thank you!
wow this is so useful thank you! nice video as well, very clear :)
Check out dill if you want to export objects as files.
dill was a very useful module for me.
Could you please remove that constant bass-hum in audacity or something? This is quite unpleasant to listen to
Sorry about that, I noticed it a couple of videos too late, it will be removed soon!
@@Indently thank you for listening to the audio feedback!
I now always have my EarPods in for monitoring to avoid unpleasant surprises like this. I was editing the video without headphones so I didn't even notice that humming until I put my headphones on a couple of days later 🥲
@@Indently understandable, it's a very easy trap to fall into lol
Sounds like a good project for you Mulakula, an active low pass filter so you don't have to ask content creators to change how they record videos.
underrated ngl
Id personally use it on a vm or public computer if it was a random pickle file
Will this work if the python script loading the pickle file doesn't have access to the class definition?
From the top of my memory, no.
@@Indently that's unfortunate.... it seems like I will have to install sklearn on my SBC. Thanks for the content, very helpful and informative
great video, like the ones you have done before
Why can't we just put the object's .__dict__ into a JSON instead?
Nice
Good content, but can you look into the constant buzzing noise in the audio?
Yeah unfortunately that happened for 2 videos, I learned it was because my iPhone was connected to my adapter and for some reason that created this noise, it will be gone in a couple videos.
TH-cam suggested the video at very right time.
I was trying to save a pandas dataframe in database and I think this will do the job.
Whole dataframe can be saved in single cell in a pickle I guess
Can you show this "binary" file, please ?
No
@@Indently can you show it to ME? Im fairly trustworthy.
Pickle Rick
from food import pickle
from food.buns import *
from food import fried_patty
from food import salad
from food.sauces import mayo,mustard,ketchup
from food import onion,tomatoes
friedpattyc=food.fried_patty.config(beef=True,ground=True)
# NOTE: its a joke
You know, there is actually module called Flask-WTF. WTF is WTF?)
Why the hell it's called "piclkle?" Why the hell web server is called "flask?" Why the hell xml library is called "beautifulsoup?"
Why the hell people should taky Python seriously, if python's naming conventions are so infantile?
I love python, but this is just what I don't get.
Personally, I think you should sway away from that mentality, I would relate Python to being a social language that everyone can understand. If you want to really get serious, then I recommend just using C or a lower level language.
The comedy that comes with Python is what makes Python, Python 😉
@@Indently I know, I am using multiple languages, I just don't get this "forced cringe." Fortunately this awesome language pushed old baddies like pascal out of Universities (yes, in 2013 as I was here, they learned Pascal as a main language 😞 )
@@IndentlyIndeed, even Python is arguably an odd name for a computer language. Who would name their premier software masterpiece after a TV comedy? Guido apparently has an offbeat sense of humor, and surely his followers could be excused for continuing in a similar vein.
Thank you.