OUTLINE: 0:00 - Introduction 1:10 - Sponsor: Weights & Biases 3:20 - How Hugging Face models are loaded 5:30 - From PyTorch to pickle 7:10 - Understanding how pickle saves data 13:00 - Executing arbitrary code 15:05 - The final code 17:25 - How can you protect yourself? Try the model: huggingface.co/ykilcher/totally-harmless-model Get the code: github.com/yk/patch-torch-save Sponsor: Weights & Biases Go here: wandb.me/yannic
Lovely.... I already downloaded the Stable Diffusion open source v1.4 off Hugging Face a few days ago. Now I'm worried. Anything I can do after the fact?
This is awesome. in one video you have not only explained how you went about finding the exploit but also went full circle of how to give back to open source!
Thanks for the video. I've used pickle endless times and was never aware of this pitfall. Also, thanks for the very understandable explanation of the technical details.
Thank you. I had been meaning to look at pickle properly for a while now. This has been a wonderful rundown, and explains some of the warnings I saw when people were discussing pickle, yet I hadn't realized precisely why it was so powerful/dangerous. Edit: possibly we need a text extractor for pickled data that doesn't try and do anything other than reveal the contents in a human readable format. Edit 2, your explanation allowed me to understand whats going on under the hood, and I was able to use this new knowledge to my advantage already, Thank you again, and 3 thumbs up.
Hey yannic, I think it would be cool if we get to see what you're working on from day to day (if possible), an example of your current projects and your day to day relationship with machine learning. Im curious because I'm at a stage where I dont know what I want to do with my current base knowledge and skills in machine learning, so this would be inspirational. Thanks.
Awareness is very much a double-edged sword, since it also gives people the idea to make the malware. It only takes one person to make malware, but it takes everyone to go out of their way to be careful to avoid these attacks.
TLDR; Old man finds out about deserialization exploits. It's surprising how few people understand how deserialization works and what sort of dangers lurks in deserializing untrusted data, great explanation!
Not as hopeless as you might think. Any medium to large sized tech company that's concerned with security can run models in a sandboxed environment. You could run something like strace to figure out what system calls the program is using and disallow most/all by default, whitelisting them one by one.
This sounds like a simple solution - strace, execute the reduce method, see what happens, then use that info to generate a file that shows user what syscalls will be made when you load that model. User can accept or deny. But even this is pretty easy to workaround, at the end of the day it is an insecure method of hosting content
My recommendation is to run untrusted packages and models in Docker containers. This limits the blast radius if you do accidentally run malicious code.
I'm now researching all the pages I looked at and I'm determining which models I downloaded are considered safe, which need to be purged, and which have a good amount of likes/downloads but are still pickles and still may need to be purged... I don't think there is any model worth risking viruses for, but there are a few models that I'll be disappointed to not have still... Nonetheless, it must be done! I get very paranoid over hackers and what not since it is so stressful having your stuff get hacked... Hopefully everyone updates everything to safetensors (and hopefully safetensors are safe... I feel like if 'safe' is in the name it better be bloody safe...)
Is this really a danger of "open source", rather than just trusting unverified code? (in which case if its open source, at least its easier to verify?)
Wow, never though about this issue! Seems like a good reason to implement __reduce__ method in a way it does some funny (but not harmful) thing like loading funny image on the screen or loud noise)
9:44 let me put my two cents on that, I was tinkering with raw CNN code and tensorflow uses hdf5 as a serializable. So, I did a mistake by not taking the input_size of the image in the ConV2D **kwargs. You know what it saved as usual. but when it comes to load_model it gave an error pointing the error specifically for me!! Yeah, hdf5 too evaluates the file while loading!!
"I feel like at some point, hugging face is going to be just full of features that they implemented because I did something stupid" Hey, I'd take Yannic breaking my code over an actual black hat hacker any day :D
This looks like the anti-cheese model of security. Bubble up the warning until someone forgets about it. Instead they should ensure that models are scanned, there is a safe way to use pickle and torch mandates that method.
Pressing F12 in VSVode to jump to the definition of some class or function never ends. Perhaps use VScode and read the docs to have docs in VScode like the website looks because it's all based on docstrings. So it's impossible to rely on the super().__reduce__()?
Hey Yannic, I am trying to work on a challenge I believe is based around this video. Is there a way I could share it with you and ask a few question but to see if I am implementing it correctly?
Any kind of sandboxing limits what arbitrary Python code can access in the local file system; but e.g. the malicious datafile's code could read whatever is in your local data directory and exfiltrate it with network calls.
Thanks for the video. Still, I did not get the point "How the malicious code got onto my computer". So, you save with a "patched version of save", but I still cannot execute it locally or why should python automatically download it; do you point to it via url like in a javascript attack?
Great information that everybody should know. But why would you make the github project? This would just make the exploit (which I agree is already easy) that much easier to be used. It just invites more people to exploit it and I see no use for the github project.
Turns out, people found solutions for this. Your demonstration now errors out during the load on the United version of KoboldAI (Only takes GPT models, so had to test with a GPT version of your example). Curious if people will be able to bypass it.
Does anyone have experience with realESRGAN? It’s a popular image upscaler often used for AI art like Stable Diffusion. It involves a .pth which uses PyTorch and Pickle. I don’t know whether the source is trustworthy and how safe it is to run
Well there is a thing we can do... don't use pickle ! Any file format that is only used to store data and doesn't contain executable code is fit to solve this security issue.
@@benjaming.8368 No, use a different saving protocol instead of pickle (torch.load calls the pickle module by default). Use one that doesn't execute arbitrary code on load.
@@search620 mbam = malwarebytes yes. Windows Defender beats out everything except BitDefender right now, but BitDefender is only a bit better. AV research tests have shown that all you need nowadays is Windows Defender and common sense for the most part.
So now that you've shown the whole world how to upload malware to hugging-face, do you think you can do enough good improving security as the harm you've done, releasing this code that helps anyone upload any malware they like with zero understanding or effort?
Wow I think you took away exactly the opposite of what he was trying to explain. The only reason to prevent misuse is by spreading awareness. If you don't talk about vulnerabilities then they will fly under the hood for a longer period of time and more people will fall victim to truly malicious code. I very much doubt that anyone with serious malicious intent would use his awareness-spreading repository. They would instead write something that's much better concealed and hidden but exploits the same vulnerabilities presented here.
Security through obscurity is no security at all. If you think threat actors weren't already looking at this vector before he shared this, I don't know what to say to you. Not reporting problems because you're scared of malicious people is how you get endemic vulnerabilities that are exploited for many years, sometimes even decades. It was responsible to spread awareness, and by construing it as "[doing] harm", the narrative presented is the same one people who don't want vulnerabilities patched use.
@@Kai-K no one forced him to release his code to the public, but he did! And since hugging face would have to replace an entire module to fix this, they probably won't. So now explain to me how we're safer.
![[TH]FFWS SEA 2024 Fall - Knockout Stage Day 3](http://i.ytimg.com/vi/LDH9JbNtjnQ/mqdefault.jpg)
OUTLINE:
0:00 - Introduction
1:10 - Sponsor: Weights & Biases
3:20 - How Hugging Face models are loaded
5:30 - From PyTorch to pickle
7:10 - Understanding how pickle saves data
13:00 - Executing arbitrary code
15:05 - The final code
17:25 - How can you protect yourself?
Try the model: huggingface.co/ykilcher/totally-harmless-model
Get the code: github.com/yk/patch-torch-save
Sponsor: Weights & Biases
Go here: wandb.me/yannic
Lovely.... I already downloaded the Stable Diffusion open source v1.4 off Hugging Face a few days ago. Now I'm worried. Anything I can do after the fact?
TGO th-cam.com/video/2Hqc2QA5aU4/w-d-xo.html🌐📊📲🛰️✅🔚✔️🚀📉🆒🕶️🛰️📸☢️Tap on a clip to paste it in the text box.
I always new pickle was unsafe, but I never understood the programmatic underpinnings that of how pickle worked behind the scenes. Thanks for sharing.
This is awesome. in one video you have not only explained how you went about finding the exploit but also went full circle of how to give back to open source!
Best AI youtube channel, hands down
Thanks for the notice, almost watched the whole ad. Pretty good open for a vulnerability video.
Thanks for the video. I've used pickle endless times and was never aware of this pitfall. Also, thanks for the very understandable explanation of the technical details.
Thank you. I had been meaning to look at pickle properly for a while now. This has been a wonderful rundown, and explains some of the warnings I saw when people were discussing pickle, yet I hadn't realized precisely why it was so powerful/dangerous.
Edit: possibly we need a text extractor for pickled data that doesn't try and do anything other than reveal the contents in a human readable format.
Edit 2, your explanation allowed me to understand whats going on under the hood, and I was able to use this new knowledge to my advantage already, Thank you again, and 3 thumbs up.
Love your videos, keep up the good work!
This is awesome! I gave a talk today at my company to raise awareness of malicious code in AI models. Thanks for sharing! very useful examples.
Thanks. Instructions are very clear. I will be loading every model I can get my hands on.
I’ve been using wandb for months nonstop 🔥
Hey yannic, I think it would be cool if we get to see what you're working on from day to day (if possible), an example of your current projects and your day to day relationship with machine learning. Im curious because I'm at a stage where I dont know what I want to do with my current base knowledge and skills in machine learning, so this would be inspirational.
Thanks.
Wow really cool that you contacted them and made them aware before releasing this video too!
Nice video, thank you! I avoided this problem by using ONNX models. Does any1 know whether there are security holes for these models?
There is a PoC of ONNX malware on GitHub repo alkaet/LobotoMl but the attack path does not appear complete yet (requires custom-op registration).
Awareness is very much a double-edged sword, since it also gives people the idea to make the malware. It only takes one person to make malware, but it takes everyone to go out of their way to be careful to avoid these attacks.
No it's not, security by obscurity doesn't work. Cybersecurity 101
TLDR; Old man finds out about deserialization exploits.
It's surprising how few people understand how deserialization works and what sort of dangers lurks in deserializing untrusted data, great explanation!
Now the next step would be a rickrolling model.
Haha, OMG I can't believe they use pickle as it is in torch (and huggingface). Pickle is not save to use. Thanks Yannic.
Not as hopeless as you might think. Any medium to large sized tech company that's concerned with security can run models in a sandboxed environment. You could run something like strace to figure out what system calls the program is using and disallow most/all by default, whitelisting them one by one.
This sounds like a simple solution - strace, execute the reduce method, see what happens, then use that info to generate a file that shows user what syscalls will be made when you load that model. User can accept or deny. But even this is pretty easy to workaround, at the end of the day it is an insecure method of hosting content
Thanks for the video Yannic!
this is just one of those occasions where I need to find some time somewhere to actually dig into the code a bit more!
A more elegant alternative to patch_torch_save seems to be a function decorator as a function factory.
U made it simple man ur subscribers are well deserved
You’re doing the Lords Work, thanks for bringing this up to the community!!
according to the youtubes, I hit the first like! yeet
You should make a hugging face model that exploits this to open this video on the victim's machine, haha
I didnt realized that pytorch uses pickle!. Thanks.
This is very interesting. I was not aware of these problems! Thanks!!!
My recommendation is to run untrusted packages and models in Docker containers. This limits the blast radius if you do accidentally run malicious code.
Thanks for doing wonderful things to the society.
Thank you for making us aware, big security risk that should be fixed
The "where did it go" part is so ryt now
Great information. Thank you for sharing this video.
Wow an ad for development tools in your vid, thats saying something.
This is so cool, props
What up buddy Long x no se en Yannic God Bless You brother
This is generally useful python knowledge. Thanks for sharing!!!!
I'm now researching all the pages I looked at and I'm determining which models I downloaded are considered safe, which need to be purged, and which have a good amount of likes/downloads but are still pickles and still may need to be purged...
I don't think there is any model worth risking viruses for, but there are a few models that I'll be disappointed to not have still... Nonetheless, it must be done!
I get very paranoid over hackers and what not since it is so stressful having your stuff get hacked...
Hopefully everyone updates everything to safetensors (and hopefully safetensors are safe... I feel like if 'safe' is in the name it better be bloody safe...)
Thanks a lot...I learn so much from your videos
Is this really a danger of "open source", rather than just trusting unverified code? (in which case if its open source, at least its easier to verify?)
Wow...Bravo man!
Wow, never though about this issue! Seems like a good reason to implement __reduce__ method in a way it does some funny (but not harmful) thing like loading funny image on the screen or loud noise)
9:44 let me put my two cents on that, I was tinkering with raw CNN code and tensorflow uses hdf5 as a serializable. So, I did a mistake by not taking the input_size of the image in the ConV2D **kwargs. You know what it saved as usual. but when it comes to load_model it gave an error pointing the error specifically for me!! Yeah, hdf5 too evaluates the file while loading!!
Some things that can be done is not loading models as administrators and maybe doing it only in contained environments like docker or VMs…
"I feel like at some point, hugging face is going to be just full of features that they implemented because I did something stupid"
Hey, I'd take Yannic breaking my code over an actual black hat hacker any day :D
stay on tune with what you want to learn, just because it's hard now, doesn't an it's impossible. It's all about ntal mindset and
Very interesting, should have crossed my mind, but didn't!
It's no ";" SQL injection, but it's pretty cool.
Thank you
This looks like the anti-cheese model of security. Bubble up the warning until someone forgets about it. Instead they should ensure that models are scanned, there is a safe way to use pickle and torch mandates that method.
Pressing F12 in VSVode to jump to the definition of some class or function never ends.
Perhaps use VScode and read the docs to have docs in VScode like the website looks because it's all based on docstrings.
So it's impossible to rely on the super().__reduce__()?
How is there not a way of locking eval and exec down in these environments?!
Is this also an issue when loading .h5 files or only for models saved as .bin?
Hey Yannic, I am trying to work on a challenge I believe is based around this video. Is there a way I could share it with you and ask a few question but to see if I am implementing it correctly?
Sounds strange to me place in one sentence: hidden and open source! :-D
What is the music at the end?
When are you gonna do eye reveal :)
Thank you in advance
Is there a way to sandbox pickle initially so as to detect executable code or malware?
Would virtualization like running off a docker image protect you from this?
Any kind of sandboxing limits what arbitrary Python code can access in the local file system; but e.g. the malicious datafile's code could read whatever is in your local data directory and exfiltrate it with network calls.
love your videos
Thanks for the video. Still, I did not get the point "How the malicious code got onto my computer". So, you save with a "patched version of save", but I still cannot execute it locally or why should python automatically download it; do you point to it via url like in a javascript attack?
Usually, making things run on other people's machine is quite hard
Will there be a library to unpickle with reduced administrative rights?
Are PyPI, Tensorflow hub, Pytorch hub just as dangerous as HF?
Pickle trick!
Excellent
This is why we can't have nice things.
Do you work at meta? If so, then what is it like working with Yann LeCun?
thanks bro
very interesting
Great information that everybody should know.
But why would you make the github project? This would just make the exploit (which I agree is already easy) that much easier to be used. It just invites more people to exploit it and I see no use for the github project.
Could also just do eval('__import__("webbrowser")....')
Welp hacking people has never been this easy
Turns out, people found solutions for this. Your demonstration now errors out during the load on the United version of KoboldAI (Only takes GPT models, so had to test with a GPT version of your example). Curious if people will be able to bypass it.
Hey dude could you give me a bit more details about it? (Working on the Open Assistant project)
Huh?
"open source is dangerous"
*turns around*
"here look at the source code and see how dangerous it is"
😕
me getting feelings of a hacker
Ah yes my friend my fellow awareness giga chad. Once we make everyone aware, one of them will find the solution for us :)
pandas has a built in json that doesn't have the pickle security holes going on, cmon now.
How is your learning progression? I'm tNice tutorialnking about getting the Signature version soon, people like you are really persuading to give
Does anyone have experience with realESRGAN? It’s a popular image upscaler often used for AI art like Stable Diffusion. It involves a .pth which uses PyTorch and Pickle. I don’t know whether the source is trustworthy and how safe it is to run
Well there is a thing we can do... don't use pickle ! Any file format that is only used to store data and doesn't contain executable code is fit to solve this security issue.
dont use torch.load?
@@benjaming.8368 No, use a different saving protocol instead of pickle (torch.load calls the pickle module by default). Use one that doesn't execute arbitrary code on load.
Any low level function with such a non descriptive name such as pickle I'm always weary of.
pandas has a built in json loader that doesn't have the pickle exploits, cmon now.
Dynamic typing and its consequences have been a disaster for the human race.
You can't run these models on a container can you? Because of the GPU requirements... Well this sucks
Malwarebytes and nod 32? not recognize it?
Mbam is weaker than windows defender
@@xbon1
Mbam = malwarebytes?
what about nod32?
@@search620 mbam = malwarebytes yes. Windows Defender beats out everything except BitDefender right now, but BitDefender is only a bit better. AV research tests have shown that all you need nowadays is Windows Defender and common sense for the most part.
Maybe they should just not use pickle...?
"Does thinking", whilst this could be used for bad it could be used for good. Sigh. If only people leaned towards 'good' by default.
Java has a far superior serialization subsystem.
I never liked pickle..
So now that you've shown the whole world how to upload malware to hugging-face, do you think you can do enough good improving security as the harm you've done, releasing this code that helps anyone upload any malware they like with zero understanding or effort?
Wow I think you took away exactly the opposite of what he was trying to explain.
The only reason to prevent misuse is by spreading awareness. If you don't talk about vulnerabilities then they will fly under the hood for a longer period of time and more people will fall victim to truly malicious code.
I very much doubt that anyone with serious malicious intent would use his awareness-spreading repository. They would instead write something that's much better concealed and hidden but exploits the same vulnerabilities presented here.
@@PeterSeres nonsense. No one reads through the code when they download from hugging-face. Any exploit is as hidden as any other.
Security through obscurity is no security at all. If you think threat actors weren't already looking at this vector before he shared this, I don't know what to say to you.
Not reporting problems because you're scared of malicious people is how you get endemic vulnerabilities that are exploited for many years, sometimes even decades. It was responsible to spread awareness, and by construing it as "[doing] harm", the narrative presented is the same one people who don't want vulnerabilities patched use.
@@Kai-K no one forced him to release his code to the public, but he did!
And since hugging face would have to replace an entire module to fix this, they probably won't.
So now explain to me how we're safer.