Trust No Format: How ML Models Get Weaponized

11 and we are going to start the next presentation. So hello everyone and welcome to trust no format. I am Octavian and with my colleague and friend Dennis we'll explore how something as simple as model files uh can become an attack surface. So the title is literal and every file format uh carries assumptions, shortcuts and trust boundaries that uh attackers love to cross. And um if you remember one thing uh from today's presentation, make it this that a model file is executable input until proven otherwise. Now uh raise your hand if you've ever downloaded a model from the internet and just execute it. Exactly. So that tiny moment of trust is where our story will begin today.

So I'm Octavian Purkaro, the person you want to have on your side when uh things go sideways, but you also want to have uh Dennis on your side. >> Hello, my name is Dennis. I'm a former CTF player and I also have a few certifications under my belt like the OCP and the OSDA. Uh so if you are new to this field u make sure to come to me uh and I can make you some great recommendations for getting into this field. Uh that's all from me and I'll get back to my colleague to show you the agenda for today. So everything we we're going to share with you today has the goal to uh make

you think the the silent trust built into your model stack. So now now let's get oriented with the agenda. Uh so here is how we'll break this down. Uh first we'll talk about serialization methods. Uh what they actually do and what why they matter to you. And then we'll walk you through the uh core risks and vulnerabilities in pickle and ggf. And we'll follow that with u some of the vulnerabilities uh that actually manifest in practice. And we'll finish this with uh an actionable plan. So this is uh a simple structure. Understand it see it and then uh defend it. All right. Uh let's start with the fundamentals. what are ple and ggf and uh what what's

the difference between them? So serialization is the process of turning uh uh live live objects into bytes and then back again. So, Pickle uh is the Python native serializer and it can capture almost anything like u classes uh functions uh tensors but uh as you as you probably imagine this flexibility is convenient but at the same time is also uh dangerous because uh the data becomes uh code during loading. So uh that's pickle. Uh now ggf built it's built for large language models. It's efficient. Um it's binary and metadata rich but it doesn't execute code like pickle but what it stores in metadata can influence the runtime behavior. So we have two formats and two

philosophies. uh one executes on uh load and the other one ggf executes uh it influences uh uh the data during use. So keep this uh try to keep this distinction in your mind because it's the foundation we'll uh discuss later. Um now zooming out a bit uh we'll notice that these issues are on the front page. Uh we have the whis research that exposed malicious AI models in uh replicate service. Also data bricks and uh the uh hacker news that ran a story on model payloads uh uh through injected model payloads that got in hugging phase. So different different stories and the same basically the same message model files uh have become a serious infection vector. The now the takeaway

is quite simple uh protecting AI means uh protecting the the formats that AI uses. So with this uh context in mind let's break down exactly where uh the risks uh hide. So um as I said pickled pickle strength is u flexibility but this is also its downfall cuz it can serialize uh anything. So when unpickled the python uh rebuilds the uh objects by calling the functions and uh the constructors they they define. So the typical risks here are arbitrary code execution in the serialization and uh the main takeaway here is to don't uh to not um the serialize um data that you did not generate yourself. And uh also very important never do it uh in

an unprivileged environment. Uh now now let's uh talk about mitigations and uh uh yeah now let's talk about mitigations. Um GGF uh shifts uh the thread model from load time to run time as I said earlier and its danger lies in the metadata that it defines basically the fields uh that define how model interacts with prompts and the chat templates. So if your application loads and uh renders the metadata directly especially using uh template engines like Ginga which my colleague is going to show later it will effectively execute the the code created by the attacker. So think of it as uh delayed execution. Pickle uh attacks happen when you load the file and GGF happen when you when you use use it. So

I let Dennis continue. Okay. So uh I've chosen two CVs for today to to present and for the first example I've chosen a CV in invokei which is a known software used for uh generative AI. I've also attached the source code for you guys to see if you can spot the bug. Uh this bug actually uh was patched and then uh they realized it was still flawed and you were still able to get RC through it. So the first uh bug happened because the scan files was set to defa to false by default. And after the pack this bug was patched pretty quick. uh there was still a flaw because pickle scan uh didn't actually scan uh wasn't actually secure

when scanning pickle files and uh machine learning models that were imported in invoke AI and it introduced new bugs to the loading of malicious pickle files. A user was able to bypass the can and gain RC through this. As you've previously seen, because a file file scan is done using PL scan, we can use store safe to save our model and execute code. This can be exploited by importing some built-in function from numpy library that wasn't picked up by pickle scan and which indirectly called some dangerous f function like exec which was later used to import the os library which you may know that you can use it to run uh system commands like getting reverse shell and um the

attacker can later use to import this type of library and gain RC through is for our proof of concept. We define our class and the red reduce reduce function used by pickler which return returns either a string or a taple. We can abuse this with the rank string function from npi testing private utils to execute the return string which won't be picked up by pickle scan. For our test we uh we used curl to see if we can get a call back to our uh HTTP server. I've also attached a video for the proof of concept where you can see the full exploitation path. Uh now for the analysis part uh we can see that I used Python script to

disassemble the whole pickle file and we can see that the main focus in is actually on the global and reduced op codes. The global op code the C numpy testing private utils run string tells the Python pickle machine to import the Python the function run string from the module. Uh next we observe the pi bin uni code import OS system which is actually our reverse shell command. Uh then tpple 2 builds the tpple of argument the embedded command and an empty dictionary and reduce actually calls the previous imported run string with those arguments thus executing the command. With this we are uh able to get the call back to our server. Now for the second CV. This is actually

a pretty significant CV because it affected uh Llama CPP not that long ago. As you can see, because a chain of vulnerabilities to be exploited can be actually covered in one screenshot. I'll go over the vulnerability itself. As can be seen, when chat format and the chat handler is none, it goes through to the guess chat format method to try and actually guess the chat format from the metadata. It will then be parsed into ginger to chat formatter as parameters to chat handler. Now for where the vulnerability itself lies, the code instantiates a plain ginga 2 environment without a sandbox that is the critical part. Then renders the attacker control template. Ginga 2 compiles this template in the call

constructor allowing RC's or denial of services since Ginga 2's render evaluates embedded codes like eel enabling arbitrary code execution for a proof of concept we are going to abuse this and see if we can get a reverse shell to our VPS onto the exploitation part because the ggfjet template is read from model metadata and passed to the un sandbox ginger twin environment attacker control template expressions are evaluated at render time. In this configuration, Ginger 2 permits attribute reversal and evocation of callables available via the render context, allowing a crafted template to reach Python objects and execute uh arbitrary code during prompt construction. Basically meaning we can remote code execution. I've also attached the proof of concept for this.

And in this next slide, we'll be delving more in the payload used to gain RC through this command. Uh for the analysis part, I'm going to explain in better thoroughly the functionality of the template. The expression class base subasses is actually used to retrieve a list of all subasses of the object basic object class and then we check if it's warning in x.name. Uh now we'll access each modules via the module attribute and access the methods builtins functions through built-ins and we'll use import function from there to import the OS library and then we'll use the popen function which is used to execute commands on this system and we'll use uh named pime named pipe uh

reverse shell to get our uh reverse shell to our VPS. Yes. Uh to summarize all our examples, we can see that this can seriously affect all of us especially because it can also happen in a big platforms like hugging face. Whereas uh although they have strong uh and strong security because they implement AV scanning for machine learning models when uploading them to the site, we there are also platforms which do not implement this and uh they might contain back doors. As you can see in the picture, there are lots of shady models being uploaded to platforms like hugging face every day. Make sure to either download them from an official source or dive in depth before you

decide to use them. Now I'm going to let my colleague talk about the remediation steps for each of the uh machine learning models. And that was all for me. Yeah. So let's close with uh the defensive side. So for pickle, please avoid uh it altogether whenever possible. And if you must use it, um I'd say that you should restrict the the serialization to safe objects and uh safe object types and to use the as Dennis mentioned the weights only true in PyTorch and you should also run the serialization in a sandbox environment with all the uh network access uh restraint and also the the rest of the privileges. is uh uh uh set up and also to enforce the the

resource limits. And for GGF uh you should uh probably render templates from a model metadata. uh I mean to never do that but uh you should also treat the metadata uh purely as data and to validate it and use um hardened templating uh runtime if and if rendering is uh absolutely uh required and also you should all disable automatic uh rendering uh path in production and across both please enforce U origin checks. The these are uh the most important uh things that you should do uh especially during today's um compromise compromises that are in the headlines. You should require uh signatures for every artifact that enters production and integrate model scanning in your uh CI and also make

every uh machine learning engineer understand the that model files are basically as sensitive as executable code. So final thought here, you should build systems that fail safely whenever they face malicious artifacts. And that's trust no format. Thank you.

Now if you have questions guys. >> Yeah, please. I was curious about how to enforce those.

>> Yeah. Uh to answer this question, you should have conversations periodically with uh the developers that use these models and as you imagine the the attacks now evolve super fast. So you have these periodic checks. You should also have uh as I mentioned earlier the checks inside your pipeline in the continuous in the CI/CD pipelines. So yeah you you should have periodic conversations. Now Dennis want to say something else here? Uh no, I think my colleague mentioned almost everything. If you guys have any other questions, if not, I would also like to add the uh the thing that there are lots of CTF challenges that cover PL serialization and I think there are very very

interesting ones uh where you have to bypass different types of uh uh limitations about the functions that you can use to get uh to call the reduce function and actually gain RC on different machines like heck the box uh machines or challenges and stuff. So if you're interested, I can give you some recommendations on that as well. If not, that was all from us. >> Yeah. Thank you guys. Please reach out to us if you have other questions. We are happy to take them. Thank you.