Lucija Valentić | The Hidden Threat: Unmasking Malware in Machine Learning Models

so uh welcome back we finally have some refresh after three male presenters so next in line is Luca Valen from reversing Labs but don't let sweet face of Lua fools you Lua is pretty dangerous but luckily not for us but for malare hting in the npm packages pip packages so I'll Leta do the talk about the hidden threat in unmasking malver in machine learning models uh thank you uh Miss love hello uh thank you everyone for coming I'm here to talk about uh how there is a way to hide malicious software inside machine learning models and how the way they are saved is the reason for it uh for this presentation I'm going to focus on pickle and we're

going to show some examples and maybe tools uh with which you can detect those uh malicious uh pickle files uh this presentation was already held before but I have added a few slides that are new uh with new research that uh we at reversing Labs actually did uh but first let me introduce myself uh my name is Lua valich uh and I'm a threat researcher at reversing Labs uh there in everyday job I I'm uh searching for um malicious software malicious npm packages up until uh last couple of months I've been searching on vcad Marketplace and ruby gems um so let's start of course in the last couple of years we have seen an increased interest

in uh artificial intelligence with large language models being popularized of course GPT 4 that PO GPT being one of the most famous ones we can now also seen some open- Source um lll models such as deeps um I think also from uh French uh we they also uh relase some open- Source uh llms and so on and of course companies and users alike are start including those llms in their own applications and using them uh so of course people have starting training models being more interested in them um and they're trying to use them in their own applications so of course repositories um such as hugging face or pyos Hub were created uh which sort of

serve sort of uh repositories where you can open source repositories where you can store your models you can share them with other people maybe if you want to just use them and not write models um you can find there uh but of course like with any open- Source uh play Place uh potentially there are also some risks and vulnerabilities found there um and two years ago Spang did a research uh on risks and vulnerabilities on hugging phase um they focus on hugging phas because hugging phas is currently the biggest one uh repository we can store malware and what they found out among other things is that majority of models found there are pickle models and that is a bit concerning since

pickle is inherently unsafe and they are very they're it's very simple to abuse it um to make it execute arbitrary code and we will show some of the examples uh but first uh we will cover what are ml models we're going to cover how they're trained what they're used for uh very very briefly and then we'll uh say where comes piin of that uh so ml models are type of mathematical models that are used for various function they are used for image classification for uh for example CL file classification for detecting malware for price prediction and so on they're trained with uh using data sets called training sets and with each iteration uh weights of the module are

being improved improved and they are being better and better at its function and once the model is trained once it performs well that function it is saved without a question um and there are many reasons why you want to save your model for example maybe you are creating a big model on your training in on big data sets so it takes of course time a lot of time and computational power to train it so of of course when you need it you're not going to you know do it once and then scrape it and then do it again and then scrape it you're going to save it once and then each time you need it

you're just going to reuse it of course sometimes maybe you want to share it with your team members because you are the person who trains ml moduls but other will use it in their own functions their own applications whatever you also maybe want to share it with Community you want to upload it somewhere for example on hugging face because you really want to work with ML modules you really like math behind it you really want to train it but you don't want to use it or you just want to be a good person give it to community so they can you know use for their own applications and you know whatever and of course while you saving that model you

sterilize it and um like with anything you are using ml Frameworks because you're not going to write everything by yourself and different demal Frameworks have different libraries for sterilization and the sterilization and depending on which kind of model you're working with which kind which kind of modu you're training you use different EML frams different um libraries for sterilization and the sterilization uh and here comes the most famous library that I have already mentioned um and that is pickle which uh is used by by torch uh but what is exactly pickle and how can you find yourself in a pickle so pickle is a python library for sterilization and D sterilization like I have already said um and here we

encounter two uh Concepts pickling and un pickling or serialization and distalization and it's very simple so in the process of pickling or sterilization um you take a python object hierarchy and you turn it into a binary system pickle and in the process of unpickling or distalization you take that binary system and you turn it back into um python object hierarchy um of course if we want to understand why exactly pickle isn't safe because it's not very uh now known uh we want to look at its source code we want to look exactly what happens in the process of pickling in the various extensive commentary in its source code uh but basically in the process of

serialization that python object hierarchy is turned into a series of op codes that represent that uh python object hierarchy and that is called pickle and in the process of unpickling something called pickle machine that has stack and memo is interpreting that op codes until us op code is reached stop and of course there are various sub codes that do various things uh some putting things on the stack some are taking it off the stack but two op codes can be a bit dangerous they're normal op codes but they can be abused um to execute arbitrary code and those are reduced and Global so they in the process of unpickling they invoke uh function reduce which is again a normal function

um in normal unpickling it returns all necessary information uh for some object to be distalized to be unpickled but because of how pickle works and how op code Works uh redu can also be abused to execute arbitrary code um normally reduce um has a normal function so basically if you want to um pickle your custom class perhaps pickle won't pickle it serialize it um in a way you are satisfied with or an error will get thrown because pickle has list of objects that pickles well but sometimes you know your custom class won't get through so what you do you tell people how you want your custom class how you want your objects of that custom class

to be serialized um and that can be abuse which we actually see in a um example here I have made very very very simple example just to Showcase how easy peopleo is to be abused uh so I have made um an evil class that I told people how I want that class object of that class um to be serialized and deserialized so what I did I of course pickled it unpickled it on Linux um and I got a host name of that computer very very easily um and that happened because I told pickle with function reduce that I want in the process of film pickling that he returns to me function OS system uh with

arguments host name and that is again very simple example uh B Showcase and safe iness of pickle and something very similar actually happened in real life uh so we're going to look at next three examples how faction reduce were was abused in real life or proof of concept to Showcase unsafe of pickle and how there are uh Malay seml moduls on hugging face uh so I think last year uh J frog found malicious seml model on hugging face that had RSE shell inside so of course when victim would use that machine learning model when it would upload it load it use it uh reverse shell would get executed and of course malitia sector would without victim

knowing have an access to victim's computer um and it was very easy uh to inject that malicious payad inside machine learning model with just using function reduce uh this example is another very simple example very reminiscent to my previous one uh but is a real life example that was found on hugging face uh the next two examples we going to look at are more proof of Concepts um but they still use abuse function reduce um uh to execute to uh inject malicious payload so malicious uh so llm models are turned malicious so in this proof of concept uh researchers at H layer uh found a way uh using stenography to inject R someware inside of the weights

of model and then using function reduce they injected malicious uh script that would once model was used once it was loaded uh that malici malicious script uh would uh actually piece together on where it was using stenograph it would extract a back from weights it would piece it together and then in the end it would execute it so basically uh they of course have written like three uh scripts and uh only two commands are necessary for uh machine learning model that is normal uh to be injected with um run someware that would run once ml model was used once it was loaded the last example we're going to look at uh of malitia seml models and

how pickle is very unsafe um is a little bit different so researchers at trob bits uh they did a pro concept once again uh but they didn't try per se to inject anything malicious inside machine learning model uh so usually large language models among other things are used for for example generative AI so of course when people are using them they expect that you know informations are at least a little bit correct or not harmful uh but what researchers at trob bids did they again abusing unsafe of people uh and abusing fun reduce they injected uh script that would once large language model was used that script would change weights of the model ever so slightly just a little bit uh so once

the model was used in some application or something um it would actually give out uh harmful information and of course you know drinking bleach as uh for health is very easy to detect uh but that may not always be the case and as we can see in this example sple is very ins safe and this leaves us with the questions um can you detect those stuff can you detect if you think some uh pickle for example is malicious or some ml model and of course there is there are systems and tools put in place uh so the first tool that can be used uh is Fickling the developed by researchers at trailer bits uh even though Fickling

is a little bit more on verse engineering side so if you you know want to play with pickle you can inject malicious payload inside you can do static analysis of it whatever uh you can also technically use it to detect if a pickle file is malicious with option check safely so if that pickle is something is a miss that pickle is perhaps malicious uh Fickling will alert you that you know one function can be used to execute arbitrary code and that fle is mostly malicious and here we have very again simple example that uh showcase the evidence of that uh another tool that can be used that again it's not strictly for detecting malicious pickle files but it can be used um is

pickle tools so pickle tools is a python Library um and it usually used uh for developers that work with pickle because it gives you extensive commentary on pickle and pickle functions and how your pickle actually looks like uh but I can also give you a symbolic disassembly of that pickle so it gives you a list of op codes that you know will get interpreted again you still have to do your due diligence because it's not going to alert you but you can perhaps catch if something is a miss another thing uh was actually set in place by huging phase so if you're using huging phas there are scanning for malicious pickle files of course they

have Mal scanning where you upload like any open source repository I think at least uh they whenever you upload machine learning modules they scan it for malware uh but they also have pickle scanning so whenever you upload a machine learning model or you update it they scan pickle files uh for all the Imports and they route write out all the Imports that are being used inside however um you still need to check all those Imports of course they alert you if they know that some import can be um used to execute arbitrary code as we can see in this example function evil uh but sometimes they do not actually check all those Imports so for

example if some user wrote pii package and pickle file used it in its import of course pickle scanning won't alert you to it uh pickle scanning uh you can the tool that is used for pickle scanning is a pickle scan that you can uh use locally you can find it on GitHub uh but with pickle scanning when we you know did a research uh we found uh another uh malicious ml model that actually vaded that pickle scanning and I did lie when I said those examples were the only one I'm going to show you uh another example that we found that uh my colleague Carlo did um and he wrote an amazing blog about it uh but basically he found uh

malicious model on hugging face um and inside uh had reverse shell it wasn't detected by pickle scanning uh because probably it was compressed with 7z uh so P torch load couldn't load it so it wasn't detected uh but we also have noticed um one interesting thing about pickle files that had malicious payload inside and this showcase another reason and how easy uh pickle is abused so those pickle files that had malicious payad inside that had reverse shell were broken and you would think that if they were loaded if they were used nothing will happen because you know it didn't when you scan it with pickle scan nothing returns and there are returns but um it shows that

there were no malicious files and if you try to load it again and error returns but that is wrong because it has everything to do how pickles scan work works and how actually un pickling works so pickle scan first it uh checks if the pickle file is valid or not if it's valid if it's not of course Broken then it scans it everything's fine however if it's broken it's not valid and an error gets thrown and that's it but it still returns that no malicious files were found it still has that output and if you try to unpickle it because unpickling that pickle machine works as interpreter it doesn't check first if pickle is valid or not it just starts to

interpret those OP codes until last op code is reached or until of course some op code that is doesn't exist is reached which means that if malicious payload is before that breakage it will still get executed no matter if unpickling returned error so what Carl did and it's explained um in his blog basically he made um an example very simple example that had malicious pilad inside that is not actually malicious it just creates file and he tried to scan it with pickle scan um of course because it's not valid pickle file it he broke it um of course an error was thrown but it also had an in in output that there are no infected

files inside even though that was incorrect and when he tried to unpickle it as you can see U again an error was thrown so you would think that nothing happened but but as I said of course all those OP codes were interpreted so malicious payad was still executed uh so that file that he created you know he tried to create with that malicious payload was still created um and this is another example how people is very very unsafe and how easily uh it is to abuse it and to use it to execute any arbitrary uh code and of course that again leaves us with question can you be sure that you know your models your pickle files if

you download it from somewhere can you be sure that it's not malicious can you be sure that if you download it from open source that is 100% safe well like with any open source software if you're not going to write yourself even in some cases that can also be tampered with uh you're not always 100% sure that uh those models those pickle files that you download are safe so what you can do is you know avoid um loading models using mod models or pickle files from untrusted sources even though we have seen that you know hugging phase is very uh popular open source uh model M llm model repository but still had malicious files another

thing how you can uh mitigate unsafe iness of people is to use some Alternatives some maybe Alternatives libraries or maybe some Frameworks that do not use PE for example open oral Network Exchange change uh uses protuff uh which is safer than pickle not uh 100% foolproof but it is safer than pickle because pickle is very easily uh abused another alternative to pickle uh are safe tensors it is a fast and safe alternative to pickle uh developed by hugging face and of course hugging face they really did their due diligence so they have of course Mal scanning they have pickle scanning uh they are trying to mitigate unsafe iness of pickle they also have uh safe tensors and they also offer a way

for you to convert your models that are you know using uh pickle into a safe version of that that are using safe tensors um another thing uh that can be used is uh um perhaps if you have you know uh listened in the beginning all those malicious payloads all those malicious models uh malicious payload was executed once the model was loaded once it was used so if there is a way which there is uh to customize unpar you can maybe stop or catch that something is a miss and there is uh for example you have Krypton and Pickler that does just that um you can override function fine class uh and restrict all globals and imports that

are being being called because fine class um in the process of unpickling it returns um it gets invoked whenever some Global is being imported so of course if you write un Pickler by yourself if you uh maybe restrict or B some globals uh you can maybe perhaps catch in the process of unpickling if something uh will happen and if something is a Miss however uh I haven't mentioned it during the presentation but all of those tools and all of those systems um are not 100% foolproof and I have told you you know some alternatives to pickle but they are also not 100% foolproof uh because I have just showcased couple of examples how pickle is unsafe and how it

can be abused for example with function reduce very easily uh we have also seen that we have found uh broken pickle files just abusing the way pickle works it would also you know bypass pickle scan but it would execute arbitrary code even though person would think that an err was thrown and nothing happened um those conversion to Safe tensor for example there they there has been research uh I'm not sure if that was patched but that uh that conversion can be used to tamper with other models on the side so that fact that I said even though if you write it yourself um your own models can be still tampered with uh protuff it's not 100% foolproof

uh it is safer than pickle of course because pickle is very very unsafe uh it's unsafe that uh even if you go to uh official documentation of pickle python uh itself have have wrote that pickle isn't safe um but what I want you to leave you with is that um all those open source models or open source software uh that I usually see every day you have to be mindful be before you use it before you included somewhere um there are tools there are ways to mitigate uh risk risks vulnerabilities uh their tools and you know there are other people with expertise and you know just in the end be safe when you use open source

software um and that is all for me thank you for listening and if you have any questions uh feel free to ask I'll try to answer them best way I [Applause] can so is there any question for luia on this topic

uh the open search uh I I should answer in English right yeah uh so for uh they uh they use open source pickle models for developers they don't but if they use uh models it's most likely that those models are using pickle so because pickle is very very used and the most OP Source models that can be found are on hugging face and as I said like most models are people F so it's just is there any machine learning model to detect malicious machine learning models well technically in house yes we do have it

okay hi uh so if machine models are mainly data why are we packing code uh uh we are packing it uh because I'm not sure but I think because we want to like compress it and reduce it in size I'm not sure because I'm really not working with machine learning models so you have to ask people who work with machine learning models right I'm just focusing on pickle here so yeah I mean the pickle is basically just taking the whole object yeah with the code and data but the Machine model is basically just data so you could you know compress the data and leave the object aside yeah I'm I'm really not sure I have to check

because I I'm not working with machine learning models so I'm not sure like everything I said about machine learning models I checked with our machine learning team so I'm really not sure about that I'm sorry thank you so any other questions uh do you think that it makes sense to dynamically analyze pickle files like have a specific type of sandbox for uh for like importing a pickle file running it and seeing what happens on on the on the whole system or is that something that already exists or does it even make sense uh I'm not sure if it exists but it could make sense because you know for example Fickling can maybe alert you uh but of course

there are you know because uh malicious actors are very they're developing always new things how to bypass some detection uh so of course if you dynamical analysis do a dynamic analysis of it maybe you can you know catch something what happens and so on but I'm not sure if it exist maybe it does okay any more questions so if that's it of questions again thank you luia [Applause]