GenAI Application Security: Not Just Prompt Injection

we are going to uh hear about generate genai risks. So last but not least, Ahmed is going to send us home with few more nuggets of wisdom. I'm very interested in hearing what he has to say. Ahmed, take it away. Round of applause for Ahmed. Thank you. Thank you everyone. Uh well, first of all, thank you so much for hang for sticking around. I know it's everyone is almost done, right? We want to go home. Um, and I feel like this talk could be a very good way to finish the conversation. We're going to go back a little bit, right? We're going to build on top of other talks that were done. Uh, there will be a few things

that were discussed. We're going to just talk a little bit high level in there. Uh, with that, my name is Ahmed and uh, I've been doing cyber security work for the past 18 years or so. Um, my full-time job is with Forinet. I'm also working with cyber dojo which is what we it's an organization we do research right uh security research with AI and I'm also teaching for SANS I teach two classes sik 5440 and sik 545 uh please feel free to connect I really like LinkedIn it's the easiest way uh I promise you this is not a malicious QR at all and I'm pretty sure you have heard this joke many many times at this

point um with that my friends I come to you today with an argument and the argument is when we talk about security uh in specific with ji it's not going to be a lot different than we are typically used to do on the high level the details of it are going to be different of course um and u the reason for that is because hey we we have new type of technology and we need to understand like how does that work one of the things that I heard over and over uh in many different ways is the fact that we cannot secure what we do not understand and because of that what I'm going to do

today is break this session into two sections. The first one is that I'm going to go back and discuss some of the key concepts about generative AI. Where does it come from? Where does it fit? What do we expect to see in the future? And then based on that, we're going to move to security, right? We're going to talk, hey, like are there going to be any risks associated with that? What should we do uh with it? I'm going to start from the very very beginning in the beginning of time, right? In the 1950s. Well, I'm kidding. I'm not going to start there but what what what this slide is here to tell us is that we have

been talking about AI or in other words we have been hearing about AI for a very long time right um but it wasn't until few years back I would say maybe five to 10 when AI was became such a hot topic and in specific three years ago when chat GBT came to life and all of a sudden everyone technical and nontechnical now understands what AI is but let me take you back a little before that. So there was the always this issue of generating context text as an example right so generating text has always been a challenge I remember when I was in college there was an AI class and uh people started dropping from that class

nobody would take it because it didn't have a lot of applications back then obviously things has changed the reason for that is that we didn't have the needed technology now obviously over the past few years this technology came about and one of the key components was the Transformers technology transformers was created by Google uh June 20 2017 uh there's a link right there right and that links take you to to a paper that is called attention is all you need anyone read that paper oh great so we have many people that's that's awesome so just to recap right so transformer came up um I found this on hugging face website and it basically shows how many

different technologies were built based on transformers that we have. So what are transformers? Well, it's very basic on a very high level. What we are going to do or what transformers do is that they take a string of words, right? And then they try to predict what's the next thing that they need to generate. Let's just switch that into a little bit of a technical language. What's the next token that's going to be produced or generated? Um we have a very obvious example in here, right? You go to Google and you try to search. You type the first few words and it automatically tries to predict what is the next thing that you are going to do. This by itself

might sound like a very trivial thing like the use cases of it might not be obvious at the beginning. Just keep in mind that this can be used to generate text as in language but it can also be used to generate instructions which means that we will have an engine and that engine could produce different types of outputs including instructing meaning taking actions on other things as we will see uh later on. Now this has created well what we now know as LLMs and the obvious example is the list of GPTs or chat GPT being one of them. Let me pause here. So chat GBT itself is just an application right it's a chat application but at the heart of it or

let's say at the back end it has an engine that's an LLM based engine a large language model you would communicate with it right you would send instructions via this web uh for HTML format right it's uh it's very basic but then at the back end your instructions or your prompts are going to go into this engine that's going to do some magic and then response you whatever you know the answers that We are all used to it. This is great. And when this happened, everyone was like, "Oh, wow. This is a gamecher because now it can help me write emails. It can help me do my work. It can help me do research, summarizing things." And u that was

great. But that's not not even the beginning of it, right? The other thing that OpenAI allows us to do is to integrate with their APIs, with their models through their APIs. Take a look at this example that we have for you right here. Let me see. So, if you guys look at line number I can't see it from here. I'm going to use this one. So, if we look at line number five, you can see that uh this is a request an API request that's going to the API the OpenAI API endpoint and there are certain parameters. One of them is the model itself. Now, this is Python, right? Underneath that there's also authentication. So, another thing that's

going to be sent is the key. And the reason why we're saying this is because this is a security uh conference, right? So there's a key authentication that's going to be happening that's going to be sent as well. And then the rest of it is this JSON output or JSON input that's going to say this is what I need you to do for me with some instructions as well. This means that you can take this and you can put it inside an application. It would look something like this. So now it seems like our applications are changing where previously you had an inter an application that has many components. So for example it has a web interface. It

could also have some sort of an SQL database right. It could also have some other components right different libraries being pulled maybe a backend etc. It seems like now what we are going to see in our organizations uh is that these applications would have another component that is the genai component. That's the engine we're talking about. So let me again pause and say why where are we going to see this? We all work at different companies and um we are going to maybe buy software that has Genai components. Maybe we're going to develop our own. We have heard many talks today about what people do with AI and most of it is going to be hey I'm going to write

some code with this geni component at the heart of it. This also means that the logic of this application, so this could be a chat, right? But not necessarily a chat. Chat is just one application. But the logic is going to be not just if statements and loops, right? What we do with Python, but there will be also logic that's being created in a way that's not, you know, static, if you will, more dynamic. All right, that's interesting. We we all understand this at this point. These models while they are great they have limitations and one of the limitations is the knowledge that they have. There are ways that we can enhance this knowledge. Um and I

would like to point out that all of this could change then within the next year or so right so everything here might be you know outdated very very quickly but one of the things that we can do to enhance the knowledge is for example adding some instructions to this prompt. So you know for example we're asking the chat GPT to give us answer about whatever topic the answer is not exactly what we need. So we would give it more context maybe we would give it uh more explanation and I'm pretty sure everyone has done something similar to that. That's what we call prompt engineering. Right? All right. So that's one way. Another way to do that is to take one

model and then fine-tune it. That's good. Right. We can also have external sources of information feeding this model at real time. We're going to talk about some of these. So fine-tuning. So what is fine-tuning? We would have a pre-trained model. So that would be your GPT4, GPT 3.5, whatever your cloud, whatever. These models are already trained on specific data. And these models were created by whatever company, group, organization that they created. You might need to tailor this model to your specific use case, right? What that means is that we you might want to have this model to produce an output in a specific format. Think about a model that would help you write SQL statements, right? You can go to chatgbt

and say, can you write me a statement that does that selects all tables, whatever, right? And then drop them maybe, right? what that what the output might look like is something like an explanation what SQL is and then the statement itself and then maybe some other notes and so on so on forth that's great if your application is a chatb based application but if you want to take that output and then automatically apply it somewhere else so maybe you have an agent and that agent's job is to go into this database and fetch information for it you can't have just like whatever random data the output has to be an SQL statement nothing more nothing less and

it has to work so one way to do that is to fine-tune your model so that the output is different all right this is good another way to do that is by using vector databases and that's another concept that has been discussed many many times today and yesterday so a vector database is a type of database and it's called vector because it stores vectors and we will talk about that in a minute but the short story or the high level story is that we will have bunch of data more recent data maybe that we are going to store in this database and then every time we ask the LLM for or like the application for some

information that it doesn't have it will go check with that database it will go to that database and say hey do you have anything relevant to the question that I have and if there are things that are relevant then these things are going to go back and they're going to be combined with your original question into a prompt and then that will be sent into the LLM. Now the LLM has more information and it can answer that question based on factual information that we should have vetted and underline and should have right okay this is what we call rack that's what the rack process is this is great awesome this is this technology is we

can achieve this technology because we have embeddings and embeddings is basically the process of taking words like I don't know just characters, words, etc. and converting them into numeric values and then into vectors. So what does this mean? I went into Chad GBT and I was like all right can you create like some sort of a graph that simplifies how embeddings work and it needed to be a 2D a two dimensional array right just because it's easier for us to imagine and it came up with this. So take a look at this one. So we have we have apples and we have car and this is just like random like it doesn't mean anything right but this is a vector and

that's another vector and if you look at these you can see that you can place them on this two-dimensional array and you can place them in a way where the relationships between these different words or tokens are going to be very very relevant. It's going to cluster tokens that are related to other to each other together. We have few examples, right? We have all these fruits down below. We have bike and car and you get the idea, right? Now, obviously in reality, we're not going to have two dimensional array. It's going to be an array, but it's going to be have it's going to have hundreds of dimensions. It's very difficult for us to imagine,

but that's how um embeddings work. We're going to take these numbers and then we're going to store them in this vector database. Our application now looks a little bit more comp complicated. It might look something like this. So check this out. We have some sort of an interface. Could be a chat interface. Notice that we still have communication with users, right? We still have the whatever we had traditionally like the traditional code that we have previously. We do have some models. So LM models, right? We also have embeddings models. So now we have a distinction. We have more than one type of model within our application. You could also have agents. And notice that in this graph the agents are within the

application. That's not necessarily the case all the time, right? But you could have it this way. And then you can also have external resources. So the external roses, we know what databases are. We could also have agents to connect to different things. So search engines for finding us more recent information. We can have it to connect to basically anything that has an API interface. Anyone is scared already from a security perspective. Raise your hand if you are. All right, we only have few. I'm I'm very disappointed. No, I'm kidding. All right, so I think this is a good start because now we can look at this and we can ask ourselves certain questions such as are there going to be

any risks that are associated with this newish design? And obviously the answer is yes. And uh luckily for us we have an organization called WASP. And has created the top 10. I'm pretty sure many of us are familiar with it. they create the uh top 10 web applications etc. One of the documents that they produce is the OASP top 10 attacks for LLMs. Now we took those and then we tried to broke them into you know where does this affect which part of the application right so for example let's take like few uh I think everyone knows what prompt injection injections are right and they have a direct effect on everything that is a model so it could be the LLM model

as an example right let's take another example here let's see so we have uh misinformation right here right now I don't know if this is like 100% accurate this is the best we could come up with, right? But I feel like it gives the the idea. Um, have you guys noticed the access keys over there all the way to the top, right? Is this AI related? Not really, right? But it's heavily used within the AI. Back to the argument that we're going to look at the same security controls that we have, but we are going to adapt them for how the applications are changing. All right. Um, so let's talk briefly about prompt injections. And I think uh we're not going to spend

a lot of time in here. You have a model. The model has multiple areas in which input can come from to, right? So you have direct access from the user. The user is asking certain questions, sending these different uh prompts. Some of them might have injection attacks. That's one way to do it. Another way to do it is to do it indirectly. For example, you have a vector database and someone up uploaded certain data and that data has instructions, right? And those instructions are going to affect your model. Another way to do to to do that is by looking at the different external resources. Take the following example. You have some sort of a model,

some sort of an application that goes online and searches for information. For example, financial informations from blogs. In the blogs, there might be credible information. Is there a comment section? usually. So if someone goes into this comment section and adds certain does that go into your LLM as well. The point is there are multiple entry points either one of them could have certain malicious input that could be one form or another of prompt injection. So what can we do to combat this? Welcome to the ugliest slide in your lives. This one. So we're not going to talk about all of this, right? But I want to draw attention to some of some things that we

have and uh in specific the very first one. One of the first lines of defense is to apply filtering. And this idea is not new, right? SQL injections. That's basically what we do. We apply filters. Um the question is where do we apply the filters? This is why in the previous discussion we said there are the inputs. We need to identify where the inputs are coming and based on that we're going to add the filtering in there. How do we add these filters? We do have guard rails. We do have practical guarders. So for example, the AWS bedrock has has mechanisms to create guarders that are very very good. We also have other solutions. So for example, we have llama

guard. Llama guard is a model that is fine-tuned. Llama model that is fine-tuned to look for security issues, prompt injections and other things as well, right? It will classify the input and then identify certain aspects uh that it has. You could also use models like other type of models your own for example to look for things such as trickery. So like sometimes that becomes very difficult right because if you think about SQL injections there is very specific way in which you have to craft your query but you can be more creative uh fighting static with finding dynamic with static is not very easy right so you can rely on LLMs to do that as well.

The other thing that could there we go. The other thing that uh that we could suffer from is poisoning this these different knowledge sources that we have. Does it makes perfect sense that if you have a vector database that has a lot of good information important information that's going to be a target right so does it also make sense that you if you have an external access that will also could become a target such as the example that we have given. I think in the previous talk there was an there was an example about emails. Someone sending some sort of malicious code poisoning the email and so on. Poisoning the data is not just about tricking the model into doing

something else but also affecting the integrity of this data itself. Does that mean that we need to think about how do we ingest data? Maybe look at the process and the overall process. Absolutely. This is one of the um one of the uh ways we can prevent attacks that are happening in here. Take a look at this list right here. Right. And I would like to point out few things and uh look at this. So we have where is this? I'm going to use uh so implementing network controls and authentication. Does it make sense? Like there's nothing new here. But now that we understand where these components and what they do, we can apply these different u models as

well. Take a look at this one. My favorite ML secops. Are you guys familiar with dev sec ops? Many of us are right. So that's taking the devops process injecting security into it. We're doing going to do the same thing on ML secs as well. We will get back to this in a little bit more details. In addition to that we have given the example of open AAI and then we said that hey you can have integration you can send requests to open AAI. With that comes some privacy questions, right? Uh but these models could be executed or they can be hosted locally as well. Does that make sense? So how can we do that? Anyone played

with Olama before? That's I think the easiest way. We have like a few people. That's great. Uh so we have that's I think that's the easiest way. Do you guys agree? All you have to do is download this piece of software and then you open it in the terminal and you would say something like lama run whatever llama version 3.1 or deepseek r1 whatever you want and what that will do is that it will reach out to the lama library it will pull a model a specific version of that model and it will store it and run it locally on your system. The green screen in here is obviously the server Olama X in a server client

mode and um to my surprise when I first ran this right this is running on this laptop actually which is an M1 laptop and it runs it's running the smallest Llama model I think it's the 8 billion parameter one and it runs pretty quick and you can connect to it using this client right you can see like a lama run and you can just like ask it whatever it has a chat interface it also has an API interface which means you can integrate that with your own applications as well. That's not your only option. There are more, right? So, hugging face is a is a hub, right, that has many many many models and other things as well. And so,

that's these are thousands of models. So, for example, meta could create uh models and then they could publish them on hacking face. I can also create models and I can publish them on hacking face. Anyone sees an issue with that? If I reach out to you and say, "Hey, do you want to use my cool model that I just wrote, right? Is everyone going to use it?" Of course, because you trust me, right? Obviously. Um, so that's one thing we can do. So, we have thousands of models. Everyone can push these models in and then anyone could use them. Does that scream supply chain attacks? Of course, right? Uh, so that's one thing. There is more to hugging face

than that. One of the things that we are going to talk about a little bit is the transformers library that you see in here. All right. So, hugging face itself because of this because it has a lot of important assets, important models and data sets and other things as well. They have been applying a lot of security controls and they are growing. So look at these right. You have you can you access a hugging face by API which means you should have some sort of control and you do when you create a an API key you would choose exactly what that API key can do. It can also scan for malware and secrets as well. So what does that mean?

Like why do you need to scan for malar as an example? Well remember hugging face is a place where people upload stuff. Could they upload malware? Absolutely. The foundation of hugging face the the the repo is based on git. Now obviously it's bigger than that but the basic of it is git. So someone could push some sort of malicious whatever including models themselves. That means that these models that we have are practically everywhere right? We have them everywhere. We have researchers in our organizations who are very very curious. Are AI researchers proficient at security? Right. I see people laughing. So, yeah. So, does that mean that they will be curious when I send them that

email saying, "Hey, can you take a look at my uh my model?" It's pretty cool. It does like cool stuff and they would download it and they would do that, right? So, how does that work? So, hugging face offers us the transformers library, right? And the transformers library has many classes in it. Um, one of them is this pipeline. Where is that? There you go. So what this line does, so pipeline equals whatever and there's a model name or what appears to be a model name. What this line will do if you run it is that it's going to download this specific model from Facebook AI, right? And it's going to just load it into your

memory. As simple as that. So that's pretty cool. That's basically how we can actually have someone to use your own model. Because of this, over the past couple of years, I would say I would say we have been seeing things like this. So many different attacks. So in particular, look at the very first one basically saying uh hey there are thousands of malicious models around you know hugging face and other platforms as well. And then there are different stories about what can we do to combat that. One of them is about AI bomb uh bill of material right and uh basically whenever we need to build we need to list all of the different components in

them etc etc. But why is that such an important thing? It seems like these models can be treated as executables. They're not by themselves executables, right? I don't think that's the tech that the right technical way to say it, but they should be treated as executables because they will be loaded into your memory and they can execute things. Let's take an example. One of the formats. So these models are files, right? And you can store these files in different formats. One of them is the pickle format. The pickle format is not an AI thing by itself. It's a Python thing. It's a Python way to represent Python objects in a serialized way, right? So, it's an ASKI representation,

a byte stream representation of whatever Python object that you have. It has many advantages, right? One of them is that it makes working with these um with these objects much much easier. Well, guess what? The AI models are also objects. So you could for example serialize and des serialize these models. So what that means is that you can take them from one format and then uh have them represented in the desk or in memory in a different way. You can also append to them certain code. Anyone sees an issue with that? So could you for example find a model and then download that model and then write some ninja code that will append malicious code into this model? You can

absolutely do that. So that's what we call the serialization des serialization attacks. That's how it works. What can we do about that? Well, because this is new, right? And um we have we always need the theory. We need to understand how this works. But it would be also awesome if we have some tools that can help us work with that. So luckily for us, we do have certain tools. So for example, model scan. Um model scan was created by Protect AI. It's a free open source project and it's pretty cool. What it does is simple. You basically say, "Hey, model scan, can you scan this model for me?" And that's it. And it will go through this model. It will go

through read the code and uh it would look something like this. I think this is the second worst slide that you guys have, right? Uh but at least it's it's readable. So look at this. So you're running it against a specific model. Where is the name of the model? I think that's it. I I'll just use this one. So you have um this one is the model file which is like the file name and the tool is going to go through it and it's going to tell you hey there is a critical issue and the issue is you're using Python system to execute commands so like bash commands on the whatever right so what that means is that this model could have

something like a backd dooror which can be executed every time you run this model and it's going to be transparent to the user the model is going to execute it's going to run perfectly with a back door associated to it. All right, so this is great. Is anyone in favor of running manual scans? No, of course not. Right, we need to find a way to automate this whole thing. Uh, and we can do that because this is where MLOps comes into play. In the near future, if you are if your organizations are building different applications, one of the things that they could do is that they could download a pre-trained model. They could train that model. They could make

changes to it. You could maybe build your own from scratch and then you're going to use it within your own applications. That's one scenario, right? You can also add data to it, right? So the training data is going to come from somewhere. It needs some processing. All of this should fall under the ML ops. That's the operations and that's where you can actually apply a lot of the security that we are talking about. Take a look at this example right here. This is a very simplified example. Right? What this is is a GitLab CI/CD pipeline and uh you can see that I am building an application here. So this step build is actually building the application. That

actually means building a container and the container will have code and it will have the actual model file itself. Now this is a very small application. So this is not an LLM. You can tell from the name it's a sentiment API application. A sentiment model is a class of these different models that basically makes classification. It will tell you, hey, this is positive or negative or like whatever, right? So, it's a very small model that can fit specific use cases. The next step is going to be deploy, right? So once you build this uh docker file, you need to deploy it somewhere. In this example, this is like Kubernetes, right? Doesn't have to be, but this is just the example that we

have. What we can do is add a step before that very similarly to what we do with div secops right and in the step we can scan. So what we can say is that hey if you detect something wrong with this model just break this pipeline do not push this into production. Are we going to apply this only to scanning models? Could we apply this to other things as well? So for example maybe running a checksum maybe verifying a signature right? So maybe we need to whenever we create whenever the a pipeline creates a model maybe we can sign it and associate a signature to it and then we can verify that signature in here. Right? All right. All of this

leads us to threat modeling. So one of the things about threat modeling is that um we would understand what is the good behavior and then we would ask certain questions. If only this would work. There you go. And these questions should be things like uh hey um what could go wrong right and how and then we're going to try to address these issues. This is obviously a big and developing topic, right? So, we're not going to go into a lot of details here. Uh, but I was looking around and trying to see who has done some good work on that and I found this. Check this out very complicated graph, right? But not really. It's

actually very similar to what we have shown you earlier. Um, you have different components for this application and at each one of them something could go wrong. You can see that we're mapping. So, for example, L LM03 that comes from OASP, right? So training data poisoning. So this could affect this aspect of this application. Let's look at another example. Let's see what we have here. Uh we have supply is that supply chain and it could happen at this stage and so on and so forth. It's always great for us to find a way to visualize uh what could go wrong. If you ever find yourself in a situation where you have a local team that's building

this, maybe these are good way to ask questions like hey are you guys uh looking into prompt injections? Where are you applying your checks? Are you applying your checks on the interface, forgetting about the vector database, or are you applying your controls or your checks as close to the LLM as possible? Right? So, different architecture and design questions that we have. MITER has also created a lot of good work. MITER Atlas is the version that basically, you know, we're mapping what could go wrong. Again, I would urge you to look into this, right? Obviously a lot of details and if you look closely you will see that the vast major majority of them are not LLM well maybe not the vast majority

but a lot of them are not LLM specific some of them are and some of them are not just like a combination of both this is this was a lot right I have one more slide for you so we haven't talked about MCP right and uh I think uh many people know about MCP right but uh MCP is uh this new protocol and the protocol is basically going to help us into building a server an agent server and then everyone could create their own. I can create my own. I can come to you and say hey would you like to use my MCP server and you will happily do that right? So does that raise any supply

chain issues? Maybe not yet but maybe soon enough as well. Um there was this document I will get to it in a second. Um and this document talks about MCP threat modeling and what are some of the things that done that could happen and if you look at it it's also very similar to what we have been talking about which is controls that are traditional controls that we are applying to this in this case the MCP servers uh feel free to check this out I have a link for it later on actually it's right here so just to summarize right if we are to go back and say hey this is a new topic I

am a security person in my organization. Where should I start? I would recommend the following four different uh sections. We already talked extensively about um OASP right that's so that was one of the first ones. SANS also has their own. They have recently created that. We have two documents. The multi-agent tech system threat modeling guide and the inter enterprise guide for MCP security. This is a very recent one. I would also encourage you to go and take a look at that. With that my friend, thank you so much and I hope you enjoyed this one. Uh please let me know if you have any questions. Big round of applause for Ahmed. Thank you so much. Great information. Uh so

I'm looking at uh the slideo. I don't see any questions pouring in. I hope uh that's not because of any technical difficulties but be that as it may, we do have a few minutes. Uh if anyone has question in the room certainly please raise them. I think the topic is very easy. You blew everybody's mind. They are stunned. Any questions at all? Thank you so much. Well, if not we going to call it a wrap. That's besides 2025. Thank you everyone. Thank you so much. Enjoy RSA if you are here this week. Yeah. I'm going. Thank you.