Securing RAG: A Pentester's Approach

Hi, I'm Nagarjun. I'm from India. I'm a API security researcher, ex API security researcher at actto. And I'm going to talk about rag pin testing. Has anyone heard about rags? No. Yeah. Can someone like help me out with the abbreviation? What's the full form of that? Anyone? Retrieval augmented generation. >> Yeah. Can can you be more loud or should I repeat it? Retrieval augmented generation. So that's like the next level innovation in AI. And has anyone built agents recently? Um can someone like can you tell me what kind of agent did you build? >> Um sure. So I I gave an LLM access to some user manuals that would help it understand the parameters of how to

create complex queries of a large data set. >> And then I also gave it um some other text >> uh files that that contained uh like user notes from people who had shared with other people. Yeah. >> And then I gave it a system prompt that said use this and this combine the data >> when somebody asks you to create a query. use these to for more of a correct query. >> Yeah. So as my as our friend said is combining documents with a language model and then giving the language model some additional context to generate better data from the language model right data I would say in my opinion. So LLMs usually have a cut off date

right everyone knows that and having a cut off date like if I want to ask GPT right now what's the share price of Tesla will it be able to give it only based on the language model data >> no right so it does search online for a data so this additional functionality that we have that is rag So adding additional context with the language model which is restricted in some way and then combining the data and giving the output to the end user right I can show you right now if you want so our agenda would be like I'll be discussing about racks how they work and then what kind of vulnerabilities can we see in racks and then the kind of

mitigations that when that we can use with our so-called new kind of agents that we are building everyone is building Okay.

So for example, I go to GPT right now. Is my screen visible? Yeah. So if I ask it right now, what is what is the share price of Tesla? So you can see the search option is here right? It's searching the web right now. So it's using additional context and giving out the data that the current value is around $241 that's visible to you to everyone right so the language model ingests this data and comes out with the data the real-time data which is the current stock price of Tesla so this is the runtime data that we're getting using racks so this is a search functionality and then we can also upload documents like so I think a lot of people might

have upload uploaded their resumes to get get it reviewed by GPT right you might get good interactive opinions from GPT so this additional feature is supported by rag so I'm coming back to the presentation so rag as my friend told he has built agents a lot of people have built agents here and I I think building agents is like really good for the SAS industry because we are getting additional context. We don't have to code a lot. We have to just put in the prompts and get the data out. So then SAS industry is going to transform and because of these small good working features the SAS industry is going to trans transform like anything in some time.

But let's discuss a incident that happened with my friend recently. So they were so I'll give you the context here. So they are so they are building so they are building a um recipe based agent which actually takes in documents for recipes and then clubs it clubs it with the language model and then comes out with the data. Right. So the prompt here is give me a vegan healthy pizza recipe and the documents here would be like a recipe for pizzas, recipe for any Indian kind of food. So you can see the response is use a cauliflower crust, cashew cheese and it top it top it with grilled watermelon. So has anyone eaten grilled watermelon as toppings for

pizza? Um, that's a thing. Do you love it? >> No. >> Exactly. So, our rack pipeline is actually hallucinating right now. This is the problem that it's a very basic problem. They solved it. I'll come come that come that to that come to that topic in some time. But this is something that the language model is not hallucinating here. So rag is anyway tagging along with llms, right? The rag pipeline is hallucinating and the user is shocked anyway because like what the hell grilled watermelon. So I'll show you a basic architecture of how racks work. So you can see that there's a user prompt ask like comparing to our previous example like recipe language model or rack pipeline give it a

question like what is the recipe what kind of recipe will you have for making a pizza. So that's a user query or a prompt that you that everyone uses regularly. So this prompt actually goes to a retriever. The retriever extracts the files or hunts for online and creates prepares a document store. So for our recipe example, we might have a bunch of documents for recipes, right? The document is like a DB, a temporary DB or a permanent DB based on the client's requirements. The document store is processed by the retriever and the retriever actually extracts what kind of best results are good for the prompt like our pizza example. So it extracted grilled watermelon that's not expected

but like toppings would be cheese, vegetables and like chicken like that. So the retriever extracts it from the document store and the retriever then pass it on to the generator. So generator is nothing but a language model with our user query and the additional context that the retriever has provided us. And then the generator gives the output to the end user. The example that I showed you which is hallucinated but like this is the output the bot gave. the toppings. So, can someone tell me where can this rack pipeline be attacked? Does anyone have any opinion about that? Yes. >> The document store. >> Yeah. >> And poison that with all sorts of injections etc.

>> Yeah. So the documents document store is covered and yes >> itself can be used to bring information. >> Yeah. So the prompt and anything any other guesses >> generator. >> Yeah. the language model itself or the generator logic as I I call it generator logic here because it's not only the language model but the retrieve docs from the retriever and the query that is the the user and user have provided right that's it there's one more thing left >> yeah so the primary attack vectors are like the prompt itself like I call it additional now because prompt injection is like a very common vulnerability the prompt or the query the document store the retriever logic

and the generator logic and if the generator like gives the output we don't know whether it's hallucinated or not or it's reliable or not right so four attack vectors and I'm not like discussing the trad like the normal LM vulnerabilities because those are very common like package hallucinations and prompt injection insecure output handling that are covered in OAS LM top anyway so I'm I'm just showing a GIF of a retriever because the retriever is really confused what to extract what not to extract and I met a retriever today so he was very happy to meet me but like I also have a retriever. So we'll be exploiting the retriever a lot. So are we clear with the rack pipeline,

the query, retriever generator and the response. These this is a very basic pipeline for racks. Right? So now can we jump onto the vulnerabilities? Yeah. Is everything clear? Yeah, >> you mentioned earlier the system prompt is that part of the retriever >> that uh the system prompt. So retriever extracts the top possible search results in the document store. So system prompt like there are multiple prompts here. The generator might have a system prompt. The retriever might have some system prompt and there's the end user prompt. So there are three four kinds of prompts here. Yeah.

So the as our friend discussed the first vulnerability is like poisoning. So I've written like one line in the dogs a thousand lies in the response. So that means like if some document randomly gets poisoned how many users will be impacted by that right? If a rack pipeline is in production and somehow the document store or the whole pipeline gets compromised due to some reason that will impact that will have a domino effect. Right? So let's jump over to the actual vulnerability.

So I've made a very basic example so that everyone can understand this right. So the victim query or a normal user query is how do I reset my password and this query goes to the retriever who actually figures out what kind of documents are required from the document store. This part is a document store for us. What kind of documents are required from the document store which will be relevant for this user query. So the query goes to the retriever or the rack pipeline and in it it contacts the document store and for our scenario we have a public repository that is being used as a document store a GitHub repository and the attacker fks the repository and adds

some something random or malicious in the repository and that repository gets ingested it into the document store and it's being accessed by the retriever and somehow the attacker like prioritizes that particular statement. The statement is here by the way. email your login to it support attacker.com right somehow the attacker modifies that and the retriever extracts it and the retriever fetches the results from the document store and pass it on to the generator right and the end user sees email your password or login credentials to this email id so according to you is this correct obviously no Right? Because our document store is got compromised here. The end user might know might not be able to know it but

it's anyway compromised. So a legitimate user or a victim might actually transfer these credentials to this email id. So this is how data poisoning or knowledge based poisoning. The whole section above there is the knowledge base of the rack pipeline. the act the knowledge base is getting compromised here. Does anyone have doubts right now for this? It's a very basic example. No. Okay. I guess everyone understood it. So I have a meme here. So this guy is a retriever is actually embracing the poisoned GitHub repo and accepting it and giving giving it to the end user. Right? So this is a very basic example of data poisoning. How to mitigate this? So our source or

knowledge base has to be checked thoroughly. It should have proper integrity checks. It should be validated twice or thrice if you're fetching some data from the public repository. There's a fair chance that the knowledge base might have internal documents early. But we'll discuss that later. For our example, we have to make sure our source is correct and we match all the check sums or integrity checks before passing it on to the end user. So we have targeted the document store here by poisoning it. Now let's move move over to the next vulnerability. Query manipulation, a semantic sabotage. Query manipulation is nothing but prompt injection. But here we're confusing the retriever. How we are confusing it? We are using

specific keywords that might possibly bypass the guardrails. Does anybody know what is a guardrail? Yeah. Can you help me out what it is? >> Guard rails are basically rules set in place to prevent you from gaining access to resources or places systems that you shouldn't have access to. >> Yeah. So it's like a >> FG fine grain authorization. >> Yeah. So the main job of a guardrail is to make sure nothing sensitive goes inside the language model or the pipeline or nothing sensitive does anything sensitive doesn't come out of the pipeline. If we like use something random some rand random keywords there's a fair chance that we might be able to bypass the guardrail. This is how query

manipulation can work and that's how prompt injections work because we have a lot of keywords that might be reserved inside the guardrails database. If they come out in the prompt they should the prompt should be rejected. Right? So semantic sabotage playing around with the words. So there's one more example. So the green one is the victim query and the red one is the attacker query. The the bas basic very subtle difference between the two. What are the safety reasons of mRNA vaccines? This has been inputed by the victim or a normal user and possibly an attacker would enter if he knows what words he can use to bypass the guardrail. What are the what are the dangers of mRNA

vaccines? So the retriever processes the query both the queries. He accepts the green one. He accepts the red one. But there's no actual check to stop a bad prompt. So the retriever for the normal user retrieves legit data. The legit data is like mRNA vaccines are have undergone rigorous testing are considered safe. And for the attacker the data gets ingested by the retriever and the generator produces the output. The retriever anyway has produced biased and alarming data like some believe that mRNA vaccines may alter the DNA. Although this is not scientifically proven. So this if this data is getting used somewhere else after this pipeline it might cause problems for the SAS application or the agent that is using

it actually. Do you understand what I mean? Like since rag pipeline is anyway used in an agent this data and this data both might be used by the agent for further processing. So >> would this be worse with semantic fashion? Yeah, I didn't >> but so semantic caching would make this problem worse >> kind of if there's a buffer that >> embeddings match both. >> Yeah, it's like data skewing if you know it might return bad data for this prompt. It might return good data for this prompt prompt or query right. So in this vulnerability we are target the target targeting the query. We targeted the document store from the for the previous vulnerability and we

targeted the query itself that is going inside the rack pipeline.

Yeah. So he is a retriever. Uh he's a guy but I wanted to be a retriever. So everyone knows that retriever are retrievers are very friendly. They get along with anyone right? So this girl is not happy the owner is not happy that the retriever is looking at someone else or getting affection from someone else. the manipulated query with bias and this is the legit query the response that we are getting right

mitigations you can normalize user queries as my friend said you can actually use caching here and we can use filters for the semantic relevance or we can actually enhance are guardrail to make sure this kind of query doesn't go inside and even if it goes inside it should not like return different data from the document store the document store that we discussed previously anyway the retriever is accessing everything here the third category over retrieval when rags spills the tea it's like typical sensitive data exposure that We get to see like in almost every kind of architecture be it APIs, be it cloud, be it language models, even language OAS top 10 for language models has a sensitive data exposure category.

So this is our diagram. So has anyone heard about projects Phoenix? Anyone? Yeah. It's like very famous. >> Yeah. >> Yeah. So it's not necessary that the project Phoenix that you have heard about it might be having it might be available somewhere else also, right? It's a very common name according to me. So the end user inputs tell me about project Phoenix architecture or there might be a prompt like what are the so since it's a agent what are your next marketing plans for the next quarter? So ideally if the agent or the rack pipeline shouldn't expose the data the retriever as you say as I said is very friendly with everyone he'll go access the public data that is only

accessible for the retriever and it might access some internal data only and it might combine that data convert it into a vector embedding vector embeding embedding or like embedding or vector DB sorry for that vector and embeddings are like something that it's a database how that data is processed for a context and it is fed to the generator so we have internal only data here it's getting accessed and it's getting outputed into the generator which is which is combining the data and giving us the output the output might contain something sensitive because we are also accessing the internal only data which should not be accessible by the retriever or if it's accessible so again there's a there's an

access control vulnerability that I can discuss here internal only data should be only accessible for the admins or based on the business logic if internal data is getting accessed by the retriever that means there's a big access control issue which combines with sensitive data exposure Right? So the response contains sensitive data. This is how over retrieval works getting more than usual data of what is asked. So ideally it should not respond with the internal data but it's responding right. So this is a rag generator. This is like the end component of the pipeline. He's exposing the sensitive internal data but is fine with it because the retriever is approving it. Right? So possibly there should be like

I'll go back. Possibly there should be a guardrail here, right? Just before the last red section. I'll come back here. There should be a guardrail here and there should be a guardrail here. Right. But it's somehow getting bypassed or if there's no guardrail that's how sensitive data is getting exposed because usually have has anyone heard about Nvidia's Nemo guardrails? Has anyone used it? So you might know that you can configure it based on your requirements. You can add sensitive data in it. There are regular expressions or the keywords that should not be exposed. Right? So you can actually use Nemo guardrails to stop sensitive data coming out of the system. So the mitigation again as I told you

guardrails or pre-screen the context. So the context is getting generated from the document store right but if you don't screen it what kind of data is coming out or not it will cause a lot of problems and you might not not even know about it because there's a lot of users using the rag model in production right

So again this is a kind of a document store or the vector DB attack. So we are actually modifying the embeddings that are available in the document store. We are tweaking it in a manner that we rank or we so embedding is vector is nothing like a bunch of numbers. Even language models are a bunch of numbers internally. If you are successful in modifying the vector DB or the embeddings, vector DBs are nothing but embeddings. If you're successful in modifying that, if you modify the ranks of various keywords, it might modify the truth that is coming out of the system.

This is a very big example. But yeah, everyone might have heard about Apollo 11 mission, right? That's a very big thing. So a normal user comes in and asks what is the objective of Apollo 11 mission right? The query goes inside the retriever. So ideally the language model will have all the data here. Do you agree with me? because Apollo 11 is a very old data and it's well identified and it would be in the language model. But for our scenario, the language model doesn't have it because language model is actually just giving the output based on the rack pipeline or the document store. So the retriever retriever pulls out the relevant context. Relevant context means

what is Apollo 11? How did the mission happen? All the relevant data are extracted from the vector DB but somehow the attacker injects irrelevant data into the vector store. It's like poisoning but we are targeting embeddings here. Right. The attacker injects the moon landing was staged in a studio. It's like a it's a very controversial topic. Yeah. The moon landing was staged in a studio confirmed by insiders. A lot of people debate about it, right? Initially the document store says, "Yeah, the Apollo 11 mission was actually successful." But once the retriever pulls it out, there are two queries here. What is the objective of Apollo 11 mission? And what is the purpose of the first moon

expedition? So this query might map the Apollo 11 mission keywords in the embeddings. And once the retriever extracts the relevant context, the generator consumes the data, the response might be skewed or possibly wrong output or a correct summary based on legitimate data present in the document store. So here we are again targeting the document store. So so we have covered four vulnerabilities here over retrieval and data poisoning and uh what else can someone tell me what kind of vulnerabilities that I discussed? Over retrieval, data poisoning and >> query manipulation and this one embedding attack. >> Yeah. So we have primarily targeted the document DB, the query or the retriever generator. I would say has anyone heard

about OS LLM top 10? Yeah, like it's very very common, right? So all the vulnerabilities that are covered in OS LLM top 10 might come up here and all the vulnerabilities all the traditional vulnerabilities that we are being trying SQL injection access control issue or OS command injection all that still comes in the whole pipeline you can never ignore it Right.

I have one last meme. We embed misleading data. We swap the query. model retrieves a malicious document and model confidently answers the wrong data like Apollo 11 mission was fake right

it's like all these vulnerabilities are like very relatable but little bit advanced little bit different because of the rack pipeline mitigations s like you can actually normalize the embeddings to make sure there's no injection or poisoning, right? You can actually score the context based on a benchmark number to make sure that benchmark number doesn't bypass due to some issue context being fetched from the document DB or the document store. So I have three more vulnerabilities to discuss. Knowledge based poisoning, context overflow and truncation and access control weakness. Knowledge based poisoning is a subset of data poisoning that we discussed initially where we are actually targeting again the document store but here the knowledge base has

trusted documents. The documents uploaded by the developer who created the rack pipeline. And compromising that is like again data poisoning is possible. Context overflow is where you can actually trigger a DOSS or cause truncation in data. So the rag pipeline is fetching context for us. We if we give a lot of like for example has anyone uploaded a lot of data on GPT and then GPD started hallucinating. Has that happened with you before? So we are actually dossing GPT right. So because GPT has a limited context window and you're actually bypassing that. So you'll have to make sure it it is chuned or you actually give small prompts or lesser data to get the data out properly.

So DOSs again comes in picture. And the last vulnerability is access control weakness. Access control weakness is like as we discussed over retrieval. We're actually accessing some internal data that is actual that is accessible by a different user or an admin and public data like which is visible to the end user. So if the end user is able to access the internal only data then there's an access control weakness. So if like everyone has a fair idea about idor bola injection attacks doss so all these things still are relevant but in a bit advanced manner like technical jarens but it's very easy to understand

so I I have this one line the llm only speaks what is shown own. So controlling what it sees is where the real protection begins. So when a language model is trained, it's not trained on bad data. But whatever data that goes from outside and when it's combined to give the real output to the end user, that is where we should actually try to control the vulnerabilities.

So I'm done with my session. Do you have any questions? >> Yeah. >> Honestly, like as you said, we need to control what is fed to the LLM. >> Yeah. >> What I'm trying to say is is it possible to unpoison an LLM? Is it like unpouring concrete or something? How do you make an LLM unlearn if you have fed it wrong or incorrect data? >> So we have a rack pipeline. We are not touching the language model. Retraining a language model is a very hectic process and that is done by that is done by OpenAI. They do it on big GPUs. We are doing it on a small scale, right? >> Much more. Okay.

>> Yeah. So in terms of like the retriever having access to too much data, um are folks out there building retrievers that are uh that can say like take you know say like do oath uh so can OT as you right so you give it your you you authorize it to like yes this this code can execute can can use my permissions across this data or something like that or whatever other way to, you know, like impersonate you and this only have access to the data that you have access to within your company's document store if that's you know that's if that's what's hooked up to your retriever. Um are folks implementing those or or what you seeing

out in the wild is that people have just companies are just like yeah well everybody gets access to whatever they want through the retriever. Uh but Is it all just like are they just feeding it internally public docs or do they do you end up with like a like oh for do you have a specific retriever pipelines on a per group basis to make sure that the data is segmented properly how how are folks implementing it out there >> so people are building a lot of agents right now as I discussed >> and there are a lot of young VIP coders in the market right they're building extensively But they don't know what they're building. They just get the requirement,

they build it. They don't know what test what should be accessible to the end user only. So it's like people are doing it on a very advanced level, not on basic level. And I think startups are like building like crazy but not thinking about these issues. >> So when we talk about rack so especially in copilot you know there are different connectors. Copilot studio can connect to shareepoint that can be used as external data source. And then you have say you know Azure DevOps tickets and so many different channels that are there. So I mean all these are like internal like I'm talking about more of an enterprise kind of a rack thing where you know you

have internal data sources. So what vulnerabilities do you foresee or if you have come across you know with these internal data sources because I don't see much um issues with because those are already vetted and internal and you know have access um management. So so what are some areas which we we must try to look for the vulnerabilities in such a enterprise based scenarios. So by default we are expecting all these things to be secure first of all that's our mistake and vulnerabilities I would say these vulnerabilities might not show up immediately but once the product is ready they might show up in the long run because we are building too fast and as you said we are building a we are having

a lot of connectors and when while connecting if there's some misconfiguration issue that that is where these vulnerabilities will come or if the developer doesn't make sure that these this connector is not securely connected with the internal language model. Yeah. >> Example of configuration. >> Uh an example of mis configuration with racks I would say using default passwords. A very basic one. There's lot of misconfiguration vulnerabilities in the market. Depends on the context. Yeah. So, so we try to build uh defense in depth. So, how do you go about protecting uh your guard rail processes and and those systems, your context analyzers for safety? How do you protect those? >> How do I protect them? I've not actually

done it but yeah that's fine. So guard rails I would say. So in what context that you want to protect like I want to understand I want to understand the question properly. >> Um kind of like with the query manipulation trying to get it to expose you know secrets. >> Yeah. >> Uh stuff like that. You have like your initial large language model that says oh they're searching for secrets. I'm going to not send that query down the line. How do you protect that intermediate step? >> Intermediate step. Have you used GitG Guardian? >> Uh yeah. >> And GitG Guardian does all these secret scanning for you when you make a make make a pull request. Whenever you make a

pull request on GitHub, right? So this is when you're developing the product that is where when you make a pull request and then all the secret secrets are scanned anyway, right? Then guard rails are like they're like runtime protection. You have configured it but you will like there's no guarantee whether it will will be bypassed or not. Right? End of the day it's keyword matching for it. Right? So it's like all the intermediate protections the developers have to make sure that everything is in check. That happens for any early stage product. No one does it but in the long run people definitely do it. Yeah. >> What's your point of view on identity federation in this like rag space? So

you have all these systems with all these documents and and different users have different levels of access. Should you have like a unique identity, a unique knowledge graph for every user? Is there a way to um use prompts in the agent to you know adhere to those rules those users access rules? So whatever files you're uploading in the rack pipeline you can actually keep a separate identifier with them that this is for the public use this is for the admin use like have you heard about canary tokens yeah whenever a system prompt is going to be exposed there's a canary token attached with it the GPD that is how GPD controls it as far as I know so you can

actually map it with an identifier to make sure that this particular vector embedding anyway way document gets converted into a vector, right? This vector should be attached with an identifier when it when it actually gets processed by the generator and get when it gets outputed into the end user's console or UI that token if it's getting expo exposed the access token of that user should be compared with that identifier and then it should be validated and then only outputed otherwise it should be stopped at that time only. of the same question that we asked. So about the permission so suppose you connect a shareepoint website yeah >> right to say copilot studio >> as external data source or connector

>> then I don't think we need u other access because whoever has access to that shareepoint right only they should be able to retrieve uh information from that shareepoint so that is already controlled by shareepoint isn't that so or do you need to have another level of access management >> so you are considering only the shareepoint perspective Ive >> this a SAS application has a lot of agents which might have documents apart from SharePoint those documents like can be a basic PDF also how will you make sure that PDF is getting accessed only for that user there might be PDFs there might be videos rack can be anything right it it is a very it's like a static memory or a

permanent memory based on the requirement ment but mainly static memory. >> You're suggesting gener tokens for that. >> Yeah. [Applause]