Using AI to Secure AI

uh so hello everyone good afternoon uh hope you all enjoyed your lunch and had a good first session after lunch uh so if you not sleepy at all then we'll talk about the next topic uh before we start how many of you have heard of geni oh very few of you man that's sad uh but yeah it's it's Talk of the Tower and everybody's talking about things and uh but how many of you think that it has an actual impact just few okay the number is getting lower and lower uh uh so I I just wanted to briefly talk about few things uh one is we talk about uh how it's impacting the security landscape in both ways which is uh one

is offense one is defense and also little bit touch touch based upon uh the different use cases which exist in the uh in the security space uh where you can use uh leverage Andi in U different ways so yeah uh I'm J uh I'm one of the founding members of Trace AI one of my teammates was here uh he recently joined us in the morning so uh I I I lead the product management at tble uh been doing that for around five years now I'm also AWS machine learning hero uh then uh I I graduated from Stanford uh still doing some AI research at the Stanford uh been involved in some of this uh llm NLP

related things since uh last six seven years uh if you want if you can find me on Twitter now it's called X on J it uh one so uh another thing this is this is Star Wars theme presentation why uh I have watched one yeah one episode of Star Wars in my entire life but this came into picture when uh we were supposed to do like uh one of so Disney's one of our customers and we were supposed to do a presentation for them and that's where they had a good idea that you should do a Star Wars team presentation me and my CTO both unaware of all the things around Star Wars we had somebody create the presentation for

us now it has lot of Star Wars references which I'm aware of to some extent but not a lot uh but hopefully like how many of you are Star Wars fans okay I'm offending lot of people at this point uh but cool uh I I hope you get the references uh unlike I do but uh this is what the whole thing is about cool the yeah so uh you all PMS we have seen uh this evolv in front of our eyes most of you you were around for last 20 years I assume so uh most of this happened in the recent years uh IGN know 1967 to 1997 nothing much happened in that period but it was significant in enough

Improvement for the times they were uh since last few years at least in 2022 2023 we have seen uh the llms getting into a picture and that's where things got interesting because now you can now we have the generalized understanding of languages and pictures and different of data sets and we can do bunch of different things with it again we are not doing much rather than writing fancy emails but we will in some time so this is again goes through some of these details where uh Rec ual networks were uh the first things which came into picture which actually originated into uh reall into the Transformers which we use for all the things llms at this point so everything

you see around the uh uh generative AI actually us just Transformers at the uh in the core uh yeah then what all it did in last few years uh people are claiming that there's we can do safer brain surgeries using AI automatic automatic autonomous flying vehicles uh how many of you have uh been to uh San Francisco in recent times and tried the wayo oh yeah so that that's a pretty cool experience because uh it's it's completely autonomous and it's it just feels like future to most extent so uh we already like seeing some of the uh some of the things which are happening around artificial intelligence or using like generat and non llms and everything

else but uh it's a it's a double A SW like there there are two different every single thing has two different sides and this also has two different sides uh one is the lighter side of it which is uh it can be used to develop powerful security tools for detection analysis uh automated incident response but at the at the another end it's also being used for things like defix uh launching fishing campaigns uh automating attacks uh so we'll talk about both the sides and we'll see like how can we leverage it for the better uh version of it than on the darkx side so yeah uh the Star Wars theme the dangers of LMS so uh

for okay before we get into those details uh one of the uh important things from be the before llm era was if you want to perform an attack on any application or any API or any system for that matter you need to understand the system so now you will spend time in understanding the system understanding all the nuances before you actually go and like perform and sophisticated attack nowadays you don't have to if you find the open API spec in the morning session uh another guy was talking about it so finding the open API specs or API documentation is not that hard but and if you do you can just give give it to llm and it will find all the loopholes

for you it's it's it's literally that easy so uh it's uh and also access to the information so now if I find the loopholes what can I do with those loopholes that information is easily access accessible with llms and exploiting that uh using that information to exploit the applications is also becoming easier uh that's where the bad part comes into picture and uh uh automating some of these attacks is easier creating defix is easier uh yeah so H and at the end of the day everything every application we are dealing with when we start integrating uh gen into the application we are adding one more external API call so in most of the cases as a traditional

application looks like uh you have the apis right and then you're making one more API call to a third party endpoint which is not controlled by you if that introduces some compliance issues or that has some uh security vulnerabilities you are also you are also uh susceptible to those those challenges that's where uh I I told the story to uh in one of my talks yesterday as well so we were dealing with um so I was talking I I was in Singapore in uh September I was talking to one of the banks and uh one guy told an interesting story that they release and chatbot for the internal use cases uh for the sales people where they can go to the

Salesforce or twas to the leads and then find uh what's interesting what's not what's qualified what's not and then do the reach out so the chatbot was supposed to do all of those things it was based on chat gbt uh literally the open uh API right nothing nothing much but uh somebody like eventually somebody in the sales team realized that this can do way more than just reaging to the leads uh they started sending credit card information they started sending account information started sending bunch of other things to the chatbot assuming that it's an internal implementation it was not it was using open at then and uh By the time somebody saw those patterns in the traffic and

caught it and starts uh reporting it it was already too late uh so those kind of things can happen if you don't uh put the Right Guard Wills in place if you don't protect the implementation you are dealing with and uh that's where as you're dealing with one more third party integration now it becomes important uh to be aware of the challenges Al uh llm as a threat actor uh it uh because it's dealing with lot of sensitive information is dealing with lot of uh input uh in in a way and then also the output is generating is being fed to the different system it can lead to damage at a large scale won't go into details so there

maybe we'll come back to that yeah uh then orm topt uh this these are all the prominent attacks which can happen on LM as an attack surface uh I'm I'm one of the contributors to llm top 10 and also reviewers for that uh also for the API top 10 so that's where this list is coming from uh we have been working on this uh since quite some time now uh there's a second version of this which will be released next month uh but yeah this is the first one which came out last year uh you can see bunch of different things here uh which can which which which are pretty easier to replicate to be honest uh prompt

injection being one where uh all you have to do is craft a very uh uh nice prompt which can expose things like passwords or maybe give you things like API keys and uh you'll be surprised that because uh lot of the llms in the in the early stages like let's say for the opening atbt or uh some uh gups co-pilot were trained on actual code uh earlier in the days the repositories which had real API Keys which are still still available out there uh you can get get those API Keys some of those work uh again it's it's getting better and better as we go through the process but uh it's still susceptible to some of the

things which are uh which are easy to perform and also easy to avoid to be honest so this is where uh this Bank assistant is giving access to an API key and now somebody can maybe use this API key to perform some operations transactions which is problematic uh another example of the same bot and this is is Real Deal where uh it's giving you a sensitive information uh like email ID date of birth passwords balance in the account uh Adar number for the Indian origin so all of those things right and it's just it just uh sad so that uh and while talking about this one of the uh so we work with lot of uh US Banks and

financial organizations and that's where um they are little reluctant to adopt uh llms for some of the customer facing use cases because of same thing we're dealing with the sensitive information all the time but uh in other regions where uh people are actually adopting it you start seeing the issues and that's where uh you need to do a better job at avoiding these issues then uh yeah so in prompt injection what happens is when you provide the instruction data you try to modify the instruction in a way that you can get sensitive information out of the llms or any information which you can use against uh the end user and uh if if uh you are

able to craft a really good prompt you can get uh an uh like the malicious data sensitive data out of the uh application which can be used for malicious uh activities then uh like we we can could talk about all of this but it will take time I will give a very simple example where this is very impactful right uh how many of you have received fishing emails and past 10 years yeah they're always beautifully written there will be some there will be some uh uh spelling mistakes here and there or maybe something is wrong with the fishing emails uh nowadays they have gotten really better like literally as good as the original ones uh purely

because because of llms and that's where I think llms are shining a lot that's a real use case there so uh people are using llms for social engineering and uh writing better fishing emails uh and Performing those fishing attempts at scale that's where it's being used uh if this these are some of the older ones like the the one on the top and one on the uh I don't understand left and right so probably left uh are the older ones where you can find some spelling mistakes or something's wrong with it which which can tell you that this is a fishing fishing attempt uh this is something which uh which is again a recent attempt on on our own

organization somebody used my colleagues uh name and all to modify uh and to send the email to our financial team so they can modify the payroll information and uh and and this was bad right this is if you see this one this is very badly written email the comma has a space and then everything else as a space uh the indentations are not right I can identify some fishing attempt but recently the same thing happened to me like somebody tried tochange my payroll information and this is the message I received from our CFO saying uh I received a question in the morning to change the pay and it looks suspiciously good so uh again thanks to

LMS uh yeah so this is this is the bad side of it where people are trying to use it in different different ways all you can do to be uh to protect yourself from that side is uh like at least uh have something in place to test against the oasm Topton and protect yourself Fromm topon that gives you an edge to some extent again it won't protect you from everything like Evol it's it's it's an evolving attack vectors people will try to craft beautiful attacks as we go through the process but uh they did definitely uh puts you forward in the process then yeah uh how can we use it for a G good uh so if you take a look at

this this will just look like a random screen but I'll explain what it is uh this is an uh credit abuse attack where somebody is trying to so one of our one of our customers it's Cloud company uh cloud service provider uh so that's where somebody is trying to just create different accounts from different IPS and different locations to get more get more and more credits and they using those credits for uh in in most of the cases for Mining and that has been a huge problem for them like it it used to cost them millions of dollar every single month in those free credit abuse attacks uh and that's where this the screen comes into picture so if you look

at the uh emails there it's it's that one email plus 37 others which are listed down here and it's coming from uh 31 IP addresses mostly from Singapore if you go down like if I scroll down there are a bunch of others from uh India and few different countries as well and uh yeah so they got $493 as a balance uh created 38 pent accounts over the period and from four user devices so uh detecting this kind of attacks takes a lot of time and energy so uh we have been doing this professionally since five years at this point and developing models to detect this is pretty hard uh we have been working on statistical models for a

while uh but it's it's it's different every single time it it's different for this kind of scenarios where somebody's trying to create so many accounts and get uh access to inform get access to credits and in some other cases somebody is trying to do uh same thing for uh in one of the telecom companies we we working with uh people are trying to get different numbers so they can send random messages as in the fishing attempts they can randomize those messages using the numbers and SIM cards they are procuring and that's happening at scale like there are thousands of SIM cards being used or thousands of SIM cards being generated at uh in every single day uh in some cases they are

very uh intelligent so they do low and slow attacks which is this scenario so if you notice uh there's a somebody goes disables the notifications and that's when they transfer the money and this is uh kind of on once or twice in a DAT type scenario where uh each day on this particular user account um some threat actor will come just uh disable the notifications send the random amount to one specific account and then again enable the notifications this this is this is continuously going on for last 15 20 days and uh tracking this kind of scenario and then uh making sure that you alert you uh the end user in this case a bank which is running this uh uh

whole online activity uh alert that bank or at the end alert the user that something is happening on their end is important but doing this using statistical models or even any kind of modeling for that matter pretty hard uh and that's where LS come come into picture because uh what all of these problems need is in context if you provide the right context and if you provide the right data uh llm model can learn well from it it can identify this patterns it can even uh detect some of the scenarios early in the cycle than later in the cycle when it when the attack actually happened that's why I talk about these two things which is uh

security with geni which is what we want to talk about right now and we talked about security for Gen earlier so in with Gen there are different set of use cases uh are this uh one of the one of the first use cases is five minutes threat detection and response uh so uh the scenario I was talking about earlier the uh that that was more of a fraud or uh the fraud kind of a scenario but there are traditional attacks which are like Bolas and bufas of the world just broken object level authorization function level authorization uh even to simulate those things or even to find out those attacks uh we can use llms because uh it gives

you uh it can identify those patterns pretty well and it can uh detect those attacks early in the life cycle and also helps with the mitigation aspect of it uh because uh because it has the context of how application works it can provide a first line of mitigation and then you can always build on top of it uh fraud detection side uh the attacks we talked about the credit abuse or sign of flow uh those are again those are being solved by clustering traditional but now uh you can use the power you have with llms to actually not just rely on clustering completely but can detect uh feeding those patterns to the to an llm can detect can help you detect those

attacks early in the cycle as well uh vulnerability identification this is an interesting one because uh how many of you have dealt with Dash tools nobody okay okay one person I I I I can't really complete the story now because there's one person but uh but Dash tools are known for uh producing Dash is like Dynamic application security testing so uh what you do is like you provide API documentation or uh application URL and then somebody uh the tool goes and calls to the application or just sends a random malicious request on application and sees if it returns a valid response back or not so uh it's hard hard problem and then it produces a lot of false

positives because you are sending very random requests fuzzing the data it it might not be valid data most of the time so even if it returns 200 uh as in response it might not be a vulnerability so uh that's where the vulnerability identification part comes into picture where we can uh and when we have been experimenting a lot with that uh if we have a valid API request from the past we can craft a valid malicious request and send it uh to an application which looks like a legit request most of the times and uh it does produces pretty high efficacy so we have been able to achieve like more than 95% accuracy in most of the cases

by shifting to uh generating these malicious requests using llms than traditionally using just fuzzing or other techniques right so uh that's one of the interesting things then behavioral Biometrics traditionally banks used to rely heavily on that where you click on particular part your mouse movements your where going after after this particular click all of those things tracking those things again pattern identification uh good use case for llms policy enforcements uh how many of you have used any kind of WPS firewalls any firewalls okay few people uh so uh as we uh like as we deal with waps waps are not intelligent by Nature waps are enforcers by Nature uh they don't they don't have an exact

application or API context to decide what to block what not to block that's where policies come into picture uh but based on the application context we can develop the better policies to block the right threat at the right Point uh then secure code reviews um intellig already released something called intell code or intell sense I don't remember the exact name but uh that does does a similar thing where it runs in your ID while you're writing the code and it tells you if you're writing secure code or not I classify this uh whole thing as like four stages design develop distri uh deploy and distribute and uh uh when it comes to all the four stages the

security touches all the four things right while you're designing the apis while you're designing your system the architecture should be secure by Design uh while you're developing should write the secure code like as in you should use the uh best authentication you should you shouldn't rely on the basic o in these days uh and all the other things which come into that then while deploying making sure that the uh you you protect uh yourself from things like all the all the vulnerabilities that can exist in containers or your ic's and then distribution is all the apis application specific vulnerabilities so uh uh this this all of these things can definitely touch one of these areas

where uh uh putting like some kind of llm salt on every single stage can can help you get towards a better security posture and that's the thing we talked about so these are the different use cases at different layer uh for Discovery you can discover the shadow uh AIS uh again this is for specific gen security but uh even in using disc using AI like using llms or gen you can discover the uh so one of the important aspects when it comes to API um API security is visibility uh lot of these organ like lot of the compliance standards and uh uh regulations want you to have a detailed inventory of your apis and it's hard to

create an inventory uh the box and that's where uh where like we we we do that as one of the services but identifying this third party end points uh identifying the partner partner API identifying all of your apis then uh providing that a risk core all of those things become important uh and specific to gen like uh discovering the Gen end points uh Shadow a end points uh like having the right restrictions in place like you are not feeding sensitive data to the LMS you are not uh uh Pro like just getting the output and output and providing it to some other uh third uh third party system which you're going to consume from uh that can result into issues then

uh compliance violations then doing the security testing for oasm top 10 we talked about uh at one time like detecting the attacks mitigating those attacks all of those uh yeah important things I uh thank you I had to rush to it because I I had 25 minutes to be honest but uh uh any questions concerns uh anything you have okay I'm not going to ask the yesterday's question just you think that's lm5 is can be a good solution for I mean that reduce the risk when uh which chain AI yeah um it's it it's definitely a good solution uh the only carat is uh same as traditional I I will say having one more firewall

doesn't help and that's why I I feel most of the traditional firewalls will also add up to this new things now so and even the firewalls we have right now are still not say good enough uh so that's why I'm waiting for more Evolution on that side where we actually get good tools out of the process we will over the period there's no uh another problem with the whole uh thing is there are lot of people talking about it there's a lot of chatter but there are no production deployments and that's what we have seen across the uh Enterprises that no enterpris is actually using llms in production yet uh they are exploring use cases but there's

no uh major adoption which can trigger some of these things like LM firewalls uh it will take some time but we'll we'll get there I I saw that there some product uh I mean that's for LM firewor they can um Pro protect from BR injections yeah and uh for for ra and for respond okay yeah so mean that's I don't know that if the this solution will be in the future for yeah it it's definitely interesting it will be helpful the only thing is uh adoption all of these things are really tied to adoption so once the adoption happens I think that will pick up as well no questions no questions thank you and uh Star Wars fan uh was the star

were the Star Wars references right no didn't like them cool uh thank you everyone uh if you have anything you can you can grab me around as uh another huge round of applause for JS for everyone