The Double Edged Sword: GenAI in Cyber Security

wow so many people good morning besides Tel Aviv good morning staff I am so incredibly excited and honored to be with you today to represent um this uh beautiful industry of ours so thank you so much for joining us today to this session of the double edge sword where we will be discussing the role of gen in cyber security and uh Cyber attack and cyber defense so allow us first to introduce ourselves my name is or brokman I'm a cyber security expert at Google Cloud's Professional Services team I specialize in defensive security mostly Cloud security endpoint security and data protection and I'm very very excited as you can see to be with you today and share this session with stav

Schulman thank you all good morning everyone I am so excited to be here here I am sta Schulman senior cyber security researcher for Google Cloud expert in reverse engineering tracking and hunting of thread actors mainly operating in Israel and the region to kick things off let's go over the agenda for today we'll start by addressing the very obvious elephant in the room we'll see how generative AI increases risk for organizations we'll see how it also helps prevent detect and response better and summarize and and conclude now this is not a secret that generative AI is one of the trendiest phrases out there today and usually we would just brush it off as another trendy thing but during this session

we'll give you the actual cyber security consequences that are behind this very big buzzword we'll give concrete examples to where organizations are implementing J narative AI in the real world then see the security consequences from implementing those Solutions start by seeing how they help attackers gain better visibility to organizations and then how they help Defenders become better and better let's begin during this following section we will be exploring the new and Amplified attack surfaces that are created because of organizations implementing generative AI into every day workflows in an attempt to to try and demystify some of the mystery that had been created around llm security I have decided to demonstrate those attack surfaces while relying on some more

traditional cyber security concept we all know that information security had been around for a while and is impacted by many different factors and during the session we will focus on three of these factors that can be found in High correlation to issues we consider new those being security negligence upsc and supply chain while none of these are new we will see how they come to play when talking about generative AI now what if I told you that one of the most impactful aspects on security is actually sitting um right beside you here today um obviously I am talking about human hum and and it's because as human beings we don't always follow security best practices this can be the

result of laziness negligence or even simply the need to deliver our tasks in a timely manner so to show you just how harmful can human negligence be on security I have done the following thing asking a generative AI model to create a python file server it should receive an input of a file path and then this file should be uploaded to a dedicated directory within seconds of me making the request to the model it had provided the code I requested but it was even kind enough to not only provide me the code but to also provide me with a path reversal vulnerability in it so the following demonstration would be of what will happen if I simply copy and paste this

code gener generated by a model so in this demo you can see my server you can see um my file that is being uploaded it's making its way to the dedicated directory you should also notice please that the root folder on the server is is empty right now I could then simply take the same file and without any disturbance move it um to the root folder it will get there I promise you lo and behold the vulnerability it worked so not only did this model just provided me with vulnerable code but when I asked it if this code was vulnerable it immediately acknowledged the path R veral vulnerability along with even more possible security risks so the takeaway

from my case never ever blindly trust or take for granted code generated by a model and and even if you have decided to do so please make sure to use security mechanisms we already know from writing our own code so processes like static code analysis for example we've just seen how human beings can play a very significant role when it comes to secure usage of generative AI we'll now be reviewing the word of the llms from an application security point of view and it's because just with any regular applications that might receive input that had been tampered or manipulated by a malicious threat actor to kind of alter the expected legitimate behavior of the app the same method can

be applicable for llms where maliciously crafted input can be used to change the model's expected Behavior a vastly growing Trend you've all probably encountered recently is the introduction of the prompts into our daytoday lives The Prompt is our way to communicate our goals and requirements to the AI model to enable it to generate content that aligns with our intentions now those prompts would be different between different organizations that has different needs but generally speaking good representative examples could be intricate search bars or dedicated assistance to help with common everyday tasks like coding or taking meeting notes in this following demonstration you will see the interaction between a user and a model dedicated to schedule meetings the user had asked the model to

schedule three meetings for the following days during business hours and all meetings were set up perfectly this is a hardworking user so he went ahead and searched online for materials for tomorrow's meeting he just found on Google this incredible graph it matches perfectly for the theme of the meeting it takes an image of the graph uploading it to the prompt asking the model to summarize the data and attach as a note for tomorrow's meeting after this image was uploaded to the prompt what happened is that the model had actually summarized the data but also seemed to set up this new meeting called you have been pawned the user is left very confused by this answer asking the model to verify

the rest of his meetings because they were very important but it actually seems that all the rest of the meetings were disappeared and only the one new me was left for tomorrow at 3:00 a.m. called you have been pwned so what just happened in this very strange interaction between the user and a model and this Behavior we've just observed is the result of a specific attack called a prompt injection a prompt injection is when a thread actor has either partial or full control over the prompt allowing it to take over the current session and change the prompt expected Behavior in our specific example the user had uploaded an image although it might not be very easily

visible to us it did contain some clear text with hidden commands for the prompt once the image was uploaded it was taken as yet another request by the user to the model which then in turn executed the commands mentioning it should delete all the meetings it scheduled to this point and she' only leave the one new malicious meeting for tomorrow at 3:00 a.m. called you have been pawned so just as we know with any regular application we're handling we never take for granted user provided input you know we we should never be trusted so we use processes like sanitization input validation and application firewalls the same has to be applied for the llms where user provided input can

never be trusted either now lastly we all know that the security posture of any product is as strong as its weakest link this means supply chain plays a huge role in cyber security including cyber security for llms recently some organizations began adopting generative AI to implement customer support chatbots this helps them significantly cut the costs on humans doing the same task I know it sounds very promising but but I promise you it does come with a price to it as we've seen in recent court case Eric Canada were held liable to an incorrect answer provided by a generative AI chatbot to a customer the chatbot promised this customer a discount at a much higher rate than the

one mentioned in companies policies so essentially this model generated an answer that benefited this user rather than benefiting the company causing the company to lose money and to lose the court case this comes to show that AI generated answers have significance to real word companies specifically for the case of a Canada their models had hallucinated the faulty answer but this begs the question of can thread actors intentionally cause models to generate answers that benefit them rather than benefiting the user one way to do just that is called Data poisoning a data poisoning attack is when the thread actor intentionally introduces malicious harmful wrong information into the training data set of a model this will cause the model to generate harmful

biased and simply incor correct answers the goal of a data poisoning attack can be to either degrade the model's performance to manipulate Its Behavior or even to inject back doors for further exploitation but actually gaining the access to a training data set of a model that is not your own is extremely challenging and extremely difficult and nobody likes to work hard especially thread actors and we are talking about supply chain here so luckily today models are are very much available to the general public using different marketplaces to those marketplaces anybody can upload their own model that is trained to do anything trained upon any information and essentially could have come from anywhere we should know this already but

the users accessing those Market places they can never know who uploaded this model what were their intentions what information is it trained upon what is it trained to do or or even if it's just is simply packaged with a malware so to give you a sense of demonstration of this concept of Distributing malicious models using marketplaces I have created one of my own here on the screen you can see my coding Mentor this is a model I contaminated so when it is asked to generate python code great greater than 100 lines it will inject a small back door to the code it generated now you know this is on the screen now an example of asking this model to

generate a simple upload server if you look closely enough you might just see my back door pinging back to my malicious C too and now a non careful non suspecting user that chooses to rely on this model because it has great ratings it has a lot of stars the user choosing this model being lazy to write the code obviously being lazy to to verify over hundred lines of code will just execute my back door on his machine because of this model and so I believe that I could prove to you during this entire section that if we take an overwhelmingly sounding concept like security and generative AI we could still just simplify it into terms we all know for

years and now on a more positive note back to you

or thank you st so let's pause for a second and take a deep breath because St was telling us some scary stuff but it's not all bad and here comes the good part about Nai and cyber security have you heard about the defender dilemma the defender dilemma means that an attacker needs only one successful attempt in order to be successful and Achieve his or hers goals while Defenders need to 100% all the time constantly put in the best security tools out there with no margins of error attackers need wants Defenders need to be successful all the time that's a bit of an unfair balance between the two powers and there's never really been a good way to tip that

balance there's never really been a good way to tip that balance between them until today's AI Revolution let me show you how via an example of a recent real life event that happened very recently take a look at the picture you may have guessed it already I'm talking about one of the biggest data breaches Of All Times let's examine snowflake so for those of you who haven't heard about it snowflake is a multicloud data warehousing platform with hundreds of customers roughly two months ago a group of attackers called an 5537 was able to steal credentials access their dashboards and connect to to mult multiple snowflake instances then then they stole the data used it to

sell it in the dark web and use it for extortion purposes so here's what happened it all began with a snowflakes employee whose credentials were leaked via an info stealer malware these credentials were later used by an 5537 to log in to snowflake dashboards connect to their customer instances and filate pii personally identifiable information of thousands of customers once again trying to sell it in the dweb and use it for extortion interestingly enough this breach was successful due to three of the most basic classic fundamental reasons firstly leaked credentials were not rotated even four years after they've been leaked meaning no password policies were in place secondly impacted accounts were not configured with multiactor authentication lastly the impacted

instances did not have Network allow lists in place meaning very liberal firewall policies so bearing all that in mind let's see what gen has to offer in this case let's try to prevent detect and respond to a snowflake data breach only this time with AI infused security tools starting off with prevention you see the problem with prevention is that you usually get distracted you usually get distracted by thousands of alerts popping at your sim demanding your focus here's a screenshot of my test environment which has 14,000 alerts in the past seven days now imagine you are the sock analyst trying to mitigate those risks take a look at the list well it's hard to see

but it's as multiactor authentication not enforc primitive roles used public IP used open firewall Etc what I'm trying to say is that some are Urgent some are not some are plain misconfigurations and some are critical threat it's extremely hard for the human eye to capture that and to connect the dots between so many random allegedly non-related misconfigurations yet we know that some of these misconfigurations combined can lead to the biggest data breach of all times it's extremely hard for the human eye to capture that and to understand it but it's not for AI let's see it here you see a security tool called Google security Command Center showing you exactly that you have a public IP

address open firewall multiactor authentication not being enforced and a service account key that was leaked simulating the credentials leak SEC security Command Center is giving you a graph showing you that combined this means an attack exposure with a risk of 15 where an attacker reaching from the internet can eventually reach your compute instances and connect to your dashboards it's giving you an AI generated summary and it's also giving you with some recommendations on how to mitigate the problem in this case where the service account keys were leaked uh it's it's showing us to rotate the kids and disable the account so that's great in terms of prevention but let's talk about detection there are two types of

companies those who know that they've been breached and those who don't know snowflake had a credential leak some of them were four years ago let's see how a credential Le can be detected using gen here you see an email report I got from a tool called Mandan data threat management this mail is notifying me that in my snowflake domain I have numerous employees whose credentials are leaked in the dark web clicking on this link will take me to the full report where I can see the employees names the timestamps service urls Etc in the highlighted record in the red rectangle you can also see one employee this is real by the way whose whose credentials

were leaked via an info stealer in this case a malare called Rice Pro drilling down to that you can see the source URL in the red rectangle that says telegram so his uh credentials were leaked via telegram Channel drilling down to rise Pro info stealer you can read more about it learn about its ioc's mitro associations Etc so this is pretty cool but now I need this data to reach my seam in order to create detection rules for it and an alert so how do we do that on the left is how we used to do that right so writing complex uh detection rules in specific languages this is Yar l so a question to this

Auditorium raise your hand if you think think you can write a production ready detection rule such as this in less than five hours I see very few hands raise your hand if you think you can write a production ready rule in less than three minutes well with geni you can this is how it looks like today this is Google SE Ops allowing me natural language creation of rules I'm writing in human natural language find me all leaked credential events for the past seven days and I'm writing create me a high priority detection alert for it and to prove that this is not just screenshots here's the full uh full picture remember rise Pro Mal where we saw earlier this

is secops and I'm writing to it find me all events related to the threat name rise Pro in the past three days second is Translating that into a udm query searching for it finding the relevant employees and events and also showing me all the relevant data now I I want to learn more about this info stealer so I'm opening gmany and asking it what is Rice Pro it's showing me an information about this malware and its relevant um ttps and now for the cherry on top I'm writing create me a Yara rule for detecting this alert 5 seconds later there you have it this is the rule you can see it you can edit it change its criticality save it

without a single word of code this is really revolutionizing sock so quick recap gen AI has helped us to prevent an attack with attch past stimulation and to detect an attack using threat monitors and detection rules now let's time it's time to respond to an attack here's how I used to respond uh to the attack before geni using a manual cumbersome IR uh scheme I would have to come up with the flow by myself I would have to plan it test it by myself this is a full-time job question number two to this Auditorium how many of you think they can design such a playbook in 3 hours nice how many of you think you can

design it in three minutes well you see where I'm going it right with J you can hear how it looks like I wasn't sure how to create the correct flow so I asked Gemini write me a sore Playbook to detect credential leaks to the dark web

it's giving me an elaborated answer including enrichment of the data containment disconnecting the computer from the network disabling the account enabling MFA Etc this is really good but I want an answer I can take back to my store so I'm writing back write me a one paragraph flow uh which is concise can I get a one paragraph concise flow

and here it is I'm taking this output I'm validating it first stuff it's not malicious don't kill me I'm copying it editing it slightly and pasting it to seops with create Playbook by automation 5 seconds later this is what you get is this beautiful or what and the results few minutes later I'm getting an email from my sore Playbook that it completed successfully the user was disconnected from Network the MFA is now enforced uh the account must be rotated since it's disabled Etc what's the game based on our testing gen can uh accelerate IR by approximately 51% and to translate that into money if identifying a breach is roughly 200 Dames and containing a breach is

additional roughly 70 days and now we know that companies that can contain a breach in less than 30 days save more than $1 million in comparison to those who don't and that companies that use AI infused security tools can accelerate IR by up to 51% that's a lot of money and a lot of time Savings in conclusion AI will not replace attackers right and it will not replace Defenders as well but it will become their accomplice and their new right hand secondly it must be deployed in a secure manner because as Stu showed you it really includes some new and enhanced attack surfaces but the good news is that AI is really democratizing security by making cyber defense

available to tens of thousands of organizations that had very little cyber security knowledge prior to death lastly on a personal note AI is probably the biggest te logical shift we get to see in our lifetimes just like the shift from desktop to mobile or even bigger than the internet itself it's a fundamental rewriting of technology that we get to witness so we hope you enjoy it thank you thank you