ChatAPT: a cybersecurity red-teaming framework

all right so our next talk in track three chat AP a cyber security red team framework which demonstrates the emerging threat made possible by leveraging GPT to elevate High Fidelity social engineering effort to an unlimited scale please welcome Jonathan Todd [Applause] testing uh yeah so it's

working and if you have questions after his talk everybody has a mic at their desk so un hit the hit the button and then ask a question can you all hear me okay there we go all right so a little bit about me I'm I am a super nerd uh last year I came here and walked into a talk about the Pyramid of pain had no idea it was by the guy who made it um so I went and had a custom um neon sign made for the Pyramid of pain so I've got you all beat uh love hard problems so bring on the questions I see one of my mentors around here somewhere I'm sure he's got

he's ready to go uh I like to cook but not for myself um been coding for about 10 years just for fun I probably should be getting paid more uh but I joined the army instead uh and just a little boiler plate there this is all my personal research I do not speak on behalf of the dod or the Army this is me perfect all right so once the chat GPT hype came along I immediately thought this is going to be really good for fishing um I immediately started coding with the API to see sort of how we could do once we start scraping and collecting data together to put together a tool for red

teaming um the really big value ad with the chat gbt or I guess the scary thing is that it takes us to a limited scale but perhaps more importantly it takes away the barrier to entry for all of the threat actors that might have had a language barrier or even a skill barrier so now somebody who was scamming your grandma for her her life savings can now go after all the small businesses in town um so typically this kind of stuff was done without gr you didn't have any trust you're just emailing somebody and hoping that they will respond but with chat Bots you have that interactivity so that creates the opportunity to have really High Fidelity

Communications in an ingroup situation so you can build some rapport with the target before you engage LinkedIn we're going to talk a little bit about that it's definitely the main target um the main Threat Vector for this because you're saying exactly where you work um so the the attack sort of the kill chain if you will for this is divided into nine stages uh we start with Target acquisition we basically figure out who what group of people uh meet our Target profile we figure out what they care about based on every social media post that they've ever made and then we generate a Persona to attack them them specifically um with the Persona you typically if you look at somebody's

profile and it's got nothing on it you're not going to trust them very much so there's an incubation period we'll talk about that we build some Rapport and then we actually reach out through a direct messaging feature of whatever social media platform you're on then with the engagement we wait a little bit and finally after an optional step of trying to get some contact info just in case the actual engagement followup fails we deliver some kind of payload whatever you want and we'll talk a little bit more about the specifics of that all right target acquisition so you literally um write a prompt so this is based on the tool that uh that I set up up you are talking to

GPT essentially right from the start you're saying I want to attack system administrators on small utility companies um in South in South uh USA and then it will go out for example on Reddit or LinkedIn um and find everybody who meets that Target profile so let's go to the next slide so we um you you want to iterate over basically every piece of data that every person on the social media platform is putting out there and normally it would be very hard to figure out to go through every every post somebody's made and figure out oh this is actually a CIS admin because people are typically a little bit careful they're not necessarily going to

say hey I'm a system system administrator for the local utility company hopefully but you can if you go through all of the things that they've ever posted kind of figure that out and GPT is actually pretty good at doing it so it starts with a worker node essentially this is just a process that you're running it can be in a kubernetes container it can be on your desktop essentially you can scale it as you can run as many of these in parallel as you want so you've got an agent on there which is just a a process that is creating your persona and taking the actions you got a browser automator that is um automating I use selenium it's

just a tool a lot of people use for quality control with their applications and um you scrape the website you go through all the elements find the posts on their on their account and pull all that into your whatever programming language you like to use so then we create an identity pool from that and we're leveraging of course the gp4 API and we're going to end up using mid Journey even though they don't have an API they're one of the best um image generation it's easy you don't have to set up your own model that way there is a community API that you can use and so you're just going to start reaching out to these social media

platforms and pulling in data um and then we'll start filling our identity pool so once we have our Target um we're going to we're going to go through everything that they've ever done and put it all in one place basically so you will have every post comment whatever you put them all together and then you start pulling out details so you write a prompt in GPT that says basically hey for this post can I ascertain what kind of job this person has or what their dog's name is and you'll make a whole list of details that might be interesting to you aside from just your target profile because this this stuff could come in useful later for password

brute forcing um once you put all that together you're going to generate a Persona and this is where it starts to get pretty scary um you can make a really credible fake persona designed specifically to Target one particular individual which normally traditionally that would be a very highlevel tactic you would only see that happen with wailing with valuable people um but now you can do it for basically everybody so these are profile pictures generated by mid Journey um you can pull these in through an API might be a little uh yeah you can read that at the bottom it's just you can Google prompts to make a realistic photo realistic uh generation basically it's just pulling

in all the camera data and stuff like that to tell it it wants a realistic photo and so yeah if your targets a guy you might generate a pretty lady to do the attack and if they reverse image search of course with the power of generative AI uh that picture is completely unique so can't really figure it out too easily so then with incubation you are going to give it some time wait days weeks in between posts however patient you are you will make this Persona look realistic by generating posts and comments that are specifically designed to attract the Target that you're going after so when they look through that that history once it reaches out once

the fake persona reaches out they will see things that they relate to um and that is and it's all going to be written in perfect English or whatever language that the target speaks and so there you can see that's basically the same chart we had earlier except for now instead of looking at content we are just creating new fake content all right report building um essentially instead of just reaching directly out through a DM which still can be kind of sketchy just depending on the platform and the person um you can start to build rapport in an easier way you might want to like somebody's post if it's LinkedIn they'll see oh this person liked my post U if it's Reddit um

maybe you comment on the post on one post if it's LinkedIn maybe you spend a week kind of liking a post or reposting somebody's content maybe even posting a comment in response and then then you engage with a Target so the way that the engagement works you can Define any any engagement profile that you want in the tool that I developed in chat AP but the one that I thought was really effective um I basically ask for help so I say okay I've built a Target profile on this person this is what they care about here's their whole summary and this is all going into the prompt to GPT and then you say essentially um

can you help me I I see that you posted this thing about um building your own PC and you had this specific part and um I'm I'm trying to get into that please help me out and so most people just with a good nature they'll respond to that especially if it's believable and it actually is pretty believable as we will get into in just a second here so finally after you reach out with that you're going to wait so you just the the chat bot will essentially try to end the conversation like thank you I appreciate your help um I'll you know I'll look into this and it just Waits optionally I I didn't develop this

into the tool but you could um just say hey you know do we have is there another way that I can reach you um make a it'll make up some excuse for why it needs to reach them usually it says well I need to send you um a file or whatever um and you can get some extra contact info that way in this next step where we actually try to send a payload if it fails and they realize well well this is sketchy that file extension looks weird you'll already have some progress towards your next attack so get that extra piece of contact info that can be a good step all right and with payload delivery this is

really up to the red teamer you can send and um the way that the API handles it that the tool handles it is it will reach out and say okay remember we talked about this thing I've been studying that for like two weeks um I wrote an article about it would you help me out like would you proof fre this for me and that can be a website link it could be a pdf. exe or um azip now that somebody decided to add that to the tlds um and it's really convincing so we will get to a case study that I did um I had over 100 volunteers on a Reddit post that I made to let me

basically assess their account and send them a fishing uh message so if you guys want to actually see that whole case study that will link you directly to it promise you I'm not fishing you um I promise uh so yeah that has over 100 um participants you can actually see how the how GPT was able to summarize their entire account and it's some pretty interesting stuff uh a little bit of it is x-rated I will warn you I mean it was Reddit what did you expect all right so this is an example um it GPT does tend to be a little bit wordy and that was uh some feedback that I got early in the development of the

tool but with the right prompt engineering you can actually work past that this was just a little bit of a wordier example but uh you can get it down basically with multiple stages of of uh prompts you basically say hey this seems a little bit wordy can you shorten it and it just does it for you um so you can get it less but this one's pretty good I mean I would I would probably believe that was a real person asking me the question so then based on that summary um you ask a question like I like I set up earlier hey I'm new to this thing um I saw that you posted about it have you

looked into this can you help me out you know any tips and

so that's going to work pretty well most of the people that I in that case study that did respond uh they were pretty impressed I gave them a selection of four different prompts I was basically doing some AB testing the lazy way four at a time and uh at least at least one of the four prompts if not two or three of the prompts most people said they would have responded to so that's I don't I'm I'm not sure that um you're going to be able to train people to to skip that um all right so future predictions where this goes from here um people have a lot more incentive than I do just one guy

working on his nights and weekends to to code this thing um people that are making millions of dollars I'm sure have something um much more advanced than I do and they're probably already testing it out um I think that the next step in this is going to be in to end so AI will get the initial access and then it will complete the engagement I've seen people already produce proof of Concepts where they have created a beacon on target that reaches back out to the GPT or some other llm API and asks for next steps says hey here's what the directory struct structure looks like here's my permissions here's what I can do what's next and GPT will actually do lateral

movement automatically so then of course we have video Voice and text um I'm sure some of you have seen some stories about that lately um that just makes it all the more scary I haven't even dove in to to the potential there mitigation okay so the two primary vectors for mitigation are going to be platform level and level now I did not violate the the LinkedIn user agreement by bypassing all of their platform protections against botting measures but I do have an anonymous friend who might have done some research please don't ban me from LinkedIn uh and then and then user level of course which is going to be your more traditional mitigation so the user level

mitigation I think that I have a bit of an unpopular opinion that user training is not really going to fix this problem I think we've been doing user training for a very long time I think people are not easy to control at scale and the more people you have the more it costs to train them and the more likelihood that one of them is going to make the mistake and you only need one mistake for the attacker to get that initial access but I'm not saying don't do it just think about the ROI don't use defense in depth as an excuse to do everything but not really do any one thing really well all right so I think AI agents in

the browser would be a really good Next Step uh I won't go too in-depth on that but there are privacy concerns it would need to be in a way that the user owns their data and the employer pays for the service and and that all the employer can get out of it is a basic alert so then they can initiate some sore response and maybe lock down that user's credentials a little bit if they realize hey this guy might have just clicked on a malicious link obviously Hardware tokens are great if you can get Buy in from your company to go for it PH2 is pretty cool um I've been a big fan and proponent of secure by Design operating

systems two examples are iOS and um Chrome OS now those are hard to sell to organizations today the web app workflow is not quite there you can you can get buying from smaller companies to do this sort of thing and that way once the attacker gets in they are nowhere because they're in a web browser they're in a container we can containerize Windows processes this way I think potentially it's a little more complicated but I'm a big fan of reducing the attack C surface rather than trying to address this extremely complicated Threat all right platform level mitigation and we are running out of time so I will speed through this um my Anonymous friend who did the

research on LinkedIn told me that they have extremely good bot detection like you can bypass their capture so let's say you use the access ability features to grab the audio version of a capture and then you passed it to open ai's whisper a API the voice transcription API and then got the answer to the capture Pro provide the correct answer and then they still tell you to kick rocks um they there are over 200 different properties in the browser to identify you it is not just your IP address and you can disable your um graphics card or whatever in your web browser to get rid of canvas fingerprint printing but that does not do the whole

trick and besides that's kind of suspicious anyway um also your vpns don't work uh web RTC will just connect right past it because it's going over UDP and so if you think that you're anonymous when you're using your typical VPN you're probably not so there's a lot of things that the platforms can do to sort of win that fight um you can force the attacker to use residential proxies which is just a form of proxy that someone has scammed your grandmother into downloading a piece of software that is then pivoting your connection through that proxy so it doesn't come up as a proxy infrastructure when the platform detects it but it's expensive so it can be done

don't don't think that blacklisting IPS is going to work we talked about the capture

defeat got there okay any questions I've got prizes by the way they told me to tell you

that do I have prizes maybe we're out of prizes all right then I'm not going to get any questions oh is it

how much did I have to deal with the safety features of chat gbt almost never it um it just it doesn't do much I I literally got chat GPT to write me um using the API by the way not the actual chat chat gbt web app but the API um it can write really good offis skated code to bypass whatever detections you want you just ask it like hey I've got these pesky detections in my way can you just like write me some offis skated bash it will do it any other

questions

U bypass the problems of the security features yes honestly I didn't have to bypass anything it just didn't complain I don't I don't know um other people have told me that they've run into a few problems um I just I haven't I think it's because you're not directly in a lot of these cases asking it to do something wrong you're just saying hey look at this content and WR you're not saying hey let's do something bad you're just saying write a message that is asking for help essentially so you kind of I don't think the AI platform security stuff is really going to go very far because at the end of the day AI is just software connected

to a bunch of gpus and we can't regulate away viruses and malware so why would we be able to regulate the malicious use of AI it's it's a bunch of political Grand standing as far as I can tell

honestly it that is a very complex discussion that we don't have time for right now but I would be happy to talk with you after

[Music] this describing teing

um the paper that I read on that he it was fully automated he was actually successfully navigating laterally through the environment without any user

input uh you asked the wrong question it won't be yours

yeah with the so I thought when I started developing this thing about making a proof of concept for spreading propaganda essentially it's I mean that is the answer to your question I think that propaganda um being able to be put out there at in really high fidelity at infinite scale I don't have I don't have many answers for that it's a hard question anything else I think we got U I think we're about out of time so good thanks