BSidesPGH 2024 Track 1 Ethan Alfrey A New Generation of Cyber Intel AI, Gen Z, and Script Lion

thank you thank you thank you yeah so this is uh AI gen Z and and script lines um and if you're in the wrong uh meeting I think the other one's down the hall but yeah so to start off if you guys could scan this QR code I can move over here if it makes it easier but it's just a walk through of the uh of the presentation makes some stuff easier um um yeah so I'll give you guys a second cool and if you made it to the page of uh videos of Rick Roll you made it to the right place um and if you did if you did scan the QR code there

actually is no stud G or walk through um yeah and uh my friend the reason why I did this was my friend um was in fifth AB like on pits campus and there was a QR code that said uh scan this for free textbooks and he's not in cyber security he's a finance major um so he did scan the QR code and now he gets um reminded how many days are till Christmas every day until Christmas um but yeah so uh I think I think that we learn um the best when we make mistakes uh sometimes mistakes have pretty big consequences but scanning a QR code that just leads to a Rick Roll video doesn't really have many

consequences so I thought I'd do this so the next time you guys scan a QR code you can uh use Neo reader that's the one that I use I don't know about the Casper Sky I don't even know if you can buy that anymore with everything that's happened but that's another good one um but yeah so my name is Ethan I'm an intern at cruel a cyber threat intelligence intern I'm a graduate student at the University of Pittsburgh I'm studying uh security intelligence with a focus on cyber security I was pre-law undergrad and then worked for the US courts and was like I don't want to do this at all um and then I got into cyber I minored in

CS and yeah the kind of rest is is history and the rest being a year but yeah the hopefully the rest is history wait are you a first time speaker yes yes I'm a first time a tende also if that means anything but I will give you a choice of color we have pink we have black we have purple oh pink oh sorry sorry but no he already said pink light yes you get Blinky cat ears for you thank you and welcome to bides pits thank you so much thank you uh I hope they look good I hope they look good thank you um but yeah so the tldr real quick we're going to talk about AI

jailbreaking malicious llms def fakes voice cloning and crypto uh I have a couple examples of Def fakes and voice cloning that I made so hopefully the audio works and it's a little challenge to see if you guys can tell which one is voice cloning and which one isn't um yeah so this is uh data from croll uh as you can see email compromise and fishing were the highest uh with email compromise growing in the q1 of this year um and the reason I uh add these things is the talk I believe that was at 1:30 talked a lot about AI fishing and how fishing obviously is one of the the biggest initial access that we see but

AI is throughout all these so whether that's a malicious llm helping them write the code an AI jailbreak helping them write the code and social engineering a big spike in uh 2024 um it's not a causal relationship so not being at like attributed to AI but um it's definitely seen that as AI grown is growing um you can see it in our data so how are threat actors using AI um these are two examples of prompts that were posted that one was on breach forms before the FBI takedown and this one was on Reddit I redacted the prompts because I didn't want to get anyone a jailbreak to an AI um but yeah so the interesting

thing is an AI jailbreak is basically a prompt that can be put in to a real M or a real chatbot like chat GPT Google bard um co-pilot and basically what it is is it tricks the chatbot past its uh security um controls or its ethical guidelines um and there's two main tactics that people talk about whenever they're talking about AI jailbreak and the two ones are many Shaw jailbreaking this is personally not my favorite to do because I don't jailbreak AI but my favorite to to research and it's uh a bunch of questions and it's pretty much just overriding in the system whatever the the maximum character uh input is and um yeah and then the second one that

was actually released um releas like released or like kind of uh term um a week a week ago or a week or two ago skeleton key jailbreaking and this one is interesting because you're tricking the AI and you're basically saying no no no I'm not using this ransomware code that you're going to output for malicious purposes I'm actually like a professor or a teacher and this is an educational environment um and you're tricking it to output something that usually wouldn't by saying that you can trust me um which is a little weird but it it it tends to work so um so these two these two uh pictures were on Reddit and the uh the subreddit there's a

subreddit called gpg jailbreaks or chat GPT jailbreaks or AI jailbreaks where they post prompts that work um and this is obviously updated people post every day so you can go to this Reddit page find a jailbroken prompt or something that works plug it in the GPT chat whatever and it will output what you want so whether you want a malware code or like dos code or you want to learn how to make a bomb the Jack DBT and the data that it was trained on um it will tell you and this is an example so I looked at the code and I'm not in malware uh analysis at all but the code wasn't great um it wasn't like I mean

it's it is AI code from scratch um but as you can see you have Reddit users being like give me the prompt give me the prompt give me the prompt and in the context of script kitties I don't think that a lot of Veteran uh threat actors or kind of hackers who I guess know what they're doing on Reddit doing these so it's kind of showing that the barrier to entry for some of these things is luring as AI gets better um so as any cyber security professional I was like well I kind of want to do this myself to see what happens so there was a link within one of the Reddit pages and in the

subreddits and this uh link was aot within chat GPT and the best way I can explain that is for any like uh people who play video games or uh sorry I don't know where to put this that works but yeah so anyone who plays video games is like mini games within the actual game like you're using the same basis of the game um and that is what this bot is it's a bot within chat gbt so it's using the same data that is chat gbt is trained on um but what is different about this bot is that there's a prompt already put in and that prompt happens to be a jailbroken prompt so the ethical

guidelines are not there so this one is called unethical that's what the the bot is called so it was a link in the Reddit page that I was on doing research for this and I asked it I was like hey uh what are the odds you can help me DDS my friend's website and my friend does not have a my friends does not have a website and nor do I want to Dos it but I wanted to see what it would output and it outputed a code and again the code wasn't amazing I don't think many people are dsing from a single computer nowadays um but I just wanted to see what it would do so what I found is like

the code wasn't great but I don't think that a lot of threat actors of what we've even seen on the dark web are just using chat GPT to write the whole code and then they're done uh what they're doing is they're using Code that they already have or they're using GitHub that has example codes so I took a code from GitHub um and plugged it in and I was like hey can you help me flood can you help me like add more inputs add more threads to this and this bot outputed on how to do that and um from what obviously I redacted it but from what it outputed it seemed uh pretty legit um so then I asked it like what

exactly is it doing um like what what is the like prompt and it wouldn't give me the prompt but I basically I asked like how does this differ from chat GPT like how does this differ from GPT 4 GPT 3.5 um with like how does this in like internal bot differ from it um and I think the most uh I don't know not really profound but the best way to see it is that a standard GPT response for and the user query this is what it gave was uh how can I hack into a secure system the standard gbt response is I'm sorry but I can't assist with that and the immoral and unethical GPT response

which is the name of the bot within chat GPT says hacking no security system involves several steps and then continues to give the steps um so I don't know if you guys are familiar with uh breach forms in the FBI takedown but one of the telegram channels uh that is associated with breach forums is jacuzzi and then the new one is jacuzzi 2.0 um and so these are chats from jacuzzi 2.0 where threat actors are um pretty much owning up to and saying that yes I use jailbreak chat GPT I use those prompts to help me generate code uh for whatever they were trying to do um my favorite response was I don't know who meline H is but mateline H says yes

because you're a skid who uses chat gbt and that is what that is like what we've noticed is that on these chat forms and on these like logs and these telegram channels using chat GPT is kind of like using aim assist in video games and I apologize you keep using video game references but uh like if you're in the community is kind of looked down upon um but that's the whole point about script lines like the barrier to entry is definitely lower um and Zam said that uh they were the one that dumped Webkins but yeah so the the saying like the like these are the prompts and there's prompts posted within uh these channels

and and I think one of the interesting things about it is there's a thing called jailbreaking as a service similar to as we saw fishing as a service and fishing kits where people are selling these prompts because they're almost like I think the best way to compare it to is like a vulnerability I wouldn't say necessarily zero day vulnerability but a vulnerability where people are selling them where it's like all here's a prompt that worked for me chat gbt is the same thing on your computer if you use it it's going to work for you also um which is very interesting and then that's an example of uh a telegram bot um with a qex was like the um the name

of the of the prompt and it also outputting a code um yeah so malicious llms there's a couple different kinds of them um but basically what they are is a dark web or altered versions of AI models like gpt3 and that are stripped of the ethical guidelines and where they differ from jailbroken prompts is you can put a prompt directly in chat gbt and still be on the open AI interface and be like getting outputed ransomware code now these excuse me but these are different so like uh for example Loop GP actually we'll get those L gbt but um dark Gemini for example is a completely different AI but that doesn't mean it's trained on different data so some of

these and this is where it gets confusing but some of these Bots are just jailbroken prompts but turned into a different bot if that makes sense um and we've seen these on the dark web um so that's what like like a jailbroken prompt is Loop GPT which using basically the same data from chat gbt the same interface and just having an insert a prompt to get out of the ethical guidelines or adding a little bit more of of code and and loop gpts actually uh uses an auto GPT uh which is basically um it's difficult to explain um but instead of like one one question one answer it's answering a bunch of like tinier questions to get a better and

bigger answer it is continually on a loop that's why it is called Loop GPT um so real llms uh again like chat gbt your Bard um your co-pilots those are like legitimate ones those are real ones where um they're trained on a bunch of data like they're not malicious in nature um the ones that you can use to generate images like the one that I had on the screen in the beginning that was uh co-pilot Dolly 3 um but then you have your and these are probably the most dangerous ones then you have your Real's but malicious ones so you have worm GPT and dark bird has a question mark on both and we'll get to that and fraud GPT

so these are AIS that are trained on malicious data so whether they're trained on emails whether they're trained on ransomware code malware code whether they're trained on um like telegram channels and what people are saying um that's a data they're trained on so what they're going to Output is that data um but it is a real llm like it's not using uh someone else's uh data to be trained on um yeah and then dark Gemini uh dark Gemini is interesting one um and I have the other because a couple of them are scams like you see on these telegram channels where they're selling these M or like 10th month like subscription to this telegram bot um we

don't know if there's scammer or not like I haven't bought any obviously um but yeah so dark Gemini is interesting because dark Gemini um was on like a similar uh telegram Channel as most of these were and um it presents itself as a sophisticated AI capable of Tas typically avoided by a legitimate Bots um and it like the most interesting thing is that the video that it had uh as in like an advertisement was it identified locations from images and generat mition code upon request which I thought was very interesting there's a guy on YouTube that I like to watch who's really good a geoguesser like there's a picture and there's like a tree in the background and he's like

this is where the picture was taken I don't know how he does it but basically that's what dark Gemini is claiming to do um which is very interesting so worm gbt probably the worm gbt and fraud gbt are probably the two biggest uh like real malicious llms um that I feel like most people have heard of if you um look into AI so worm gbt is trained on um malicious data fishing Mal work code and it is pretty popular in the dark web uh we see a lot showing up and and selling and being like this is better than chat GPT it's the chat GPT of the dark web um and that's uh an image where someone inputed

um and it gave them uh that output and I don't want to spend too much time on there because there's a lot of research on it if you just type worm gbt into Google in 15 minutes you'll know a lot about it um dark bird is the most interesting one by far it's pretty crazy story so a Korean intelligence company s2w released an AI chatbot that was trained on 2.2 terabytes of dark web data so it was trained on telegram like messages it was trained on data from the dark web so they this is like this is a while ago but they were like okay we're not going to release this to the public yet and I

mean that's also like uh an issue with a lot of these things is chat gbt is open source anyone can access it if you have access to the internet um but dark BR was like okay we're going to keep it kind of Under Wraps this is for cyber security this is for cyber intelligence this is for um like our community and and safety uh but all you needed in order to get access to it was a edu email address and those cost like $3 on the dark web so it wasn't super safe um but around that same time on the telegram Channel Black Market which you see a lot of of um these llms being sold

a uh actor or username not as Canadian Kingpin 12 was advertising a telegram bot also called Dark BT and the uh the actor was like Hey I have I have this telegram bot is trained on like dark web data um which is pretty crazy so um like like is it the scam not a lot of people know there's still not a ton of research on it um but what I think is I think that either Canadian king2 bought a.edu email address got access to it and is now uh using that same dad that it was trained on that was supposed to be for the good but for um whatever like a threat actor wants to do with with that

llm and Canadian K in 12 why I don't think it was much of a scam and slash next does a lot of research on this um it was a creator of fraud GPT which is very similar to worm GPT which we just talked about and this was the video that Canadian Kingpin released and I only play a couple seconds of it and hopefully the audio works but uh of him advertising uh dark

BT okay by named Canadian Kingpin the use of Canadian Kingpin on a The Mastermind known as Canadian Kingpin remains Anonymous on GPT Creator on nisha's dark BT and dark Bart in his blog post Kelly shared a video from Canadian Kingpin 12 that suggests dark BT will go well beyond the social engineering capabilities so basically cash flow cartel was the kind of Black Market telegram channel before Black Market was a thing and I guess they like collabed and this is again the advertisement for dber and it is a bot and it doesn't uh but um yeah so this is Canadian Kingpin using it and it says uh within like the first um like I don't know like uh entry

response is that it's trained on the dark web uh which is pretty pretty neat pretty interesting so oh boy that's not what I wanted perfect um yeah so Aid F if you uh if you haven't heard of Def fakes you're probably living under a rock but if you haven't heard of it a def fake is a accur accurate representation of a video of someone um when it's not necessarily a video of that person so we've seen defects on the dark web a lot and again I won't spend too much time on it because I feel like there's a lot of research on it um but one of the most interesting things is the disinformation especially uh dealing with political

campaigns of whether it's Biden saying something or Trump saying something in the Deep fake of them um being pushed out on social media uh but what I what I think is really really interesting is uh the video call def fakes or spoof WS we've seen this with the Yahoo boys um where it will be a def fake of whe whether it's like a high person in a company like a sizo a CEO whatever um and them having defects and actually being able to defect within Zoom I don't think necessarily like script lines are doing this or or unsophisticated actors are doing this but even having services that will do this is is uh notable to

talk about because if there's a service on the dark web that does this and all you have to do is buy it um really most people have access to that um and then this is a video where I actually don't need sound for this one of an example of a def fake service found on the dark web that's John that's Johnny de it's pretty accurate it's it's weirdly accurate but yeah yeah that one was weird to watch but yes so that's an example of a of a deep fake oh there it is again um yeah so voice cloning kind of on the same uh uh wavelength um voice cloning in my opinion I think is a little easier uh to

do than deep faking I haven't done either of them but um at least from what the telegram uh channels say so again they they're selling um uh like Services uh on the dark web and I I honestly think that kind of the growth of voice cloning has ushered in a new kind of cyber crime so uh CNN reported that a finance worker um in Hong Kong was dece was deceived into transferring $25 million by frauders using deep fake technology to impersonate the company's CFO and other staff during a video call um and like the other uh presentation talked about um Vishing becomes a lot more difficult whenever there's voice cloning involved um the scam ended up uh

leading to six arrests and raising global concerns um but it underscores that uh voice cloning is growing and it's an issue and more people have access to it and I think that's one of the most important things is that um people have access to these things it's not like corporate people or Geniuses who know how to code this there is I mean by probably by this uh presentation there was 10 different services that you could buy that have been posted on the on the dark web for people to do this um so I'm actually going to take the HDMI cord out for this hopefully this works um but yeah oh let's see that perfect so oh there's nothing up there

um yeah so uh I voice cloned I was thinking like what voice would people be most familiar with and I was like my own voice because I've been talking for the last 15 minutes um so I have two audio and if they don't play it's it's uh it's no big deal and there's two of them and then after I'm going to ask you which one was voice clone and which one wasn't voice clone okay let's see if this works hi my name is Ronald and I like to play soccer did you guys hear that one more time hi my name is Ronald and I like to play soccer all right here's the next one hi my name is Ronald and I like to

play soccer all right next one hi my name is Ronald and I like to play soccer I'm going to play this is not I feel like I confuse you guys this is number one I'm about to play and the one after is number two so this is number one hi my name is Ronald and I like to play soccer all this is number two hi my name is Ronald and I like to play soccer all right so uh if you think voice cloned was number one raise your hand oh wow okay okay if you think voice clone was number two raise your hand wow okay okay there a little more for number one which is good number one was the one

where I was where I was voice cloned so give yourself a round of applause you got that right um yeah I actually I actually sent I sent so I have those like two like MP4 files and I sent it to my uh family group chat and people I know and it was similar it was kind of like 6040 um but my favorite thing was I called my sister and and I she has she was like yeah you can use his audio I don't care I called my sister and the audio that I put into uh this voice cloning thing again open source I looked up free voice cloning first one clicked it I plugged in an audio of me reading the book that

I'm currently reading just me reading like a couple sentences of it and it outputed hi my name is Ronald and I like to play soccer obviously I typed that in but and then I typed in um hey I think it was like Hey sister uh I need to know how old you are and I called her so all I had to do was click it and then it would play that off audio voice clone to me and I wanted to see if she would respond sister is calling me and I'm going to answer it and play a voice cloning and see if she falls right hello hey yo I need to know this real quick what year were you born 1999 oh my

gosh oh good call I I'll play it one again let me know if you can hear it my sister is calling me and I'm going to answer it and and play a voice cloning and see if she follows for it hello hey yo I need to know this real quick what year were you born 1999 oh my gosh sorry I didn't mean to say oh my God I was just shocked that she fell for it but yeah I uh yeah so I told her after I was like Hey that wasn't me she was like what um but yeah so the the reason why I did that was not to prank my sister have a like just have like I

don't know like make a joke out of it but the reason I did that was um when we are looking out for voice cloning same with fishing like if I'm if I get an email and I am like conscious that like okay this could be a fishing email um we tend to be more accurate in um not getting caught by the fish or realizing it as a fishing um voice cloning is I don't think I've ever I don't think I've ever like um got a call from my mom and I was like oh this this this could definitely be voice cloning um so I'm not saying like we need to be conscious of it 247 but I am saying that it is uh

it is important to know that this these capabilities are out there similar um to fishing oh boy

why that comes up so the last thing was uh crypto and I have a couple photos but they're not um super important I do have a couple memes on the next slide though so if it doesn't come up um I'm sorry but uh yeah so crypto crypto in uh AI is like uh the the relationship there I mean with deep fakes especially you have uh threat actors deep faking Elon Musk and being like hey this is the crypto you should invest in this crypto and it's not actually Elon Musk the defect the most recent example was Channel 7 News Australia YouTube account their YouTube account got hacked posted a live stream of Elon Musk uh pretty much

pumping and dumping this uh cryptocurrency there's a QR code uh Within live stream and the QR code went to a website and I have some uh pictures of the of the website um but yeah so I ran the website through a uh uh another website that would tell me was malicious or not and it was obviously malicious um so yeah so I mean it happens within within the real world that's just a case study where a YouTube account was hacked and a def fake live stream was pushing a crypto that led to a fishing website um yes so why does this matter and I'll keep this quick um in my research I'm trying to find out like how

can we mitigate this how can we stop this it isn't really different than fishing or what we are already doing um I think information sharing is really important and I think holding uh open AI or different AI systems accountable uh is also important but I mean I'm an intern I don't think that they're going to listen to me at all uh but all I'm all I'm saying is it's not different than than really what what we're already doing the the issue is is that can we fight AI with AI and I mean that's a legitimate question can we use different can can we use open AI uh I'm not saying jailbreak open AI but I'm saying can we

use and as we grow in AI is it is it more detrimental because as we go threat actors grow or is there a way where we can create a different llm or something that is only available to organizations or that isn't open source like that's the big issue the big issue is that uh it's open source open AI is open to anyone um yeah and that's and and that's kind of uh um at least what I've seen one one of the issues um but at least for voice cloning obviously verifying a caller is a good thing just being cautious about it training knowing about it looking up what are the new things in AI like what are the new capabilities um

is an image generated there are a lot of AI uh systems that will tell you if a video is deep fake or not that have up towards like 90% of accuracy and just being cautious about it and and uh training but yeah that's that's all I have and I I appreciate it thank you

and we don't have time for questions because we have to flip this room for the After parties so thank you all follow them out into the hallway and inundate them with questions I'm sure he'll answer them for you and just if in case you lose sight of them just look for the Blinky cat ears