On Your Oceans 11 Team We are the AI Guys (Technically Girls) | Harriet Farlow & Chantelle Ralevska

okay well good morning everyone and welcome to bside Sydney 2024 we excited here to provide two dedicated talk rooms throughout today this is the first theater lunch will be held on level three capture the flag careers and the second talk theater will be on level five and workshops and networking later this evening will be on level six without further ado let's start by introducing our first speakers Harriet farow is the CEO of AI security company meva security labs and a PhD candidate in machine learning security she is the creative mind behind the YouTube channel Harriet hacks and Chantel Reesa is the founder and CEO of cyber a startup delivering cyber security consulting services and security awareness training to

businesses across Australia presenting a talk titled on your Oceans 11 team where the AI guys technically girls please welcome Harriet and

Chantel R Group

[Music]

11 we might just skip that it was working in the test but of course course now we're live it's not working doesn't that always happen basically the end of the video exposes who the real hackers were behind the MGM Casino breach and it was in fact Harriet and myself uh so spoiler alert fortunately for us black cat SP scattered spider and a bunch of other a groups willingly took the blame for the attack so here we are one year later ready to Target our next victim a bit of introduction into who I am uh my name is Chantel and I'm incredibly passionate about the people side of cyber security that is so distracting I can hear myself anyway um

yes so I run cyber we're very much focused on cyber security awareness and training so reshaping how people view cyber security because ultimately it's not just something that we're responsible for in the cyber security industry but people outside of it the rest of the organization so really empowering the those people to understand cyber and be passionate about it like all of us are so today we're going to focus very much on the the I guess the Cyber attack kill chain and the psychology of cyber security and the end user versus just the technical components I hand over to you Harriet thank you Chantel everyone can hear me okay all right um we don't need cyber

security instance when we already have bad U internet connection so that's okay um thank you everyone for being here on a Saturday morning at 9:00 a.m. in Meadow Bank um I uh we really appreciate it the last time or the first time I delivered my portion of the talk was at Defcon this year which was by far the scariest most stressful thing I've ever done so being able to deliver like the the content to you know people like you and and an audience in a venue where I sort of think back to bside Sydney a few years ago as one of the first talks in security that I did so it feels really special and really nice to be here with

you uh this early in the morning um but my name is har farow as we heard in the intro I run the AI security company maleva security Labs um and my PhD is in machine learning security and I also have content on Harriet hacks on YouTube and other socials that's a bit average but it's getting better so if you stick with it it'll eventually get better um but I've been working at the intersection of AI and security for about a decade I started in Consulting mostly on defense projects I spent a year living in Darwin doing data Remediation in the Navy which is fascinating um I worked at a startup in New York I worked in the government um

my latest role before starting the company was as an acting technical director at the Australian signals directorate so I very much come from seeing the problem space of all of the ways that AI can introduce risks into our businesses our organizations our societies and now the work I do at maleva is trying to like Stop Those risks from coming

about building the suspense okay so um I originally had this idea for a talk of wanting to focus on casinos because we all love a good heist movie and any good sort of security related um topic it helps to link it back to uh some some sort of narrative so can I get a show of hands if you're familiar with the oceans franchise of highest films great okay um and we all get the idea anyway basically there's you know a group of cool people so that's us too um trying to get into a casino we use all sorts of um different means to get in there and steal the information that we're looking for and

in this case we're focusing specifically on the cyber security the AI security element so sorry shant places so these are our objectives for today we're going to Rob the casino um just like we did in in Vegas before um we're going to focus on looking at um some oan and social engineering that Chantel is going to be looking at um I'll be looking at AI security so how to actually compromise or hack the AI system itself looking at facial recognition in particular but you know really any computer vision system is vulnerable to this kind of attack and then we'll be looking at the implication of deep fakes and the idea is to profit of course um but also to actually come

away with some lessons that we can apply to our own organizations or our own teams or our own roles as well uh so some disclaimers obviously this the presentation and the narrative is fictional um it is based out of like real research and real attacks but that's not necessarily the point of how we're presenting it to you we want it to be interesting um and we did work with a number of organizations to conduct This research in different ways so we're very very grateful for for them to be able to do that so I want to start off with some statistics just to paint the picture because we're focusing on a couple of different intersections of AI and

security Chantel is very much looking at you know how AI can be used for security purposes and then I tend to focus on the security of AI systems themselves and looking at AI as an attack surface so you might be interested to know that in 2024 77% of organizations actually reported AI related security incidents and this kind of statistic is scary because it's very real and impacting businesses and I know as Security Professionals you're already Keen to all sorts of security risks but most of the time I'm speaking to people who aren't from security and especially when it comes to AI systems and whether they represent real security targets um people just don't care you

know it's totally disregarded a lot of the time um but it is real and so I have a prize for the people who can guess what percentage increase this statistic was from 2023 and because obviously you have to have a challenge coin so the first two people who come to me with the closest correct answers based on what I remember um either after the talk Depending on time and not wanting to take up time from the next speakers or during the break um we'll get a challenge coin so please do that um but I'm going to let you take it from here Shantel awesome yeah so we're using the MGM hack as the basis for our attack on our next Target

which is another casino here in Australia so for context on the MGM hack does anyone know a lot about how it actually occurred yeah a few people so haret and myself using our favorite platform LinkedIn were able to identify a Target who was an IT specialist with privileged access so we used ENT to gain as much information on this target as possible and then we called the MGM help desk and we're on the phone to them for about 20 minutes back and forth trickling pieces of information that we knew about our IT specialist and getting them to reset the password which of course gave us access to the mgm's network because this IT specialist had access to both Azura

tenant environments and OCTA as well so once we were in uh there's reports that we stole about 6 terabytes of data including full name dates of birth passport information Social Security numbers the list goes on and on and once the casino detected that we were on the network that's when they shut down their OCTA sync servers as well as essential infrastructure which was the cause of you would have seen in the video even though it was really choppy uh the slot machines not working digital room keys stopped working the reservation system and that was because of their reaction to the attack so it cost the MGM $100 million and they were shut down for about 10 days and despite that they

refused to pay our Ransom so the saying what happens in Vegas stays in Vegas wasn't so true because it actually impacted MGM Resorts in New York Ohio Michigan and so on now this attack is definitely not unique it sits at the Nexus of several attack Trends one of which being the increasing targeting of people and we know that 98% of all cyber attacks are the result of social engineering which is why empowering the end user is so critical so these are the three stages of our attack today at first of course we want to infiltrate the network so infiltrating through reconnaissance and social engineering the second is escalation we want to gather credentials so that we can gather more data and then

the third is of course exfiltrating that data without detection we are using ENT for reconnaissance so obviously reconnaissance is so critical and a lot of sophisticated threat actors leverage reconnaissance in order to make their social engineering attempts so much more sophisticated that you as the victim and the subject are so much more likely to click so we're using ent ent or open- Source intelligence is the process of gathering publicly in available information either online through the surface web or the dark web or offline as well and then using that to answer specific questions or to form specific conclusions there are multiple tools that you can use for ENT social media is a prime example um we've got internet

search engines the dark web platforms like breached forums public records and government records now why oent so traditionally used in law enforcement ENT has now expanded into a range of different Industries organizations being used for a range of different purposes so ENT helps you to identify vulnerabilities identify potential security threats and then on the other side of that as an organization it helps you to actually bolster those threats either personally or professionally so oin can be used for we we do oin at Cyber um we do it for executive protection but it's also used for a range of other purposes like theft prevention uh internal investigations cyber threat intelligence goes on and on and it's such a valuable tool that

actually they estimate by 2033 that the oan industry will be worth $ 58 billion I had to remind myself so um obviously the Dark Side of ocean is that anything that can be found by us as cyber security practitioners can also be found by threat actors so we are mirroring on from the MGM hack we're leveraging LinkedIn and luckily for us our Casino doesn't have nowhere near as many employees as the MGM uh MGM Resorts did so we're able to locate our Target very simply which you can't really see on the screen but his name is Henry goolan so he's the director of security and surveillance at this casino I thought I put my laptop on do

not disturb I'll do that again going back so when we want to uh do Recon and our Target there's a few different different pieces of information that are really valuable in profiling that Target of course things like emails phone numbers date of birth passwords home address and family who where their kids go to school all of these pieces of information that typically leveraging social media is really easy to together so using a data broker platform like contact out I was able to find Henry's personal and Corporate email address basically instantly as well as his phone number now what I'm doing with the email address is I feel like once you have an email address there's so many different

pieces of information that you can find from that and of course passwords are one of them and exposed. LOL is this platform that I have a screenshot of here and it's really similar to have I been pawned in the sense that if you put your email address in it will tell you what breaches that email is in however unlike have I been pwned it actually exposes the passwords used so uh you don't even need to as a threat actor you don't even need to purchase the data breaches to to gain access to this person's passwords so from that we were able to find several credentials linked to Henry now social media a an attackers prime hunting ground for information so

Henry we're validating that this is actually Henry because a lot of the pieces of information linked to his LinkedIn so for example where he went to school that he lives in CRA and so on and from his social media I've been able to gather so much information so I've located uh who his two children are Lily and Grace I got a bit more information about his interest he loves caravanning and he recently sold a caravan and his daughter posted about his 55th birthday which you can't see either um but yeah his we we know his date of birth as well so all of these pieces of information just by scouring social media and actually that's a really

interesting trend in the UK as well is that they're delivering training not just for individuals in an organization but actually for family and a lot of the time the information that exposes us our dates of birth where we live all of this sort of information is information that is posted by our family or our friends and I'm sure a lot of us can relate to that so that's a really interesting Trend out of the UK now we want to find Henry's home address because that's one of the pieces of the puzzle that is missing so there's a range of different ways you can find a home address leveraging oent things like da approvals uh but in this instance Henry is

actually a runner so we've got his phone number and we've put that into straa to be able to locate his home because a lot of people that leverage that use straa well straa by default is public um so often that can actually expose your home address so we've been able to find where Henry lives as well so we've got all of these pieces of information we've got his passwords his contact details we know his family Dynamic and we also know where he lives so using all of this there are two different attack vectors that we can execute the first is compromising Henry's home Wi-Fi network so because I know where Henry lives I was recently in

camra I thought how convenient I'm going to go to Henry's house sit outside for a few hours and try to crack his home Wi-Fi network so to execute this attack I used the air crack NG Swit and an alpha network adapter which I left at home I should have brought it with me and with both of those I'm able to see all the networks within vicinity by being outside of Henry's house and of course I only care about Henry's Network so I'm able to locate his BSS ID which is his network name because Henry's network name was Goen family which makes it really easy um and the whole objective with this is to crack that

four-way handshake which is basically encrypted password that connects your device to the access point so using the alpha network adapter I do what is called a deauthentication attack which forces which sends an authentication frame to the network forcing all devices on that Network to De authenticate and then automatically reauthenticate and in the process I've been able to intercept and capture that four-way handshake which enables me to go home and then crack those packets and this happens in the background it's a glitch that that no one would even no one would even notice or recognize so once I've gone home I've cracked those packets I've been able to gain access to Henry's Network and from that there's so many things that I can

do I can sniff all of Henry's traffic and decrypt everything I can conduct a man- in the-middle attack and compromise other devices on the network including Internet of Things devices enabling me to persistently listen to everything that goes on in Henry's house I can compromise families his kids devices but most importantly I can steal his credentials and his session cookies which enables me to gain access to the mg not MGM sorry to the casino and uh yeah and impersonate Henry to steal data and access apps so of course home networks are nowhere near as secure as corporate networks and it does make it a lot easier for an attacker to compromise um and typically Wi-Fi credentials are the

most typical generic configurations how many of us in the room have actually changed uh changed our username and password on our Wi-Fi network making them so much easier to crack as well now I'm looking for the path of lease resistance so say I'm not in Australia I didn't visit Cambra the second attack Vector that I can execute is a targeted fishing campaign so there's two ways that I can do this I can either attach a malicious attachment with macros that will run a malicious code when open or I can use a malicious URL which would conduct a man in the browser attack because through our oent we also found that Henry is operating on a vulnerable

browser so because I know Henry loves Caravans I've crafted this beautiful Caravan newsletter about secondhand Caravans in the area actually very close to Henry's home address and when we run these targeted fishing campaigns that are so highly curated to the person nine times out of 10 people click on them so I'm fairly confident that in this instance Henry is going to click and I've decided to opt for a malicious link because now Microsoft alerts users when enabling macros which makes that attack Vector less stealthy which probably why URLs are now the most popular delivery mechanism for cyber crime as well so once Henry clicks on my fishing email he's going to be redirected to this Caravan site and in the background

it's going to uh execute a malicious JavaScript exploit that gives me complete control over his browser and he's not even going to recognize that this is happening in the background and of course browsers have so many permissions so once you're in the browser there's so much that you can do you can move laterally you can install malware onto the vi the device install a key logger and track his keystrokes and most importantly again steal credentials steal session cookies to be able to access impersonate Henry and access the casino's Assets Now hand over to you Harriet if you want to swap with me we can well yeah okay we're changing Tac a little bit now so we have all this useful

information about Henry and now we're going to exploit it but we're going to exploit it in a slightly different way you're right I can keep hearing the vi by me okay so I wanted to do bit of research into casinos in general so we have we have this information about Henry now we actually want to be able to walk into a casino and gain some extra information there and I was sort of interested in the AI uses in casinos so I went into this project and I sort of assumed that artificial intelligence would mostly be used by casinos for gameplay monitoring and I was actually surprised I found that the most important use of artificial intelligence in the casino

setting was facial recognition and person detection through the facial recognition so you know casinos might be looking for known Persons of Interest but they would do that by um putting that person's information into the the sort of tool that they use and then it connects to all of the cameras in the casino and if that person is uh is identified then they're flagged and the security person goes and goes and takes them out so we know have about Target it the most most useful Target in Casino format is the facial recognition detection so we're not using AI for hacking just to clarify we're actually looking at how to hack the AI system itself we're looking at how to disrupt

or manipulate the facial recognition AI system so you might have heard of this concept called adversarial machine learning and the idea is basically that you can um disrupt deceive or disclose information from a machine Learning System a cool part of an artificial intelligence by adding specially crafted patterns or materials to the object itself so in this example this is thinking of an autonomous vehicle use case that's using its computer vision system to be able to identify stop signs and so the first stop sign we see is a normal stop sign it's able to recognize it is one with 99% accuracy and then the next one is quoted in a special adversarial paint or adversarial pattern so that it

instead recognizes a sports ball at 80% and this has been shown to work in lab environments it's often been seen as a bit a bit theoretical a bit academic but the field has moved quite a lot in the last few years and recently I think it was a few weeks ago actually a research group published a report on all the ways that they were able to hack real Tesla vehicles so that they couldn't recognize Lane markers stop signs all this kind of stuff but in the real world this is the original adversarial example so if you're not already familiar with this field this is the picture you should take note of and then you speak to AML people and they will

think that you're one of them basically this panda is superimposed with adversarial noise the noise is not random but it's crafted based specifically on the Target Model that someone is trying to compromise so that when it's superimposed um we instead recognize a given the the AI system recognizes a given instead so there's now over um you know 100 different attacks like this since that original example was created in 2013 and they happen at all stages of the artificial intelligence sort of attack cycle so if you think about an AI and it's its core technical capability being a machine learning model there are lots of different kinds of machine learning models some do computer vision so being able to recognize objects in

the environment some do natural language processing so being able to understand humans generate text like Transformer based large language models there are so many different kinds of attacks that can do different things depending on the model architecture at different stages of the life cycle whether it's in the training process or the inference phase which is sort of when you're making a prediction or generating an output and a good way of thinking about them is either disrupt deceive or disclosure based attacks I don't want to try and talk through you know dozen or more different kinds of attacks um but you can think about them being able to do one of three things they either disrupt the model so it doesn't work as

intended this is kind of like a Dos attack in cyber world but for an AI system um it could be a deceive based attack so like the stop sign example you're deceiving that model so it doesn't just not work but you want it to instead recognize a specific speed sign for example um or a disclosure based attack so you're leaking private or confidential information about the training data or the model parameters imagine being able to steal you know all the model weights of gpt3 um or GPT 4 um which has been done from um very from earlier forms of the model so people have been able to replicate GPT too for example so if we think about the idea of

trying to hack an AI system I mean the idea of hacking a model has existed as long as models have themselves that panda an example was released in a paper in 2013 and it you know it exploits the inherent architecture of the machine learning process but the idea of algorithm hacking so just hacking an algorithm has been um has a lot of history to it and a really cool Casino based example is this one um involving random number generators so in the 1990s um Ron Harris who was employed by the Nevada gaming Authority board was able to get access to the seed behind the random number generator at some of the Kino machines and was able to from

his hotel room and using a sort of person based in the casino still $150,000 which in this economy doesn't sound worth it honestly but um I guess they could have um taken more but they were found another cool example is card counting itself so the idea of you know I mean a blackjack game is an algorithm right you have a specific uh way of of playing that can be more strategic and it's a game that is inherently advantageous to the house so any any game at the casino um has between a 2 and 25% um uh favor towards the casino and the 25% cap is for slot machines and that's only imposed because of law

otherwise it would be it'd be higher but something like a blackjack game is the lowest of around 2% and if you play Perfect strategy you keep it to that minimum 2% and then if you're able to implement card counting on top of that then you can lower that even further so that's a way of hacking that algorithm too now the idea of adversarial machine learning is taking that a step further from just hacking an algorithm to actually hacking the way that machine learning models work in the first place which at its very core despite all the different kinds of machine learning models we have these days and how complicated they're getting you're basically taking input

data and then you're finding a way to map that to a prediction that is as accurate as possible and the way that you map that is by building a an architecture a model architecture it's trained over many iterations very quickly and the way that it's able to go from sort of not well performing to very accurate is by the idea of a gradient so you're constantly tracking this gradient function which indicates how accurate a model is over the trading course so you go from if you're thinking about this as a sort of hyperdimensional space that's detailed by all of the parameters in a model you go from this area of red which is inaccurate and then over many

training iterations it becomes more accurate and sort of moves into this blue area so the idea of adversary machine learning is that by using the information about the gradients instead of making the model more accurate you're identifying what you need to do to the model what kind of data you need to feed to the model to make it less accurate to move over a prediction boundary so this is how it might work in a convolutional neuron Network so you might have your the image you go through the training process there's feature extraction classification and then you're trying to predict at the end if um if you're looking at for example Sarah cona now most of the the ways these

attacks work is by perturbing the actual Target itself so in that stop sign example you're actually changing the stop sign in the Pander example you're actually changing the image of the panda but that can be a constraint for real world applications and I wanted to experiment with instead of actually changing the The Thing by adding other objects around that thing to be able to cause a misclassification that are sort of dynamic but you don't actually have to change the object itself so um I created this method called distributed adversarial regions so you can distribute the adversarial regions around the target to cause a disrupt or a deceive attack to either make the model not work or to make it uh classify

or predict a specific thing based on something you've predetermined and because I used to work in defense the original idea I was sort of imagining was like if you could have these adversarial Boys around a sort of military platform to act as camouflage so by placing these objects and no I know they're not real boys but you know forgive me I didn't have enough funding to put real boys around a around OT or something um but you could cause another machine learning model to look at that military platform and basically not see it's there or maybe have it predict that it's like a sailboat instead of a military platform so I wanted to apply this to the casino

context and I'll talk through this code which is not always the most exciting demonstration to have um but basically if we're looking at a facial recognition model uh I wanted to see how I could add objects to my face for example so that the model that's looking at me either would just not recognize me or would maybe think that I'm actually someone else and be convinced that I'm someone else so basically what I'm doing is I I tested a lot of different open-source uh facial recognition models these tend to be celebrity classifiers and things that I then fine-tuned to recognize my face I take all of this information about the embedding space of that model so this is understanding

basically what makes it accurate using this information I can understand what what uh noise I would need to add to specific regions sort of optimized regions of this image to make it less accurate and then I um I have to figure out you know what is the boundary of inaccurate before it's too too wild or too obvious I'm just going to skip through this because we're sort of uh started a bit late so not tracking for time but basically the the best way of hacking a facial recognition model right is is jewelry there's not many other objects that I would be able to change around my face but something like these adversarial patterns encoded in earrings

is something that does work and so I tested these and this is a day I should have it was a cold day in CRA and from canra I should have done my hair but this is me approaching a facial recognition model um there's no match found and then if I add these sort of adversarial earrings um this is just digital mind you um be match found so depending of course on whether I want this to be a disrupt based attack so um maybe not recognizing me or a specific Target or having it recognize someone else we're able to sort of hack the casino I guess but what does it really mean to hack the casino

in a facial recognition context what we actually found was that for all the open source models we tested it reduced the classification confidence by 40.4% so usually in a machine learning model you'd have a boundary of maybe 80% that says um or 90% that says if the confidence that this face is har is 5% it's over that boundary then it's a yes and if it's maybe 80% it could be a no so this is how it can be quite different between cyber and AI based attacks because AI systems are inherently probabilistic so the idea of hacking an AI system is more about changing that classification boundary and this of course is a computer vision based attack

usually when we hear about AI hacking it's in chat Bots and prompt engineering and and prompt injection things like that but I think this is a really underexplored topic that's also really important because computer vision is used all around us in our phones surveillance systems that are used for all sorts of purposes um you know airports now for for security monitoring um this is something important to to think about back over to you Shantel I might just stand here too because that is so distracting um so we're moving on to deep bakes uh which which ties in nicely to what Harriet was talking about and actually there was research that came out by Gartner recently that within a matter of two

years 30% of organ maybe it was I think 50% of organizations will no longer trust facial recognition as well which is really interesting so a lot of the context around deep fake has been in regards to scamming organizations or scamming individuals into giving away X number of dollars but what I want to explore today is the use of deep fakes to disrupt political and economic Landscapes which I think is uh an under disc topic as well so the world economic Forum ranks AI generated misinformation and disinformation as the biggest threat to our Humanity over the next two years which is pretty scary um oh I'll go back so yeah for for a lot of the organizations that we

probably work for a lot of larger organizations something like defect technology can be used to impact that organization's market value so for example creating a deep fake of a CEO of a publicly listed company saying something incredibly outlandish so that the value of that company goes down and then the malicious adversary can purchase shares at a lower price and so that's a really what we want to use deep fake for in this context is to actually deep fake the CEO of this casino and tell people to stop going there so that they can start spending their money at other casinos for when we target another casino next which is what we've been doing because we went to

Singapore we went to Cambra we're just on a casino hacking Journey so that's our attention so I've created this deep fake very quickly so it's not perfect but it's of the CEO of our Casino Nicole lidman if it plays which it probably but when we we do work we then have to go it's going to be safey oh it's not working it's okay I will I will skip over that um it's okay anyway we're just having lots of technical difficulties this morning so then how can we protect our organization well the first one is around tailored Education and Training so many organizations have generic training for all staff and we know that each employees relationship with cyber

security is very different and if we look at the MGM attack on the IT Help Desk having tailored training that the roles like an IT Help Desk know how to detect social engineering that they don't just need a full name and pieces of information to actually validate that this person Works in this organization and is who they say they are the second is of course being careful what we share on social media obviously social media is so right for for stealing data and being able to Target you so specifically the next one is don't trust WiFi even at home I don't do my online banking on my Wi-Fi and always update your browser because attackers exploit vulnerable

browsers and they can sit on there forever and you wouldn't even notice and then the last one is beware of tailored fishing campaigns I recently went to a restaurant and then the next day I received an email saying thanks so much for coming to the restaurant last night here's a 10% discount code and I looked into it and I realized that wasn't even from the restaurant so a lot of these online booking platforms they're either compromised or they're selling data or they're doing both and tailored fishing campaigns are becoming so much more rampant so really being careful of that as well I'll hand over to you thanks Shentel we're on time so I just want to

end by saying take AI security seriously a lot of the research that we're doing part of my work now is actually uncovering incidents related to specific AI security attacks that go undiscovered because people just aren't aware it's a real thing but it happens a lot more than you realize and it has significant consequences for organizations and their personnel in and of itself um and now we actually have laws and compliance um obligations through ISO and N standards related specifically to AI security so ensuring that AI systems are robust so if there wasn't an incentive before for organizations to do security of AI now at least they'll be fined to make sure that they do that um but I think it's a

great space to get in so as Security Professionals I really encourage you to donate your expertise to the AI space because as a previous data scientist there often is a surprising lack of intersection between the security and the data science disciplines and data scientists are the ones building these models but often don't know a whole lot about security principles so I think it's fantastic that you're here please lend your expertise to AI take it seriously help your organizations understand what to do and uh contribute so on that note thank you thank you for being Eno