MITRE ATLAS: Exploring AI Vulnerabilities

right let's begin so it's estimated that by 2027 90% of digital communication interfaces will be poed by AI what I want to know is where's the 10% cuz I want to speak to a human it's uh awfully annoying when you contact customer support and it's a chat spot and if you actually do man manag to somehow find a phone number and you ring them a phone bot but as annoying as it is we as Security Professionals are task to defend these systems and this talk I hope will help you now how did MIT Atlas come to be well it's all started in 2013 the miter attack framework a wonderful piece of tool a wonderful piece of Kit close to a decade

later we got two new Frameworks defense for the blue team and Atlas focus on talk St let's play a game here spots the difference I'll wait no one one exactly and that's because M Atlas much like M defend is built directly on top of M attack and that's for a reason my attack works we all know how to use it I hope most of us have some exposure to it if not then you definitely should mitus great tool for AI what is AI with respect to miter and MIT frers AI enabled systems are this quote here feel free to read it but the tldr is that it's an algorithm which uses logic and if you can break that logic

then you can quote and quote hack [Music] AI here are some common attacks on AI myasus defines three such vors first AI access time now this can be either training or inference training is well I can't swear but there's a common phrase in Big Data bad in bad out so if you feed your model bad data well I have a bad model inference the infamous prompt injection we all know it we've all tried it it's fun sometimes it works it's fun to jailbreak llms like chpt too but there are malicious purposes too AI access points so there's a fairly standard stuff like digital API attacks there's also physical now this is very interesting a physical attack on an AI

access point would be for example well when's the last time you use chat GPT you ask the question what does it tend to do query the internet it searches Google Now we all know about SEO search engine optimization if a bad actor can have his website jump to the top of the search results then his website is queried and you will receive those results those malicious results there's also system knowledge so the the culture around AI so to speak it's very open and honest there's lots of documentation white papers this is a good thing for knowledge sharing but it's a bad thing for um for bad actes because they also read your white papers they also read your

documentation and figure out how to break your systems let's examine some case studies now what I love about myus is it's focus on case studies there are a great many but I've sampled two one has a good actor one has a bad actor this case study is an example of a good actor German Master students Ludwig ferdinant stomp I may have butched the name but uh moving on so this German Master student attacks math GPT and this happened last year by the way um math GPT used GPT 3 for his back end and it was executing python codes that was directly generated by gpt3 Ling further The Stomp was able to infer this send its malicious prompts

grab the host systems environment variables and the unsecured gpt3 API key VI prompt injection attack he delivers a proof of concept showing that he could have incurred excessive costs and executed denial of service attack so this flowchart here contains a number of tactics and techniques tactics are the why why did this act of do this thing techniques are the how so the first tactic reconnaissance amlt O2 this is directly mapped from the MIT attack framework the next tasic ml model access Tao is unique to M ratus and then you have the techniques and so on and so forth now we can see his uh his methodology here he he served for publicly available resources he did some iterative testing

which led him to believe the math GPT was directly executed python code and then he carried on with experimenting with the prompts eventually he was able to grab the unsecured API key and the systems environment variables he was a good actor because he kept in contact with math GPT and he delivered it his for results he didn't really act maliciously more of um Natural Curiosity which I think we all do now the next case is a very interesting one quite ingenious actually in the sense that wasn't very sophisticated in fact I believe any of us here today could probably replicate this attack and that's scary because of the scope and the scale of the damages two individual threat taxes in

China were able to siphon off $77 million us let me repeat that 77 million us over the span of two and a half years by exploiting the shangai tax authori mobile web app how did they do this well first they bought victim information of dark Mar well that wasn't first sorry first they bought customized lower mobile phones they then load those mobile phones with custom Android ROMs and a virtual camera app next they obtain software that could turn static photos into videos and add realistic effects such as turning of the head blinking of the eyes lighting and so on they then accessed black markets to purchase pii personally identifiable information of individuals within China and

abroad this this information included everything necessary to register for the Shanghai tax Authority including and this is key for this attack high resolution facial photographs they were then able to use these photo use that information to make accounts and to use those photographs to evade the machine learning based facial recognition Sur service by which they verify themselves as these victims with the shell company which they had set up they then funneled fake invoices and through that method were able to over two and a half years extract 77 million us I think we can all agree that you don't need a PhD to execute this attack it is an incredibly technical or sophisticated these Tas accessed developed markets developed

black markets for information they bought off-the-shelf Hardware off the-shelf software and this quite damaging attack are there any mitigations the answer thankfully is yes mitas gives us 25 such mitigations these mitigations cover three key areas ml machine learning cyber and policy each of these mitigations have an Associated ID each ID has sub techniques so one of the mitigations is to limit the public release of information I did mention this earlier but it is quite a difficult topic some would say that this borders on security by obscurity and we know historically that doesn't seem to work others would say that it's just best practice it's difficult to say as with all things these need to be

applied to your own unique environment key takeaways these hacks are often less sophisticated than you think think so again the shankai tax Authority attack there was no coding marathons it wasn't a huge team of nation state actors it wasn't a bunch of you know teenagers fueled by caffeine it was just 2 Tas accessing off the-shelf Hardware off the sh of software and employing an ingenious attack Financial losses can be huge $77 million us and that's just a tip of the iceberg this is just what's disclosed remember there are companies which are attacked which won't disclose such information and the thing with AI security is that we see how pervasive it is we see how it's

accelerating and the adoption of AI enabled systems into environments increases attack surfaces a framework does exist to help defend against these attacks it's called myatlas adoption will accelerate further it is inevitable so do stay ahead of the curve and do not neglect the security of your AI systems now the question I'm often asked is this is great and all but where do I begin well my answer to that is my fatless websit take a look at the case studies there are a number of them I think them 24 or so one of them may have some mappings to to your own [Music] environment take a read through the flowchart there take a look at the

tactics and the techniques app app it to the Matrix now I know we haven't discussed the Matrix we don't have much time to do that but you can apply these case studies to The Matrix much like you would the miter attack framework so you can visualize these tactics and techniques you can visualize the attacks and the attack chain and once you can do that you can begin to craft a theoretical attack on your unique environment and whether you do that using tactics and techniques or the Matrix ever's easiest as long as there's some some good outcome from there thank you everyone for listening if there are any questions please do reach out to me I'm available

on LinkedIn and my email is there I set it up just yesterday forward to my email um and a special thanks to James B my mentor who helped me to deliver this speech gave me some brilliant advice and to Tom blue a good friend of mine who uh urged me to to apply for this speech if he hadn't then I wouldn't have and last but not least thank you to all of you the audience your patience thank you