Running up Chat Hill: A beginner's guide to LLMs for Cybersecurity

month um okay so the um thanks to the people at besides Perth for inviting me to this which is my first b-sides talk ever um the main title is running up chat Hill a beginner's guide to chat GPT uh the alternative title is that AI won't steal your job but Stephen prandall just stole mine um yeah actually like our our talks despite I've never met Stefan Stevens never met me but I think there's an awful lot in common uh Stephens looked at it from very much a conceptual Viewpoint of AI whereas mine and why it's awful and mine is more about the practice of AI and why it in many cases is awful okay so the general structure of the

talk is going to be personally I'll start and talk about how do AI systems specifically llms work in in a little more detail then look at where the llms that are available now can be useful in the cyber security context and when and why they aren't uh and then talk a bit more about where the Field's been where the Field's gone and my guesses as to the future of AI inside security I'm particularly saying my guesses because it really isn't said enough um like any future scientific Endeavor anyone that's telling you where AI is going to go is at some base level guessing uh and that's a point that seems to get lost sometimes so who am I

um I um started I after my engineering green Computer Engineering I went and did a PhD in artificial intelligence specializing in biologically inspired Ai and Robotics uh yeah old manuals that cloud was the other alternative title for this talk um then I did 15 years of research combining cyber security and AI for defense Science and Technology Group in Adelaide about a year and a half ago I ended up living there and becoming a security architect for lydos all the opinions expressed in this talk are solely my own and do not reflect the views of my current or past employers so I'll start off by discussing Ai and large language models uh so AI problems um much more comprehensive definition

earlier from Stefan a little more of sort of the the working definition that has kind of Gone by uh in terms of research is really just problems that can't be solved by the analytic approaches that you normally take when you're programming answers to problems this is a very nebulous moving Target of a definition uh some of the general themes where this has been used problems that are too complicated to be reduced to a a programmatic response so particularly anything interacting with the real world where you're getting computer vision and processing visual inputs where you're interacting with the world in terms of Robotics and such are cases where you run into problems very quickly that aren't reduced to existing

algorithms and so you have to take other approaches anywhere where you're dealing with a human partner or conversely a human opponent is an area where a direct sort of a programmatic response isn't necessarily going to address what the other person will do in games uh once you get to any sort of game with any sort of complexity even quite formally defined games that combat that computational explosion leads to a problem that you can't just solve analytically or cases where the solution isn't really fully defined an element of you know it when you see it now um so AI Solutions um a lot of the solutions to this problem boil down to a very particular workflow and llms are a particular

example of this uh which is um classification or supervised learning what this really boils down to is you get a massive set of data and you get some people to label it and tell you for example the classic one what is does this picture have a traffic light in it you get a bunch of humans to go through and tell you whether there's a traffic light in it possibly as a way of getting into a logging into a particular service you then use some form of General statistical model where you can feed in the data and get a prediction out and you can use something to feed it lots of data and gradually iterate through until

you have a prediction that is sufficiently accurate um as a point that Stefan made before is that the sizing content of this data set is extremely important I'll be harping on that a little more later on but the general idea is that the algorithm is going to optimize around what it finds it the easiest solution which may not actually be what you were trying to solve the classic example is a tank detector that was meant to detect distinguish Russian tanks from American what it turned out it was doing was telling whether there was snow there because all the Russian tanks were pictures in snow and all the American tanks were pictures and grass so large language models follow this

model the data is pretty much every piece of the written word they could get their hands on whether they were worried about the license agreement or have it or not as is now coming out in court um the label in their case rather than have a human label because that means you have to hire people uh what they did was doing the next word prediction was to block out words in the text and test it on its ability to guess what the next word was the learning goal is to for starters just given any piece of text predict what an appropriate next word is and long-term goal of that is you do it enough times and you train it enough you

can get systems that are generating entire documents based on a series of input that's provided and the model that's already been trained so there are a couple of um sort of structures that are involved in this in the in the Transformer diagram that Stefan put up earlier one of these is feed board neural networks uh what these are is basically a line of summing nodes the hidden layer that have random weights as to the input to them the input is a set of real valued vectors so whatever input you have you have to digitize so a picture gets digitized into sort of individual pixel values you then have a hidden layer that weights the sums of the inputs they then

pass the inputs through to an output layer which is again a weighted sum um the technique it uses to learn is called back propagation what that is is you feed in one input the neural net pushes out its yes as to what the answer is you then check your the answer you already had and depending on whether you're right or wrong you adjust you adjust the inputs if it was right you reinforce the inputs that gave you that right answer if it's wrong you de-emphasize the ones that gave you the wrong answer uh over time you gradually adjust the weight so hopefully you're getting a reasonable uh probability that the input layer is producing the right output

layer um the other part that works in an llm is What's called the self-attention mechanism um as I said what you need to do in this is put in a set of vectors and what uh what llms do is they use either words or what are they called tokens which is either a word or a part of a word like half of a compound word they represent each of these as a location in a really big high dimensional space so they're all like points in and dimensional space and they've designed this space so that words that are close to each other have similarity in some aspect it's a very big dimensional space so it's not

necessarily just spelling it can be other Concepts that are captured in that tension mechanism performs similarity uh computations to connect those inputs together so to sort of take a couple of examples from Arnold Schwarzenegger movies um the silk retention mechanism if you gave it if it bleeds we can kill blank would pull from information that's already in the context so if it bleeds we can kill well it already mentions it so that might be it in that case you can just bring it forward from previous whereas the neural Nets are through the training are connecting words to The Wider set of sentences that have been trained on not that specific context so if we go to the Arnold Schwarzenegger

movie Batman and Robin uh he says what killed the dinosaurs and then the neural net could pull in okay well the ice age is an appropriate response in that case and it does this in sequence over a massive number of um massive number of layers I think 96 is the guess for at least one of the main like chat GPT producing millions of Weights that are dealing with I think about at a sort of Corpus of words far bigger than anyone's going to read in their lives and gradually through training and produce something that will predict the next word that fits um that being said um as Stefan pointed out the people that made it don't know exactly what's going

where a lot of the information about how it works is actually other researchers going in and doing research into how you can pull that information out uh this can cause issues uh one issue that can happen here is that uh the people who created it basically say oh this information there is no way X could happen with this like there's no way that it doesn't store whole images in uh Dali or stable diffusion and then some other researchers go and ask the right question and just pull out a picture from National Geographic that happened who have been used enough time since it's in there so llms uh generate chains of text to match the context in the data

set that text is coherent if it's the rules of language the grammar is good it sounds like a general I guess high high school level kind of speaker generally um and because this neural net has so many learned node weights in it ends up being able to store uh information about grammar and also facts that it can pull out to help put in an answer that makes sense now we'll go into a bit more sort of how llms relates to security so I I kind of assume just in case someone hasn't actually used it um chat.openai.com is how you can get access to uh chat gpt3 uh also being if you go to Bing it gives you lots of very

clear instructions on how to access Bing chat um and you can start asking a question so here is uh its argument for why hot files should have won the Oscar for Best Picture which it should have um but um so it's I've just asked that it's come out with a detailed bullet pointed if kind of generic argument as to why off I should one last picture Oscar um you can because it works on the context it's generated that includes its own work so you can ask it to refine Solutions do them in different formats you can tell it that it's got it wrong and it says sorry and it kinda tries to do it again but that's a bit

because of the nature of it completing sentences it doesn't really absorb it and you can input a whole heap of text beforehand um to guide the response um most of these programs are doing that anyway uh the controls on say not saying offensive things or that are built in to form the input are basically full sentences that have included before you ever put anything in which is interesting from a cyber security perspective as well so as just some quick examples uh here's kind of The Good the Bad and the Ugly of trying to get information out of these systems so I've asked uh what is the Australian information security manuals advice on selecting passwords so one version of

chat of um Bing has given me an answer which is correct but only a small part and kind of devoid of context um it's actually cited the government website it's given it said that passphrases are recommended it's suggested the 14 character long past phrase which is kind of one of the recommendations but it's not all of them it doesn't mention say if you go and read the ism that talks about single Factor versus multi-factor it talks about the different level of security of the network that you're setting the password for and also it's not giving you hard rules it's a set of suggested controls for a system that you do a risk assessment process that's all useful

information but they're not going to give you it then if you ask chat GPT chat gbt can't search the internet whereas being can so it gives you a very general response it says that really you need to read it yourself to find out as if that wasn't what I asked it for um and the security can involve over time but it doesn't give you a hard number then you get this other one that asks where I've drilled down and asked is there a specific minimum password requirement and it said oh it's 13 alphabetic characters according to ASD and it cited a website that isn't a government website that's giving you the 2016 recommendation stripped of all context and just told so

in this case it is very confidently giving you a wrong answer and it's cited its wrong answer do not ask chat gbt to cite things because it makes up the sites um and there's if you haven't read about it already there's a lawyer in the US who used it to generate entire case histories for a case and it went very it is currently going very badly for him um the nature of llms is you can also ask them to generate code for you so here's a quick example I just did three circles in paint uh can you write some code for me to tell me where the centers are um I then asked it to redo it

immediately so it just gave me the coordinate rather than giving me a new picture um so it happily gives you a set of code that gives you the three centers of those um looks good then you run it and it gives you oh I didn't count but probably about 15 centers of circles and it got two out of the three um sorry yeah um I told it oh I found too many circles so I gave me a big description of why this could have been a tricky problem thresholding stuff like that she didn't really apply to this issue because I just painted circles um and it gave me a whole nother approach by code uh this one gave me one of the circles

right but then broke and I had to quit the terminal um so I guess this is an example of what it does in terms of code if you ask for say in map scripts and stuff because the Corpus has a bunch of information on how to run in map and stuff it'll give you pretty good guesses as to what how to run those scripts so do the as you've seen they give you it very cheerfully gives you answers they aren't necessarily accurate and Stefan hit a little on this but yeah the the thing is that as we've said um what you train on and how you design an AI system is particularly important um neural net systems are sometimes

described as brittle in that if it's matching the data and it's matching the problem it'll do great as soon as it deviates outside of the proposed problem or the design or what the data set covers it falls off extremely quickly um I don't personally um people use the term hallucination but I personally don't like that because for me um I guess here's the example uh I went out and played Paul with some people last night and say I'm playing pool and I hit the cue ball and it pops off the table and into someone's beer saying an error from Chachi he's like a Hallucination is like saying me doing that and then saying ah my elbow slipped

and that's why the Cuba went off the table when everyone knows it's just I'm a bad pool player and there's a fair chance to when I hit that ball is going to end up in someone's beer it's it's inventing an extra explanation for something which is actually intrinsic to the model the model has kind of a hit rate is gonna hit things it's Gonna Get Right a bunch of the time but it's also going to get it wrong a bunch of the time in terms of coding as soon as you get into any coding that involves mathematics or any tracking of complex State um the accuracy is going to start to break down pretty quickly there's

already a couple of good studies and Technical reports out um someone evaluated for use in detecting vulnerabilities in code which you can do you can show it a bunch of code and say where it thinks that there are potential vulnerabilities in but they found that I think a dummy classifier where you're doing like binary classification is uh just as good as using chat gbt um another example people compared the effectiveness of asking chat gbt something versus stack Overflow they actually found stacked overflow was more accurate in their study but the people using touchy PT trusted it more because it sounds more confident now all these issues regarding accuracy are also kind of apply to AI art systems

like Dali and stable diffusion because they are fundamentally doing the same thing except rather than predicting the next word they're predicting the adjacent pixel and uh and they're trained on a massive body of work of art similar to Chachi BT being trained on a massive body of text and so you get some of the same similar issues so the issues you have that Chachi BT uh can't really do maths that well uh or that it starts to get errors as you go on or complex code appear in terms of the perspective and shadows in a work so this is off uh Dali's web page this is one of their examples of what their system can produce and just you

can quickly notice that there are Shadows on the front of the horse but no Shadows on the back of course and if you look particularly if you want to see whether want to guess whether a piece of art is AI generated looking at the shadows in buildings and where they've actually matched the shape of the building is one good way to do it because um because these systems aren't that great at the map it's required the issues with fact Rapport and analyzing the world come up in terms of the anatomy as you see this one this astronaut has these weird mittens the Bridle doesn't work right the horse appears to have some sort of badge or

label on its side which is strange but um yeah the anatomy thing that I think lots of people would have seen that you can if you want to know uh if a picture of a person is generated by AI you check if the fingers are the right number and uh and they're in the proper configuration kind of like you're in a fantasy story and like you're dealing with some sort of like illusion um and also things like three-dimensional objects don't match the space that they're in uh houses don't have the kind of doors and windows we'd expect at the house all the all the things that humans learn via experience greater experience with these objects

rather than just learning from art or words so that's a bit of a coverage of some of the use cases of um llms as they are now so the last bit of the talk I'll talk about the general field and the future of cyber security AI so one thing with AI problems in general and it's one thing that ties into a lot of the predictions that you would um encounter in The Wider Media or on LinkedIn or on the discussions of these kind of things um is that as I mentioned before AI techniques are really dependent on the data both the size and the quality of it in terms of its pertinence to the problem

and once you get outside of the bounds of that the performance falls off really quickly um tailoring gin techniques um so all of these things like neural Nets and stuff like that are General techniques that then tailored by data to a particular problem and that's difficult um in it's as difficult for AI as for any other situation where someone has a proposed General solution to a problem that has never actually gone into field and tried it on the actual problem they're solving which I think in in many Industries if you're dealing with oh well this is this solution we use is used in another industry and I'm applying it to yours there's some work fitting it to that industry a lot of the

time and a lot of the predictions around AI seem to assume this is Trivial in spite of the wealth of evidence both in Ai and outside but this is a really hard problem that sometimes doesn't work and the reason I've got that picture up is a particular example of um of machine learning which is quite recent and hilarious which is that there was a group working with the Marines in the U.S to do a visual recognition system to detect humans walking towards it they spend a whole bunch of time they got a group of Marines in to um to train watching them walk and generating detectors that could then pick in any piece of film whether there

was someone moving around and I mean um then they went and let the Marines on the final day testing Try and defeat the system and um they did a range of techniques uh one of them did somersaults rather than walked and that the camera didn't pick that up I think uh someone actually got a bunch of brush a bunch of leaves and like walked slowly holding them like a tree and a couple of guys a couple of Marines who I will buy a beer if I ever get to meet them got a cardboard box and hid under it like in Metal Gear Solid and look towards the thing and the camera didn't notice sorry so that's a a great example of how it's

designed for this very narrow problem which can break down as soon as you're trying to apply it to a broader issue um another example uh one of the sort of big uh about 10 15 years ago IBM had a system called Watson uh which won Jeopardy uh against a couple of the the best human players Brad Rutter and Ken Jennings took them on in TV Jeopardy was able to analyze the questions come out with answers um Watson was then sold um to a lot of companies as a general solution for AI like for using AI to manage knowledge and they did particularly a really big focus on the medical field and if you Google the

stories about this it did really really badly because it turned out that the data inputs for the medical problems were quite different to whatever it had for Jeopardy and that the answers required were quite different to just answering a quiz so they were doing very badly I think by about 2021 they'd started to get at least a little bit of Headway there but then Chachi and stuff came along so I'm not sure how that would have gone um so uh going to um cyber Ai and and here here I will slightly diverge from the previous speaker uh in that I think there are like some bits of hope uh I agree that um Stefan talking about

um GPT enabled waifus is terrifying um in all sorts of ways um but um one thing about the cyber security problem domain is that it's actually by the standards of all the domains you can apply i i a I2 extraordinarily hard um the way most AI techniques work including llms is they apply General model to a massive set of data and particularly for llms they're looking for the most common thing the most um something that is broadly acceptable and trusted um this has some other issues with knowledge recognition that I didn't touch on before um if you have one authoritative source and 10 General people discussing something and you ask Chachi PT for the result it's going to tend to Veer

towards the general answer rather than the specific correct one but the issue as well for cyber security is that finding the general answer um particularly in areas like exploit development and stuff what you're doing is not trying to find the general common Nazi you're trying to find where the general answer breaks down you're trying to find edge cases uh and that isn't something that these kind of AI systems are designed to do they're designed to go where the data is not where the data isn't I think when you look at exploits like Ro Hammer that are reading Ram based on physical pollocation of things in the chip that's not something that you're going to pull out of computer

data you've got to find the data that tells you that before you can then have any AI that's going to be able to even approach that in some way um also because data is so essential in cyber security both attackers and Defenders are dealing with restricting data anyway attackers lie all the time because that's what works um defense in response limits access to data when defense occasionally doesn't lie themselves in sort of deception based techniques so a data-driven approach has the trouble that it's in a domain where everyone is trying to manipulate and restrict information and really the sort of the the ultimate Point here is that any cyber AI system is going to be targeted and manipulated

because that's part of cyber security uh there's already I think a good 10 years of research into adversarial machine learning or AML which is the approaches of how you can undermine and deceive machine learning the classic example for for most of that period it's really focused on visual systems and audio so here we have a picture of a duck or picture identified as a duck by a visual recognition system put in some noise looks like a horse um ditto for audio and there are already a whole there's already a whole heap of work on how to deceive llms um llmattacks.org is a website discussing Research into automatically generating counters to llms also if you want to try it yourself again

AI thanks Michael for pointing that one out to me is one where you can can try it yourself and there is a a little a small LM that is meant to not tell your password and you can use more and more elaborate prompts to get to get that password there was a particular attack could do anything now which was a big block of text where you could basically bully an llm into telling you whatever it wanted rather than what it was required to do and this is this isn't the natural most domains that AI goes into people designing AI cars aren't designing it around the rest of the road trying rest of the drivers actively

trying to drive them off the road because that wouldn't happen and there are other controls around that whereas cyber security is an area where trying to sort of trying to undermine other people is what a fair number of people get paid to do another big issue with the idea that um that AI um can take over a lot of the jobs people say it does um has a really great example in autonomous truck driving um autonomous trucks were meant to be replacing human driven trucks already if you look at say a lot of the these uh papers that get produced where they analyze how many jobs are going to be taken over by AI they have a habit of

saying these whole blocks of Industry will be taken and one of the ones that was a really big Target was truck driving uh when people when waymo really started gearing up its work on autonomous car driving sort of truck driving was moved in there as oh this will actually probably be easier and there's a lot of money in it um but waymo uh who are really the preeminent um sort of company in this area has just announced about a month ago that they are basically putting on hold any autonomous truck driving research um it's just the same problem scaled up fundamentally the they still have a battery of sensors that can do it there's um potential scope for very

large profit um they haven't gone out and told people why um as someone who's worked in this area I have some hunches and I think it ties into basically how truck drivers are like scizos um one of the fundamental hardest bits in autonomous driving is high speed merging so merging on highways so firstly it requires a lot of assumptions about what other people are doing um but I think fundamentally one thing that is underplayed as a problem is that as soon as you're driving alongside other cars at 100 kilometers an hour there's a bunch of risk you are now accepting um there are there are so many moving pieces and other people can move faster

or break or whatever that there is no anal there is no guaranteed solution that eliminates possible risk thanks um when you build that up to a large uh a b double traveling at that speed then that risk is going to be even far far is going to be far higher with a massive potential impact on anyone else at the road if you get it wrong I think potentially when you're dealing with AI problems like this you run into the issue that there is no real way of eliminating that risk with an algorithm um there is always going to be a risk that that goes badly and that's both the problem in terms of this the the micro level of Designing a

system that's going to work in it and it's a problem at a macro level for a company people have talked about the best thing about one of the advantages of robotic drivers is that you won't have accidents anymore the flip side of that is if you design one of those systems and it does crash then at some um that robot was doing what it was designed to do and you're accountable for it so one issue that these companies haven't really been able to grasp yet that humans do because we have it as part of our jobs is to accept that level of risk and the consequences for it and truck drivers as well as accountants

lawyers electricians up to c-suite have as a key part of their job not only am I supplying advice or doing a task but I'm also standing by my work if it goes badly I'm the liable person and I think that's a big issue that robotics hasn't actually grasped I guess it could grasp it if the company assumed all the risk but they really don't want to do that I imagine what I guess what they have done in terms of that is basically put the risk back on the human driving uh or just avoid it so I just do some final conclusions and let everyone uh let us get to the wrap-up so just some quick takeaways for today uh chat

gbt is good at what it is designed to do which is analyze writing and produce new writing on a variety of topics well written um chat gbt gives you the common answer whether that's good or bad depends on your question it has built in that it is going to get things wrong a fair bit of the time because it is not built to be correct it's built to write good English which are different things and so if you are going to use it in any way that has to be introduced at the start or the end um whether it be quite better sort of pre-design or by having someone accountable at the other end um and don't put proprietary company

data in it in case you're thinking just very quickly takeaway to the Future cyber security isn't particularly nasty domain for AI which is I guess some hope if you want to push back on it um pumpkin risk is a really hard problem that AI companies will find very different to handle uh just to briefly get on my soapbox claiming your invention will replace an entire field of endeavor like they did for radiology back in 2016 is super lean even if it were true and when you're just kind of guessing and it turns out you're wrong that is pretty horribly mean um tailoring AI to domains is much harder than people let on and one other

little sign of Hope is all the well-established cyber security practices like segmentation patching multi-factor things like that um AI is not a wizard it cannot magically work out a way around around really well tested and verified techniques um so those techniques will work just as well against AI as well as the work against command um yeah and just briefly for a more detailed discussion about llm's work uh this has technical article a jargon-free explanation of how AI large language models work is a really good one and um or more discussion in detail of the problems with AI predictions uh Rodney Brooks was head of the MIT AI lab and computer science lab one of the leading

lights in autonomous Robotics and he writes very regularly about a lot of the problems with um AI predictions from someone who is very much inside that industry that thank you very much any questions if we have any time

thank you so much because I forgot to talk about it it's a really nasty problem it's a fundamental issue in the data sets and anyone talking about existential well some people in AI are talking about oh well we should worry more about the existential threat we shouldn't we should worry about the actual problems that are in it here and now and those are those problems yep um

what's revealing on if that thing is

I'd say I would say the person well I'm not a lawyer I think the lawyers that's it I would I would imagine that the person if the point of the triangle was to mess up the speed limit and the speed limit messing up led someone dying then I think the person who put the triangle on there would probably be answering some questions from the police anybody yeah I am yeah yeah I guess I think the law would try and work their way around it but yeah yeah I think in cyber security it's a much bigger issue like I think one thing for that issue is that it's not like people aren't going around trying to like crash cars that often uh because

of all the negative results for it but cyber security messing with other people is kind of what happens all the time so

that's how I felt like about halfway through your choice

um how do we get to the next AI rooms in the past tremendous question uh whoa I think I guess I'm a kind of a testing and measurement walk so I think more focus and more publicity for actually testing these systems and showing the problems um apart from that I think the more in I guess particularly in our field the more people start to actively break them that might and the more visible that gets that might accelerate it a bit keep reaching for that dream

s