Patrick Perry - Exploit Kits Machine Learning

thank you can everyone hear me all right all right okay so I'm going to talk to you about um a little bit of machine learning today and uh how I'm trying to make that pertain to exploit kits um at the outset I want to be sure I clarify that I'm not really an expert in exploit kits I mean it's not like terribly complicated stuff going on there um but I think it's important to to say that um and as much as I'm not an expert on exploit kits I'm less so of one on machine learning just sort of something I've played around with for the last three or four months so um of course this seemed like a really good idea

three or four months ago but now now you get to see um what I in fact know about this stuff something I like to do when I give a talk is I like to give people a few options so they can sort of choose their own adventure this is I think good for the second one and then [Music] um I've got 20 minutes to get into some like pretty complicated topics um I mean 20 minutes 45 minutes there's never enough time when you start talking about things like this um so this talk is geared to be high level and hopefully give you some ideas um but definitely not to be a record of scientific fact

these are mostly my own musings from the work I've done the last few months uh as Mark said I'm a systems engineer at firey I've been there for about 2 and a half years now before that I worked on the Geer out in Michigan go Blue uh before that I worked as a uh host forensics consultant uh for an undisclosed vendor um out of out of Boston and if anyone else uh has been a consultant before then you might feel like I do where the the best skill I feel like I learned as a consultant was that in the time it takes to get a call to go somewhere where you have to show up on site and be an expert and teaching

yourself on the plane what you need to know to sound like an expert it's a really useful skill um and something that that helps because it's nice to not sound like a okay uh I worked for Chris Sanders uh who already spoke today and spoke yesterday he wrote uh applied network security monitoring um and also I'd like to give a shout out quickly uh to the rural Tech fund which he also uh founded which does a lot of great work um you can check it out at I think it's rtech fund.org is that right Chris okay yeah so check that out uh and then I like to throw in fun facts when I can a fun fact here is uh

you'll see Jason Smith is a co-author on this book I wouldn't go so far as to say Jason's my friend um but one time he did abandon me in a dark parking lot on the side of I95 in Connecticut as far as I know that was an accident okay did uh show a hands does anyone uh remember me speaking last year about the digital FF wall okay a few people yeah I remember remember a couple here um so I just wanted to quickly uh provide an update to that story uh because I've seen a few people around here that have been asking me about it because they were really curious to know about the exciting

conclusion so anyways I'm just going to go through a couple slides real quickly um and this has nothing to do with anything um but some people were interested so I'm going to share it quickly uh what we see here is my house is the blue one um in the distance and there's a big 5030 50 by 30 foot wall lurking in the foreground there um very sinisterly that was uh my my neighbor put that up last year because he didn't like looking at my house uh so someone uh this really good out of control I was like on national news and stuff with this story last year um anyways what I like to say here is you

should never kick a man when he is down and that is not something that Chris Sanders believes yeah thanks thanks Chris this is this hurt a lot and okay so anyways I gave this talk last year all about the thatford wall and what happened to me there and um if you go on Google you can find all sorts of crazy stuff about it uh but about 30 days after talking here last year um the sort of situ the situation resolved itself when the state of Vermont told my neighbor who's also almost the governor of Vermont at one point um that she needed to remove the wall or they were going to do it for her so uh the Tweet

up there was the last I saw of the fford wall and that's all I've got to say about that okay so now our story today really begins um back when I was a lad in high school I guess and what happened was uh at the time this was in the early 90s I really wanted to be uh I guess it was called like a hotspot doctor back then I wanted to sort of be someone that was like dropped into remote areas to respond to things like Ebola um I've just always had a passion for infectious disease so I went to college and I majored in Biochemistry well I started doing that I ended up switching to a computer science

um after a while but I'm actually still about a semester short of a biochemist degree um as part of like a a biochem education one of the things you learn a lot about is evolution now uh I uh personally think that evolution is kind of topped poorly um in a lot of schools um in our country for a lot of different reasons um so if you're not familiar with it um just wanted to go through a couple things here because it sort of relates to what I'm talking about I think a good way to think about evolution is in some ways to see some examples uh that aren't right so before uh Charles Darwin there was a guy uh

Jean Baptist Lamar and his sort of famous example of how Evolution worked is he believed in this uh idea of use and disuse so essentially he thought like the way giraffes sort of evolved were that they had to continually reach for uh food on higher levels of trees and things and it was just through the act of like one generation of things sort of a needing to stretch and whatever the next generation of that organism would uh incorporate that and everything would just sort of get uh would adapt in that way not really based on anything other than use and disuse the similar thing would be like uh he believed that like if he cut off his toe

and then his kid cut off his toe and then his kid's kid cut off his toe eventually you get a person that didn't have a toe there and that's not how Evolution works but at the beginning of the 1800s this was sort of groundbreaking because he was trying to think about well how do things change now darwinian Evolution um is predicated on natural selection and this is uh a really famous example that you'll see a lot it sort of explains the idea uh simply um and elegantly and what you have here is uh this is the peppered moth uh which had this this natural selection occurred in the industrial revolution um in England uh so late 1800s but what happened is

the peppered moth uh sort of started as it was on the left where it was white uh and black uh speckled and that was fine before the Industrial Revolution because it would land on trees and that camouflage would protect it what happened during the Industrial Revolution is that the trees um there was so much pollution the trees were covered with soot and Ash so that the uh peppered moth would actually stand out and they would get picked off by Predators much more easily so what happened with a variation in the gene pool is that natural selection uh sort of allowed the the progression of the peppered moth to become um more solidly black because it allowed it to uh evade

detection by its Predators um when it was on these sort of ashy trees so that's like a very clear example of how um Evolution sort of works one that's simple to understand um but the big thing with evolution is that it's very incremental right so it's something that happens overnight it takes several several several iterations or Generations um to be able to accumulate these different changes that occur and uh finally I like to show this just because I think it's so cool but um moths are neat because they can adapt really quickly to a new environment based on genetic variation and the relatively short lifespan so this is the Emperor moth and I'll ask you real

quickly can anyone tell what its adaptation is that makes it successful here yeah kind of makes it face yeah right exactly makes a face um and I actually have a better example that on the next page so what you see here this is the same moth but what's cool is that um it's actually evolved to look like the face of an owl because an owl is a predator of the birds that pre on the moth so it scares its Predators away um and it's even to the point where you can see it's uh the spots on the wings that look like eyes they're not perfectly round because ution isn't about perfection it's about uh being good enough so that you can

survive to reproduce another day which there's like that's a s of interesting idea but um what's neat to me about this what I just think is amazing is you can even see how um the that's not actually reflected light in the spots on the wings that's that's actually how it's evolved to look like reflected lights of of on uh eyeballs which I just think is amazing [Music] so I love science so all this idea of evolution and stuff I I've always I think about this a lot it's sort of like the lens I view the world through now and um what happened a few months ago is there's a guy in my team uh his name is duson and

I was sad for him because he was like on the 20-some iteration of of a rule he was writing for the angr uh exploit kit and if you know anything about exploit kits um the popular ones they'll change frequently and it's uh it's hard when you're writing rules to detect this stuff um to sort of keep up with it uh what I have here also are just some other exploit kits uh that were sort of popular in the last uh year or two so this is what really gave me the idea of I was thinking you know if these exploit kits are changing uh so rapidly uh as the sort of malware authors are trying to evade detection I

wondered if over time that is something that could be uh predicted sort of like evolution is it evolving in a way that I could guess how it would change before it changed and that was sort of the idea uh behind what I was doing here with my research uh I'd like to give a shout out to to Brad uh Duncan from malware traffic analysis. net here because that's where I get all of my uh well mostly all my samples from um it's just it's a great resource for people in our field uh if you're not using this site you should be um but it's just invaluable for studying exploit kits so definitely check this out um and just uh I assume that most

people here are familiar with uh exploit kits but high level a victim visits a website whose server has been uh you know compromised by cyber criminals victim is typically redirected through various uh intermediary servers the victim lands at a rogue server hosting the explo exploit kit exploit kit gathers information on the victim uh and determines the proper exploit to deliver based on the software the victim is running on their machine an exploit um is then delivered based on that information and if it's uh successful then a malicious payload gets downloaded and and executed that's the 20,000 foot view of uh exploit kits so I wanted to sort of take the idea of this an evolution and I thought

well this sounds sort of a lot like what I'm hearing about um machine learning um these days which is sort of the new buzzword the hot thing so uh again sort of high level but some some ideas and how machine learning works it's a it's a branch of AI um but it's a very I would say probabilistic based approach uh so I also think of it as very analytics is although I've not heard um everyone refer to it um that way but machine learning it relies on examples and experience so with what I'm familiar with there are three sort of different kinds of machine learning and it's uh you know people might might have some other

subcategories or whatever but this is sort of my understanding all right so there's reinforcement unsupervised and supervised learning uh to give you an idea of how reinforcement works think of Chess uh reinforcement is actually a subset of supervised learning um so when you think of Chess uh and you think about a computer game that is playing someone you know if at the end of that game someone uh if the computer loses then it can take that and say okay that's a bad outcome and learn from that that's sort of the idea of reinforcement if it wins then it did some right things it can also use that um as a way to sort of feed feed the

algorithm feed itself thank you uh so unsupervised is really you get uh the thing that stands out to me with unsupervised is the idea of clustering and that's essentially you have some data um and if you just I really should have put up a slide for this but I didn't but uh data might Clump somewhere on a graph in different regions right you might have like a think of something simple like file size or something uh usually numeric that's easy to plot but if you see clusters of things um today's algorithms with machine learning are really good at recognizing that and deciding okay that should belong in a a a special category from this other clump of things and that's

that's sort of the basis of unsupervised learning the sort of thing I was doing um with my example was what's called supervised learning and with supervised learning the way it works is you essentially already know the answer so what you have to do is you train the data um that you are looking to be able to detect so in my case if I've got a bunch of exploit kits or a pcaps from exploit kits I can run that through bro I can generate some data I know that with this particular data set um that you know these are malicious so I can use these to train um my model and uh I can also then take uh benign data and

sort of do the same thing to teach it okay this is uh this is stuff that is not malicious so what you do uh with supervised learning it's it's really a very the high level simple three-step process you collect training data you have to train a classifier and then you make predictions and that's really it so like I said I used bro uh to get HTTP proxy data and essentially what you're doing is you're kind of creating a simple function you feed data into it um and you get a label out and in my case my lab out was going to be this is malicious or this is benign okay so an example uh for

training my data this was done with the magnitude uh exploit kit and something you can say uh this is uh as as I'm training it I've got uh HTTP method user agent refer cookie vs URI length U string and label so uh I've got two malicious samples there and two 2 B9 and with today's uh algorithms that are available that really does all this stuff under the covers and you don't even have to be an expert um in this stuff to do uh you can feed this into uh into these these different uh Frameworks like s kit learning and sort of over time um with lots of samples train uh train your program to be able to

detect um you know what what malicious or what benign looks like so there's a couple of issues here um that I'll talk about quickly one of the things is when you're doing like machine learning you really want to keep everything um in terms of numbers because that's how sort of math happens right so if you have like a a string like get or post um you need to somehow convert that uh to a numerical value and there's different ways of doing that like you could say if you want to get okay if it equals get that's a one if it's something other than a get then it's a zero it could be that simple um in the case of magnitude

a lot of times with the payload you're looking for a 32 um uh a 32 character string so if you can actually look for the length of the string that can be a good numerical okay 32 that's a number I can use 32 is is uh going to be good anything other than 32 it's not going to be uh it's not going to be malicious um and I'll talk about the problems with your I strings in just a second okay so once we sort of collect and start to train this data um we need to be able to sort of uh have our program do uh ex execute this decision tree and while this looks

pretty complicated this is like literally a line or two of python code um that that's really pretty basic so I'm just taking those uh those classifiers that I showed you from before those labels uh the data that we feed in and we just take it one step at a time so if I give it a new uh a new sample this is a new log a new HTTP proxy log uh that comes in well first I check you know is it a get if it's not a get then I know that it's benign for this magnitude EK thing that I'm looking at right um but if it is indeed a get then it just goes on and it it Cascades

on down until if it meets all the conditions then we can say with a reasonable um amount of certainty that it is in fact malicious and that's just an example of how a decision tree Works um in machine learning and that's really what your goal is you're trying to get your data into a way that you can sort of give it you know yes or no questions that's very binary one or zero sort of things um real quickly I just like bag of words I think it's fun to say but we have this issue with machine learning where you doesn't work really well with strings so you just have to figure out a way to get those strings um to think of

them in terms of uh numerical so like I just used an example of a user agent you could do something like okay if I know that this is a bad user agent this one is not this is normal but if it was I could say okay I'm going to assign a numerical one or zero for every uh every like token in the string um there might be things that are sort of Contra indicators like if it's uh apple or something that you don't want in there you know that that's um legit then you might just assign that a zero in the training data um and then quickly Anaconda uh and python Anaconda is like pip if you if

you've used it it's a package manager um for python uh for scientific and math packages um so you can do simple commands like cond install a package and the packages are things like uh numpy scipi s kit learn which is what I used um for my work matplot live and pandas so all really good stuff and then finally um my results what I have here uh it just shows the samples up at the top that I tested and then what I found for my uh uh success rate on the bottom uh so for the rig EK is at like a 34% success rate angler Point uh 22 or 22% angler angler can go to hell and

die nutrino was better I think because of uh the good samples I had for that and magnitude um was sort of a unique thing it was a kind of easier to detect and I think that's why I had some better success there um so those were sort of the results that I had there are problems with that and I'll talk about I just have a minute or two left here so uh this isn't anything like scientific that I pulled out of this this is um this is my own thing that I I want to share as sort of like a learning uh a learning thing that I experienced here and I think um with with this kind of uh machine learning at

least as it applies to to exploit kits and things that are constantly changing what happens is uh you have this problem of exponential decay and what that means is if we have like false positive rate on the left 100% And on the on the uh x-axis we have think of it as effort that you put into um the problem well with very little effort 10% effort we can get the false positive rate down quite a bit but what happens is you have to put a lot of effort into continually get that false positive rate down and that's a problem when you're trying to do um I think detection at scale um for for a lot of people because one% of a big

number is still a big number so like when I'm showing rates of 30% 40% 50% success um it's a step in the right direction but I still think it's it's kind of terrible okay uh so I think one of my issues might have been my sample size is likely problematic just pretty small um I already made this point um again if you don't have a lot uh to look at in your board which I don't know a lot of security practitioners that don't have anything going on um then you know maybe 60% rates are good for you but I don't think it would it work for me um again this is my own thought I think

that machine learning is probably better for variations incrementally dissimilar patterns the problem that I saw a lot with expolit kits is that from one version to another there can actually be a significant change in the URI or something that I'm trying to key on and that makes it really hard um to detect uh I saw some white papers out there I think University of Toronto and Chicago where they claim to have 99% success in detecting eks um oh but what what I think this had to do with as much of anything was uh they were looking at specific examples where they had the source code to the malware and I think they were looking at different

stages of the exploit kits uh for instance I was very interested in looking at the final stage where we had malicious payload being delivered because I spent too much time in a console um and I know that it's really important to uh narrow in on that ladder stage for the guys in the console because if you start investigating everything at the the front end of an exploit kit uh possible uh you know exploit being delivered whatever it's uh you'll spend all day chasing this stuff down uh like I said exponential decay might be a deal breaker because you're putting a lot of effort in to kind of tweak things down to probably where it needs to

be and going back to the evolution idea evolution is very incremental um often large variance between exploit kit samples which I think is part of the reasons why it didn't work as well as I would have hoped so had to buzz through that at the end but uh does anyone have any

questions excellent questions all right there are any questions Patrick do you have a question that you can excuse me pose to the audience for a valuable prize and today we're going to be going um person who answers the question is going to get a rubber ducky we actually have two questions um yeah we're get a rubber ducky and uh we'll have an iOS um application book so if do you have two questions sure uh okay there were three kinds of uh machine learning that I talked about uh give me one of the three in the orange right here uh

okay you said you want one more yeah one more okay um let's see this is a good one it's a sports ball but I don't care um who's the greatest quarterback in the NFL thank you

okay I think