BSidesTO2017 Nir Yosha

hi everyone but so statistics that's a very good time for you guys to get a nap no no that's that's not the most most interesting part but we'll make it interesting I hope and excuse me also for my voice a little bit um I think I'm getting a cold getting a little bit open and sexy yes yeah so that's that's not my well whatever so I'm going to talk about threat intelligence it's it's I just told the organizers first of all I'm very impressed with this besides I've been inaudible sides but also the order of the talks makes a lot of sense because keys here just talked earlier about risk from an impact perspective and I'm going to talk on the threat it's

a perspective and if you look at the risk in total it's probably made out of these risk made out of threat and also vulnerabilities which obviously we all aware of so and and it's talking about statistics I'm kind of an anomaly because I'm originally from Israel and I live in New York and I ended up talking here in Toronto so but I couldn't have this talk in u.s. because 12 out of 10 Americans don't believe in statistics I have added jokes but I'm gonna get myself in trouble so so what are we going to talk some I'm gonna I'm a threat intelligence engineer so um I want to give you my perspective on threat intelligence first and in a

disclaimer I mean threat intelligence engineer not an analyst which means that there are much smarter people denied that actually do the research and going into the dark web and and and monitoring the adversaries my role is to gather all this threat information and make sense out of it so I'm the simple stupid guy that depends on on those threat intelligence analysts we will look at how statistics can help us with rate data what's the problem today with rate information spoiler big data and finally I'll give some example of scoring again it's rate part of the score so like I mentioned I'm originally from Israel anyone from Israel well Shalom and I started my career actually in the Israeli army in the

Intelligence Corp so I'm very familiar with intelligence physical intelligence I moved here to the US fifteen years ago and worked for user behavior analytics security networks and obviously threat intelligence platform so just to get you guys know who knows by a raise of a hand what IO CSR okay what is it indicators of compromised correct a couple of more questions what's the second phase of the kill chain areas of a hand your loud it's actually weaponization so deliveries after is the third one happens and what's 50 P stands for anyone yes number three tactics techniques and procedures okay so let's go over a little bit those terms I'm just trying to get a feel out of the

audience from a an indicators of compromised perspective what is it it's very simple think about a list of informations collected from incidents that most likely indicates a breach so the two types of indicators host pace like IP address or host pace would be hash files file names registry keys those kind of things and then network are IP addresses URL domain and so on and so each one of those indicators can help us identify a breach and they could be related to each and every place for example in a kill chain so for example a an email address is an indicator that might relate to the delivery mechanism spearfishing an IP address could be related to the command and control a

hash file could be related to the malware sample and so on and so forth so those are little evidence just like in any other cybercrime that can help us solve a mystery now not all indicators were created equal anyone heard about the Pyramid of the pain right so this is a this is a post by a guy by name David Bianco which which is a friend by the way great guy and he realized that the adversaries have continuously trying to change their indicators so the defenders I would not be able to catch them however the higher you go in the Pyramid of pain the harder it is for the adversary to change those indicators so let me give you an example

hash values for malware's are very easy to change right it's just simply you can add a couple of bytes and you get a different hash value IP addresses are relatively easy to change you go higher when you go into the network and hosts which are the actual infrastructure used by the adversaries those are much more expensive and harder to change and finally pools are reused mauers have versions just like any other product and TTP's tactics techniques and procedures are the behavioral in the that we are straight intelligence analysts trying to find so this is a rough description of what threat intelligence is the way I like to look at it is that there's two groups there

are the good guys and the bad guys right the defenders and the attackers threat intelligence is about us the defenders trying to learn as much as we can about the attackers so that's why we started with those terms as TTP's IOC s we're trying to get as much information as we can and make sense of it so this is this is a known problem threat information is not necessarily threat Intel so the threatening information is everything that you get today specifically from open sources is very ro IP addresses with no association you don't get any context around them this is not intelligence there's not really anything practical you can do with them Megan though you cannot just send them into a

firewall and use them to block them on the other hand a threat Intel is when you get the context you understand where is it coming from how is it being used and what can I do with it and so there are a lot of vendors out there that trying to sell us dirty intelligence and they always each one of them will tell you he's the best they have the best threat information and the funny part is they're all selling you different things each one is telling you that this is the most important threat information but if you compare them you'll see that they're all different also using a lot of different terminologies and so the

problem the bay challenge is and yeah I decided to go with old movies photos don't ask me why no reason at all but I do like this this airplane that you're trying to fight and it's it reminds me a the concept of a threat intelligence analyst trying to fight all those data sources and make sense out of it there's a lot of noise a lot of things doesn't make any sense and how do you deal with that with this type of noise

so here's here's an example on court different sources coming in into a specific company the names are not really matter here but what you can see is the amount of indicators that each source is adding or removing with time so the blue here means that there was a inflammation that was removed sorry added and the the oranges was added so for example if you can see here that the top left chart you can see that there are a lot of spikes so what happened is a lot of the search teams are starting to look for an information-gathering it making a report out of it and sending it to you but it's not on a timely manner

so theoretically adversaries keep on working keep on attacking should be some kind of a constant flow and not something that looks like that with a beacon you know wanna cry might generate a peak but generally speaking that shouldn't be the flow was read information on the other hand you can see here in a specific on the top right a specific chart that shows a threat intelligence source that never retires their indicators they keep on adding to the list if this keeps on adding and by Nature threat intelligence indicators have a life cycle at some point they're going to be not relevant anymore here's an example might be a better example where the amount of retired and added

indicators are the same so so that's that's one problem that we're trying to solve here's the second problem so like I mentioned earlier a lot of great intelligence sources if I put them against themselves you can see here in the middle the diagonal line is dark which means that there's a 100% overlap between those indicates obviously you compare it to themselves but if you look at each one compares to the other there's a lot of white or light areas there what that means what I'm trying to say here is every threat intelligence source giving us different type of information so how do we know which one is the right one right and lastly the problem is with terminology there's no

specific terms that are dominating the market today there is a there's a an effort an open-source effort from oh as this CTI group which called Styx and taxi and speaks and taxis trying to do exactly that they trying to get a standard of terms and relationship between threatened intelligence objects so when you describe it either from machine to human or from machine to machine it's all in the same language because what you see here is we cannot really normalize the data we're getting different terms from different sources and we cannot cross correlate it because each one is calling the same thing in a different name right so that's a real challenge with the threat intelligence okay so now we're

going to the boring thing right statistics everyone doesn't sleep already so how's the t6 can help us so basically statistics has two branches right there's the descriptive statistics and inferential statistics one of the things sadistic scan helped us is just by helping us all visualize the data if you look at the threat intelligence feed it's pretty boring and honestly unreadable you cannot you cannot really get anything out of it so the first thing is you know making it more visualize or make it visualize the other part of it which is the more interesting is trying to make sense of it and potentially even predict behavior and associate attributes to specific fsor we're trying to find a mystery we're

trying to find out from that specific site who could be the suspect that eventually committed the crime all right so unfortunately I need to get you back into a statistics 101 and we try to do that fast so there's it an interesting term that crosses both the statistics and threat intelligence domains which is confidence right well when we describe current intelligence is always confidence associated with it and kids showed it earlier where you can say in a low medium high confidence on that specific impact same thing can go with your source now it's important to understand that confidence is not impact right when I'm talking about impact for example I can say it's gonna rain

tomorrow there's gonna be any Urich how you say that or we can sir or again tomorrow or it's tsunami which is another beautiful word and each one of them is a different impact however for each one of them you can have different level of confidence right and by the way there's another parameter here you don't see which is relevance could be a hurricane tomorrow in Texas but what do you care right here in Toronto so we're going to look at confidence impact was covered beautifully by the previous talk so how we look when we look at confidence from a statistics that's that's pretty clear it's less intuitive it's more scientific we're looking at confidence and literally it's

the area under the normal distribution curve that's that's what it is and anything that is on the side it's called the p-value and you can see here that if there's a high margin of error we don't want we don't like high margin of error right we want a small margin

okay so we know what's confidence is in statistics how we can apply it into threat intelligence so the main things that we want to calculate when we're looking at threat intelligence with regards to confidence will be what is our confidence in the source is it is it someone that I trust is it at first I evidence obviously if I saw it that that's gonna be high confidence right but if it's someone open source list that is somewhere for free in the Internet not sure if it's a commercial one that I pay for it traditionally maybe I'll give it a little bit more confidence otherwise they're going to get out of business right and the links between the big

object also should have a confidence right let's think about an adversary let's say ATP 28 and a malware let's say extreme rat what is the confidence of this malware associated with this adversary so the link that this network that you're trying to build has to have some confidence as well so this is an example of confidence so just taking it from from 56 101 and the question is very simple let's say I have a file right there malicious file and I'm running this on virus virustotal and virus total run this file against 20 different antivirus machines each one of them is independent and the results I'm getting is 14 out of 20 of them engines

are telling me that it's malware the rest are telling me it's not Maru and the question is what is the confidence that I have of this being an hour so I'm not going to get into by the way is there any mathematicians here where is over hand okay so so I'm not a mathematician not even close so not going to explain everything here I'm just gonna make some points and the idea here is the confidence cannot just go by itself it has to go with the margin of error because I can say with a hundred percent confidence that sometimes I'll get back into New York this week but if I want to be more specific

then my confidence is going to go down so confidence and margin of error has some kind of relation now using this formula of margin of error if I'm looking at and exactly the same ratio I just round it against 200 engines against 200 engines I keeps on getting to 70% ratio all of the sudden the margin of error is going down that makes sense am I too simple or too complicated with good so what is the conclusion here pretty simple and intuitive but statistics shows us that size does matter only statistics shows us slides does matter with statistics all right next let's look at probabilities how we can use probability when we're looking at threat intelligence the probability

is relatively simple when it comes to what's the probability of event happening it's the number of time this event can happen divided by the total events and if I have two events that are not related to each other just multiply multiplying one by the other however with threat intelligence there's a lot of dependency so we're looking at probability that has dependency

so this is the formula for probability with dependency anyone have seen this before so this is the notation probability of event a given P equals the probability of avail event a plus B divided by probability of B and now it really really sounds like a statistics professor or student better and so if I'm looking at at that I can create a scenario where I have a delivery mechanism and its location and I can look at what's the probability of me exploiting a mobile device based on that it came through a spear phishing for example just as an example right and then you can plug in the numbers and find out what is the probability now

another thing that you can do is if you get more information in other words you can find out the actual probability of P a and B based on actual experiment that's going to change the data and my point here I know it's kind of kind of mathematically may be a little bit complicated but the point is that with time we're going to gain more and more experience the physics is telling us the more and more information you get you can get better with your predictions very important part with with certain targets is correlation so the whole idea is to try to correlate right we correlate the threat with our seam we correlate the actors with malware with

correlate the actors with the actors it's all about trying to correlate so unfortunately not all the time it's a linear correlation and it's not easy but one of the things we're trying to figure out is for example we can look at our events coming into our seam and we can use correlation in order to understand why those events are coming or what's the chances of those event keeping on the same trend I'm not going to get into details of all the techniques just to mention theaters there's a simple moving average which is basically an average there's a weight moving average which gives more weight to the latest activities rather than the oldest ones which intuition which makes sense

right and then you can use exponential smoothing which very relevant for threatening tangent so from our experience all the exponential activities related to spread of a malware for example or the effectiveness of a specific Peaks they're more exponential than linear and then there's a seasons seasons are very relevant for certain intelligence because actors are people that lives in countries that have holidays and they keep on having some patterns you can see more activities or less activity depending on where they are in their calendar you can see when the hours you can see the actors coming in from Russia working in the Russian time zone you can see the actors coming in from China don't work on the Chinese

New Year we actually can see those indicators and the next step is trying to take all those data points and somehow make sense out of them now machine learning is the next step there's a lot of ideas behind how to use machine learning and threat intelligence honestly nobody is doing it today and nobody's doing it the right way at least a lot of people say when machine learning is a nice is a nice word for co-marketing people but it it's not really working today with threat intelligence why because there's still too much mass in the data you need to organize the data before you can really make sense out of it and to organize it

I suggest classification so machine learning has much a supervised and unsupervised we're far from unsupervised learning with machine learning we are in here where we can look at specific attributes relevant to the adversaries to the specific process in the kill chain or to our internal environment which is the best indicator right if I find an indicator within my seem like an IP address and that IP address is being accessed by one of my hosts at the same time this IP address shows up in a high fidelity as an indicator of compromise that's a problem all right we're getting close to the end and what's best all right so let's real close get into scoring so this is what

we do we take all the threat intelligence we duplicate normalize aggregate it and start to score it based on specific attributes like the attack phase malware families etc we look we don't look at the impact to the impact we already talked about we look at this rate based on the threat of geographical areas this rate of verticals so there are specific hackers that going after emphasize NIC or a certain specific hackers that going after financed specific hackers that going only after healthcare so if I'm in finance I'm interested in the threat intelligence related to violence and motivation there's different type of motivations basically you start to score even the capability of with adversaries how well

are there how train there what do I know about them that can be relevant to me locations like I said an attack vector and finally I come up with a score sheet and at that point if I have a threat intelligence platform I can start make it actionable I can start pushing it [Music] one of the things you want to do and Keith mentioned it in his doc as well is to use a feedback loop using false positives so if I'm having a score and I'm looking at a specific era let's say seven and up and from seven and up I think it's relevant for me I need to check and see in reality it's going to

be something like that they're going to be false positives country false negatives and I can use that in order to improve my model so putting it all together we're going to end up with an actionable threat intelligence which is driven by statistics cross correlated with the seam and specifically using probability confidence level and like I mentioned earlier scoring and we are more than getting close to the end I think we are at the end so those are just a couple of bullet points - how do we start it's starting to be very relevant for great intelligence teams to have a data scientist with the team in order to help cut through the noise and make sense out

of the data you still need to use your threat intelligence skills because you using your intuition will be able to identify what is important and what is not and finally the sleeper must awaken anyone knows where is this from doing so I like this this is this is kind of a side of my talk this is more of as a security professional industry trying to keep on get better that's why I like this this those besides meetings when we all sit together and think how we can make it better because the adversaries are doing the same thing they keep on changing their TTP's we should do the same thing so I you for your time have a great day

[Applause]