Gordon Mackay - Murder Mystery - BSides San Diego 2017

um we're already a touch late and I understand that we have happy hour after this anyway I thought I'd get us happy by playing a little game which I call a murder mystery but first a little bit about me I'll spend 60 seconds uh my name is Gordon maai I'm Chief technology officer of company called digital defense uh we do vulnerability management I've been doing that for many years I'm a software developer by we also do penetration testing so I've learned a little bit about that through through osmosis and then security awareness training um so enough about digital defense this talk isn't really about them but then again they made my trip and so I thought I would share that

um so I was born and raised in Montreal Canada I speak English French Ching learn some Spanish right now um I graduated from Mill University if you're familiar with that in engineering and uh started off my career in software development uh in Brockville Ontario uh for a small company called Northern Telecom Nell you've heard of it uh they they're out of business now but they're a telecommunications company and because I was in Telecom I thought well um where's all the action going on where's all the opportunity and uh I quickly realized that was in Dallas Texas so I moved my family my small family and myself to Dallas and uh I started working there for a company

called digital switching Corporation and uh doing switching processing those kind of you know types of software um and then Telecom started going downhill around 2000 time frame maybe a little bit before then uh and then towards 2002 U I was on a very large project and towards 2002 we were we were coming from 300 people to 10 um and then anyway long story short I was being laid off I started looking around and I found this company digital back in November 2002 um the vice president at the time who's now our CEO said hey Gordon why don't you come on board and uh help us out with with this new incarnation of vulnerability management and so I did I

learned about it learned about security and I learned a lot about vulnerability management specifically um an issue or a challenge within that domain that uh that this talk is based on um so I'm going to share a little bit of research um sort of a mystery if you will in my mind it's a mystery as to why people um it's it's funny because I've given this talk at different venues before in different formats and my original format shared uh sort of a step by step you know little bit of background Etc until finally three qus of the way through in the presentation I shared here's what the problem is um and then some folks said you know that that's almost like a

murder mystery why don't you share that in the form of a murder mystery hence that's what this is has anyone played clue before the traditional game yes excellent okay so I mean I'm not going to explain the rules but you have a set of characters these are the this is the classic you know game set characters Mr Etc we got a set of weapons right the wrench the Rope knife revolver Etc and then you have a set of rooms and uh you have a board where you have rooms laid out and you're a player and you move into a different room and uh essentially what happens is is at the beginning of the game you split the

cards up into three categories the run the weapons and the people you shuffle them up turn them around and you take one of those cards from each category and put it in an envelope and that envelope is the mystery right and so your job as a player is to ultimately come to a point where you can make an accusation that's correct right it's for example Mrs White committed the murder uh in the ballroom with a knife and if you're correct you win the game right so this is sort of modeled after that all right so um overview we're going to go over a crime scene I'm going to share a use case to set the stage uh we're then

going to talk a little bit about what I call detective tools a little bit of background of vulnerability management scanning uh methodologies we'll cover that um and we're we're going to get into circumstantial evidence you'll see what I mean by that um specifically tracking endpoints or host across time is the challenge we'll talk about Mrs gard's testimony uh we'll also talk about a study that I perform myself and my team perform uh to track endpoint characteristics you'll see what I mean across time and how often do they change it's relevant and then who done it this is the point in time where I shared here's who actually committed the murder what's the problem what's what what are

the details behind this problem Etc victim's consequences and then finally avoiding future TRS now as I go through this if you if you guess it lift your hand up and I'll call on you and uh and if you say hey Gordon you know explain explain the problem you don't have to actually say okay it's Mrs Garett who committed the murder in such and such place but just explain here's what you're talking about Gordon here's here's what I think this problem is and uh and if you guess this right you get recognition which is pretty good I don't really have a prize but anyway this slide's going to be quick it's just it's just a slide to share

we're entering a crime scene years ago right years ago most security Technologies or all security Technologies perhaps with the exception of sim operated within their own silos right so they were solving specific use cases so vulnerability management is one of those um identity access management data loss prevention Etc I'm not going to name them all but essentially they they would operate within their own silos and they wouldn't communicate information to to those other silos to solve risk use cases But as time as evolved we've evolved in our thinking right as people um and also the attackers have been more advanced as time has gone on so we've we've been forced to move into and to think about how do we handle more

Advanced use cases and so we've shared we've started sharing more information within this what I call an integrated security ecosystem and so what I've done is I mapped these different characters to the different security Technologies right Mrs gett to vulnerability management Mr Green to IM Colonel uh mustard to blp Etc one of these Technologies is the culprit of this and refer to and what's happening is one of these Technologies is sharing information into the security ecosystem in such a way that's it's often poisonous it's inaccurate and we'll talk about why and it's causing for example your Security leaders your cisos Etc to drink poison so it's not just the fact that the security use cases is is inaccurate due to the poon

data but the people who are making decisions based upon these disinformation unfortunately they're making they're making right decisions but based on inaccurate information hypothetical risk use case um now please don't go out of this room and share Gordon just released the zero day because that's not what I'm doing I'm sharing a hypothetical risk use case just to set the stage now imagine that we just learned that a new uh a new vulnerability has emerged for Apache Apache web server and that this vulnerability only impacts releases 2.4.0 to 2.4.2 4 but not the most recently released version of 2.4.2 five doeses that make sense what would you do as a security professional to understand you know to

to handle this information and to understand what your risk is in your organization well certainly one of the things you would do is you'd want to query your vulnerability management system to understand where do I have deployed uh the a an Apache vulnerable version throughout my Enterprise and you you likely could actually query your system because these vulnerability management scanners sure they detect vulnerabilities um but they also detect you know software inversions Etc so even though your vulnerability management vendor may not have just emerged with a detection for this issue you probably can still query your system to understand hey where are these vulnerable versions but likely you'd want to run a recent scan anyway right

so this diagram is sharing uh right so on the I guess from your perspective right hand side of the diagram sharing a network diagram that's showing what are the end points the hosts that where Apache is installed at the vulnerable version so the Red Dot represents the vulnerable version the black dots represent that don't have that issue and you be like aha I know where my present risks are this is great but really that's not enough a lot of Security Professionals would stop there but there's more information that you can Le and what I mean by this is and this is important because it brings in the dimension of time if you look into the past and ask yourself well what

if like were there instances where I had this vulnerability on hosts even though it might not be there today if what if it was there in the past is that important and it's it could be important because the bad guys and bad girls might have known about this issue even though you didn't know and they may have attacked you already right so you may have you may have had past instances of vulnerabilities even though they're not there today and you may have been compromised in the past and perhaps your incident response programs didn't detect this so this is a pretty cool use case right I have other use cases too but this is a pretty cool use case that

shares vulnerability intelligence where the past is important if that makes sense and you can feed these candidates right so there's still one candidate that's the same across time but there's two others that were there in the past they're not there in the present and you can take that information feed it into your instent response and investigate you know have I been compromised probably not but it's good information right so this is just sharing the the value of time if you will from a perspective so at this point in time of the game you're you know you've you've concluded Mrs Scarlet is the guilty party you don't know the other two components you don't know the weapon yet

you don't know the rooms but you know Mrs scari vulnerability management or I'm sharing this with you vulner ility management Technologies and uh many vendors suffer from this uh suffer from a flaw if you will or a limitation uh where they're feeding information into your security ecosystem in such a way that they're they're sharing inaccurate data often or what I refer to as poisonous data okay does that make sense so I'm going to get into a little bit of uh what I call Mrs discarded or vulnerability management weakness or challenge so I often use this diagram where uh the challenge is assessing host across time so if you look at the bottom part of the diagram

and I often fantasize that I'm Doc in the movie you know Back to the Future Mary on the Whiteboard know I don't know anyway so in the bottom part of the diagram these are the real world assets and they may be virtual but essentially what I'm sharing is you're running specific software using specific os's on these assets that you can touch I know it's here right from from a mental perspective but and the top part of the diagram we sharing with you this is how a vulnerability scanning solution actually perceives these assets at different points in time right so the red asset of you know for a scan at week I'm just using that as an

example uh sees it in a specific fashion and its job is to correctly correlate how it sees it at a different point in time in order to satisfy that use case that I just U that I just mentioned so that's a challenge I think I think you might be gleaning what I'm sharing as far as the claw here so let's talk about vulnerability scanning Technologies there's I often share it this way there's more than three scanning methodologies but I'm putting up here three different scanning methodologies that pretty much cover the bases right I'm not showing passive scanning if you're familiar with that but essentially let's talk about these so there's agent based scanning this is

where uh the vendor comes so so first of all vendors provide at least two sometimes even three of these Solutions and as clients we'll use a combination of these for various reasons so let's dive in agent-based technology scanning is where uh you have to deploy in some form or fashion a program or an agent onto the endpoints onto the hosts and of course there'll be a centralized scanning management solution that messages these endpoints likely to say hey I want you to start scanning and they'll they'll run on the endpoint itself and collect information and determine vulnerabilities Etc it's got pros and cons of course it's on the endpoint so it should be very accurate but the problem is vendors

don't typically emerge with uh an a program for every single type of endo out there right Windows of course they cover often many Linux systems and even still deploying it is cumbersome right and costly so uh it's great but often it's used for cases where if it's easy to deploy great um or for um for um cases where you have or you know that you have uh high value assets right this is where my data is so I need something that's more in death remote unauthenticated base scanning this is a this is a type of scan all vendors offer this where the scanning engine is it it's not on the endpoint it's remote to that end point

right it's running and it will send messages it'll you know pain sweeps Etc to try and discover are the are what are the assets that are out there in the ranges um what are the ports that are open what type of os are these things what are its vulnerabilities Etc right and I use the candlestick because the Candlestick is like a flashlight so it lights what it can see so typically the light is not on the end point if that makes sense I had to use one of the weapons I only had so many I didn't have a flashlight um so that's remote unauthenticated base scanning very easy to deploy or easier than the others um

in fact the endpoints don't even need to know and they don't typically know that they're being scan right um and it's quite accurate the problem is of course it it it doesn't get as much as say agent based or credential based which we'll talk in a second because it's not on the end point but never the less it's going to get you enough information that overwhelms you anyway to the point where you have to prioritize you know what fixes you want to do and then credential based scanning is sort of a mix between the two and that is where you have a scanning engine that's remote but it will authenticate using credentials that you have to set

up to these end points that it's scanning uh and it it would then detect pretty much almost as much as agent based will detect if that makes sense um so it's very accurate uh but it does come with overhead right you have to administer credentials you're not going to want those credentials to be long lived um and then the other point is it's it doesn't cover all devices right okay so that's that so those are the three different vulnerability scanning methodologies circumstan cial evidence so before we get into this most organizations what they'll do we serve as many organizations so I know this um what they'll do is they'll they'll set up unauthenticated scanning to sweep

their networks by and large and then on different sections of their Network they'll they'll either use credential based or agent based so to supplement that but by and large unauthenticated remote scanning is is the technology that's used right circumstantial evidence so one question that you may ask is if you recall the the diagram the previous diagram where I where I had the assets and the scanning that's happening at different times and how how it's related is how do these vendors actually what what do they do to actually figure out hey the asset that I saw at one point in time is the same one as I see at a different one so I have two

different scans I have posts that have been discovered in one scan posts that have been discovered in another scan how does the system know keep in mind the the scanning methodology that I'm talking about is remote unauthenticated right so it's not on the end point it's it's remote and so all it could do is in terms of relating those two and determining ah the red asset is really the red asset is it could only use what it can see and what is it that it can see in terms of characteristics is there anything that it can see that never changes and is permanent and that's that's the real question and you know a lot of people

will say well things don't really change that much I mean servers for example I mean how often could their IP addresses change yeah maybe a little bit but probably not that much so we're probably good um and anyway so the vendors out there use a VAR various different characteristics to match post that they see from one point in time to another they'll use IP address various types of post name Mac address is great if they can get it um You Can't Always Get that I've received a lot of questions on this we can talk about it post type and other types of characteristics one of the largest vendors out there has an algorithm and I'm sharing

with you this is how they solve this problem they use one of the three possible characteristics listed here IP address DNS host name or bias host name so you can go into the vulnerability management and administration as the administrator and specify from such and such range to such and such range I'm going to use IP address to track that host but on a different range maybe I have clients there right maybe maybe I'm in a DHCP range I'm not going to use IP address I'll use uh net hosting Etc so it's flexible in that you can choose which characteristic you want to track on but it's limited because you only have one characteristic um and so is that enough

that's the question does anyone want to venture a guess terms of what I'm talking about is terms of this murder mystery challenge what the problem is that's okay okay so my team and I performed a study to understand how often do these characteristics that these vendors use to track how often do they change in it environments and you could say well maybe in the past maybe not as much nowadays and virtualization and bring your own device Etc I wonder what that is right so we performed this study we can talk about the logistics of the study but essentially we looked at very large time frame keep in mind we're a cloud-based uh vulnerability management system right so we could actually see

all this data in our data center even though it may be encrypted and anonymized we can still Gan a lot of information from it and this is partly how we came up with this stud and uh what we've determined is across a three month time frame and so what we did is we broke down the study into different device types client machines server machines printers Etc so I'm just showing you here a cross-section of different device types I'm showing you servers and client machines so for Server type machines database servers web servers you know application servers where you would think that IP address doesn't change that much or at least maybe you know DNS the characteristics actually change

at surprising rates I was surprised by this we actually performed this study multiple times just to confirm for example IP addresses of servers we found across three months change at a rate of 4% so in other words if you have 100,000 servers in your organization 4% of those or 4,000 would have changed your IP address within three months could be twice that they change it but we only count it it you know if it changed twice we count it it once so that's pretty dramatic and obviously in client M client type machines you'll notice that their IP address rate is higher but that's normal right because they may be in dhtp ranges great so what's the takeaway the

takeaway here is uh end points or host change not all of them but a certain subset do across time and if you're looking at these things at different points in time and relying on characteristics to match but those characteristics are changing you're going to be that makes sense right so that's what this diagram shows and I'm mapping the different rooms like the library and the doyard room to the different IP addresses so this is interesting because in this situation even though I'm using the analogy of clue where someone committed the murder in one of these rooms this is this problem is even more complex because their murderer commits the murder in different rooms because Chang if that Mak sense

right so who done it revealed this is what I'm talking about we're sharing that in vulnerability the various vulnerability management vendors out there are using simplistic algorithms to track end points across time this only is really applicable using unauthenticated uh scanning methodology because with agent based you could actually assign a unique identif to your agent and you can use that unique identifier as a tracking mechanism so agent based technology or scanning methodology is not prone to this issue that makes sense credential based same thing as you're doing a credential based scan when you're authenticating to a host you could actually drop something onto it unique identifier so that the next time you come back and do another unauthenticated

scan you'll see oh yeah there's there's an ID there I'm going to use that as a tracking mechan but for unauthenticated which is the most prevalently used uh vulnerability scanning technique that's not the case uh and these vendors actually use very limited checking the time uh very limited algorithms if you will one characteristic in the case of that large vendor I was talking about even though you have a choice um or maybe two or three but that's it so it's sort of like the analogy I use is if you look at your fingerprint the vendors are actually using very unlimited ridges as a pattern m a matching mechanism to match post across time and they're going to get it

wrong not all the time but enough that's that's sort of my pitch right consequences so we revealed the murder mystery but what does it mean to us two things asset duplication and asset mismatch So within the vulnerability management Silo these issues exist right so if you just look at vulnerability management you're going to get into situations where we'll go over it you're going to get into situations where uh for example suppose I do a scan in week one and I detect three devices which are these three assets but then suppose somewhere between week one and week two it churn has happened and IP addresses have changed other characteristics too but I'm just using IP addresses an example and if your

vendor is using limited tracking mechanisms very possibly they're going to get this mismatched such that the system believes that that asset up there that yellow one is actually the red one but obviously it's not so just within your own vulnerability management Silo you're going to get into a situation where assistant communicating to you but let's make the assumption that the vulnerabilities detected for for the red asset are completely orthogonal to those of the yellow one nothing in common just to make it easy the system's going to declare to you wow all of the red vulnerabilities have been fixed they're gone what a great job job you did yeah you got some new ones from the yellow

but you're still doing a pretty good job just solve the yellow ones so the more you mismatch the better it looks but it's not true right see that's the problem uh so that's just one example um and then duplication is a situation I don't have that up here is a situation where instead of actually mismatching it to different asset it says oh I've never seen this asset before I can't match it to anything must be a new one let's add it to the asset view so now you actually have a duplicate um seen this a lot and that causes other issues so the key M so what are the impacts right and you can imagine this now I have a few listed but

sort of two different levels within its own Silo you're going to have issues right just as I explained in terms of vulnerabilities declared as being fixed when really they're not or vulnerabilities having been declared as brand new on your asset when of course you're gonna you're GNA start perhaps assigning these vulnerabilities out to your team and keep in mind in large Enterprises you don't just have simplistic teams uh that have you know your security team and the owners of the assets in the same teams often that's completely different teams and so often what has happened and we see this and Prospects that have come to us is where you have a situation where the the

Security Professionals are assigning out these vulnerabilities to different teams those members are actually doing investigation on the vulnerabilities so they're learning that's good I know how to solve it let me go find the machine and solve it now only to to determine after quite some time this isn't even our machine what what were you communicating and then they pick it back and then they finally figure it out so there's a lot of Wast of time right that costs money that's only within vulnerability management when you talk about sharing information outside and into the security ecosystem those security those other security silos are looking at that data and assuming that it's correct and if it's not correct you're you're also

wasting time and your security gauges are off so what's the solution don't rely on circumstantial evidence right now even though traditionally fingerprints are considered circumstantial we're talking about a ridge of a fingerprint as compared to a complete fingerprint what I mean by this oh by the way I forgot to actually open up the envelope and show you the suspects where you have Miss Scarlet who represents vulnerability management unauthenticated vulnerability scanning which is the Candlestick and I actually put two rooms in here as opposed to one room because of the room shift right so what is the solution the solution is look if we can if we can actually see and view all of these characteristics

and all of these vulnerability scanning Technologies can from all of venders they can see IP address they can see Mac address to some extent they can see and determine OS they can see what ports are open they can see what vulnerabilities exist why don't we use all of that information of course we know what's going to change but if you the more information you use the more it's kind of analogous to real fingerprint that is one solution there many other Solutions but that's one of them are there many vendors out there that use that kind of a solution yes so that's you know what I recommend is when you're evaluating vulnerability Management systems and often in the past

what has been done is hey let's just run you know one scan and compare vendor to vendor to vendor and see what what they get and what they miss but they don't really look at little about across time and that's extremely important so obviously the ideal scan to scan post correlation is to get this right even though the red asset and the yellow asset for example have experienced the change across how do you do this once again with fingerprint matching technology that Mak sense murder myy solved I think we' did this in record time which is great uh once again there are there are use cases out there you need to consider the time Factor but what does that mean that

means you know uh and also Network endpoints change across time they're not static so you need something that's intelligent enough to detect that change but problem is most vulnerability man Solutions are very simplistic in how they solve this problem and many talks that you know many times when I give this a lot of people come up to me afterwards and say wow I just assumed that it was just working I never really thought of it right but in fact this is a challenge that plagues pretty much all vulnerability management vendors uh and I say that because it it it also plays digital defense but we understood this years ago and we come up with an algorithm that

uses all of those ridges as I mentioned plus we're able to measure when we're off by comparing ourselves with our authenticated scance if that Mak sense use your own endpoint correlation technology if you can but of course that's not simple takes time um there are some tools out there apparently that pull in data from vulnerability Management systems and they do have more advanced correlation so that's another option I rest my case murder mystery solve questions let get on to the happy hour thank you