Wait, there can't be only one?

well i guess i'll start captain people filtering the filter in um you two already know who i am i'm michael kafka otherwise known as shecky um i've been in the field for around seven years professionally full-time i started off in it about 25 years ago and i was doing it well before i started doing it as a living i helped organize chicago's uh one of chicago's burb sex i'm officially the organizer for burbsec north i'll help out with some of the other burb sex as needed as we all tend to flip and float around with the decentralized systems whoops i also am a hack for kids volunteer love working with the kids love seeing them go ahead and come up through

everything that aha moment when they're doing something especially this last one that we had up in milwaukee where somebody took a part in the destruction village something like so that's how that works there's my website i blog rather infrequently i keep saying i'm going to do it more frequently and i get into about a month and a half two months time frame where i'm vlogging regularly and then it just sort of drifts off to the side where you can really find me hanging out is on the twitter verse i do a lot of tweeting and a lot of retweets out there so what we're going to talk about today is what defense and depth is and

how it goes in a different way sometimes than you might originally think uh the basic idea it's what what is it we we know that it's the idea of using multiple layers to increase security it's not something that's new it's something that's gone on back to prehistoric times of building moats around castles and before that building the sand rock around it or finding a spot where you had better defense because you were up higher and you had trees and then you had natural water in front of you just to be able to give yourself a better defensive position the today's talk is going to focus more on multiple vendors as a type of difference in depth

and i'm going to be showing this through an actual real world scenario that i went through last year in the year before last year for the most part started the year before that during covid times with a couple of simple questions as to why things were the way they were graphics yes we have graphics um i hate doing slime guns i hate slides but what we see here is typically what you think of as defense and depth you have your data application your host your internal network your rings going on down through all the way on to the point of where what you're protecting what's getting protected by what where it's going and where your layers are sitting and what

you need is that border in between each of the layers for each of the defense so theoretically your data is supposed to be the most secure thing that you've got going on out because you've got to get through more stuff and i say theoretically because we know that's not true but that's the basic idea behind defense and depth we tend to um it doesn't give a depth to the idea so what you're seeing is just basically action we tend to skip or lighten beyond areas in that and a lot of times it's due to lack of resources maybe sometimes like a budget this day and age with how much we're spending on it i doubt budget really

falls into it as much as resources do why would you still budget there's still budget but the budget winds up being more on hiring the right people or getting somebody into the position that they need to get into as opposed to buying a blinky box everybody likes to go ahead and buy a blinking box but i think the key thing that we have to take away from all this is layers just like onions they have layers just like ogres have layers now i know you might not like ogres or onions you might prefer a cake or a parfait but layers are what we need to talk about and it isn't always so easy and it

still isn't we we have to think about layering within the layers so if you've got a section of cake you can layer inside of that section of cake you can't layer inside of a section of onion it's a whole thing but you can layer inside of sections inside of the world that we live in you can call it sub layering some people have done that people try and do it with the same product and it doesn't work the same way you really need to go ahead and do that sort of sub layering need multiple products that work together and don't work against each other and we've all seen back in the day the semantic versus mcafee situation where

people would toss two antiviruses on their machine and their machine would become unusable because all the resources were eaten up that's not what we're talking about what we're talking about and there are a lot better the vendors out there a lot better about it is actually integrating to a point where you can use stuff that use do a similar type thing but for one it's a primary for one it's a secondary but that back up there can be very very different now that's not to say that the interference between products doesn't happen anymore it does in the same breath it doesn't happen quite as frequently we all know the situation you're out on the net it came from the net you've got

your web blocking software in there you've got your anti-virus on your machine everything's there you go to a website and it was blocked i ran into this situation last year where a site known as mail.ru you can go ahead and look look it up there was get and its companies were being alerted on by microsoft security tools in most in a bunch of machines and by cisco umbrella on other machines and the and it was driving my bosses completely crazy what software is not working and why is it not working the same across the whole environment we've got microsoft defender blocking we've got umbrella blocking umbrella should theoretically from what we understood because it's dns based and

everything should have been blocking everything beforehand well we'll get to why that didn't happen as we talk along yeah this was our first thought was why what what's going on here both technologies did the blocks but there needed to be a reason why one was chosen over the other and male i are you overall especially at the time i found like other generic sites like yahoo like even nowadays there's what's happening with am with stuff on aws or azure has both good and bad things on it mail.ru has legitimate purposes for email addresses and other stuff but it's been known at times for hosting a lot of malware and for hosting a lot of c2 content

so some threat feeds rated high and some don't rate it so high so i went aha there's got to be the first problem there it's just the threat feed rating well sort of but not quite what you find out was there were more layers inside of it it was not just because of the technology themselves but it's the actual layering of the technologies that we ran into you have multiple t's pieces of tech that can have similar results but do you know where they're actually hitting and where they're actually working inside of the whole process so what's going first let's go going second what's going third they say well what would that matter in forensics that completely

matters should that breach happen you need to go ahead and know where did the breach actually occur why did this one fail and where was this in the order of things that they were able to bypass it so the tekken play over here you'll see a typical screen from mail.ru on cisco umbrella and over here kathy knows this very well is microsoft defender for endpoint now you'll notice right now that there's this block list in there and this says custom indicator after everything was said and done when i'd done my research on mail.ru i went ahead and said i don't care what the other threat feeds are we know we don't have any use for

mail.ru in our organization i put in a block into both of them so that way should it come off the threat feed which very well could because virustotal said yeah it's fine it was alien vault that said oh yeah all this stuff has happened with it and i said yeah we don't need it let's just put a block list in there that didn't change the fact that we were still only getting some machines hitting it with cisco umbrella and some machines with microsoft defender we decided it was in our best interest to make it done so what we wound up with here is this is a key word there network filter lookup service that was the start of where i started

looking at things when i found finally figured it out so we've got cisco umbrella we've got microsoft exploit guard which is what this network filter is and during my testing i also wound up going ahead and hitting up on you block origin which we went across which was not blocking the site because it's not an ad site per se or if it was blocking stuff it was only the ads that were being blocked off of it that were escaping everything else so why did it need to be manual blocked list in umbrella and the reason i say that was because originally umbrella was not blocking it at all it was not coming back as malicious or

anything it's mailed out are you you would think with cisco's talos threaten talents under they would at least put some sort of warning on it but it wasn't hitting any of the block categories it's microsoft's threat until initially was going ahead and saying yes this is a bad thing we are blocking it so the first time that i ever saw any block from mail.ru was through microsoft which led to the question initially of why isn't cisco blocking this microsoft was going ahead and doing things through their things umbrella itself is interesting because it only blocks from its own blacklist from its own categorizations so it doesn't do any proactive stuff it needs to go into one of those categories

or it's dumped into initially the newly seen domains if it's never been seen out there which is a whole other mess in its own right now we had some machines getting a black message from umbrella and others came blocked by microsoft it was an edge if it was edge in microsoft you would get their smart screen flat out it would you i'm sure you some of you have seen what the smart screen is it's a big red screen that comes up and says sorry we're not allowing this through in the same breath what was driving my bosses crazy was the fact that most people don't use edge they were using chrome and they were using firefox

more chrome than anything else and on either of those two all you would get is a white page with up in the left hand corner it's saying access denied no reason no nothing so this keeps going ahead and driving my bosses crazier and i'm going i'm sure this is the way that it works but you really want me to dig into this more i will dig into it more so i decided it had to have been an order problem security is knowledge knowing what should be doing what if you do not know what your products what your what your stuff that you're using for security does you're dead in the water somebody needs to know the order and has

to have that documented for everybody whether it be for a sock analyst so that way they can make the right call whether it needs to go up or for a forensics person when you get breached to go ahead and backtrack where the breach actually occurred it all needs to be done so i started doing tests to figure out the reasoning why one tech was happening over the other and in this case it was timing of running into this problem that made my life easier because it made me learn a bit more about the two technologies in play during my test i'll say flat out you block origin would be the first thing to block but that was only in the case of

what was in its block list it does its thing it does its thing well it blocks ads that's that so i was able to go ahead and just say well it's not blocking it because it's not in the you you origin block list easy enough but did find out by testing other sites that yes you block origin would hit first what happens with umbrella his umbrella itself is actually a it was formally open dns it allows successful dns lookups before blocking and it uses its own intel to categorize sites what happens with umbrella is that it actually would be called a machine in the middle protection just like a machine in the middle attack it sits there in between everything

and it's evident that that happens when the root certificate expires or doesn't get installed properly because then all of a sudden you can't get to any site whatsoever because umbrella's certificate has been rewriting whatever certificates out there and your computer goes in and says not happening it also means that you get a dns lookup before any sort of blocking happens and i confirmed this with umbrellas people themselves when i was going through this i'm going well why am i sh why why are we showing that well it's got to hit dns before it can go ahead and be blocked this way so it allowed me to understand why inside of the umbrella interface i was seeing allowed for dns

but then i wasn't seeing anything else beyond that on the machines that were being blocked by microsoft and i was using machines that i had i was using a virtual machine and my own laptop to do all this testing with and it just kept on going like that and the second i put it in the block list i was still going ahead and getting a discrepancy so it still didn't make complete sense to me microsoft network protection we all know how much we love to hate on microsoft microsoft hate is really misplaced in this day and age a lot of it stunned from a lot of fear uncertainty and doubt and fear of microsoft itself and not being able to

see into certain things microsoft network protection is part of exploit is part of their exploit guard system it's turned off by default so you do not have microsoft network protection turned on without turning it on a secondary way machine coming straight out of the box and plugged into a network is not going to have microsoft network network protection on it it is enabled through either group policy or intune so you can very easily enable it without having to touch any machine this is also a very key factor in what wound up happening and we'll get to that as we move onward exploit guard itself is part of microsoft's attack surface reduction or asr rules asr you've heard i'm sure pen testers

complain about asr because it makes their life more difficult again microsoft is trying to do things right and they're coming up with some good technologies out there it all uses microsoft's threat and tell whatever their magic sauce in the background is which i know hits up on other stuff and they just bought risk iq to help with their threat to make their threat until even better network protection does give that smart screen message if you're using the edge browser so figured out why it was hitting on the edge but what was going on with the edge browser it was hitting that network protection and there's the other thing that i found which was just that access to denied on

non-microsoft browsers if i tested on mail.ru that for some odd reason did not have network protection turned on i got the umbrella block screen real nice and simple it came up and it showed if if that if i did it on a microsoft on a machine that i could see that that microsoft network protection was on network protection blocked it and i would get an alert in the back end on mde and everybody would start going why are we getting this alert

it alerts and when you've got a couple of people that are going to sites that just happen to have mail.ru ads on them that are not blocked by you block origin it alerts frequently and we still get those alerts because network protection is blocking so the reasoning behind you blocks position which was the initial thing that would block stuff was that it compares the url and turned against its blacklist before the network traffic is even looked at it just looks at the name that's sitting there and it's right inside the browser anyway so just it's the fastest and shortest path very simple very easy microsoft's protection works similar in the sense that it runs off of a pure

ip or dns response before you actually do a full lookup when the second it sees an ip address when you go to do dns is when it hits before it goes to look at any categories or anything like that so it's looking at a block list in the threat until on its own end but it hits before any sort of one blocking categorization it says there's always some network traffic involved in it finally with cisco cisco was the third layer in it so if the other two are not working then cisco hits so it starts to make sense and i turn around and i tell my bosses that this is what's happening and the more i thought about it it's

well that's an ingenious way of looking at it i now have three technologies there and one of the three should block between the three of them they should block almost all the majority of the bad things out there for the end user and based off of that i should be able to tell by what is hitting and what is not hitting what's going on well the question is is why was the group policy not being applied to everywhere well it turned out at that point in time we had not finished getting all the asr rules rolled out the machines that were actually getting getting the microsoft block were machines that were using the on-prem domain

so it was being pushed out through gpo and when i went into intune they hadn't set it up yet so all of our in now we are dealing with cohen so the majority of people are either home or they're going in through a vdi computer so those that were on vdi were getting hit by microsoft and we're getting the alert through microsoft and if you were on your home laptop unless you were on using microsoft edge and its smart screen which uses the same technology you were able to get through to the new block to the umbrella black screen on it all so what i found out and figured out after all this time was that the defense

in depth actually does work it was a proof of concept that i never really thought i would see because everybody talks about yeah you need all these multiple layers but when you actually see it work in such a fashion you start to understand where it comes through um what bugs me still is that the dns lookup is allowed through umbrella with everything that umbrella touts that it can do to allow a full back and forth connection to say okay yes here is this here's the dns request and we're going to give you this information and then we're going to block you doesn't seem like that's the right way to do it but i'm not a designer

but the more layers that you've got the more complicated it becomes when you have to backtrack on everything so the best way of figuring out where your layers are sitting is usually to ask your vendor itself what's going on do a little bit of research understand how the technology works because the second you understand how that technology works you go okay so umbrella takes up takes the xyz it does a dns lookup it comes back and then it checks against a blacklist great microsoft network protection goes ahead and talk checks immediately the second that you go there and it adjusts changes dns to ip before it ever comes back to you it says oh this is bad

that's the way that it does it it's layers upon layers upon layers there are general layers big outer rings the moat around the boat the firewall sitting in front of your uh internet your internal network there are sub layers your ips ids email protection systems there are some layers inside of cell layers the layers themselves can be differing technologies network protection versus web content filtering versus ad blocking but all three of those wind up overlapping to some degree

the idea on it all is to have each of the layers complement each other instead of working against each other and we've seen at times where things can go ahead and work against each other a lot of it has to do with secret sauces a lot of it has to do with improper implementations misconfigurations again all stuff that can be most of that stuff can be easily avoided with a little bit of knowledge behind what the topic behind what the technology is doing

and i sort of lie there can be only one but the one is the order of the technology and not just a single blinky box you can have multiple blinky boxes lined up behind multiple it will line up in front of multiple software things and software put in between each of those blinky boxes if you don't know where you're going with all of that if you don't design it out properly or understand how it's all working together what good is it really going to do you yeah it might go ahead and keep you protected for some point but we all know that any sort of protection can be bypassed eventually there is no perfect protection

there is no perfect solution um kathy said it best and i've been saying the same thing as you did for years security is a journey it's not an end point and we're constantly having to change in fact i've gotten to the point where i dislike telling people that they need better security i tell them they need better risk management because that's really in the end what we are doing is managing the risk level down to something that's more acceptable and easy to deal with as opposed to being able to lock everything out if there was perfect security half the wars and that we've had over the years not just in the 20th century but going

way back to ancient times probably wouldn't have happened because they would have figured out how to block it and that could have gone ahead and gone forward in time to go into the world of technology just like social engineering has been the same over in anything else gets brought over into technology so it's a matter of working together in the end technology companies tend to keep a lot secret from us going through just figuring this out took me a couple of months because i wasn't we put in initially a request with cisco a support ticket with cisco about it and they came back to us with yeah dns should be allowed through like that yeah everything's working as

intended that didn't help me out all that much outside of letting me know that dns would always get would always would never get blocked the end result was that we were in the midst of rolling out x point guard all across the company it was it was part of a initiative that we were doing to try and cut down on the chances of wasting if i ran somewhere uh an initiative set up by the ceo of the company because at the time you had all the big name ransomware stuff that was going out and going around everything was working as intended there was no issue which was my boss's biggest concern my cso's biggest concern was

that something was not working the way that it was supposed to be working but with better and simpler documentation on all the products so that we could see what their interoperability would be or lack thereof we would have come to this conclusion a lot faster we can design and implement better defenses with that same sort of idea simpler documentation better training materials out there not as much fun it all comes in together seated gastronomy is prevalent in our tools how many people here know what c-tech astronomy is how many people here have ever seen the movie sneakers how many people now remember what c-tech astronomy is no more secrets oh it does it's it's prevalent in in our

field it does hinder us it does help us in some ways it's understandable because companies want to go ahead and have an advantage over their competitors but if they really want to be part of the information security field and not just an information security vendor they need to find ways of sharing more information out with us because the people that wind up paying for it in the end are us on the front lines

and with that i'll take questions it's not goodbye it's i'll seal you later any questions

feel free to hit me up on twitter if you ever want to how did your uh boss like once you figured it out and you sat down and explained it to them like what was their reaction angry at cisco because they didn't realize the way that cisco umbrella was actually working grateful that we had the multiple layers in it all a new understanding said okay now we know if it gets past cisco it's gotten past a and b also so we could go ahead and tune stuff up a little bit better make some detection technologies to go ahead and help out with that i mean as you saw in the uh one slide it's now custom detection inside of

microsoft defender i wrote up the detection myself i wrote put that in myself and i have custom detections inside some of it based off of that some of it based off of other things i've seen run and go through just so that way something's going to get through but the odds of it not being detected start to go down and detection is going to be the key going closer but yeah he took it he took it with a thank you michael thank you for all of your hard work i'm not happy with cisco about this but microsoft's web blocking technology is not anywhere their categorization everything's not anywhere near where cisco's is right now and i don't see us

switching over to something like z scalar or blue code i'd worked with bluecoat in a prior company and it was fantastic and especially on the newly seen domains situation where i could contact them and have something cleared within a couple hours whereas with cisco umbrella i turn around and i have to put in it put in a site into an actual white list which we forget about for months on end and then i have to go back through and try and clean up the white list and then it starts all over again so when you're design when you've got a section designing websites for your clients and everything you run into newly seen domains all the time

any other questions well i'll give you back the extra 10 minutes here all right thank you thank you all for attending [Applause]