BSidesTO 2015 - Allan Stojanovic & David Auclair - MOAR Logs and Tactical Defence

hmm

my name is alan stojanovic this is david eau claire we work at the university of toronto uh and uh this is actually a sequel to the talk from last year logs and tactical defense did anybody do their homework well a few that's awesome that's better than expected uh so i'm gonna quickly do a couple of bits and pieces uh as we get into the new stuff but first uh re the recap um so our environment is huge it's very very big we have six uh b class worth of public ip addresses a lot of them go directly to the desktops we've got now a 32 ipv6 public address space that is an internet's worth of 64s

we have over 600 departments we have over 450 000 users in our identity management and we are effectively a city unto ourselves we have every service that every city has except we don't actually have our own official fire department but we do have a fire uh fire prevention team so our motivators uh uh in trying to actually find some security and protection on this network is around the fact that we have a network that is mostly open required to be so uh we have all of our departments are mostly autonomous uh they have the ability and they have different goals and different capabilities that they go through we have a lot of intellectual property i

mean we are a research institution by by the very definition of the word and we don't have a big security budget but from everything i hear nobody really does and we have that extra wrinkle called academic freedom now quick note on academic freedom because i'm finding that the more people i talk to that aren't in higher ed they don't quite seem to understand it academic freedom doesn't exactly mean they get to do anything they want what it means is they can propose to do anything they want but they still got to go through a committee to get approval so the idea is they can't do things that would obviously harm individuals or groups either physically or financially

or any of that kind of stuff so that's a an intro an intriguing and interesting point an extra wrinkle to all of our stuff we have i like as i like to say we have every make we have every model we have every vintage and we have every skill level across the entire university and we have multiples of them to the point where some of these people they could be running the same thing and it's ancient and they don't even know each other now uh the recap of what we talked about last uh last time just to make just to bring everybody up to page uh uh bring everybody the same page so we came up with uh six original what we

called recipes very fast things to try to figure out what and what the intent is of network traffic on our on our network specifically internet traffic inbound we had uh an idea around something called that we call the trial by firewall so it's very simple concept if you have a let's pretend that you're entirely a windows shop and there's no reason for you to ever have ssh listening on the outside world let's pretend if somebody touches port 22 on your firewall and is denied that could go to intent and if it does then why not deny them all access so rather now the second one is uh that we dealt with was known as dr bad touch same idea except

now actually putting a honey ports program out there so you put an unadvertised ip address you wait for see if somebody touches it and it goes again to intent are they scanning the network should you allow them access to the stuff that you allow if they've already shown intent to do something questionable the blatant 404 was logs coming in on your web servers when people are hitting up a whole bunch of web pages that don't exist the big example there was when they go through phpmyadmin and he's checking every version that you got to try to figure out whether you've got a vulnerable version right there you've got an opportunity to take an action because

that's pretty blatant for intent right the impossible multi-auth so you find yourself in a situation where you've got users around the world or cross larger pieces if somebody logs in from from china and from toronto at the same time it's worth looking at the questionable single source was another one where it's similar except the uh kind of from the other direction when you've got multiple users logging in from one place uh and it's very quick and very fast and more importantly when you see that a single ip address doing the fail fail fail fail fail fail success maybe that's something you might want to look at as well and then finally the the last one that

we came up with back then was uh the fake fishing attacks so when you see a fish come in one of the things that uh that we've been playing with is that we actually fill in the fishing forms with fake credentials see where they come back from and see who else they come back as and potentially action anything that we see from there from blocking the original attacker all the way through to resetting any passwords that were actually successful from that source so those six recipes led us to some combination of actions the original actions that we had at the time was basically based around the idea that bad intentions deserve to be denied if you

can if you can figure out what the intent is of somebody coming into a network maybe you can deal with it sooner the sooner you can deal with it the better off theoretically you should be um the id attempt to block the attacker not just the attack i mean sometimes you'll be successful sometimes you won't but that was our ultimate goal of course white list where appropriate uh you don't want to necessarily block your vulnerability scanner especially if you're automating all this and you don't have to worry as much about false positives you're going to have false positives so what you've got to do is make certain that you can deal with them quickly so that they don't become a major

problem sadly they're a part of life because well you know hackers they try to hide all their tracks anyway and of course investigate and test everything aggregating ip address sources was a big one this was very very useful so the ability to say well i've got a network over here that's a complete asn that's done nothing but bad things and is completely um attack an attack vector and has never had a valid login well think about blocking the entire ip space um investigating the repeated compromises either user or machine so you get the same machine compromised multiple times or you get the same user clicking the same phishing multiple times it's a re-education opportunity and hotspots of compromise

so being able to track where people are getting compromised for us it seemed to be airports hospitals uh there was a couple of more escape me at the moment but you get the general idea so keep in mind those an eye on those people logging in from those locations to make certain that they're not compromised on the fly it gets you kind of a little bit in front of the problem and then of course the research and sharing we are a research institution so we have the ability to share a little bit more than maybe you do but uh the ability to be able to just research out these ip addresses that you see coming in what are they where are

they what's the behavior what's the signature set look like look them up find out what else is going on then we had the original reaction set what do you do with it we have the the obvious options quarantine or block block if you're going to block block permanent if you're going to quarantine pick a nice uh appropriate period of time and you know work your way from there you can do it at a sub network layer so like whatever your where your critical infrastructure is or you can do it for the entire organization or any combination thereof you can also potentially do redirection to safer places uh honeypots is great for that so if you identify particular

types of traffic that are bad you can redirect them to honeypot to find out what they're really after white lists still important here is uh one of the things that we discovered was the opportunity to white list certain client machines like your boss's machine so when somebody calls your boss complaining about the fact that you've blocked websites he's looking at that website going well it works for me the all404 so i was playing with some patchy scripts that allowed me to take a particular source ip address if that ip address came back i could make an entire website disappear every page looked like a 404. so suddenly like tools do really weird things when suddenly all they've

got is an apache container and yet there's still just enough logs going so that you can see you can determine some very very basic stuff i mean normally you could do this under other circumstances as well but here you can almost automate it is it an automated tool or is it done manually is it some combination thereof and how are they modifying their attacks especially when it looks like there's no website at all anymore uh be prepared to release your quarantine on your block quickly that was a very very important thing and of course log everything you do as well as what they do because you can actually start running reports and all that kind of stuff and

it gives you the statistics you need to be able to push up through management and find out yet even more interesting stuff the new stuff is more about generation of these uh these recipes and we're going to get into that next oh sorry oh i'm sorry we still got so we had some failures around a lot of these recipes and this is kind of new this is what we've been doing and dealing with for the the past year the biggest one that was popping up a lot was the goip location detection failure so we found ourselves in a situation where especially around suspicious login activity because we have users worldwide we have researchers that travel all the time

that go ip information was getting less and less accurate now i talked about last year the idea that a particular goip database they will actually introduce errors on purpose to find out if somebody's stealing their goip database because they want to be able to charge for it but what we found on top of that was an ipv4 churn was causing these geoip databases to actually get even farther out of sync so for instance with the lack of ipv4 space and people have started selling buying and selling more and more of this these networks so the one that popped up that was blatantly obvious for us was that we noticed the tech savvy ip addresses started showing up as it from indonesia

and it as far as we can gather it looks like tech savvy took over an ipv space from indonesia and it was just taking the uh the the providers a little longer to catch up now the reason that we actually caught that thanks to tech savvy was that the reverse lookups still said tech savvy so but that's a local isp so we know them well we caught that pretty quick we could deal with it appropriately but imagine if that was in a completely different country from one isp to another that you have no idea about so now you've actually got to watch out for your own space uh or how much of you rely upon these geoip numbers and then

there's the the proxies and the tour the proxy specifically around people trying to get around geo fencing especially because you know you want the us netflix netflix right um and and all that layer of stuff now one of the things that we've been dealing with to try to figure out whether we can actually get around this is we've been playing with the idea of de-anonymizing uh at least web logins so if we had the browser tell us where you really are not only who you are but where you are then maybe we can use that as a more accurate prediction or description of what's going on however we're running into a little bit of the politics side

there and we're still trying to explore what this means will this be viewed as too much surveillance will our user base rebel against it so on and so forth another limit we ran into is the device limit now i don't know if you've ever seen this but there is a top end to the number of firewall rules that you can actually have so if you're automatically blocking if you're putting in firewall rule denies it does weird things when it reaches the top a lot of them do uh so firewall rules the number of firewall rule objects some number of objects can be associated with a firewall rule number of quarantines so like if you've got the

ips or the utm style firewalls uh that are taking a quarantine list there is a top end that it's capable of a number of iptable denies rules if you're dealing with things at host based this one was interesting because there's no i couldn't find a an actual hard limit or at least i couldn't reach it what i was reaching was a practical limit where when i wanted to process what was already there the machine bogged right down because it locks the table and then nothing else was getting added to it while that table is locked so there's a practical limit there's a theoretical limit and you got to watch out for both of them you could

reach them if you're doing this on an individual ip basis device capabilities failures as usual so the fail open versus fail closed uh i this you know this and yet the one that happens is the one you didn't build for right so you're hoping you put in a system hoping for a fail open it fails closed or vice versa happens often enough uh and it does it under under weird circumstances because you're reaching the edges of its spec it's not even about you know about reaching a practice or a clear limit that they've outlined in their documentation it's some other limit somewhere else that they didn't even consider a leaky rule set you'll find yourself in

a situation where you're putting in these rules and they're not either not activating fast enough or under certain load circumstances they're not being honored or any of that kind of stuff so we've hit those limits as well uh there's some operational limitations like some devices when you actually change the config on the device it requires an entire config reload and during that time is it fail open or is it fail closed well we've had situations with a little bit of both and then there's also of course all the usual problems the memory storage bandwidth all that layer of stuff causing wonderful stuff to just fall right over false positives so we have the false positives um

which is in one particular example that we come across is as we're track tracking suspicious account activity if a user logs in to uh let's say rdp's into their work machine and then rdp's from there into a production server depending on the circumstances sometimes it looks like two different logins from two different countries so this is a technical problem this is something that we recognize and we're trying we're dealing with it uh and it is a actual false positive but the other one that we come across is when the users or the people that were reporting this kind of stuff when they're telling us it's a false positive and the big one is oh yeah that login from from local

was my assistant using my password to read my email while i'm in paris it's i'm sorry it's a false positive well actually no it's not especially since here's our policy that says you're not allowed to share your password and go get yourself a delegated mailbox that's an organizational problem but it is a opportunity for a for re-education so being able to sit down with them and say this is why it's not a false positive

hey guys um so we'd like to introduce this to talk about something a little bit different here so um we've all been through this in one way or another in the five stages of infosec um but how much free credit modern do we really need there's a lot of painful lessons here and maybe if we improve things we can actually learn from this without going through the painful part and just get to the acceptance stage we can learn from this we can do better because we are constantly being fantastic free of charge through the pen test army um the only problem is they don't give us reports uh we're constantly under attack and um they have all levels of skill and uh

they they do sleep but they're in various time zones so um from our perspective 24 7 under attack um but they do cleverly hide the reports in our logs so if we die to mine our own logs analog all the things um we can actually get the reports uh so um we can skip those painful steps and and go right to accept this so back to more logs and tactile defense um i'd like to talk about um what we've been working on uh our visibility project um the goal is to monitor all the things we are monitoring systems and network traffic including netflow metadata and in some circumstances pcops and definitely system logs um so one of

the beautiful things about this is not everybody logs to us so by monitoring the network traffic we can we can actually get logs like http request logs and stuff like that even if they're not giving it to us we can mine out of our own traffic um like a visibility architecture it doesn't need to be hard it can be pretty simple it can be as simple as a multi-port tap in your internet connection feeding a stack of analysis boxes but maybe your architecture isn't this simple i know ours isn't it can be uh accommodating though i mean uh we have two different gateways uh kilometers apart and uh two different uh internet service providers uh so

things like asymmetric routing happen so uh traffic might go out one path might come back another uh so if you're if you're only tapping any of those given links you might not have the full conversation really it needs to be all reassembled so there's a type of device out there called a network packet broker and basically what it does is it aggregates all the feeds it can filter them and they can load balance the outputs so the other thing is you can scale up your analysis if if you have boxes filling over at 10 gigs add a few more boxes no problem um so really though if you're monitoring all the things you're going to have a huge

amount of traffic a huge amount of logs really what you need is anomaly detection you need to pick the interesting bits out from all those logs and you need to do it in an automated fashion um so you can pick up on things like protocol anomalies so ssh it is encrypted yes but before it goes encrypted there's a handshake and the client exchanges a client version identifier and the server exchanges a server identifier what you can see is if you if a specific client identifier is being more aggressive than it should be um that is that is a thing you can identify it by um the other thing is rdp uh again it's encrypted but uh in the

handshake there is there's quite a bit of details that go back and forth including things like screen resolution uh we were seeing tons of just impossibly low screen resolution uh coming through like 10 by 17 which you know that's that's that's not a realistic screen resolution but these are these are protocol anomalies you can key on and uh you know feed your anomaly detection based on that um also the usual you know timing irregularities if things are happening faster than a user could possibly do then um you know that that's a good anomaly as well um even simple things like case irregularities so if a user always logs in lowercase username but suddenly you're seeing maybe from a different

country even capital or all caps or mixed caps i mean these are interesting things that you can key on um so once you're sure i mean obviously uh you need a certain degree of confidence you're not gonna block based on a minor thing but um once you're sure you block the host or if you're not quite sure but you you think they're up to something sketchy you can redirect them to anybody um just be careful not to block you know the googles the bings yeah index baidu the usual um i'd just like to note here i'll get back to this in a second but automation isn't fire and forget you do need to kind of

monitor and maintain uh because attackers are intelligent and work around obstacles so any any defense you put in their path they will try to work around so as part of the anomaly detection um there are attack tools right so i mean you could if you can classify specific attack tools that's a very good anomaly detection so you can you can build a set of fingerprints um some tools announce their presence like ncrack or tt ipv6 or like medusa a lot of them just they just say hi i'm ngrak um other tools they have a characteristic pattern like nmap um and sometimes these patterns they um they shift by attack stage so you might see you might see an attack tool

enumerating hosts you might see it enumerating pages on say like a web server you might say it uh probing for weaknesses in those pages and you might even see it i mean hopefully you cut them off before this point but you might even see them enumerating your database or exfoliating data so um also there's there's common attack patterns like sql injection you can develop kind of generic detection um but sometimes what you can do is based on those generic detections you can you can actually develop custom signatures so um for instance if you see a sql injection attack you can build um a unknown tool signature based on that right so uh it's it's even if you don't know what tool it is

it's still useful because you know definitely it's an attack right yeah even if it's listed as in in your um signatures even if it's listed as unknown attack tool 23 as alan said you don't know what tool it is but you know it's an attack right as long as it's a strong signature and run the tools so basically if you can run tools against yourself collect those logs collect those fingerprints then you you can develop a pretty good set of fingerprints and uh of known attack tools right so once you detect an attack i mean block them as usual um but go back and review their actions what exactly did they do what exactly did they touch and and the responses so

if your servers throwing back 404s maybe they didn't really get anywhere if they're throwing back internal server area or or 200 like http okay um then maybe they got a little further than you'd like but now is a good time to test or hopefully retest the attack to target determine are they actually vulnerable to what they're being pro before um so use your tool of choice um something you trust but also maybe in a sandbox because some of the hacker tools can be a little sketchy maybe it would be useful if you know what tool it is to run that against yourself as well just just to validate what your findings are and uh it's actually really really good

if you're logging all things keep an appropriate event history for you that might be a year's worth of logs it might be a quarters worth of vlogs maybe a month whatever um you guys decide that for yourselves uh what's great though is once you've developed a new signature once you've developed a new indicator of compromise review that new data or sorry review that new signature against the old data so you can you can discover oh well maybe we're actually attacked with this before an example of this is the angler command and control details that were recently posted by talos security um they they published quite a bit of details they published uh command and control ips command and control domains

so if you can go back through your data maybe you only have netflow you can still review your data against the command control eyepiece that's that's a pretty good hit i mean if you if you have hits on that it's a fairly fairly good indicator but if you're actually logging more data like metadata full requests and all that you can actually see hits against command control domains which is a better indicator if you have full peak apps however you're doing great because um you can then actually go back and review exactly what happened so if they hit the the exploit page maybe that's bad but if they didn't deploy the payload because maybe the exploits failed um

you at least know there's no there's no more uncertainty you can identify definitely what happened whether whether they got exploited or not so if you're doing this you will have so much data um really you need automated analysis there's no way you can keep up with this manually so what we've come up with is we monitor our metadata flow in real time and we deal with about currently about half a billion events a day um and dealing with it in real time isn't that bad i mean you guys can do it no problem um so we we detect anomalies on the fly uh based on this metadata we use something called a leaky bucket mechanism i mean

it's if you haven't heard of it it's a pretty simple concept um events contribute score to the leaky bucket uh weak indicators contribute a little bit of score strong indicators contribute more score but the score kind of decays overtime so if they are contributing enough events that with enough score to the to the leaky bucket um it will overflow the bucket and trigger your trigger your action so in our in our case we are using this uh to do automated blocking um and we are we are tracking the ip blocking history as well so if we've blocked these guys before we'll actually block them faster so we'll have a score multiplier um one thing to be careful of though is

avoiding false positive loops if you do flag something as a false positive um it might if you are letting the score grow faster and faster based on ipv blocking history you might block them more and more so it might kind of snowball so we we capped the ip blocking history score contribution at like 2x multiplier or something like that but i mean whatever you guys whatever get us aside for your traffic that's fine um one thing we have noticed is attackers know they they know when your staff's not there they um they wait i mean they they fish all week and then they play all weekend right they know when you've gone off on holiday or

whatever um so having an automated system that never takes a break that never sleeps that never gets bored of watching logs um it's great because it can keep up with the attackers in real time so i mentioned earlier uh it's not fire and forget you do need to take care of and maintain your automated systems so as you're monitoring events you will notice things that need a little bit more investigation so as part of that investigation if you notice that maybe maybe you're missing rules that you should have or maybe you could have picked up on something earlier had you noticed you know that's the chance to go back and integrate those changes into your

monitoring system and as you kind of cycle through that again and again your systems will improve over time

i'm short one of the last points just here uh for those of you that have come like i've talked to a few of you that say where do i start monitor okay because you start by just trying to learn what you're looking at and then work your way from there uh anyway one of the things i wanted to cover off a little bit was how you can actually take all this stuff so we're talking about some really low end stuff you're buried in the wires trying to figure out what's going on uh and in the meantime you've got some management that's actually trying to put together policy sets or find the supports uh to be able to get to the

point of being a security mature organization and whatnot and a lot of the stuff that you've done here especially if you're in front of them uh can actually feed into that information uh the big and obvious one that pops up quite often is asset management at least for us i mean we are so large that being able to actually even have an asset list is insane we we are the definition of byod we were byod before there was a term byod so uh so finding ourselves in a situation where we can't actually track assets and on top of that don't own the assets anyway is an extremely difficult task when you're actually monitoring the level uh at the level that we are then

you have an opportunity to be able to actually have another set of triggers that's simply feeding an asset management list of some sort you can actually have that feedback loop and then on top of that one of the things you can do is you can actually start tracking virtual assets as well so for instance if you find yourself in a situation where you got that that web server that's actually hosting 15 different websites each with its own owner you can actually track one of each one of those separately with the owner and report on them appropriately you can see that traffic if you're actually filtering it properly you can alert when a new asset shows up like that pony plug

that somebody just tried to plug into your production network or you can you know find those unfiltered and unmanaged assets and deal with them slightly separately is it in the dhcp range is it you know any of those kinds of things you can report on dorman assets how many do you have oh look this box hasn't actually seen any traffic in netflow for an extremely long time is it down or is it somebody just doesn't care about it anymore and now you can take all of that and you can cross-reference it with vulnerability management so finding a new asset determining just some level of is this my asset or somebody else's and if it is my asset

then you find that you can actually automatically do a vulnerability scan of it you can actually feed that into your vulnerability management system says this is new to me go scan this let's find out what it is what it's doing and whether it's actually vulnerable to anything so you can do kind of a just-in-time vulnerability management here too you can also on top of that with the previous setup you can have passive vulnerability databases in front of this too so you're actually monitoring network traffic trying to determine whether something is vulnerable at that level and tie that all into your vulnerability management as well you can report unreachable assets so finding yourself in a situation where a

new asset pops up and for some reason that vulnerability scanner that you tried so hard to install can't reach it is it a configuration problem or do you have another admin over there that decided that uh you're not good enough for them and they don't want their assets scanned at all this is especially good for ipv6 because as v6 starts coming up i mean the numbers are just so huge that being able to track all this is going to be really really difficult threat intelligence so this is kind of where i've been playing for the past year quite a bit i've been doing a lot of research on this side and one of the things that's come up again and again is

yes there are all these threat intelligence feeds out there and having won two or three of them is actually probably pretty good but the funniest thing is uh as uh as i've learned through other people's research is that uh if you had all of them if you had every threat intelligence feed out there private and public you still wouldn't have every attack like there is very very little overlap so the one that's actually popped up from there the logical conclusion to me as a techie in the in the trenches is the only threat intelligence feed that's actually useful to me is the one i build myself for myself with my own info now that being said

still grabbing the outside feeds because it's a faster way to get context around the information that you're seeing so that you can actually take that threat intelligence that you've now built for yourself and enrich it with what you see across the other threat intelligence feeds like you see an ip address coming in and you want to find out you can go over to arbor.net and get the botnet information if you're subscribed or go to uh simon they're they're you know they're local here and they've got some really good information and aggregation as well um you hope that it'll give you some context around it anyway uh and of course incident response so when something's happened you all i'm

sure have gotten the questions management coming at you what happened when did it happen who did it what did they do what did they get out all of that stuff everything in here can feed into that imagine if you can actually generate that report in hours instead of days wouldn't that be awesome i'm not there yet either but we're trying right and then you know this but this of course does not actually solve the classic attribution problem you're still not necessarily going to know exactly who it is but maybe where it came from can get you some something close now in the larger context as let's say uh if you're in a more mature model uh

of an organization you're gonna have some sort of grc model some sort of governance risk and compliance uh mapping that you're trying to aim for and i've purposefully chosen this complex one because they can get really complex they can be really simple or anything in between but you can play almost any box in here you can support the you the creation or use or even the implementation of anything in here so like if you find yourself in a situation where you're bored nobody right then you can actually pick one of these boxes and say i can actually support that policy in implementation by doing x y and z and by spitting out these reports on an automated basis so on and

so forth what we're trying to do next so as i mentioned a couple times already ipv6 is going to be a big issue for us we have a slash 32 and trying to figure out what we're going to do with that i'm not entirely clear the biggest issue with v6 is that default install has something called privacy extensions privacy extensions means that theoretically every connection outbound from a machine can be from a new ip think about that for a moment there so the smallest network that they expect you to actually deploy is a slash 64. and then from there you're there's so many ip addresses that any machine can actually randomly generate a new ip

address and use that for x period of time that is undefined and then and now think about things like trying to actually report back which user was downloading the latest episode of walking dead to cll to match c11 compliance i don't know yet i don't know what that's going to look like yet now from uh it's also not scannable either internally or externally it's just too big so one of the things that we've been thinking about and is the idea that that dr bad touch recipe right from the very beginning so maybe instead of actually having uh unadvertised ip maybe i'll advertise this one a little bit so i'll set up a honey ports program

listening to a particular ipv6 space with no other ip addresses and give it a dns entry just to see just to see whether anybody's actually touching it because i have a feeling that one of the most obvious ways right now to recon v6 is just to actually try to do zone transfer through your dns most likely way anyway uh the latest thc ipv6 tools the latest rounds not the previous ones has built-in sigs so you could use that so some of the techniques that they use to discover v6 space you should be able to detect at least internally this is our ipv6 space

moving on software divine networking so we've got some efforts underway to that the networking crew is putting in software-defined networking this gives us an opportunity to actually do total on-the-fly flow control we could actually use our triggers to say hey there's traffic over here that i want to do something special with uh i don't know whether what they've done is actually bad for sure so let's actually redirect them in front of this other set of sensors that's really good at figuring that out but i don't have the budget to build the big sensors so it's moving so it's watching everything i can actually have all these different paths in front of different tool sets or i can

actually use it to redirect completely to a honeypot on the fly almost transparently or if i'm really really brave and i have no idea how this would actually work practically but we're exploring it is i can take the production system in the vmware clone it have some sort of mechanism that actually shuts it off from sensitive data and let them hack that and at least i will find out a little bit more about intent what are they actually after and how would they get it right talking about the limits of devices um what do you do when you outgrow your solutions i'm not entirely sure what this looks like yet because at our scales there are no better solutions i

mean the ones that we've put in barely run as they are when we win an rfp it's because it's the one that fell down the least so when you can't handle all these kinds of things when you're designing these kinds of networks uh thinking about scaling uh thinking about upgrades upgrade paths and most importantly think about your exit strategy so as you're actually putting together that rfp as you're trying to figure out what the solution looks like you should already be thinking about what the next solution is going to look like and that's going to require you to actually talk to management because they're going to have an idea of what that network needs to

look like or at the very least what the business needs to look like so now there's a big flow and basically you're standing at the bottom of a hill in front of an avalanche going stop like i said i don't exactly know what this looks like yet uh but one key piece that seems to be working really well for us is horizontal scaling so the add another box scaling that works reasonably well uh but it adds a lot more to the overhead to manage all those boxes so a quick note uh chose these guys as the patron saints of this tactical defense because these are two average joes that have been thrown into uh well an intelligence

horde of adversaries that are trying everything that they can to take these guys down and they're just trying to survive

um the highest the highest concentration of compromises from airports and hospitals uh the uh though i'm not entirely certain why the the the only thing that i got to latch onto is it just so happens that one of the people i was in as an undergrad with is now working in one of the hospitals and she had her credentials compromised this was a researcher yes this was correct this was not a visitor uh in this particular case it turned out that somebody had forged our name in a phishing attack to try to convince them to give convince her to give them her journals that they would publish for free if she just logs in over here

that's that's one anecdote that i can share i don't know enough but i know that like i can almost guarantee that it'll be everything like everything that you can think of in every way that is compromised it's probably been used i mean we just got enough of a population that we're statistically significant

because otherwise let me repeat what was at the bottom of that slide every make every model every vintage

well there's a gentleman in the second row here who's our architect who's architecting our replacement for log management uh that is supposed to scale out across the next 10 years to 4 million events a second so we log we log everything and that's just 90 days yes

at the tactical side we don't have a strategy because we're just we're just reacting so loudest first so it just so happens that on our network rdp and ssh were neck and neck as well actually that's not entirely true telnet was number one uh and after a quick dig through on that funny story there uh we could not find a single valid use of inbound telnet we did find six comp no nine compromise switches uh so that's now blocked uh sorry you were first

i think evil thoughts he does the math

not that i've noticed i've had people go away so maybe they've detected it i've had a um chaos computer club log into one of my honey pots do two quick ls's and disappear sorry there's one for you

sir

most of it's actually custom tools using uh the various apis available from the vendors so the oh sorry how do we actually manage the configuration on the vendor's devices yes i'm repeating uh so the uh we have a a stack of custom scripts that actually communicate through whatever api they give us uh not at this time but that's coming up yes sir uh

oh this is for him well okay i mean we've seen questions sorry can you remember the question in one of the slides like a collection of ip address same same ip addresses then you block them right but let's say an attacker uses a proxy and he keeps changing the ip address and then performs that app so how would you go about blocking just repeat the question so the question is um if we are aggregating ip addresses into subnets or whatever um how how do we block a single attacker who is bouncing around through different proxies it's kind of two different scenarios we've seen like weaponized subnets where all they do is attack all day long

those are the ones we roll those up and aggregate those whereas a single attacker bouncing around through different proxies it's a bit it's a bit trickier to nail them but aggressive log monitoring fingerprinting tool signatures etc so we can see their attacks tying it all back to know it's the same attacker that's an attribution problem and that's that's a little bit tricky but we can see we can see all the different places that they're attacking from we just might not necessarily be able to identify that it's the same user you would need to continue yes yeah continuously monitoring um not not manually i mean continuously monitoring the automated tools um and then we monitor the tools

manually but does that answer it more or less okay in the in the lifecycle loop as you're doing that monitoring you might find yourself with a signature set that you can actually use depending on what they're using if they're using a particular tool then you use that to block them out yes it might be one attacker it might be more than one again if it's a single proxy have with multiple attackers behind it uh is also possible in our network so you know actually aggregating behind proxy it doesn't really matter to us if we've got other techniques for blocking them make

machine learning sense otherwise to kind of put the problem on its head and define good traffic and then loosen the policy as a request have we used machine learning to flip the script to what is good and loosen rules as it comes in okay uh no not directly um mostly because i'm a curmudgeon and i can't actually see good traffic

no not at this time uh part of the uh threat intelligence stuff is leading towards that though yes

take it to the beer

like for the bigger question that i want to ask is that uh most of the exploits that are using are already patched and they're attacking people who haven't uh they're not up to the most up-to-date patch level on their pc software flash or ie or whatever

do we notify users when software's not up to date and is under attack by known tools

we inform owners of assets when those owners are u of t assets beyond yeah all right cool

and now you get to give away a 50 gift certificate for no starts for an ebook for the best question actually it's not going to be for the best question it's going to be this gentleman right here because he's sitting here taking copious amounts of notes

and maybe next year you could be up here with your presentation right there you go thank you cheers good job all right um peter and mark our next speakers if you could make your way up here and start getting set up and while that is going on uh just a few housekeeping items is craig barreto and i hope i'm getting your name craig is craig in the house craig aha there you are ben is looking for you ben's over here all right you stand there he'll come to you there's service oh he's going to talk all right how many cisps do we have in the house shame on all of you um no no hands back

up hands back up craig is trying to reboot the local isc square chapter um so he's looking for people so go talk to him craig craig right over there okay he needs help he needs money he needs venues make yourself useful and people a date too if you're free bye and i'm still on the board of records for two months more so i'm gonna just emphasize go talk to him you'll vote me in out of spite now that would be hilarious bastard all right a few things here i'd like to say thank you to uh our sponsors without that without whose support this would not have been possible i know we're gonna we said it before but i'll

say it again i'd like to say thank you to scalar e satire who is hiring so go find them uh elastica fortinet vectra who is giving away a drone so if you get your business card into the stein up at the front desk you have a chance to win that and to akamai my day job who provided the full suite of services for our website uh what else we got going on here if anybody is short of seats be sure to find one while we're getting set up here and we did craig we got the beer and right after this talk we will have lunch served so stick around

you