Solving Threat Detection - Alex Davies

thank you you guys missed it literally just a second ago he pulled the whole cup of water down why all the electrics yeah so everything may turn off at some point but yeah I'm awesome hi guys it's awesome to be here my name is Alex and today yeah we're talking a bit about solving threat detection so you know where most people I think think of threat detection they think of socks they think of screens they think of rules and all that good stuff but in fact I don't think that's actually a good threat detection I'll be honest I think there's a lot of other things which actually matter more things like collaboration things like building smart

solutions to technical problems and things like building awesome technical teams that you know aren't churning and people even all-time you know these are the things actually I want to focus on today and I think is why she makes threat detection a good thing it's a bit different actually this presentation is young I've got very sort of technical sort of focus presentations this one actually is going to focus on a few other sorts of different issues I guess you could say I'm also going to be talking about as much of the kind of experiences I can that I've learned working to counter doing threat detection sort first hand brief introduction so my name's Alex I head up

the UK's rounding team at concept so we basically do manage threat hunting for our clients on a day to day basis what that basically means in real terms is we collect the whole Lulla data we analyze that data on scale find the bad stuff respond catch the bad guys help our clients be more secure so it's it's really interesting work I get to do a lot different bits and bobs they're working on real world instance investigations attack research development it's kind of yeah pretty diverse my role previous to counter cept I was actually a pen tester and the Riaz I kind of call that offensive experience as well as well as doing more traditional source off work and

defensive work as well big bug bounty lover as well I always recommend people get involved if you haven't already great way to learn new skills and help companies at the same time feel free to follow me on Twitter etc and is a little bomb so for any of you who were here last year at be size 2017 you may remember this this was me on stage and I didn't realize I waved my hands about quite so much it wasn't you know I never realized this into last Friday when one of my colleagues at counter said we had our solve an away day and he did a presentation and had me up on the screen and was making fun of my hand waving so

thank you Luke I don't know if he's here today but basically today you can expect more hand waving I get very passionate about security I'm trying really hard you can see my hands are kind of done with my chest today buying sooner or later I'm gonna burst out here anyway it's gone with the show so the story so far so in 2018 where we are in terms of threat detection well I want to start with the kind of a bit of a I guess you could say a history of how capabilities have changed over time and you can see for the offensive people ever since new hackers first appeared you know things have gone pretty well offensive

capabilities have improved and tools to extract credentials from memory tools to move around networks tools to socially engineer or fish people there's tools to do pretty much everything and it's not just do it like a little bit it's actually doing really well it's actually during this the point where it's combination state level and for anyone who's ever used things like cam call ball strike for example you know it's kind of pointing click and you can pound pretty much any organization that's kind of pretty bad right and this is even worse the detection is kind of lag behind you know there's no secret that we've been poned pretty bad the last sort of 1020 years there's been a lot of

major breaches I think we all kind of can acknowledge that but the main reason for that is just simply because technology has lagged behind that's all really and I think over time you know technology will and will actually improve there's actually an interesting inflection point that's worth mentioning here at least by the way this is a completely made-up graph is the other thing this is based on the facts at all uh-huh so this - this 2008 point that I think is going for was like when mo saw it 0 67 was about but also is about when nation states really saw came into the frame so Russia China the US and their offensive capabilities significantly

improved throughout the naughties and attack sides hook on a main stream around that point and that was kind of a turning point and I think governments companies they began to realize that actually we need to start thinking about detection and you know improving things and that kind of caused this kind of big investment into things now you may have seen that little crossover point at the top there I think that's gonna happen I genuinely do I'm really optimistic about it yeah it's not gonna happen tomorrow it might happen in five years it might not happen in ten years but I think genuinely between sort of 10 and 20 years we may start to see this to the

point where you know attackers have already got any more room to grow where as defenders we've got lots of space to grow here so it's going to see how this kind of plays out I guess okay but why is detection so hard let's look kinda strip it back here a little bit well first of all you got logistical problems and this isn't you know specific to security actually you know these apply to all businesses all businesses are limited fundamentally by money time the people who work for them and it's interesting that in detection and security you know all blue team's probably acknowledge that these are probably the biggest issues these are the things which hold us back it's not

those kind of individual sim rules or that little bit of deployment you forgot to do it's just these bigger issues really holding us back so it's important I guess to acknowledge that and try and solve these things on the other side of things there are more fundamental I guess theoretical or technical problems and you know as as defenders we're sort of up against and especially specifically in detection at least complexity is a big issue what data do you collect where do you collect that data from how do you aggregate that data how do you do the analytics like most analytic frameworks out there at the moment a kind of rubbish right you kind of got to roll your own there's all this

kind of all these problems exist and adds so much complexity to the problem space so while an attacker simply generates a payload sends that spear phishing email gets a shell easy to do the opposite from the defensive perspective is so so much harder and that is why we are behind the curve this lots of other issues here though no knowledge sharing we're not particularly good at that I'd argue I think there's a lot of room for improvement I'll talk about the layer in terms of standards were a new industry so we've not got concrete standards yet people tend to come up with formats for IOC s or you know datasets they just come up with names and fields

and stuff you know there's no standard so we can all communicate and all talk the same language but also lack of testing you know people build stuff and they forget to test it again I'll talk about a bit later but finally keeping pace with technology is a big one as well like fundamentally we're in a fast-moving field and if you're a large slow-moving kind of organization you're going to struggle to keep up again it's a fundamental thing we kind of need to be aware of so everything is kind of on fire is what I'm basically saying here it's not looking prey and I'm not gonna try and sugarcoat this guys in 2018 we haven't gone sirs we haven't sold

throughout detection right and there's a lot of outstanding problems but you know what I think we can solve these bombs and I think already we're beginning to see some really awesome solutions come along to show that this this can actually happen we can actually stop attackers when we can't do things in the right way so let's start with the business problems now I just talk first of all about small businesses small businesses often get sort of forgotten about in all honesty you know these are the companies that you know they don't have dedicated security people you know they may not have dedicated IT people like how can you actually do threat detection if you don't even have an IT

person in your office like it's kind of bonkers to think that would ever work right and this is what I always get a bit frustrated sometimes when people in the security industry get so critical of businesses you know all businesses don't have enough resources especially small businesses and they can I get forgotten about but at the same time there are some solutions here right I don't to be all doom and gloom in particular like on the tooling side of things you know os query grr system on all these are awesome free endpoint agents you can use to collect data and use that for your monitoring also elastic as well free database solution you can look at and

things like Apache spark Kafka all these things for processing your data they'll talk a little more about later the one missing piece now as I said before is analytics there is not really a good open source solution right now to do this and people often roll their own it means a lot of wasted time as an industry the other big ones people so we all know there's a shortage right now in security in all honesty concept we struggle to hire people that's that's just the honest truth right now in London there's just none of good security people to go around and I had a lot of conversations today and with people from different companies and they

said the exact same thing so we're not alone here but the answer for me at least is to acknowledge that we don't have lots of people who'll have lots of experience you know let's instead focus on the next generation let's focus on our intern programs our graduate programs let's get that next generation coming along and again m3r we have an internship program specifically kind of focus on this stuff lastly time so we never have enough time we all wish we have more time at the end of the day but in particular for large organizations for this one people can't forget bogged down in kind of bureaucracy in meetings in emails in calls in just working on the wrong

projects and that just sounds people's time you know be smart about this stuff like look at your calendar and be brutal like shorten your meetings down you know cancer stuff that doesn't matter you focus on the things which matter here okay so it took a little about collaboration so I thought what better analogy to show you guys today for collaboration than Rocky 3 so I hope you've all seen Rocky 3 in a nutshell it's about that guy on the left rocky who wants to become a really good boxer and this guy on the right Apollo who kind of used to be a good boxer but then he kind of gets beaten and then he's kind of anyway but yeah he kind of

basically helps rocky become a bad boxer and it's a great story because these guys from different backgrounds different fighting styles different ways of training but they come together they collaborate and all some things happen and he wins and it's all great at the end of our key three why you know how does this apply to detect you might be wondering well too often I think socks and detection teams especially work in isolation and the funny thing about this is that we all use the same infrastructure let's be honest right you guys use windows right you guys use the Linux we kind of all use those infrastructures right and we all facing the same threats well yeah we kind of

are all of us are against commodity malware all of us against nation states okay so hang Namira we've got the same infrastructure we've got the same threats so we've kind of got the same solutions as well so why are we all doing our own thing and not talking and just working on one solution it's really kind of baffling for me and it's one of my biggest kind of bugbears and things that I'll rant about again and again because I feel so passionate about this the funny thing is is when people do come together amazing things happen and I've had so many good conversations even just today just begin to people and it's because we're here

all in the same room working together effectively the same goes to fur actually purple teaming you know purple teaming is where those things that's really on vogue but the reason it's so awesome is because it facilitates collaboration it brings people together from opposite sides just like kind of rocky and Apollo here you got your pen testers you've got your blue team and they come together and they achieve awesome things together but I wanted to give some more concrete solutions here I didn't want just talked about in theory so in terms of collaboration github awesome starting point here some good projects for detection here Sigma this is a rule set there's basically like gamal a yama based rule set and we

actually use this encounter cept and there's actually a public set rules up here and florian release this completely free go and use it don't invent your own rule set just use these and collaborate you know submit your your code back as well slide bulldog he's a bit of a legend in the kind of defensive and threatened community like this guy he's done so much good stuff his threatens a playbook facing maps all the mitre attack framework stuff to actual detection stuff his assessment really cool resource held cuz well it's early days for that project but it's looking at building a whole pipeline something i'll come on to a bit later and lastly a swift on secure system on

config don't try and write your own system on config there's more someone here just use that face-to-face stuff so account set we run a regular meet up we do this I think once every sort month every two months just try and get people together in in the London area right free beer free pizza like come along I would love to see more people there or run your own meetups I'm obviously we're all here to there be sides which is awesome to see but come to more conferences in terms of forums and notifications the UK government runs the CSP Paul is such a great Paul because it brings together disparate companies and it's like a forum that

both the NCRC release loads of you really interesting a useful one like a threat reports on but you can also share your own experiences like if you get a suspicious email or you see something weird and you need help just post on there there's those are people looking to help each other the same for kind of the Facebook fair exchange calm black user exchange all these places bring people together as well as that hidden Cobra alert that actually from the Department of Homeland Security which is doing an awesome job or actually publishing more content these days and the last one was Twitter like probably about 50% of everything I know I learned on Twitter like genuinely

is the most awesome resource Twitter for security knowledge he was just a great example this one from tre I've never met tre but he's an awesome guy because he released this just awesome bit of like information here he didn't have to do this he didn't have to tell us that you can use comm grids when you call him parameters when you're running w script or C script um but he did he did that and he did that for free right why aren't more of us doing this you know collaborate and use this information I also wanted to give a big shout out here to red canary and I know this is kind of a really controversial slide because I

work for counter set we are literally a direct competitor to these guys but you know what I don't care about that business side of things these guys have done awesome work and I really wanted to give a shout out to them because you know there's a lot of companies like like can fire I like Cisco tell us some people like that who do sort threat reports and you'll see ok the latest a PTO ever coming out and then of steps cool and it's useful but at the same time they never tell you how to do detection and that's the poem I'm struggling with that's the prom work loads of people are struggling with right and that's the

prom where trance all and you can see here well that's for then sharing they're showing you okay well you can look for net connections coming from CSC which is the dotnet compiler there's a rule you could try and use at the top you've got some meme account stuff we're actually just looking for me because arguments on the command line well you could just use that in your own organization and likewise they call their use cases on the Left they're cool like these guys are just share this for free I just love that and like I couldn't help but sort of give them a shout out today it was also quite interesting to see how a lot of what they're doing

brew is what we're doing a counter because again he just shows that reinventing the wheel thing it's it's kind of bizarre that we're all doing the same stuff but behind closed doors I guess obviously give them a beer I couldn't hack it out had some in in okay so getting a bit more operational so this is kind of moving more into so that the technical space I guess you say so before to point any kind of monitoring it's important to focus on I guess what attackers are doing in in real life now this slide is actually from my presentation last year you just thought I you know I've broken some kind of presenting rule by doing this but in

fact you know I I tried to kind of improve on this but in fact nothing had really changed in the space of one whole year all of these same things were just as relevant you still want to look for pretty much everything on this slide if you're doing threat detection the too little or two or three little bits that have changed there's more ways to execute code on windows these days you know with the sort of slow integration of Linux into Windows some of you may have seen bit less PowerShell a bit more c-sharp that's true office365 as well attacks are on the rise it's something we've seen a concept affect quite a few of our clients so yeah you do need to

incorporate that extra stuff but the point here in general not all has changed and especially actually for uhm there's a subtle difference as well here between like what pen testers do which is always kinda nice weird like stuff that young UGC and then what real-world attackers are doing they're kind of different sets of activity and in terms of real with attacks we still just see a lot of binaries being dropped a lot of scripts being dropped and traditional persistence being used attackers really haven't gone into this kind of weird funky stuff yeah I'm not seeing that with c-sharp for example because attackers don't need to do that why would they of course right okay I wanted

to give a sort of a top secret plan here about how to sort of do some detection on sort scale and the whole concept here is based around sort of a pipeline when I say pipeline I'm basically talking not about a literal pipeline but about moving data from kind of a to be doing some kind of processing on it and having some kind of useful output and then if you're a small company or not very mature in terms your security posture start with the simple stuff here right you can take in you know EDR data over network data and if you have a simple UI like Cabana or elastic or you just simply use your EDR UI you can actually

get a really good value out of it and for a small organization that's probably a really good starting point now for more medium or larger size organizations you kind of want to get a bit more come right because you want to actually start integrating your datasets together so they all can have actually I guess enrich each other and complement each other you can correlate them up now this again is what everyone is doing we've done this at concept this is literally our pipeline what I've drawn on the board here so I'm just gonna sharing that with you all but you know what Facebook have done this google have done this Netflix have done this everyone has done

this pretty much if you're a larger organization again it's kind of bonkers why are we all doing this why not just release you know a single framework and you know I mentioned in cyberwar dog earlier and his Hulk stack that that actually is trying to do some of this in an open-source kind of way do the midpoint of engineers in terms of your security team looking at that automation is really important like try not to have like kind of man in an investigation here the more you can speed things up the better you know having things go direct to system owners as opposed to the security team is a really nice kind of rising trend we're seeing I think

this is kind of pioneered by people like Dropbox and Netflix where instead they'll just kind of prompt users if they see a suspicious login and it won't even go to the security things like that are kind of really good for efficiency in case you're wondering and I'm not reinventing threatening today but essentially is a proactive approach to going through your data and it's focused more on either the raw data set what kind of the early stage past data set you're not really looking too much or kind of the alerts at the back end that's a bit more of a reactive approach so get a fewer less mature organization you tend to be a little bit more

reactive that's fine but as you build that maturity you want to start getting a little bit more technical get you guys to actually work with the raw data and come up with the new hypotheses and that basically match over what attackers are doing kind of here and now try not to build a house of cards here and also try not to waste too much time on this so many companies fall into this kind of this pit of trying to build this awesome infrastructure that kind of rules them all and it takes so much time it takes so much effort takes so much so many people and you get some out at the end of it that kind of half works but

doesn't really work like really trying to avoid this like focus on the quick wins focus on what gives you value straightaway and almost try and build organically instead of trying to do everything at once I guess okay so I want to give a quote a few sort of quick examples here of how this translates in in reality I guess and these are again framed in terms of maturity so basic stuff the bad guy ran a bad file if someone runs me me counts on your network and if you've got things like next-gen AV or even regular AV if you just pick this up and you can just find X is based on hash based detection or signature detection

that's the basic stuff right intermedia this kind of verge is more into the kind of the very cagey tectors stuff or what we call rules at counter sap where we're looking for specific patterns of activity occurring some of you guys might recognize this this is just a typical macro execution rule so you're seeing PowerShell coming out of wood but again this this is kind of a gray area now and we're kind of getting more gray as we move down the screen here you're going to get some false positives potentially from these kinds of rules and you've got to tune these rules over time for your organization okay now the final one this one's a little bit harder right so if we got a

bad guy he's a legitimate account to deploy a legitimate named embedded encrypted payload execute from a service that loads into memory and uses a virtual file system to load remote modules dynamically yeah it's a bit of a tricky one this one I'll be honest and you may think I'm just kind of with you guys but I'm not this is actually real world so we had a total of compromised early this year that involved Russia essentially and they their payloads you know well-documented there's actually a post down here from carbon black you know this is nothing secret going on here it's being documented already but what was kind of interested here is that the previous techniques of just using a V or really

obvious like threat intelligence kind of stuff or even intermediate rules for looking suspicious activity they were really effective against this stuff this stuff was more kind of well you find this stuff I guess with more like threatening kind of techniques you'd be looking for like that suspicious service you'd be looking for may be suspicious use of an account moving around the network even though it's a legitimate account does it usually log into these boxes all this stuff relies on more anomaly detection type stuff and this is really what's kind of still coming along in our industry in all honesty it's not being solved like people hype up ml a lot but it's really just at the start of its

journey I think the last bit here as well just in terms of memory injection stuff as well really useful techniques I mean we deploy a lot in can seven actually helped us detect some of this activity so you've built this huge pipeline you caught your data coming in you got some alerts you're thinking everything's great and you can just kind of put your feet up right that's the kind of mentality unfortunate with a lot of security teams and socks that they kind of think well there's no alerts everything is fine it couldn't be you know more far from the truth right we all like right now this is a really big issue in security and specifically in detection and the

main reason for that is because detection fails silently when detection is incorrect me you don't know about it unless you properly test stuff and luckily though in the last year we've had a whole multitude of different frameworks come out red canary again uber endgame next one which fluorine is involved with all these awesome frameworks that effectively do the same thing it's worth mentioning MWR we also built our own framework right we do want to miss out on the party so we've got our own I'm sure ahead and CC and other people have as well it's a funny one again it people reinventing the wheel both here on the kind of the offensive side as well as a defensive side which

is kind of annoying to see but again to come back to the color point it's all about testing your systems if you build something don't just assume it works like properly test it run the use cases like if you think you can detect PowerShell run some PowerShell it's not hard and actually test every single use case that you have to make sure it's properly triggered triggering that yeah it's all about the purple team in right show some love I was not to mention metrics today because it was something that really was a bugbear for me too as well and wants to avoid events per second alerts attacks on the internet face in infrastructure just generic alerts

created by generic products honestly man like these kinds of things like these have given detection such a bad name and they create such a bad impression we really want to stop using these as much as possible because they actually are harmful in many respects because detection teams end up almost getting kind of tied to them if that makes sense so if you know that okay I'm getting measured and getting my bonus depending on my alerts per day or my tickets per day well I'm going to increase the number of tickets on making like that's a really bad thing to kind of encourage people to do there are other ways to doing detection metrics though I think I think Dave you

mentioned this one maybe earlier as well was my toe coverage times by EDR coverage this is probably the number one prime metric you should be tracking right now so mitre frame doesn't know essentially is a big matrix of attack vectors essentially all types of activity that attackers can perform you want to multiply that by your coverage because again you may be able to detect all the mitre framework but if you've only got coverage on 50% your endpoints you're gonna miss out a significant proportion of State and you can be very vulnerable use this metric it's really good investigations times time spend an impact so what matters with investigations in ticketing is not quantity but quality it's way more

important that you stop that one high-impact breach there was from that you know that inside all the contractor as opposed to you know oh I stopped a ferris and you know IPS rules that got triggered or whatever like focus on the quality stuff that actually really matters here true positive false positives raised so many organizations and detection teams get bogged down in false positives make sure you're tracking when you get true positives or false positives and basically sack off the things which don't work is that simple so many teams I I sort of speak to and see they almost get kind of lumbered with things that don't work and instead of doing something about it they

just kind of continue like doing the stuff that doesn't work and it's just kind of baffling because a counter said the least we're kind of pretty ruthless if something doesn't work we just change it it's as simple as that right and the last one here is about skill levels this is more about your team over time right you want to build your team you'll build an epic team and you want to stay you don't be able to leave you don't churn like because that's a real big issue right now in Sox - I'm gonna talk about this coming more details in the next slide okay making detection sexy this is the final section study guys imagine if you

came to work tomorrow and this and this was your job your job is to look at this screen and click on these and investigating them one by one you create a ticket for each one man like I can do that I'd hate to do that I thought was my job I gotta be honest and this is I think this was arcs I and I'm not sure I don't know what Red Sox is but I think it's some kind of IPS IDs type thing and you can see like use this product before but just looking at the others I know these are extremely false positive prone your long-duration connections aren't a good indicator in most organizations you have long

duration connection from loads of applications and servers and systems and stuff in terms of threat intelligence is again a very false positive prone just based on IP you know this is a very painful set and the thing about this is that we've in some sense like created virtual sweatshops where our stock analysts are forced to kind of work their way through these really painful things day in day out you know and for me you know we have to solve the detection problem but there's gotta be a better way than this I'm sure there is right let's just have a look at this in a more detail so in terms of traditional socks you've kind of got a lot of allure

clicking that's the main thing right you're very alert focused like people don't often talk about raw data but rule there is where all the cool stuff happens what can't we look at rule data why we look at alerts all the time it's the first thing second thing the wrong date or wrong use cases like I said before don't focus on the low value stuff you know don't feel like you have to go and dig through your firewall logs continuously you know there are low value source don't use them so you know the other thing is not enough information right again to our firewall logs you don't get enough information for them so don't use them focus on

other sources because at the end of the day I think a lot of socks do become glorified help test and that's not a good position to be in right because at the end of the day right you're not gonna want to go to work all right you're gonna hate your job well let's be honest right and what we see in you know socks is churned because of this but again this is about solving these prongs on the solutions here so I can't I said we're focus on the high fidelity stuff again something red canary is really kind of pushing as well is to actually verify that how true or false positives your signatures are how effective they

are over time focus on them things heavy use of automation in ml if you do something more than two or three or four times in a day why haven't you automated it like basically a lot of the work that could be automated in socks is just done manually and that's why can I give that sweatshop reference because it can actually genuinely be automated and the same for ml again it is infancy but already in security it can really help with a normally detection and processes and things in an automated way daily collaboration so don't lock your sock away like allow them to interact with the rest of the business you might have a red team you might have a dev

team you might have an IT team let your sock actually interact with him share information understand how your business works or even just come to conferences I used to collaborate or even look at Sur comments this is something where it was trialing encounter set to the moment when I know a lot of big organizations also do this you're having people come out of your sock and actually move into a separate team for a period of time you know maybe a few weeks or a month or so they get exposure and they get a bit of a bit of salt change in their day to day work I guess and lastly R&D this is something we're actually really big on

and that we are you know all of our guys have 25 percent minimum research time by default a counter said this actually goes up to 50 or 75 percent and again this really breaks up the role and this isn't I kind of don't think of this almost as like pandering to people's like kind of wants and stuff it's actually a massive value add to our business because it means our team are actually streets ahead of all the other detection teams and also the attackers because we're doing new stuff we're reinventing stuff all the time because of this research so we for us it's actually really important as a business that we make sure this is actually

having you do all this stuff and you're gonna team up absolutely rockstars right at the end of it which is where we should all be trying to head to like make this happen in your songs all right so just to summarize guys because I'm getting okay sure off stage um here's the summary of everything I've kind of talked about today I guess to kind of wrap up collaboration make it happen you know come to conferences coming interact with people detection and focus on the endpoint it'll actually give you the biggest value so make that your priority terms of people treat your treat your team well feed them water them and build roles that actually you know are

enjoyable to do day to day but yeah I think that's it cheers guys [Music] [Applause]