Building an Effective Intrusion Detection Program

here's Jason Craig howdy y'all how you guys doing thanks Ian really appreciate that today I'm here to talk about effective intrusion detection as opposed to crappy intrusion detection I have a lot of thoughts on this topic I hope to share some of them with you now so Who am I I'm Jason Craig I'm an inch lead on the detection and response team of Dropbox the people you heard yelling or my colleagues thanks for that I have the opportunity to work with a lot of really smart people we're also hiring here's my contact info you don't have to write this down it'll come up later so firstly thanks to B sides we should all say thank you to volunteers we should say

thank you to sponsors and the organizers it's a non-profit yes this is a nonprofit community driven conference so say thanks here's he gender for the stock I'm gonna spend a lot of time on three and four not that interesting we'll go through it so disclaimers I initially had 30 minutes for this talk I have a little bit more now I think this talk is probably at least a 50 minute talk I'm not sorry about that we're gonna go fast it'll be a little hectic sorry additionally this work is a collective effort I've had the opportunity to work with a lot of really smart people over the years and these ideas are not all mine alone so I don't claim that they

are this talk is also mostly about free and/or cheap and our open source solutions there's a time and a place for commercial stuff thank you and I I definitely don't want to disqualify commercial software either we find it super useful you may as well but this will not be that talk also the slides will be posted so don't worry about writing stuff down let's talk about some assumptions firstly I I'm a strong proponent and attract driven defense I encourage you all to engage in this methodology as well Azim Lackey has great saw about it a few years old it's still great go check it out if you haven't seen it the assumption is that you

should be attacking yourself first this will lead you to be able to detect things that you wouldn't have found otherwise so you can do this daily weekly monthly quarterly you can do it with an annual penetration test you can do it with a red team choose-your-own-adventure but you should be engaging this type of activity secondly a few words about proactive hardening I'm going to assume that you are all patching your systems like this is 2017 if you're not patching your systems you are negligent and you should be sued and you should feel really sad so I'm not gonna tell that's like 15 minutes of those 93 hours it's really sad so I'm not gonna talk about that

everyone should be like this tall to ride the internet like HT Moore says you shouldn't be operating if you're not patching your stuff Chromebooks iPads so in this day and age if you are faced with malware and implants and exploits that are directed at windows and/or office why are you running Windows and our office I don't I don't hate Microsoft at all I actually like Microsoft a lot but if that's your attack space why are you not running a Chromebook or an iPad just a thought thirdly everyone should have mandatory two-factor authentication on everything if you don't that's also kind of sad we all understand this this problem we face today with password reuse we all

understand how effective phishing is if you don't have to FN everything you really need to do that it'll save you and do it as many places as you can I personally would advocate something with a push mechanism and/or a security key ie you to eff they are the only things that can't be fished if you are using Google Authenticator that's great advanced attackers will fish or 2fa codes just a thought google has open source the password alert extension for Chrome you might surmise that they've been using that for a number of years internally it's probably a great idea to do so yourselves you don't need necessarily to use the air Chrome extension a number of other

organizations have implemented their own which is a non Google version for Chrome and or Firefox it's very effective I encourage you to use that also another thought on phishing there are a number of cloud-based authentication providers that one can pay a lot of money for or a little bit of money and alleviate the password problem so for example there are now a cloud-based authentication providers whereby you no longer type in a username or password anywhere you have an application on your phone when you off to an active directory protected endpoint internally nor corporation you get a push you literally never type your password in anywhere in fact your employees don't even know their passwords just a thought a few notes

about social things be nice to people in your organization we're not known necessarily for being an approachable group of people we have a bad reputation we need to work on that collectively this will pay dividends for you in intrusion detection make friends across your org both internally and externally this will also pay dividend dividends for you for effective intrusion detection we also have a linguistic problem we are a group of people that's known for using a lot of bug where it buzzwords even amongst ourselves that are not easily understandable by people outside of our groups this is something we need to work on to be more effective you should consider every employee you have a canary they are all eyes and ears

for you and they will help you you can't possibly hope to see and hear all the things that they're privy to but if you encourage them to reach out and report suspicious things it'll be really helpful for you we all know that more advanced attackers and now pivoting onto personal accounts because they don't have the same protections the corporate accounts do there is a lot of data to suggest this has been really popular the last two years and you should take it seriously we're a big believer in slack watts I think this is not an original idea slack is probably the first one that's talked about it publicly I encourage you all to to investigate

those they will be force multipliers for your security teams they can provide timely quick and effective methods for you to crowdsource weird stuff in your environment you push the people and you ask them did you do this I saw this really dodgy SSH connection outbound and they can say yes or no and then you spin up a response if they say yes you can then do an outer bound push by some other mechanism to get them to authenticate that they actually did that I don't know of any any organization that's been effectively attacked this way yeah just the thought also make friends with your helpdesk they're your eyes and ears as well and they will see a lot more stuff than you

will ever see so please befriend them be nice and they will help you

so let's pivot into the third topic tool talk I'm gonna I'm gonna throw up a bunch of stuff a bunch of different free and/or open source and or cheap product names that I've seen work in organizations and then we'll talk through some examples after that so firstly on the topic of logging there's a lot of great commercial software out there we use commercial software we also use non-commercial software we're big proponents of both elasticsearch and Cabana plus the last alert from Yelp is a is a great stack I've seen that be very successful windows event forwarding for Microsoft also very cheap pretty close to free scales nicely if you have Microsoft environments and I encourage

you to check that out OS X is a lot more than a logging pipeline but it can do logging as well great log also nice stream work my former colleagues over at Airbnb just released this last couple of weeks at enigma that's pretty fantastic it is not exactly a sim but it's on its way to being a sim so I encourage you to check that out as well on logging some primitives I would encourage you to embrace firstly log as many things as you can you never know when it's gonna rain and you never know when you're going to need the data that you're not logging store it in one place if you can this gets really tricky as you mature as

an organization but it's a it's a good primitive to adhere to if you can keep things as long as you can we try to keep stuff for 18 months that may be a little excessive but a lot of modern breeches or how many three hundred days old now keep data far back as it fits your org but I'd encourage you to keep as much as you can on the topic of logging maturity I'd encourage you firstly to build black lists of bad behavior after you've done that after you have version zero of that I'd encourage you to start looking for abnormal things abnormal saying abnormal things will consume most of your time it's it's hard to discern grey it's easy

to discern black it's easy to discern white when you have reached a point in your maturity on logging where you need to start automating things with your sim you should take a moment and congratulate yourself because it's actually kind of a big deal if you I'm not going to throw up the HR with the maturity pyramids for instant response or any diamond models or kill chains but if you if you have that problem it means that you are very mature so take a moment and embrace that network security monitoring there's a lot of great commercial stuff out there I'm a big proponent for bro if you need to do signature based intrusion detection Sura kata or snort or good options for you

both free and or commercial that's all I'll say about that for now we'll come back to some examples later well that's not all I'm gonna say actually I'm not going to read this to you it's kind of an eye chart this was said at enigma 2016 by the head of tailored access operations arguably the world's most elite hacking unit they're worried about network taps and if they're worried about network taps arguably everyone else is too as you'll likely see over the next examples I'm a big proponent for a client centric approach to security there's still a place for NSM it's belt-and-suspenders you can argue about which one it is probably suspenders but it's it's a good thing to have after you

build client instrumentation so here are some tools that that I've seen be very successful with the Mac OS platform OS query mostly maintained by Facebook it's pretty amazing you should check it out it gives you a rich set of telemetry on an endpoint Santa which is an application whitelisting solution from Google also free gives you a rich set of telemetry about execution driven invests in your environment if you choose to you can actually whitelist apps and have things get blocked automatically if you don't choose that you can run it in a detection mode where it audits executions and it's a pretty rich set of data audit the audit framework is built in the RSX kernel they can provide you a

lot of data there as well about process executions and or network telemetry regular Apple logs you should obviously be collecting too and if you notice about custom agents so you'll see this on every OS that I'm discussing both Mac Windows and Linux there's a time and a place for a custom agent a lot of attackers that more interesting organizations face no the endpoint tools that you're running and are adept at either subverting or neutering them a custom agent is very effective at telling you a when your tooling has been subverted and or be doing things that those tools can do there's a few custom agents running around that I've seen and they're all very impactful very

effective there's also some commercial stuff out there that's pretty good depending on what you want to do there's some commercial stuff that's pretty bad that's the talk for another day in beer let's talk about Windows I think I probably annoy my colleague is a lot talking about this one I'm a big fan I've been running it since August of 2014 we've been running it for a couple years now it's pretty amazing so thanks to sysinternals it's a sysinternals mate maintain product it's free it's performant it works really well it provides execution based logging parent and child process of tracking network connection activity if you choose to configure it that way and a host of other information swift on security has

been talking about this stuff lately so that's great I'm pretty happy because she has a pretty big megaphone to talk about stuff and she's been pushing a bunch of system on configs to github I'd encourage you all to check those out log PowerShell please log PowerShell and don't log default PowerShell you need to enable scripting module logging yes it's a bunch of data yes it's kind of duplicative yes it's kind of a mess but it will save your bacon

we'll return to the many cat stuff later you should obviously be collecting regular system logs as well allegation security in system there is some commercial stuff out there that works pretty well for Windows I would argue the Windows commercial space is much better it's much better than the Mac OS face right now and I think that's all say about that custom agents for Windows - good idea Linux most of us I think run Linux in production not a lot of people run Linux and Corp whether you run it in prod or Corp whether you run CentOS RedHat Ubuntu Debian whatever these are all applicable the audit framework is really powerful OS great again maintained by

Facebook closely they might have a bunch of Linux servers they care to protect I don't know system logs you should definitely be collecting those as well and a new introduction to me at least ima ima it's pretty great you should check that out so let's let's pivot two examples now I'll give you a moment to look through this

so I don't know if you'd notice that John Podesta is a really great recipe for lobster risotto that's on the internet you go check it out for those of you that that aren't laughing John Podesta work for the DNC he was targeted by alleged Russian hackers last year this is the actual email he got there's nothing I've altered it's literally a screenshot of the email that's that you can go find if you choose to go search for it it's a really good phishing email it looks pretty legit it looks so legit in fact the DNC's internal helpdesk said it was legit so again back to hardening if you've u2f this will not hurt you yes your password

will be stolen but no one will be able to log into your account so let's effort some examples on Bob base logs I will start with Gmail then we'll pivot to Dropbox obviously so firstly I can't see anything so I won't say show hands I suspect there are a lot of people that are Google customers that don't do log ingestion and I have anecdotal data to support that and I have non anecdotal data from talking to colleagues at other companies that are Gmail shops that are like what do you mean their logs there's an API whatever your email service provider is whether it's office 365 get on your PowerShell whether it's Gmail like they allow 10 or 15 different

ways to interact with Google Apps api's as admin so so please do that here's one example that we see of me logging in not very informative it's great it's like we have a login success the type was a password good we have the user that's me the IP address I redacted but that's one of our San francisco-based I peace so if we were going to effort some analytics on this log corpus what might one do there's not a whole lot here to effort except IP address so I'll take a short aside let's talk about IP addresses that we can apply to other domains as well so firstly build a blacklist make friends trade trade war

stories about who's attacking whom and what IP is there a lot of public Oracle's one can ask about IP address quality work on that first build your blacklist and if one of these IP addresses in an off log pops up from one of those then freak out and engage your IR procedure secondly after you build a blacklist start building a suspicious list and when I mean suspicious I mean like is tor exit note tor is okay but like we might want to go look at that more secondly is it a hosting provider but is is a is my CEO logging in from OVH in France that's probably not cool like those are things that one could easily in theory catch is

it an open proxy is it a non-commercial VPN you can mark these things up and then effort analytics based on those and then thirdly start looking for anomalous behavior it's like for example am i in SFO like 24/7 365 days out of the year am i maybe in Austin but I never visit Lithuania those are good things to pivot off of so this is a great opportunity to slack waton alert push out to someone say hey we noticed you logged in from such-and-such is this really you they can say yes and we hope that the way we built that tooling that we have a strong assertion that they haven't been compromised so in addition to password phishing I

think an often neglected area for Gmail phishing analysis is authorized apps this is far easier to perform as an attacker I don't know why they don't do it more often but here you see the the Dropbox employee who I've redacted giving this app permissions to their Gmail this is an actual thing IFTTT like this is an automation provider that's an external entity that now has access to our internal email mailbox fun fact you will note two values here you will know that there is a Google user content Apps URI and there is a friendly name there is no Oracle any of you can go ask to say is this legit is this the app I know

it to be outside of performing that action yourself and seeing okay maybe that happened maybe it didn't one can afford your family aim for anything they want so watch out for that so we'll talk about Dropbox logins a little bit this is a query that I slightly modified like four four it's pretty advanced Prarie to display so for example we are pivoting off of all logging activity for drop Dropbox for business source types where the employees are us we look up an IP address database that we have internally in our sim we then search that through this suspect table and then we pivot we pipe that out to slack bot responses and we search our slack block responses to

see if an employee has said this suspect thing is actually approved or not and then we output a report says like stats list these things super super helpful so let's dig into this a little bit more just like Gmail these are what auth events look like for Dropbox the IP address is obviously internal in this case the timestamps right I read acted a few things for for readability but again apply that same IP address analytics to this login like is it blacklisted is it suspicious is it unknown and or is it approved so note about a cloud provider logging like not everyone logs equally it's it's not always a great story depending on who your cloud provider is we actually

provide mobile device IP changes so that when you have a client on your phone and you pivot to new IP and you're a business customer we expose this data that's great if for example you have an employee who's suspiciously pops up in a row country when they're actually in your office so let's talk about some fun stuff windows office macro implants these things are pretty popular these days so in spring of 2015 we were targeted by this technique we got a resume doc Lord it was pretty crappy a bunch of employees reported to our internal phishing alias our crowdsourcing training really worked well it's fantastic by the way this one by all the security solutions at the time so our

meal provider our security provider for your email our client telemetry all went right by crowdsourcing really saved their bacon here in this case it would have dropped plug X read it into that way you want we'll take a look at how we can how we can discover this type of activity on endpoints and extrapolate to a larger class of problems via syslog so this is the actual phishing email we got it looks really bad I think it was intentionally this way so that someone would click enable macros

so what do we do next so here's actual logs from sis Mon if you're if you're familiar with the Windows Event log reader this is from Windows Event log you will notice here that the parent shell or excuse me the parent process is Explorer that's your log in shell you will notice that the image is Windward exe and we're trapping the sha-1 for Windward exe there's a timestamp there's a logon good and there's a process to it you can track these goods for logging activity you can track it for process chaining it's pretty awesome in addition to that you see the command line parameter that says Mon traps to actually open this file so I'm opening

resume doc it's on my desktop it's a default or crawl to do next you will see Windward exe calling command XE / c invoking c script and c script is calling this w SF file like this is super dodgy you shouldn't see this you'll also see windward exe invoking command exe you with the /c parameter and expanding out this cab file also not something when we react C should be doing so I'm gonna skip a few steps there's probably five or six other interstitials between what I left you off with that and where we are with this one apologies this is terrible to read so at the bottom what we have is a comm object at the very bottom parent command

line is this X blah blah blah comm which is invoking this other binary called starter exe that's running out of app data local temp the end result of that is that we now have a new service installed this service is called starter and it's named Kaspersky antivirus launcher super cute so let's take this one example that we were hit with in the spring of 2015 let's extrapolate that out to the larger problem we have an office macro Maur so we have this technique that was very discreet we have other samples outside of the specific resume dock there are some consistencies amongst all those things and back to back to attack driven defense as an attacker you would want to

utilize a variety of different techniques for invoking office macros to install your implants there is not an infinite list there are only so many ways one can do this so you solve it once you solve it for this entire class or problems in your future and hopefully you should be great at the detection so a few examples like Windward shouldn't call c script it shouldn't call W script it shouldn't call cmd.exe space C it shouldn't call PowerShell that's just not a thing that happens it's incredibly rare and if you are using sis want to look for windward calling these child processes you will catch all of this garbage it's super easy so this class of

analytics covered our spring 2015 campaign and a few others so Alexa D put up this blog post right off the election about an apt 29 campaign they got a lot of traction there were over 2,000 targets it was pretty big apt 29 is pretty advanced I think by most people is logic there were five lawyers that they use somewhere zips with Ellen Kay's inside them as one other I forget and then two out of those five techniques use office stocks with macros and they called PowerShell so our spring 2015 campaign analytics framework we applied to the same problem in the fall of last year it still works it's great I say that's like pretty good way to

move on a PowerShell so invoke mimikatz at PS one so I'll explain what that is so many cats is a class of credential stealing attacks for Windows whereby one attempts to either inject or dump or interact with the memory contents of LCS on Windows systems which is where all the secrets are stored for your password sorts where your hashes are stores where your kerb tickets are stored it's been around for a while the latest hotness is a memory resident PowerShell command lit it's actually not that hard to catch it's it's pretty easy once you implement PowerShell logging with script and module logging so these are actual events from our environment this was a head test some time in the

past we caught them in the first hour they were on site they were really surprised and we didn't tell them how we caught them so we let that game play out a little bit more they thought they'd get tricky instead of using invoke mimikatz they're like hey we're gonna alter the ps1 we're gonna invoke pandas they totally won't catch this again thought that super super easy to catch once you know how to hunt for it so a few things about this first of all PowerShell is making network connections it's invoking a couple methods ie invoke expression /ix and it's calling a network connection method in the command parameter itself these are all really really helpful

things to alert for once you solve this for mimikatz you solve it for power sploit you solve it for Empire you solve it for NPS you solve it for every other PowerShell based attack language out there it is super easy to do once you do it once so again back to attack turret in defense you should be attacking yourself with PowerShell as many ways as possible once you start doing that you will discover that there are so only so many ways for one to call call out to a network connection there are only so many ways to interact with memory so let's move on to Mac stuff Mac stuff is pretty hot right now this is an actual

event from our environment this is a red team event we had some time in the past

the plist that this Apple update binary belong to is called kappa level update totally legit in this case it was a ssh based implant that got invoked from launch deep it made network connections on four four three but it wasn't ssl

in this case I think the opportunity for analytics are like a is this P less rare in my environment have I seen this hash before what kind of network connections isn't making does that make sense what's the IP addresses it's talking to so in this case I've redacted the IP that was an Amazon IP and they thought they could blend into the traffic by squawking out Amazon because we have a lot of stuff in Amazon so if you've never touched this Amazon IP before that should probably be a thing that that you look for another output from OS query that was super helpful again from the same exact thing you will notice oh s query has

trapped someone typing ssh agent - l people in finance never do that never ever and we have data to support that assertion I'm guessing I'm guessing that you probably haven't had people in your finance department lists or SSH agents either you can extrapolate this out for other demographics of people - that's a whole nother talk itself in analytics and demographic cohort analysis but it's another day let's talk a little bit about Santa and how you can use hantum for finding fun stuff so this past week on Twitter this document was really popular it basically is the first advanced ish widely available Mac OS X in plant that came from an office macro you can go check out this tweet you can

go find the sample and VT and download it and play with it if you want I think Patrick Ward will blog about it - in some detail this is actually using the Mac Empire framework which is free open source solution you can use to attack yourself with or apparently advanced adversaries can also use to attack you too so this is kind of an eye chart I'll walk you through a few things here so in this advanced talk you can download there are some default Empire things in it and you might want to hunt for default Empire things because one of these people might get sloppy or they might put this code in there intentionally and you can catch it

really easily so for example if you were to go search github you would find this path in the middle section of the eye chart path has been shell it's calling PS and then scrapping for Little Snitch in this case this implant makes a decision to back out if little sanches is detected probably why it's there

this is super easy to catch if you just know how to look for it so I've seen this somewhere today is he here anyway so the advanced threats of this week have caught up to what he was doing in 2015 so it doesn't require that you be a nation state to have the resources to attack yourself in ways that nation-states will attack you it let's move on to network monitoring we're almost done so again there are a bunch of great commercial solutions out there I'm not going to talk about them I'm a big proponent for bro-bro is a network security monitoring platform that usually works on high-speed networks pretty well it parses a lot of

interesting data off of the network traffic you can get a lot of cool stuff that you can't get a lot of other weights so for example if you enable the bro SSH processor you can get SSH client and server version logging you can put this data somewhere and then action on it so for example you see a WN SSH thing up there and it's coming out of a Mac laptop that's pretty weird you might want to go look at that so that map's directly back to that Apple update Mac implant I discussed a few slides ago super high signal if you start looking for it in the last couple years the bro framework has gotten

really good at doing file transfer analysis on encrypted traffic SSL is not magic you can still process a Cell and get meaningful data out and make assumptions around the size of the transfer that occurred it's it's not hard anymore it's great passive DNS you can take this passive DNS data and you can ask whatever Oracle you want about these domain names and make quality assertions about them it's pretty great you might get passive DNS history you might get registrant data you might get hosting providers and then be able to pivot off that stuff it's pretty fantastic it's also well equipped now to search for ssh reverse shells and tunnels in the last year so this has been implemented pretty pretty

nicely so that as an implementer one doesn't have to write a lot of stuff to get this analysis done it's almost out of the box now in addition to that broken poll certificates off the wire for SSL transactions you might want to go hunt for certificate grammar and your SSL connections one fun thing a guy named John alcohol talis did about a year ago was he was like hey I'm gonna find all the meterpreter certs so he did that and then HD more you should have patch the next day and change the logic for finding them interpreter shirts and he's like challenge accepted I want to find the new ones too so one fun thing

you can do with this is just find an interpreter real easy if this is in the framework now you don't do a lot of work to implement it in addition to other certs they're rare that have specific grammar that you want to hunt for because you know there's a bad actor behind that cert grammar this gives you the ability to do that at scale pretty quickly and efficiently another fun thing that bro-bro gives you is user agents so for example if I'm on a Mac I should not have a debian Firefox user agent coming out of my Mac super odd those are high signal things to alert off of and it came back to snore snore

answer cata there are a lot of signature feeds one can subscribe to for free there are a lot of low-cost ones there a lot of high cost ones with different degrees of efficacy you can write your own that's all I got questions comments I think we have time how do you include like a cat big so there it is

cool alright cool thanks for your time I'm gonna post these up so you don't worry about recording stuff email addresses whatever thanks again [Applause]

all right thank you very much to Jason from our sponsor Fitbit a special gift you'll never guess what it is and special thanks to our sponsors Fitbit and hacker one a reminder that at 1:10 we will have the raffle I believe it is next door we also have right now the lock-picking workshop upstairs and the resume writing workshop over at buzz works I believe the next talk is at 1:30 back on schedule so thank you very much enjoy your lock-picking lunch resumes and raffle and we will talk to you shortly

you