Hashbrowns & BAGELS

so our final speakers yeah yeah actually inaugural and world champion we should really have some like WWE music like you guys rolling in give me some John Cena these fine gentlemen are here to talk about things specifically tools that they built so Mike Chris and Andrew are all senior cyber security researchers at GE and so thank you guys for or your organization for sponsoring and you know again they're here to present on a tool that they have built and so last talk of the night and we're excited to have them so take it away guys Jack check this thing working maybe maybe no all right hey guys so the presentation that we're going to give

today is one that we've titled automating the hunt and it's about endpoint hunting and different methods that we've come up with you know on our research team over the past couple years specifically we're going to talk about two different tools that we've developed called hash browns and bagels I'm gonna go ahead and get the legal disclaimer out of the way real quick hash browns and bagels are copyright and these are tools that we have developed you know through our company GE we would like to open-source them and get this out to the public but at this time unfortunately we can't like really share source code so I'm just going to kind of break that news right off the bat what we are going

to do is we are going to talk about the techniques and we're going to give you those limits and that sort of thing and really the whole you know motivations behind this presentation is to get folks thinking about you know how can we do things better how can we you know look for bad things and how can we help ourselves by automating those processes and do the better unfortunately that means unfortunately we just can't give source code today okay so who are we we are these three fine-looking gentleman up here my name is Mike you probably recognize me because I had the best haircut as mentioned as mentioned in our quick introduction we all work for GE

power bout to be GE digital with power and we're cybersecurity researchers so we kind of look at you know the hard problems and try to come up with tools in different ways that we can do security better I'll let the other guys to cool themselves yeah yeah I'm Andrew I'm a colleague of Mike's on the G power team

and I'm Chris better I've been on the team for a couple years now prior to that lighted security patching and stuff for GE so I've been in a semi security space for a while but decided to be here sweet we won't we won't belabor that for too long all right so what we going to talk about today what does a roadmap look something like this so we're going to talk about you know a little bit of basic stuff I'm sure that a lot of folks in here going to be familiar with some of the concepts that we're going to talk about in the back to basics section but I wanted to go ahead and and you know conclude that section

is to put everybody on a level playing field second thing we're going to talk about when you talk about hunting for static IOC no talk about really what that means to us and you know how we've done it in the past and that's going to segue and see how how can we do that better how can we hunt for static IO C's in the enterprise environment better but we don't just want to look at static ion C's as several the other presenters have talked about today there's a whole different group of indicators that we can look at called behaviors which is essentially you know looking for ways that your system might be doing things maliciously by looking at the haters in

the system and then we're going to talk about how can we do that better haven't we automate that process all right back to the basics I want to start with you know what is threat hunting I think that if you ask 10 different security researchers are starting to different security professionals not just researchers you know what do they think you know right hunting means you're probably going to get about seven different answers maybe even you know more than ten actually I don't know but I like aynd of like the Wikipedia definition not because it's super easy copy and paste but because it's abroad and it kind of just gives us a you know basis to go from you know the process of

proactively and iteratively searching through network to detect and isolate of an threats that evade existing security solutions that's kind of a generic definition and and that's kind of wildlife you because you can kind of you know adapt that for whatever you mean but really just getting hunting for bad things means finding the things that are you know wrong it's it's the whole you know symbol breach mentality and then go out there and find it you think you have to think like an attacker and you think you know what would an attacker do what kind of post exploitation things what they do and then go out there and try to find it within your network how you

accomplish that there's a whole bunch of different ways to accomplish that we're going to talk about just to two different tools that we've bought up over the past you know year so this presentation to be focused on intelligent driven methods and I'm gonna talk a little bit more about what that means and upcoming slides for hunting on the input this is going to be in contrast to other approaches that are more like you know behavior analytics or anomaly detection so intelligence driven means that you have information about the bad guys you have information about what they're doing whether you gather that information from your own Intel program whether you get an Intel fee from somebody else or whether you get

that from you know partners or something like that and all of these are very valid methods of approaching it anomaly detection and behavior analytics those are awesome things to be doing and we absolutely should be doing all of these things and that's going to be kind of one of the themes that I want to talk about kind of harp on today is that there is not you know one size fits all you know one approach now I'm going to you know install this tool that does user behavior analytics and that vendor told me it was better than looking for static I OCS and therefore you know their pitch was the best so that's what we should go with no my theme and my

message is that we you know have gotten to the point where we can automate these things and we can do all of them we don't have to choose you know one method over another method we need to choose the best portions of each method combine them together to an integrated strategy that works you know across their enterprise so what is an indicator of compromise I'm not going to spend a whole lot of time on this night on this likes I think a lot of folks here probably do I know what these things are I decided to splash up here the Pyramid of pain which was developed by David Bianco who is a GE alumni and what this

what this does it kind of talks about different indicators that we can be looking for you know starting down at the bottom you know IP addresses I think he actually added another layer below this which was hashes but then you got things like IP addresses domain names network artifacts host artifacts and then as you start to get to the top of the pyramid pane you start talking about a little bit more abstract concepts like tools and TTP's or tools tactics and procedures no actual methodologies and I think this is just a good thing to look at because it kind of shows the different things that we can turn into indicators and the different you know

ways that we can approach detection and hunting so what are indicators come from well in our case they come from Intel and like I mentioned before Intel is where you know something about what the bad guys are doing there's a whole bunch of different ways that you can get that you can get that from threat beads would you pay for some of those are limited some of those are better than others you can get those from and a threat feed might just give you a whole bunch of domain names and maybe you don't know how to use those in your enterprise they might be completely useless so if you're looking at different threat feeds think

about you know where you are your maturity level you know what schools are going to go into that sort of thing you can also be part of an information sharing groups such as you know government information sharing groups like DSi II is a defense based where you know a bunch of analysts get together on a forum and they talk about attacks they've seen they share your rules they share different you know phishing campaigns that sort of thing there's industry based ones Isacc we're part of the energy is AK the financial isaac is one of the most mature ones there's also open source threat Intel all of the font color there is not showing up at this but that's a link to

a github called awesome threat Intel which is has a ton of resources if you guys have never been to that I'm not sure the guy that maintains that is the whoever it is serious kudos to them because that is a great source of open source in tell people and then I pose the question are IO C is dead and I believe if I actually have another swamp nope talking our IOC is dead this is kind of a religious debate to some information security professionals you talk to some companies and they say yeah we don't care about IOC anymore because you know you know static bytes just aren't going to cut it because you can change you can tweak something well and

all these things go out the window well sometimes just sometimes no like I said before I I don't agree with the mentality of ok well you know this approach yeah you know user behavior analytics or anomaly detection or something like that is going to be the end-all be-all I mean I think that we have to take each approach and look at the tools and what they actually provide and integrate them together I don't think I was either dead I'm going to talk a little bit more in two slides on what it is and it's a transition slide maybe it's inappropriately placed I don't know hunting for static IOC so I'm going to come back to that question why

are iOS you dead on the next slide when I talk about you know what's working in enterprise and what's not but first I want to introduce a tool that we use Gaara show of hands how many people know what your is and use it on a daily basis all right some okay cool so your is the pattern-matching swiss knife from our researchers it's kind of like grip on steroids in a very easy way of describing it but it really is a lot more powerful than something like graph it basically allows you to create your own for a lack of a better word I know somebody's going to kill me like AV signatures or something along those

lines basically what you're looking for is you're looking for byte patterns and it's super flexible it allows you to put in you know variable number of bytes allows you to put in all kinds of different wildcards it allows you to with using their modules it allows you to look in certain sections it's very smart it knows what kind of files you're looking at so with some of the later updates and that you can look at different sections of like PE headers and that kind of thing and what this does it allows you to create static signatures that are very valid for different you know malware families and very you know various tools not just malware but you know other tools

the files that are a little bit more robust in the sense that if they tweak something it's not and you know they tweak something it's going to change the hash or they tweak something it's going to change a domain name it doesn't completely invalidate your signature if it's written correctly I mean obviously you got to have you know you know well written and well the quality tested rules you can't just you know okay well you know if it contains all these strings then it's going to be a good rule I mean you got to think about you got to have you know real reverse-engineer you got to look at you know bite patterns I mean you can look

at things like you know the way that they might push bytes on to a stack to create strings so a lot of malware like the obfuscated itself and so when you run strings on it there are no strings it's all completely obfuscated but there might be a routine in there it pushes four bytes onto the stack and pops them in a certain order and then all of a sudden the way that it does that create the string well you could look for those instructions you can look for those push instructions you can look for those pop instructions you're not actually looking for string values you're looking for you know specific assembly code instructions it's not the level that you are gives

you gives you the ability to go in and actually look at that kind of thing so that's the primary tool that we use to look for what I would call static IOPS and for lack of a better definition we're talking about file contents malicious files whether it be artifacts drop by malware malware themselves that sort of thing you can also look at i/o sieve inside a memory you know memory only IO C so you can also use across the network with frameworks such as you can integrate it into frameworks like bro you can integrate it into like a box those sort of frameworks you can use this as well so you can use it on the

network you can use it on the host the project that we've been dealing with is primarily on the host all right so let's talk about the challenges let's talk about first the good things that we've been able to do and historically and let's talk about some of the challenges that we face so really you know provide motivation for what we're working on here so how is static hunting been working well within the environment Intel sources are really good I mean we've add a very good community built up over this like I said I mentioned different various ice axe I've mentioned the various guy organization there's some semi-private groups out there regard exchange is what I would consider a semi-private group

it's like it's technically restricted but I think you can just send the curator an email and just tell her who you work for she let you in so there's great communities out there and these and what that means it's basically when a new fishing campaign comes out a company gets to helping its pop baby they hop on these forums they write these signatures and then you get that information pretty much right away because of that robust infrastructure that's already set in there I can't really say the same around some of the newer technology some of the things like behavior analytics and anomaly detection you don't really see a lot of folks popping up and saying okay well you know

this is how I set up my you know machine learning base lining for this you know network so it's a little bit more vendors heavy at that point so so as far as these static i/o season using yar I mean we've been doing really good in the whole like Intel sharing world high fidelity with proper QA and proper testing the rules that you create using yarra and you know various you know very simple scanners are very good you get a hit on that it's probably going to be malicious as long as you wrote the word you can always write crap rules people do which is why we have Andrew over here working on you make code for which he

published you actually has a dirty comm presentation on it it's out there in the open for folks who are interested in that school it's a different residential email scanning email scanning is great you get emails in all the time you can quickly run it through some yard scans you run it through a whole bunch of rules you know but there's a malicious attachment or you know that you have malicious code potentially present in those attachments pretty much instantly and one thing that I want to kind of mention kind of backtrack for a second why do we even have this as opposed to you know AV just using AV AV do the same thing well yes to a certain extent but

the thing here is if we can leverage these Intel sources to create custom signatures for things they're you know bleeding edge late breaking that sort of thing which you can't do what you can't necessarily do with a Navy vet or you can't call a Navy you've interrupted saying I need a signature - Floyd now that's a difference do they do sometimes have you know helplines and commissions and that sort of thing but but and more robust programs you want the ability to create custom signatures that's that's kind of the whole reason why we do the students to begin with most important analysis so if something goes wrong something goes bad you have a whole bunch of files on a system you believe

that some of them are licious why not just scan them all and it works pretty effectively so what's not working or what has been an issue I'm not going to say this hasn't you know completely works but what have been issues full system scanning the rule sets get super big and if you don't have a very tight Intel program it's constantly weeding out you know bad rules you know that sort of thing do a full system scan across a hundred thousand endpoints has turned into a nightmare I mean it's something we've struggled with for a long time it's hard it's just hard to do it there's it's very resource intensive oh yeah you have to you have to throttle

it back because if you run these things full on if you run you know a thousand rules through five hundred thousand files on an in system computer your internet danger space heater and users tend to not like space heaters I don't like their fans going up to you know a thousand CPU I mean that's it that's one thing that I always thought was crazy you get these folks that have no you know background and I see at all but they definitely know how to open up a task manager and tell you which process is running at 90% CPU and they kill it anyway no matter what it is kill it and it'll send you an email being like why

is this running so so just getting these things accomplished I mean we have to have to you know restrict the CPU utilization we were having I meant full of numbers I didn't so I guess I'm lazy but we did this when we were doing this you know just running you know basically all of our rules through the entire system it would take a week to complete we'll say we take if we have something systems you know 20 hours plus to complete real-time scanning real-time scanning scanning is something that's hard to do with just you are by itself right I mean you can't as soon as the file is created you know it might be you

know another week before you get another full scan and we know now in days the artifacts don't generally last on a system for that long artifacts are usually pretty ephemeral what if we could stop scanning the same files over and over and over again because that's pretty much the approach we had you have your giant rule list you have your giant file list and all the files with all the rules and we do it you know once a week and then once you think about it that's a lot of duplication because you're standing the same files over and over and over again week after week after week and not only that but if you ever look at a hundred

thousand endpoints 90% of files are the same because most of them are operating system files and they're completely legit you don't need a scan to begin one and also what if we could scan files in near-real-time so those are the two things that we're looking for when we develop hash browns so the high idea behind hash browns is we're going to we're going to do distributed scanning so instead of just scanning one computer with the entire rule set we're going to break that rule set up and scan all the different files no matter where they're located in the entire network just one time we're going to use dynamic rule sets so that only the rules that need to

get deployed to a particular file are going to get deployed so like if a new rule gets created and a new file comes along and maybe this file has been seen before in the past and we scan it with half of the rules this time we're only going to scan it with the other half of them so this is a dynamic dynamic rule set and there's a couple different ways we're going to accomplish that we're going to accomplish that by first taking a full system ending fitori and what that means we're basically going to collect information and all the files in the system and so that way we know what all the systems have after we get done

with that we only have to take partial inventories we only have to take inventory of the files located on the system that have been created since the last time we've done the initial inventory and how are we going to accomplish all this where are we going to throw this data well Wow the point-seven team and we can put it in the cloud or something like that so I mean seriously though I mean we live in a great time this is what I'm proposing here essentially is taking a full system inventory taking hashing all of your files that's metadata we're not actually taking content and we're taking locations of where all those hashes are located and we're going to ship them up

to a centralized location or distribute a cluster of things data like I don't know some data scientist to figure this out they tell me that they tell me that this stuff scales but we live in a great time with what that actually is the thing is x a real thing that we can do this we can store that amount information by the way your average computer has somewhere between 200,000 and 500,000 files on it roughly somewhere in that range your if you're outside of that range anyways that's rough it's roughly the the range that these things are annual so storing that information up in the cloud allows us to track where these files are and allows

us to scan them with only the rules we need to do it there's some awesome benefits from that which I'll get to sure now benefits to that is is that by the time that you get to by the time that you finish the full inventory and you're only dealing with partial inventories you're only dealing with instead of dealing with 500,000 files you're only dealing with a couple hundred files every time you do that check and just gain a couple hundred files only takes like a second or two and so what you can get at is you can reduce the loop of how often you scan what's where you can catch newly created files I don't recommend scanning every

single file on access because then you're talking about like hooking kernel drivers and that kind of stuff and obviously there are product out there to do that but you know if you're not trying to get into kernel land which I'd I tried to stay out of kernel eggs I don't like the kernel and break things what you can do is you can get that that loop down to just you know every couple seconds check and see what new file isn't rated request you know a new rule set from the server and scan those in it's super quick so instead of waiting an entire week to determine if a new file has popped up on your system is

malicious you can get that pretty much in your real gun so let's talk about how this process actually goes I'm going to happen way through this no one can read that right I can't read that I'm standing right next to it sweet we don't have time for this anyway so we don't have time for this anyway so I'm just going to say what this is so basically the first part of it is you have to take an inventory there's a couple different ways that we take the inventory we take the inventory just by running through the entire system looking at all the different files on the system taking the hash of them and then shipping those back up to the cloud

and saying here they are next way that we take an inventory of partial inventories we say hey server says take control at a client last time you gave me an inventory was this date tell me all the files that have been modified since then then the client will go check all the files on this thing on the system we're talking about you know agents that are running on your on your client here it checks those then it sends back only the ones have been modified since that date that it gave it the third ways we actually use sis Mon and Chris and Andrew are going to talk to you a little bit more about some of the cool things you do

with sis Mon system on you can actually create a listener that listens for when new files are created and then chews them up so that way you don't even have to talk to the controller there's a couple different configurations here so you can control how often it does a check in with the controller you can control you know what information is actually stored on the machine and you can control you know how big of the - how often you want to check that queue for new files this this one is basically dumping into the queue so lesson learn here is the first time that we did this we wrote it in c-sharp and we use the

windows api is to go all the way through this using the windows api is that you know do the whole directory traversal and the whole you know it's trying to find the creation date we ran into a big problem can anybody guess what that problem would be by using windows api s to go through the directories and find out where all the files are at huge problem and that is av man hates that and the reason why a V hates that is because any time that you have an open handle using the windows API AV is going to scan it for you so we determined that when we were doing it everything was getting scanned all the time by like

Windows Defender and hips like literally every single file on your entire system it's just like totally destroyed it so don't do that either going to have to whitelist your process so that your AV doesn't do that if you're going to use this in like a higher level programming language like C sharp or something or you can or you can just get really familiar with the Master File table and learn how to use roll access and traverse the Master File table for this which is the way that we're going to go down I'll regulate the process of writing that code how our show umpire actually has a library to help out with that okay since buddy's

interested in learning how to access for all volumes read the master file tables and go through them too to find out where things are located this is a another chart that nobody can read and essentially what this is talking about is how scans are created once the once the client checks back instead taking a new scan and there's a process which is here but essentially what the process is is it says ok which files have changed it's the last time that you checked in what rules had they been scanned and there's a collection of databases I looked at those databases and it says ok these files need this rule these policy that rule we do some averages because

you know files might be all over the place so we say ok what is the mode you know what is the most common date that these things were last scan or was the most common rule and they were scanned with anything above that you know sent to this group of files anything below that into this other different files so we have a couple different database tables that we set up here we have this one called the file hash table this is where we just store all the hashes across the entire enterprise so your client checks in and checks in with this information it provides your sha-256 because that's the best Shaw it provides the last rule that that particular hash

was scanned with it provides last time the status was updated in the first time it was seen but really the only two important things up there are the hash and the last rule it was scanned with and then we have the file table so this is a totally separate table and again these things are going to be storing a lot of data so distribute that or something so the file trackers are actually storing the locations of the files and these things are linked with a foreign key to where the hash is actually and so in this case where we have a UUID for each file which is just dynamically generated there's some code there at the bottom this is a Django

model here the client that it was found on yes yeah because each tracker is specific to the client the file has which is a foreign key to that previous model the actual hash of that because the idea there is that the same files are located on multiple computers so multiple files will have the same hashes and the file path which is just a text field so you can actually find it the last thing here is we actually have the alert model so once there is an alert you just basically link it back to the file hash in the file hash through the magic of foreign keys the actual the trackers can link back to the file hashes so every time

there's alert it has this beautiful a byproduct of whenever you get an alert on a file you actually know exactly across your entire enterprise what other machines on that enter on your enterprise contain that same file because you have a foreign foreign key link from the file trackers to that hash which now has an alert also associated with it so if you identify hash as being malicious you can know you have a hundred thousand machines that that's you know that same attachment it was downloaded by you know six other users and you know five different countries which is really cool if you have a big organization like ours so this is a pretty big feature for us so folks are

interested in this this is this again I know that a lot of the details we really couldn't drill down to that some of us because of timing of this presentation some of it's because we're not able to give out source code right now so what I really just wanted to try to drive home is that the beacon we can do things automated we can hunt for rules we can distribute these these searches across the enterprise leveraging you know of centralized controller leveraging the cloud leveraging you know distributed services if we might not have been able to do a couple years ago and if folks are interested learning more please parrot everybody to come talk to me

about it and we can get into more details just real quick a couple lessons learned speed increase so before we were seeing you know alerts that we're getting hit you know maybe a week at a time that was assuming that what hands actually completed now we can see malicious file getting detected when we did it in the grant environment I think showed up one a minute after it was created and that's not even the final form we're still developing internal times of detection like I said you know we can we can see things within a minute enterprises where is the threats that's where you know you find one has a malicious you exactly the computers is located on

let's and learn is that you need to use raw volume access because you're doing your one addition Navis storing rules locally it's another thing that we're that we've there's a whole bunch of different ways to do that if you're interested talk to me and scanning newly created files of Tom analysts autonomously and what that means is whenever a new file is created you know that more than likely it's going to need to get the whole roll set so if you're storing the rules locally just go ahead and scan them right off at that and then report badge of the controller later that helps reduce some network traffic you know that sort of thing so this is a

hashbrowns has basically been an approach to take something's been around forever been around for decades simple scanning or you know by patterns and kind of modernizing it it's not dead IOC honey's absolutely not dead it's quick its effective we just need to you know come up with ways to do it better it's not the only way of doing things in fact there's a whole bunch of presentation and a huge emphasis I think in the community right now to do a here with Antilochus which is our contrast to you know static so that's kind of what the second part is presentation is so how do we go from you hunting for something that's just by patterns to how do we

actually start hunting for behaviors things that your system is doing the you know it's not supposed to be doing post exploitation so the bad guys are in your system now they're going to be doing stuff so I'm going to turn it over to Andrew he's going to give you a bunch of memes okay yeah thanks Mike yeah I'm kind of the comedic relief my my slides have a lot more funny funny jokes in them and Mike then uh yeah but uh I like to decide on that Justin you can tell me later so I like Mike said I'm very familiar with Yara I used it for a long time and kind of helped determine what the

next iteration of us using it locally would be but ultimately I recognized that we needed to do more than just bite pattern matching you know Carmen black like Sean mentioned this morning I really still just talked carbon black in Falcon host and the new guy in town what's that source project what's it called blanking on it but it's basically endpoint detection and we wanted that capability I have a lot of indicators when I read a malware write up or I produce my own maurer write up I have more than just a Yara rule when I come up with it I have all types of behavioral things that that malware did on the host and honestly I

want to be able to detect on them and in the tools that I had at the time we're not really cutting it so our art team kind of as a whole came together and said you know if if if we wanted to develop an agent like carbon black falcon host or something of the sort what kind of stuff would we actually want to monitor for on the endpoint and that kind of was how we frame this whole this whole subject is first off we had to define what is it that we actually want to look for uh and so if you ask the security guy what do you want to look for it's all the things I don't

really want to be limited I want I want basically want as much access as you can get I like more data the more data I have I can I can funnel down on what I'm looking for but if I don't have the data to begin with I can't open it up so essentially if you ask me what I want to see I want everything so we're gonna go ahead and start first I want process traces I want parent IDs and child IDs of all processes ran on computer this is like the bread and butter of carbon black or Falconhurst data use their tool being able to but but being able to view what processes start what other

processes up and down the stack network data ultimately I want I want source if nothing else I want source and IP destination and hostname locally I don't want to I don't want to depend on a network censor in my packet crossing that sensor to tell me why that packet is going it's much easier for me just to get that data from the freakin endpoint right why do I have to even bother with it with it passing a sensor in the first place one more thing I have to worry about obviously I want I want files to some data on what files created times you know all occasions all types of stuff that's a typical time stamps also

I don't even really need to mention that but that becomes really important later on for something else I want registry data registry is kind of its own weird little thing in Windows you know hive data you know it's not really on disk it's kind of goofy so I wanted to be able to monitor what kind of changes are made to the registry on the end point uh and this is one thing that I've always wanted to always kind of really upset me is that I've sort command history why can't windows give me an easy way to monitor what is typed into man law I don't really want to have to like write a giant pool to figure this

out it honestly should have been a feature from a long time ago but it's still not in there they're working on it honestly if you have the newest version of power show you can like enable PowerShell logging and command line logging but it wasn't there for the longest time and that was one of my you know dream things as an analyst I want to I want command line history of M points and then ultimately I want processor memory um our tool doesn't do that not a lot of people will let me muck around in live processor memory you know and probably with reason I don't really feel confident enough to actually do that yet and that's me I'm sad panda I

can't I never you'll get me back that's the process memory yet tools are kind of out there in there and they're getting that way and hopefully will be like that in the future but but it's one of our long-term goals but not at the time right now so we finally have our own agent what exactly we're going to do with it we should start mucking about in kernel land and like Mike said we're not going to do that we don't really have the time or the resources or the experience to create a production level kernel level or kernel monitoring driver we don't really want the responsibility for taking down large swaths of someone's production network and getting

fired so we decided to just basically stay out of kernel land and really if we had to we would have gone here when we started this project we were like Anne you want to go in kernel and Mike's like not really investing I don't know too much either and we were like well what are our other options if we don't have to go into kernel and let's not go into kernel land and so the beauty is is that we don't ask you Microsoft is actually getting really good at allowing us to gather all this information about the system in a very structured and supported method and the first being WMI a lot of people hear

about WMI I have in the past a lot of system admins use it for a long time but W a mine basically allows a way for you to query information about a system practically anything it adds its own languages Windows query language there's a lot looks a lot like sequel but basically can write out complex commands saying asking the information asking the system of information about itself you create event subscriptions it's highly supported which is another big thing of ours we don't have to reinvent the wheel there's tons of documentation like I don't even know it would take someone ten thousand years to read all the documentation out there and then we have event subscriptions which is a way to

basically not not consistently not for us to say to continually query the system but basically telling the system hey I'm going to set up this query you continually ask Windows this query and then if the data comes back that I'm looking for let me know about it and honestly it looks a lot like a signature I can ask the system for I'm telling the system I'm looking for this certain type of data and if if it comes back let me know about it so it actually worked looks a lot like a signature in a lot of ways um the way to write them can be really cryptic and honestly the windows development documentation you can get

lost in there you know I have where your 10 tabs deep and you only know where you started and you just have to close the whole browser and start again because you don't even know how you got there in the first place but the beauty is that we can also change how you actually monitor for a process being created and then the second one is basically a query based off that one so so you get the results back from the first query and then you use the parent process or ID from that data to query the information about the parent process ID that created that process so you can move forward and back and chain inputs and outputs from

WMI and that's not new we're just kind of using it you know as a way to look for security event data and then the other way is Windows Event log windows event logs are becoming their old reliable they've been around forever and Windows just keeps putting more and more features into them in a lot of ways for the advanced stuff that we want to do we really needed to get into the Windows Event log a lot of the Windows security audit logs that aren't turned on by default if you wanted to detect on really crazy log anomalies or logging anomalies you need to enable those things and then look for certain event codes in those logs that you've turned

on so we really need to get a Windows Event log in one way or another and then ultimately if we have any tool or any of our plugins on the system can write to the Windows Event log then we can use them as part of our tools which is also kind of one the nice things for us um I said we didn't create a kernel level driver and that's because someone made one forced anybody know who this is uh yeah some people know this is um he's dreamy yeah this is from his LinkedIn web page I wanted a picture up and I was looking I was like dang pretty good looking guy but uh that's mark russinovich and I got

it I got to give him give him thanks because without him our project wouldn't have really came where it was and in mainly system on I don't know if you guys have checked system on lately but it's real old-school system on has been around forever as part of this internal sweep that mark russinovich you know created on its own because windows wouldn't do it themselves and then you sold it to Microsoft for like a jillion dollars it's supported by Microsoft it's a kernel level driver in the update this was actually one of the key things about the update is the update made it so that for one it provided documentation the documentation for it was not very well

done before and you know creating a system on config was kind of like good luck buddy here you go so now they got a lot of detailed documentation and also the config file is essentially a pre-filter the system on will monitor all these different events you got eighteen different events and before there was only about ten they added about eight different ones in the last update which which increased visibility on a major way but also the config file allows you to pre-filter so so all these different registry events you don't actually have to look for every single registry event I don't know if anyone's ever done that but the Windows registry is actually really really busy all the

time so you can basically pre-filter these to only look for certain registry hives that you're interested in so essentially our scanner that parses all this data it already the data that I have to parse is already limited by what we're what we're on in the first place this is a nice free config to start with I would definitely recommend using this it's probably it's a 400 lines already I think and it gives you a nice starting point for lowering the amount of data that sis Mon is going to produce for you in the first place and the beauty about this Mont and this kernel level thumb driver is that it gives us access with a lot of the things that we

determined we wanted access to in the first place like event ID one is process creates so that will give us a process trace data on a kernel level file create times is if you ever want to detect on time stomping where a file create times are changed over lots of files in a very short period of time is not usually normal now our connections will give a source in death I pianos name you know a kernel level file creates gives us access to the file system registry gives us access to the registry this is what a system on event type 1 actually looks like and you can see tons of data this gives us tons of data almost too much

for us to go off we got users we got the process ID the parent process ID the G UID that you had your your user ID and level of security level that you had when you ran it and does anybody notice the bad stuff in here it's pretty obvious I don't know if I I think I hi I highlighted it but you also get command-line history is one of the things so this this is actually how we started looking at system on is I want to command-line history is one of the things of my requirements and system on provides that at a kernel level which is huge and actually a lot of our detection is to revolve around command line

history so so what exactly do we really have here we have visibility and power on the cheek we really didn't have to reinvent the wheel we just had to like Sean said this morning these are built into Windows half of carbon black or Falcon host is already built in Windows it's already there you just got to kind of figure out a way to use it for what you want to do so this was you know visibility and power and Chiefs kind of like a you know barrack 50 Cal you know lots of damage you got a lot of range a lot of visibility it's not super expensive WMI provides a way for you to create a

system I know you know I gave it to my buddy last night you said the same thing he's like you visited building power but that guns not cheap sounds like I yeah but um so WMI provides us a way to basically query the system for anything we want any of any if we're looking for anything in particular I can pretty much you give me enough time and I can find a way to create a WMI query to give me back that information that I'm looking for and then we also have a kernel level driver with confident complicated pre conditionals using regular expressions and conditionals - to see seven or six out of the eight requirements for what we

wanted to do in the first place in it's missing something I'm not sure if you guys kind of figured it out but all of these things kind of live on their own each of these things is like a way to detect one indicator of compromise but to develop a true signature suite of tools we have to be able to correlate these events I want to be able to say hey if this if something reaches out to this IP address and this process is started within 30 seconds of that time oh and this file is also present then I want an alert I don't want to know I don't want to learn on each one of those

things I want to be able to create a conditional it says if this and this and this all and also within this time frame that's really where the power comes from so that my my signature can actually be high fidelity and I'll have to you know wade through ten thousand false positives and that's really where the next person speaking comes in Chris Boettcher is we developed we decided that we had all these ways of getting this information we ended a way to correlate them all together locally and also through time and so that with that I'm going to pass it off to Chris who did a really great job developing a fool that he calls bagels so with that I'll

pass

Mic Check alright alright so we're back live why do we go with our own custom solution and not something off the shelf off-the-shelf products rarely allow you to use custom intelligence so you have all these sources for for intelligence that we can bring in we can modify or use and we get something off the shelf it's gonna be 30 days 60 days 90 days before any intelligence gets used in production and as they mentioned before off-the-shelf products usually it hosts external and we did not want to do that so the key features of bagels it actively monitors and endpoint machines for behaviors of interest if the persistent persistent agent runs on the machine and it allows for custom

intelligence and the core components of it there are three different pieces to this thing there's an event monitor a behavior scanner and a alert handler a dashboard so not only popular we'll skip that one for time so the event monitor is the first piece of this thing and what the event monitor does is it goes and looks at the windows event logs and it looks at the WMI queries it creates event subscriptions and collects that data from the system so you tell the event monitors the kinds of things you want to look for it I literally like Andrew said I look for process creates I want to look for processes being terminated shared drive creations USB

insertions all that stuff you can get alle WMI matching events get sent to the behavior scanner which is the next component of this but things that happen in the event of the event monitor are not typically Mauritius so these are things that happen all the time processes are created all the time people you know plug in this kind of USBs all the time all that stuff is going on all the time so the events themselves are not malicious the next slide here I've got an example of the event rule kind of hard to see just like yours was Mike I was going through real quick though so what we do here with the events is we

kind of say what do we want to look at we want to look at with the query we're going to look at W mines they give me all the processes that are created basically the event subscription is handled by WMI we're not constantly hitting it for anything the application will just sit and listen and the event will hit and come back and it'll say you know this process were created or this USB stick was inserted the attributes section we can if you go on MSDN you can look at any of the any of the different things you can query from from w mind you can see all the attributes that can come back we don't

need to get everything because there's a lot of information that comes back so you can limit the things that come back from your sequel query or your WMI query and that's the attributes section does and like Andrew mention before we can do related queries based on the first query so if the first query is a you know a process is launched and we want to know what the parent process was we can take the parent process ID from the first query do a second query and come back with all the information about what launched it which is useful information looking for stuff that you know like an acrobat readers launching some weird stuff which is common or heed a word or

other things like that the behavior scanner is the second piece of this thing and so every time an event fires it sends that event to the behavior scanner the behavior scanner is the thing that actually does the magic here it looks for the bad stuff so a process gets created it sense of the behavior scanner the behavior scanner now has has rules separate rules from the event rules that look at these incoming events and they try to match or correlate them to other events so it's a state machine that kind of looks at everything that's happening and says is something bad going on here something like you know a USB stick is inserted and then within 30

seconds the network share is created or some other malicious process is launched or something like you see run DLL 30 to execute a DLL of temp folder which is something you don't normally see from non malicious stuff which will apply to a lot of different malware's not specific malware is like a static indicator my you might look for a specific thing in that malware we're looking at more for the behaviors things the malware is doing this bed so again the behaviors are collections of specific events and not just like a single it could be a single specific event but that's more like a static into indicator anyway so the real magic here is that we can take things from all the

different sources system on Windows Event log WMI and we can correlate all those actions happening to look for malicious malicious activity and the way we do that or with the behavior profiles let's get a little more complicated but the gist of the behavior profile is that you uh you put in groups of behaviors that happen and each group can contain any number of behaviors each behavior looks for a certain event type and top one you'll see is process create and seconds actually all three these or process create so this is a real basic one it just looks at any process that is created what's like so for something that CMD execute it stopped on the

Windows Update agent which is something you're not going to normally see in a you know in your client production environment regularly so something bad is probably trying to stop that service there are other ways through system on and for WM light that you can also watch all the different services that you know are started and stopped and you can you can build these out so that they don't just look for one service stopping but maybe this service gets stopped and another bad thing happens or another processes executed for things like that at the very bottom there you'll see a groove operator in the within seconds and that's kind of some of the cooler parts what we're talking about here to

the group operator applies to the entire group of behaviors that you have there and you can nest those as deep as you want you can have different groups you know within groups and things but the the group operator in the within seconds allow you to either say one of these things has happened with an or all these have to happen or none of these have to happen and the within second says you know these things can happen over ten or fifteen seconds because this thing has to execute first and then the user might have to open and click something and another thing has to happen so we can you know slow things down and look over

a longer period of time if you put the within seconds of zero you're looking at all of these things happening in one event like this one is which is again more basic and finally we have an alert handler dashboard this is really boring I mean the last piece is once the behavior behavior scanner actually find something bad that happened it kicks that off to your cloud-based alert handler dashboard you can see here I've got some that's an example JSON out it from the from a client machine where it actually saw the bad the bad behavior happening and it lives on there an analyst can look at that kind of stuff but I mean that's that's bagels 30

seconds and free long really a lot of the stuff really looking more like a 1 to 10 seconds range and and that does happen a lot where like something will happen where it will open and it'll take a few seconds to fire up those macros and the macros needed to go do something and then you've got bad stuff going on so yeah 30 seconds is probably long for a lot of things but it can't they can do anything you want so to go get it sir one of the things that we kind of found out is that your source types like Chris kind of alluded to there's more than one way to skin a cat a lot of times so for

example if you're looking at W my queries they're looking for process Kuwait great a lot of the really ephemeral processes don't get enough information for you to actually query them quick enough but this mahn does so and that's just a case of knowing what tools are available for you so the cool thing about this tool is that the event that Chris had talked about can come from any of those places that kind of Andrew had mentioned you can come from sis Mon they could come from Windows Event log they can come from WMI event such as that's free right now those entry places those three places are well for knowledge so if you can't get it through W my Windows system

on or a Windows Event then you're going to write a totally different plugin one of the things that I did mention at the beginning that I think probably does require a little bit of clarification is that these two things that we talked about hash browns and bagels are actually two plugins of a larger project there's a larger project that we're currently working on and within our Drupal con and you off its the unify hope I'll post operations platform and essentially that's our way of creating a lightweight agent and the idea is that you know most organizations have way too many agents living on them and there's a huge push to get agents off the machine and our asses we just

want one we want one lightweight one that basically just sits there its entire purpose in life is to securely communicate and then execute plug-ins when needed and in this scenario that we've talked about the two things we talked about hash browns and bagels are both plugins to that ecosystem and so it allows you know a single single point of presence on the machine to do multiple things and so if we wanted to create down the road we want to create you know Andrews memory analyzer because he's he gets sad enough he wants to find reflective DLL to memory or something like that we could do that we just have to create another plugin for it we're definitely

interested in some of the open source projects out there like Lima Charlie and some of those things but right now it is a separate project that these things are plugging into well you know that's that's kind of one of the things about an endpoint and doing things on the end point is that the endpoints comps you really can't trust anything you get off the endpoint so I put in our case we've got about as close as we can get we're using wwu no winners query language to get information directly from a Windows service so you're like well I think we're doing a bed rate play and we talk about it yeah yeah and yeah it's a hard

problem that's a very hard problem so some of the things that we some of the other things we talked about is one depending on your environment if you got a connection to the Internet and you can get these alerts out there quicker the whole idea is that you can you can off upload these things before they could potentially be modified so that way maybe they do get modified but you can go back and since you have a source of truth like a one-way diode almost of how the alerts are flowing you know that on the server you know what had happened and what alerts you got so you can kind of deconstruct that way another way that

you can do it is you can get defensive in your in your programming scheme so if the conversations me and Chris had totally theoretical at this point about you know all the different ways that we can room persistence and essentially we decided to create malware and very real good super tech ourselves we're actually gonna put that in as a separate module maybe in some future land where we have more time but no it'sit's the interesting problem really is yeah yeah I don't think that you ever know that for sure yeah so just stop checking it I don't know the hope is that we get good enough with the behavior the behavior profiles before they have or the static

is so that we catch them as they're doing it so we get it you know we get a chance to beacon out before they can stop it you know really Southwest looking at a button a good example of that it's you know I didn't mention this is that on the list of since money vent pipes or is it listed on there because for is a special system own event type for if someone turns this model so we there are some ways where we can detect if someone's at least it means messing with our program before they ever actually fully shut it off it is definitely a cat-and-mouse game and it's one that's very difficult yeah we're

actually going to use the blockchain every single transaction that occurs very runtime makes G said that's what we're doing actually look for in the next market

yeah so I can think of a role right now so just look for open hang up handles find query what open handles if a process has you know has like like 20 different doc files open at once you know that's probably not legit that's a rule right there's you can look for so so that's another great question that goes back to one of our design principles in our company we don't do that we don't block so that nib itself would have to be another plug-in to our you hop system so bagels right now is totally passive it only alerts and that's by design one of the big reasons why we do that is because we work in

like ot environments were revealing with shop floor type of situations you know ICS data that sort of thing and that was one of the one of our design requirements that we had is that we were not going to be apprehensive system we're going to be a defensive system but you can easily modify or take the same concepts and do exactly what you said you could create a hips type of solution where you do start to kill you know that process because if you see a process that has 30 doc files oh and open handles is very unlikely unless there's certain ones that maybe you can exclude what they're you know signing certs or something putting on a whitelist user

even if it does happen to be a loophole of yeah yeah and then you could go ahead and just kill that process and fire an alert so we don't do that but you could that's the answer I guess I'm sorry what's up somebody realized memory.you mentioned email yeah so we have a whole program called fishfinder that essentially anytime that we get an email we have networks apps and just scrape them off and then they run through the same they run through a sandbox but part of that sandboxing processes they also scan through all the guard signatures I did not work on that project but I know that there are people who helped organize this it did yeah yeah so we're not

thinking I mean is really different project that actually that actually correlates with Outlook but at the same time any any attachments that you have an email that touches your endpoints is going to get written to disk and then our whole system is going to get game anytime you trigger this where we stand with our artoo will pick up on that no no you're right nonetheless not unless you actually save it yeah none of you say to your ended long you're not interacting with you only go back and rescan file change yes right what do you do with there's a change yeah no so so there are actually two different types of scans that happens there are scans that happen with the

full list of IOC s let's say you have four hundred different jar rolls that you're trying to scan for you get a brand new file that's never been seen anywhere you're going to scan those all 400 rules are going to get stand on that file let's say you add a new IOC now you have four hundred and one rules now you're going to create a new scan that just has one rule and what you're going to do is you're going to distribute that one rule across all the systems in your entire environment so that all the hashes get scanned so if there's like overlap between different machines those may be let's say two machines share 100,000 files

50,000 of those will be scanned on one machine a 50,000 of those will be scanned on another so that way it's not being scanned duplicated anywhere in your environment the other thing that I want to mention that didn't really do a good job of explaining is that after you do the full-system inventory when you do that full system inventory all you're doing is you're checking file paths and hashes you're reporting that back up to the controller and then likely because most operating systems share a lot of the same files likely you're only going to need to actually scan you know maybe 10% of those files and then you can go and distribute the rules to just those 10%

of those files you don't have to scan the other 90% and the controller will know that because it has a list of all the other hashes everywhere else in the environment and what rule that they've been scammed - so so then it can send back and say hey scan only this 10 percentage of gain is with this rule set the answer question that's definitely something that we're talking about doing we've been searching for different on you know white like big databases of safe files and hashes and things and try and look at what your habits it it's a project that we're looking at integrating that's just kind of one of those things on to-do lists there's

there's really nothing stopping us from running it on servers edges right now yeah yeah and we've actually are running in a pilot with some servers right in our beta testers on servers right now as you speak yeah I mean we're running it on servers I mean when we say endpoint or host we're talking any in foreign host and include servers but I mean not network because we're not really like looking at things on the wire and and one of the reasons why we really wanted to focus on the host it's just because networks are changing the parameter is changing in most environments I mean everybody brings their own devices you know to work their laptops can connect multiple

different networks you probably have a proxy that you go through but other than that the whole idea of you know corporate network is becoming a lot more fluid so that's why we're trying to do a little more focus on the host not saying that you know network you know analysis is not still very very valid but I think that there's been a lot more emphasis on that in the past and that's why we this project you know is focusing on on in point - clickers anybody area questions thanks for letting us be CEO [Applause]