Threat Hunting with RockNSM

okay so this time using rockets is a threatening our team that I'm a member up don't mind the elastic Brandi and I presented this elastic meetup couple months ago and and have kind to rebrand it so that's that so what been about me I work full-time for elastic as an education engineer so basically all their public trainings that we go see on their website I go around to chose for them I'm a technical advisor at a start-up first and a lot of consulting and commercial support training around threatening the rocket as soon as well and as I mentioned Azzurri animal guard that's a little bit about me before we talk about rock Edison and the passive

or before we talk about threatening it is important to define this concept of passing something because it it's very important when it comes to breakfast and at breakfast and I'm going to approach eating their data making sure that you're tipping yourself off they going to be any more than is necessary what does the past planning approach so a couple of examples here the differences in past the current active see those spy on the left there is a picture of a world war two code breakers and left apart listening a mnemonic radio transmissions from there they're not interacting with the enemy at all they're just taking a copy of everything that you hear over the radio

recording it and trying to they assessments on that the enemy has no way of knowing that they do that that this listening to transmissions that are always there whereas the active hunting approach you're out there interacting with the environment you're knocking on doors your logins workstations you're grabbing lists of domain administrators both their pilot approaches but I believe is a time in a place breeze and when it comes to rocking us in we generally start with a passive approach first so we'll refer to it as a passive person approach so why why do we need to pass the first why do I even care about that

the big thing with passive person you know the reason that we try to preach us as as an expert I remember doing the cybersecurity the primary you know thing that we do is instant response so like with the Ferguson riots you know our team spent almost three weeks of time performing as a responsible state was very that it's not only were their physical rights there are also lots of network attacks against those three state members and you know when it comes to doing this to response and you are they respond during that network and the enemy finds out that you were there you necessarily become the number one party just like as you can see here this you know addressing

parkland hunter and you know how in this requirement with the dinosaurs that's trying to you know remediate this situation is not a party because now they are a threat some landscape they're operating it so they want to you know eliminate this hunter so that they can go back to having free rein the environment for themselves so how does it work we know that we once knew custom good idea especially if you're a responder there's a new officers if you do this full-time in corporate network but especially for responders asked it was very important how you do it the big thing that you're going to need is a way of getting the network data sort of passive hunting is all about

looking at things or from a network perspective less than a coastal perspective so the back here is anything that the enemy is doing is you're going to care about it's going to happen in the perspective of the network yes they are you know they may have you know they may be compromising workstations and viola students is moving laterally but anytime they do anything that's important that were station like trying to exfil data or move laterally we're going to see that across the network and that's that's what you care about is you know when stuff happens in perspective in the network so to gain that visibility to the network you have to have some way of getting a copy of all

the network traffic just like the codebreakers you know needed a way to you know listen the radio transmissions we need a way to listen in on the network data so the expensive approach the enterprise approach here is to just get a physical tap is anybody using difficult apps in their market current here mom's very very common very expensive so you really only see it and corporate environments but nice thing about a physical tap the reason that you know if you're in a corporate environment I recommend using one is one they they fail open so if you lose power of that tap you're still going you're still going to be getting transit data transmissions in and out of it depending on the speed of

data that you're tapping you made you know step down like techniques to one day or something but they're designed available and especially when things like app cons and I think people are going to do it too I shaker traffic and said it out all directions or filter some of your traffic so if you will get specific VLANs you can you know pick and choose what you want because you know they talk about a couple stalks earlier today is sifting through the the mountain of needles in a haystack sometimes we have to be a little thinking about the deity but you can't distort everything one it can get expensive and - you're gonna overheard your analyst they're trying to figure

out how to sort through everything and if 95% of what you're storing isn't relevant and you know don't you know what's important but that's best physical tasks but going back to the home labs I'm going to talk about the previous part there's a lot of cheap ways that we do this as well so really common approach is for sending the spam court if you have a managed switch like Cisco it's super easy to or you say take all the traffic from once one through ten copy all that out support 11 it's very easy if you already have Cisco's you know there's no added cost there and there's no interruption in the network traffic so if you want to

put in a physical tap I have actually Stuber that network connection and you know do it properly it's just going to be like a few seconds period of time but that's still you know your your main data in and out of your network your Saturday night so span ports are also a good a good way to you know pull off a little bit longer to put in actual physical tasks so if you're doing it as a response to this 8 o'clock morning you can take down the network right now just a nap or till 3 o'clock at night and then put in there very properly in this window but it's there for home laughs especially the spam ports are great

couple things you have to keep in mind that is what your networks etc on spam ports so if you're sending 150 makes my directionally 10 ports so that's not going to work on a one-game court that's a transcendent back out so you have to take into account your saturation of your traffic there may be said what reports link aggregation or tools that you're sending it to when we get to the rock and assim piece can actually just accept traffic from one connections as well at NSF stick together so that's awesome approach another way of caffeinated us something that I'm doing at my full-time job rebuilding a range is using PSS you can actually do what Bridgeport it the

essence and just say everything of this network I want you that actually basically written 16 members together and it happens once to three dollars that's particularly useful for that's enough posted via word environment I don't have physical control the network in there so yes there's works great for today so very very very basic diagram of what this looks like so you have your firewall at your network boundary I going out to the Internet and then you just have here you know sure your rocket I said logo up there that's your all-seeing eye whatever your tap point is whether that's a spam cork or a physical hardware tab this is taking a copy of all the traffic on that particular

network segment we need to do some additional analysis on it and as far as the enemy is concerned they have no way of knowing this their physical tap like a different one it doesn't advertise any kind of MAC addresses on the network or to ARP there so they have no way of doing this there and what that allows you to do is actually physically signature all of your security data is separate our complaint if you're not you know actually sending logs over

okay any questions on that process for the passing of her so now we have data what do we do with it so if we you know put our cap in place what we're going to get back is this whole packet deal whole packet data it's great for troubleshooting stuff anybody use this tool up here in the right Wireshark you've done any kind of sysadmin work there security work you'll get familiar with it it's great for troubleshooting or if you have something very specific that you're looking into but it's not so great for traveling because it becomes like looking at the neighbors like how do you how do you sort through all that figure out you know what's actually if

it's just scrolling by you know tens of thousands of injuries right so now we have to do is throw and grow is I like to refer to it as a protocol analyzer so anybody in the room currently using bro we're familiar with our project and full of you okay so what bro is is it takes all of your network traffic you're all active data and it breaks it down into the two actually some human thigh vesicle information so bite at all and we'll split it out to the Ascalon in the file system is tab-delimited so you have your you know linux epic clang stamp and unique connection IP source port destination doesn't support whether it's TCP or UDP

protocol that is going over and then your actual you know handshake curve I don't know it's not my head but again I should send then you know at all or lower from whether it's responding to the or originating them I also need to call your connection information in there that's super useful and this is much more readable and interpretable and wire cutters so networks that the right direction we can start actually interpreting this network get back a little back in all this is entirely customizable so bro is a complete programming language so if you don't like the data that's being spit out here this is just a script that's saying you know if you get Deena the

initiates connection in a certain way as behaving like it's you know SSL traffic that I went through to agree to certain way I want you to send it to an SSL specific blog I want your aggressive analyzers against it maybe check the CRL certificates are being transmitted so you have a testament as some eyes ability or how you handle it information

and it's like this is not work so looking at this data from a you know there's a plain text file is it's great way to start but typically when you're dealing with network data I want to start to slice and dice that data there's some you know actual metrics out of it and we have circular to the anomaly to build some trends that's hard to do when you're just looking at a wall of connections so it's all plain that plain text asking us you're able to use all of your lettings text management utilities to the manipulate that data so over doing here is we're looking at the DNS log which is all the DNS specific

information and the throat affected were using a special utility that come from profile for OPEC which will pull out specific columns and information so you know how to write like three line ox trips to do that it will just do that one simple function for you and we're going to sort all those connections and then we're going to get the unique injuries there and sort of before we can do anything we're going to do an ounce of the unique connection we're gonna sort again no this time you're sorting the America this is again a step in the right direction but problem needs to quicker in here the problem is this still doesn't scale so if you're a limit Linux wizard this may

be a lot of fun and you don't feel like you're a black man thing all day but it's really hard to scale from there this one bringing an analyst you don't actually you have to them trainer analyst not only on how to do analysis and how to use broke the trust up into two basic Linux you know text manager so it becomes a much more cumbersome Cochran analyst to deal with my utilities so so Harkins here's some reasonable says the scale the knowledge lien analyst is a big one but by a fault bro will take all these asking laws and hourly will infuse it those and throw them in a separate folder so that if you

want to look at data that's any more than an hour now you're having to you know use tools like decap to get then go back and beat the precedents of apply and this is the system that you're trying to process roland emmerich streams on and now you're also trying to decompress pay us a lot of overhead thinking out of ten analysts all friends compressed files same time while processing change requires to be an information that's a lot of overhead so that's why we made rock so what Brock is is basically a collection utilities in a community so it's two things this collection of utilities and as a community of experts so the utilities that you have here is your roll number

stream and then we send them that stream in a few different directions first place we send in this Google Sniper a way to learn that project one person one of those sniper person so people soon I heard a really awesome project for managing all this traffic data so what it does it stores it similar to be PPAP but it connects with you so now you don't have to you know figure out ok so this this pcap file is one out of this one our for data I can just go query Google stenographers say all of the network traffic that you're sorting of this or actually using gps filters I see you say you know what

all that work traffic that had this originating IP from this time to this time and it will go back and pull all that off this and send it back to you to keep it fun so going back to the talked earlier about they get through this mountain of data you want to keep the original source data you know unmodified it's not going through this whole pipeline so if there's something that this moment you need to do if you're digging really deep punching down a thread you can go back to Google scenographer get the original overall network data and then do some deeper analysis on maybe you're not extracting plaintext passwords from great repeating that because you don't

want to have to store that in the last name but if there's a specific connection that does have that that you're interested that they go back to full house with Google stenographer I'm using more in-depth analysis in there so it's great way to just keep copy of all that information and then we have burrow which is kind of a central utility the big bettor they're very kind of walk through that and then it might seem a little redundant included a somewhat traditional ids solutions on top of also you have the option is for smart for sericata and the reason we included yes on top of burrow which also reduces my TS type stuff is you want to remove

it was much hurting from your analyst as possible so if there's a hurry fast signature for something you already know you know you have some good C CPC enemy absolutely that's an insurance terracotta the enemy does that that immediately you didn't learn that this so you you're removing that burden for me analyst and trying to you actually go on a five-step this could just easily detect a certain answer kata and I mentioned broke to do that as well we didn't survive as the cert husband purpose-built for that it's easy to pull in the streams of signatures from other sources that are automatically updated so use festival bingo another lesser no tool we have in here down at the bottom

is project called FSF FSF busy recursive it stands for file scanning framework prime twister but it's basically refers to the file scanner so it will take like a Microsoft Office document which all an office document is is a your text from the document data structure and all your images you know compresses all that together into one file enemies love to embed additional path stuff in there as well so as I set though you know take that initial office popular extract little files let's say oo inside of here is also I would extract that or just keep going down for every layer and give you all the information about every single layer of that pile and you can also drive ER

our wheels incident and runs additional protections on all the files of gross dirty network its Bravo you figure the file types don't extract everything my last job we were extracting PBS by cult and that turned out to be a really bad thing because it was a bank and every time somebody's open their bank statement on the public website we were extracting that bank statement to the possum so we turned out we don't watch this one because it was several pages an hour and - because I don't care about seeing all my customers bank statements so my brother-in-law was off the files there and then we send all the results the JSON output from SMSF synthetic nakaka sir

bro all that data is sent as JSON into cop and the separate path compactness reasonable reason not my dear in front of large stash back when we started doing this for years ago log stash was not nearly as robust as it is now and we'd send you know several thousand that's per second to it from roland emmerich strange to grow and everywhere else and log stash leaders fall over a couple times an hour so we started sending it into a optic you hear kind of buff in the data to enlarge definitely pull the data out of hopped us as quickly as we can we don't worry about the Louisiana sensitive florist activity but it also acts as a nice Penn State as

cockpit scores everything on disk so if our large staff system does vibrating reasoning i we're not losing to that state they've these persistent disk until Hoffman Airtel log staff pull them out of there so he has also after using one's great tools maybe or sending them to us and type solution as well a lot of those can extract from Kafka I said he was a senator data multiple directions maybe you want to send it to the elastic stack that you also want to send that data percent or to a separate Department additional analysis they can all just be subscribers for this popular topic city makes a nice closure data different directions as well and then we have the

whole elastic stack over here so for those not familiar with elastic law classes basically your ETL tools creaked and reduce some additional processing so some of the additional functionality that went to Brock is you know pre-built logstash filters for bridging some of a great improvement like writing gof deep filters against all the informations he gives there and just cleaning up the data and making it more presentable so we're hearing that marks tax and then the last service director sort of ended when she query everything through that's where everything eventually persists to make Obama is just a window every day US Obama is a data searching and visualization tool that's another last product South Korea analysts all your

analysts really will care about is just one piece right here everything that happens behind the scenes is really cool journalists really just incorrect Obama okay so what does this actually look like when you're doing analysis I'll just walk through a quick scenario I this actually came from one of the other Rock contributors on his home network I thought they were really nice to walk through so what we have here is a pie chart of all of our HTTP methods that we're seeing and then we have a date histogram for those same HTTP methods and now we can see those methods over time so we can see some trends and spikes in their list of our

user agents and then the post offenders or questions this is all this whole dashboard here and Kevon is all related to HP data but if you look through the list of HTTP methods in this pie for errors anything stand out was being

hi hi so hi I don't know if you're familiar with the RFC for HTTP but if you're not I have a cheat sheet for you so this is the actual RFC for HTTP these are the RFC compliant HTTP methods and as you can see hi it's not in that list so this is really what most of network analysis or a threat many comes down to is knowing what normal looks like and finding the outliers so we know that high is not our C compliant lots of people break our C's so you don't need to start spending the red light yet and say this we have a evil hack sort of the network but it's definitely something

unusual so we're going to keep digging into it so here we've you know done some additional we just pinned that one particular protocol to see if there and now HB rights that are coming back are limited to just requests that with a high method so you can see that it was all requesting this one URI right here that's also not which we would expect to see is HD or right normally those are all some slashes it makes HP s in there typically doesn't look like the long string like this so that's unusual so digging a little further so what we're looking at before was a dashboard come on us now to the discover page in kimono we're

just going to search for that same year I again actually now we search for a high method you can see what's highlighted here and now we have some additional information about what's going on here you can see the originating house so it all came from you know this the stop 30 address originated the connection and then the responder was to stop 110 dress so something internal so this was an RFC 1918 I was in backwards 1918 address and then this is internet private address so you have something inside of our network that is talking out over an IRC compliant protocol or method and it's using this weird iraq another unusual thing here this there's no data

transferred as part of the request or the response body typically if they're pulling up a web page you're I mean HTTP is all about rendering HTML pages so you would expect there to be some data and that HTML page that's being rendered that's what your request response body did just you know what was sent as are the HTTP connection weeks so that was just a to be specific so you know where you have multiple blogs so we're looking at HP long before but if we go to the con log the grow con log is basically the over the network connection metadata for every connection that happens on the network what protocol it is there's a connection

established that's going to go into the common mark it's just all the metadata about that connection so there is nothing in the question response body transmitted but the outgoing bytes for that connection was 32 megabytes so that again looks unusual there was also a little bit of data coming in an awful lot this mostly 32 Meg's it's not negligible and that's all going out of my network so from there bro generates a unique connection I need for every connection that's transmitted across the network and that allows me poorly because bro Pratap Singh's at a different log so you have your final on HP log everything that's correlated to this ID so you can then see all these new connections tie

things together across logs here where you know just grabbing that the ID produced some additional correlation but we can't really go much further from there so I mean this is really all the information that's available to us so what we're going to do at this point this is a rock 2.1 feature so this is currently in beta if you get the scripted field in elastic that allows you to generate a stenographer query directly from kobato so then that will go back and make a request astern agra for building your BPF filter based off of the connection in here so and then go grab the related data so you can going to be firm and lined up properly so we don't have

anything family go any further excited elasticsearch so we're going to grab a peek down and then we're going to open that up in Wireshark and so this is where we're going really deep on there just one interesting thing we found so we can see the actual connection here so there's our high again responsible hello so it's at least you know friendly yes so we're going back and we're looking up what happened there so we there is sorry oh so this is our actual this is that 32 things that was transmitted as part of that connection so this name go is the request body that this is what constituted during two magazine if you look at this this just looks like junk

so when you see junk like this it can really be a couple things it can be compressed data it can be encrypted data or it can be just truly random to jump so here we don't know which of those three is but it's most likely it has to be one of those three correctly compressed or disco so we at least know about what the 32 Meg's was so we'll come back and do some we don't have slide 4 here Robo what we do is pivot to that external IP address that we were sitting in 13 - and then we go and see if there was anything else in elastic that was related to that external address so

we're given our query up here that external address what we find in the logs is a CNS pretty so we're going out and looking at this Comcast server if you go to some open source investigation on that Comcast server what you'll find is that the peak speed tester so what's happening here is that the internal address was initiated in connection to a speed test server transmitting 32 Meg's of random joke and tightening the connection so this the point here is the stack allows you to do all this analysis allows you to dig into your data and do it quickly from a visual perspective using dashboards but it won't always be that so you have to you have to walk you

know excuse a stereotype if you have to walk the dog you kind of find out you know what's actually going on you can't just say you know this is not an hour so you apply this bad guy here dad could actually do some digging and find out what's actually going on run it to the ground and build you know a case for what you're seeing so no arguments that makes it easy to you know a lot easier than doing all this on the command line go and dig out that information the other big thing that I would say here that the dashboard the issue t-turn uses quicker the other thing that the dashboards give you is even if

you're you know using all the Linux command line utilities to you know pull out some nice metrics of information the thing you miss is a visual you know chart of all your data so especially when it comes to like flow data point in time metrics are not very useful by themselves where numbers become useful as where you can see trains up and down so if you see you know a huge influx of DNS queries in one specific server and we're all making this one beard query it's gonna be harder to see that from just looking at a wall of text but when you start charting that they applying it becomes a lot easier to see trends that are trying

to see the big spikes in data so that's you know big reasons you my team we used to try to stick to the command line as much as possible because it made us feel special doing all the Linux wizardry what we realized is we're just making our jobs harder easier making it more difficult for new analysts to get up to speed and to actually be able to focus on the important thing which was fighting the bad guys so we've had huge success with since we just started focused you know doing everything through Cubana it's a lot easier for the analyst and we get all the nice patterns in your data

yeah we're kind of in a state of transition right now so there was a lot of big changes between elastic five which was Black Rock to or ends on an elastic six which is what the current beta version of rock runs on so all the debt the way that she loaded dashboards and the way that they're storing that has all changed which is why it's taking so long to release that but yes all the dashboards are included so you get a lot of example ways to go through your data and at the beginning I said that rock is a framework is also a community so it's really it was very similar to second PC it's a group of people that are

passionate about hunting data in this way and it's about sharing information so if you have a great bass board that you know it's really useful to you contribute that back because it's just going to make everybody else better yeah that's all the slides I had so right seriously though guys first off round of applause [Applause] that's gotta be a first place for many tasty comments but yeah any questions I

heard somewhere not too long ago that it's possible to tap fiber optic lines have you heard of that method or is there such a thing as an IRA optic splitter and would it be a lower-cost option than any of those products which I want to number of yeah it was it has such a thing yes you can definitely have fiber optic the Giga pond I know will give it out of the box it's really just about which modules you buy with a lot of those but basically what it does is just puts the internet reflects the light to different directions yes so splitting light is actually a lot easier than splitting Ethernet because you're have the concerns I was many concerns if

you lose power it's just a mirror so yeah questions along a similar vein the starting feedbacks real bitter right now so if we go back to the network architecture slide if you don't mind real quick as quick as you can there's anything from an iPhone that's amazing right okay there we go so I imagine a fully saturated one giggling probably this to handle D is there it might bring much next closest is their concern of what network card being utilized since it looks like essentially you're creating three additional copies traffic and that's well within the PCI bus excuse me but I guess I'm curious if that can overload the actual processing ability of the network part itself if

you guys identified cards that this works very well with very importantly with so yeah that's a great question that's something I didn't go into a lot of detail on but we have done some experiment experimenting with different ways of extracting the data off the network current as well as using like dedicated capture cards blanking on the names but there's some vendors that they'll take like 40 gig they dedicated capture cards that have their own use built-in if you have those you've got the Linux drivers form absolutely use them but in general so on the network card level worries in half packet we originally started with the F ring and their driver testing process was not

very robust especially on RedHat so we had multiple incidents where we would install patches and things just wouldn't work for a week until it got the driver sorted out so we switched from the iframe to using that packet and the nice thing about half packages built into the Linux kernel so we'll have additional things we have to install but a packet copies the traffic directly it doesn't have to go up to the TCP the whole network stack to get it to grow it just copies it directly from the card to the probe CPU and then we've done some additional tuning on the growth side so so that through the amount of processor cores that you have so we just stable

hyper-threading and then you know Cercado will get its own kaida processor core and then grow basically did you know in like 16 core systems they're kind of acted like four quarters bro get like 10 cores that you leave like two cores rest the operating system you got some math in there for how it splits those out but so then you're not worrying about you know splitting time growth has to wait for sericata a you know release the cycle they always have their own dedicated core so that helps a lot with pulling the network America for us so one thing I didn't really mention here is this is all orchestrated using ansible and really a big reason that we

packaged it like this was for training our National Guard team members because there's a National Guard team you only get two days a month together to train so the important for us to have something that was easy to build so they can go and learn during the month so he is just this is all and I so that your download and it's got a line simple scripts that you run and you can just whatever Hardware you have at your house captive home network using this phantom court and you can start playing around with this at your house so that's kind of one of the main motivators for us which make something that I was built as

an all in one box for training but you know when I was working for at my previous job the bank we were tapping we had three row sensors and we were tapping it was about six big saturation on each of those boxes events it definitely holds up well that you have to do a little bit of tuning that you're almost on the same picture you say the ghouls do not refer to basically take them all captured yeah how does that work in terms of just a network i/o as you can yeah 61 Stephen send that back out the pike to Google so what we typically do select a production environment is we'll have separate separate physical disks so they're not

competing for I ops so for stenographer we might go to some SSDs because all that's doing is just writing directly to disk and you know it doesn't have to worry as much about getting backed up bro bro and Kafka we use nvme drivers typically those who've gotten a lot cheaper and they're like ten times the speed of an SSD you can get like a terabyte nvme for five or six hundred dollars which in a production environments not that much so and that's really you know Trevor better to even you know with 68 saturation really all you need for a copy up because you're not story at their long-term you just need it to be able to grab it really

quickly and then get it into the elastic stack and then we're back in a box with India needs to play as well you're your bottlenecks are typically going to be either the CPU level so make sure you have plenty of or hyper threading or at the disk i/o flow so it's not so typically we

one of the things on the grill be the data that it's towards sister ographers going to be quite a bit more data but Burrell is like a fraction of the original data expense so I think it's somewhere around 10% of the original data from Anna's what bachelors all right when we round of applause [Applause] information just because I love this sort of stuff number one my home deployment of bro I've got about eight boxes I think talking most of them around twenty four seven days where my wife is home bro logs are fourteen fifteen Giga day rolling days where she's not going it's more like seven something like that so it depends on what you're doing on your network but

that's that can be able rough and I don't have Google Fiber either by the way come out from Colombia so I have poor people internet you guys might have different experiences with your numbers there the other end global thing I learned in save is if you are looking for a cheap way to tap Jared mentioned during his talk a big winning products and a huge ability proponent the edge router X is a fifty dollar router that you can literally span every single one that works on there it is awesome for setting up a choke point for home network security monitoring again fifty bucks that's I'm pretty sure the cheapest reliable tap device that you can get highly encourage micro centers

on it have our friend here but I love so that's about it here we've got one more she did going down at 5:15 and then we gotta have to party do a fishtail so see you guys up there - yeah [Applause]