Heavy Machinery and Burly Lumberjacks and Logging! Oh My! - Dan Astor and Evan Perotti

all right everybody we're gonna get started thanks again for coming to Pisa its Pittsburgh hope you had a good lunch so this afternoon we're gonna have a talk from Dan aster and Evan Parodi they're gonna be talking about heavy machinery and burly Lumberjacks and logging oh my so let's give them a round applause cool everyone here merit all right awesome so welcome to our talk heavy machinery burly Lumberjacks and logging oh my so I'm Dan Astor I'm a senior consultant at security risk advisors I've been doing sort of offensive pen testing red teaming for about five years I'm Evan I work a lot on side Dan and I've been doing this for about two years

alright so real quick overview of what we're gonna cover today the first thing is just gonna be going over sort of what red teaming is how it differs from pen testing a little bit why we wanted to do a written sim we're gonna talk a little bit about the simp eye plane we're then gonna talk about logging and alerting within that sim and then I will cover off on sort of our setup and then some automation that we did with it okay so real quick I just kind of want to level set it with sort of what red teaming is if you're not familiar and sort of how it differs from standard pen testing and

then we'll talk about the red team infrastructure itself so with t mean the goal of the engagement is instead of a pen test where it's just sort of to go in find different vulnerabilities prove impacting access to systems and things like that it's usually time box to about a week maybe two with a red team these are how much these are much longer engagements they tend to last you know several weeks to months even and with that that the infrastructure changes to with the standard pen test you might only have maybe one c2 server for your command control you might be only doing you know fishing for a limited time you might not have a full infrastructure so

that just because you don't need it it's short engagement you know you're not necessarily worried about being detected all the time with a red team that's very different you're gonna have a lot of different infrastructure involved you're gonna have potentially multiple see to channel multiple c2 servers you're gonna have read erectors in front of those potentially payload servers this is just a quick example of what a team infrastructure may potentially look like so you have we have a primary and a secondary c2 these are behind but we're calling redirect errs and these basically will mask sort of the true destination of your c2 server so your c2 servers won't get burned if the domain gets blocked or if they find sort of the

origin of where your redirector is this just pushes the traffic back to your your c2 server the the nice thing with the redirect errs is they do give you the opportunity to sort of do some checks on hey you know where's this stuff coming from can I can I if this isn't my c2 traffic I want to send them to a different site so stuff like that for a red team you might also have a separate payload server because you don't want to be hosting your payloads which eventually beginning you know analyze things like that you don't want that to be hosted on your c2 infrastructure because if the payload server gets burned you don't want your

c2 servers to get burned and likewise with having phishing servers you know just another thing to add into the equation and you might even have you know some VPNs things like that so that's sort of the the infrastructure that we'll be talking about for this moving forward moving off of that with a red team engagement there are opportunity for the blue team to really kind of react and do things with this when we talk about the blue team from the red team perspective these are gonna be people that are either sort of active defenders like the security team in an organization they might be entering in like fake credentials to see hey you know we got this phishing campaign that

hit us we're gonna enter in some credentials and see hey what are they doing where are they coming from what behavior are they doing maybe running things in sandboxes so things like that they might also actively be interacting with your infrastructure so they might be going out trying to profile you figure out where things are hosted trying you know attribute it to a potential threat after things like that and then the flipside you have sort of some instant responders or socket list these are people that are gonna have to deal with different alerts so let's say an Evo or gets tripped or someone reports a phishing email these are going to be the people that go through and so we're

going to figure out hey is there something I need to deal with do I just need to reset credentials so they're gonna be the people doing that sort of doing the initial tier 1 triage let's say so these people are going to be dealing with sort of hundreds of cases throughout the week so quite a bit different sort of skill set in terms of what they're going to be doing so moving into sort of the the red team sim and why we wanted to do this the first thing was you have all of this infrastructure you have you know multiple servers that are out there you might even you know six to ten servers that are sort of in

your infrastructure how do you manage that and particularly how do you manage all of the the services that are running on that how do you get the logs for that so in the past it's typically been doing SSH into the servers you know grepping through blog files to see if an event hit if someone hit the server from Skype ease taylean log files to see hey or phishing credentials being entered the flipside you also have to have sort of a Red Team operator you know that's losing potential resources because they have to monitor all that stuff for you so they're potentially losing out on having to just monitor things for you instead you could you know have them be doing something

else the other thing that it gives us too is a centralized place for exporting data but also just for the data to be logged so let's say throughout the engagement you had some specific things that were done you want to go in and do a quick check to see hey when was this command first where and things like that you can go into the sim at that point do a quick search just like you would do you know for a standard you know enterprise sim go into a search figure out when those things first hit and build out your time line if you need to additionally it gives you the ability to just export all the data that's on all

of the servers it's so you don't have to worry about going in trying to script everything being zipped up compressed and then pulling it down to store it central you can just backup everything from the same at that point and you have all the data that we need the flip side of this is sort of what you use a traditional sim for alerting so with alerting this gives us the ability to do counter ir so let's say once the blue team starts to be actively sort of checking into our things maybe there are running things in sandboxes we can start to create alerts around this and so we want to know hey when is our

c2 potentially being investigated is this c2 gonna be burned soon do we need to switch to a noose to stand something else up to sort of adapt to that and then the last step is meantime to respond this is typically a term that gets used for the blue team on how quick they respond to an incident we're sort of flipping that and saying how long does it take the red team to respond to a red team event so let's say someone enters in phishing credentials how long does it take that person to sort of go in figure out they got entered in and then go and potentially use them to log into a service or the same with a c2 may be

this c2 has been running you know with the show on a potential entry point for you know several hours that person may just go home reboot their computer something might get detected and your teams will have the ability to go in and quickly respond to it establish persistence move to you know a secondary server things like that so this gives us the ability to reduce the time it takes for the red team to actually do things that we care about real quick we'll just talk about sort of what we used for the sim choice as I'm sure a lot of you guys know in the same space there are tons of different products there's commercial tools

there's open source tools there's just freeware out there that you can use all of these have different options that they support there's pros and cons to them for this we wanted to go with something that was fairly lightweight we wanted to try and keep cost down and we didn't really want to deal with licensing that much so for this we ended up going with elastic we thought that it has a pretty decent community support we like sort of what people are doing with it there's a lot of cool projects out there there's a lot of people doing things specifically in the security space we kind of like that and then additionally it's open source so

that's a nice feature too so if you aren't familiar with the elastic stack its kind of made up of some core components the main one is elastic search which handles the the search feature of your stack so it'll take in all your log data and you can search through you can distribute these systems for us since our infrastructure set is kind of smaller we went with a single host for the entire stack the next component is log stash which will be the component taking your log data and performing any type of transformations on it so if you want to parse out your log data to get custom fields to search on you can you do that

and then there's cabana which is a dashboard system you can create custom views for the data you're taking in to get kind of just quick over views of that data and then the other component that's kind of outside the stack are beets so these run on your clients so you're efficient server your c2 server and they forward the logs to your log stash and then ultimately into your elastic search so for us week focused on the two primary ones which were file beet so that's for pushing the log data itself so things like your Apache log and your cobalt strike logs and then we also had metric beets for pushing system metrics like resource utilization so we

wanted to have some kind of alerting outside of just standard security information so if your server is getting high of resource utilization you want to be alerted of that in case that's going to cause some performance issues or certain VPS providers like Alibaba cloud well hard cutoff servers if they reach certain utilizations so for us we use AWS and for for the main portion of our c2 stuff so we went with iSeries instances because they give you in terms of pricing really good memory and storage options which elke is pretty intensive in terms of memory but you could go with something like a too large so so once you get your log pipeline kind of starting to flow all the

information flowing into your ELQ stack you want to start alerting on the different types of information you're getting in it so we kind of broke this down into the three main categories the first is good alerts so these are things like a new c2 being established or getting credentials entered into a phishing server which you can log through like sequel query logs and we have bad alerts so maybe you did get a c2 but it's being run in a sandbox and you can start a learning on the the characteristics those sandbox present in terms of your infrastructure another thing we've noticed recently is some defenders entering in fake credentials into phishing sites once they get like

like a good lead on that kind of thing so being able to quickly determine if those credentials seem legitimate and the last category are informational things so like the day to day operations of your Red Team operator so what commands they're running in the c2 you can push those logs to it get alerts on it or say you want to have good insight into your operational security of your your team members so you have you - you're blocking certain commands you want to be alerted when someone tries to run one of those commands so that you're quickly able to respond to anything that might come out of that since we went with elastic for the logging portion of

it the alerting options are kind of limited to a few key areas are key products the first is X packs which is elastics first party commercial option and then the alternative is a last alert which is developed by Yelp it you can kind of generate these really simple rules which are using a yamo syntax and you can just define the fields what you want to look for how often you should be queried and then where the alerts go so this is an example of something for say credential hits so what you do is you enable query logging on your your sequel instance and push those to your elastic stack and then you can start say

searching for those insert statements into wherever you're storing your victim information then you can say push it to webhook so we use teams at our company for just chat stuff it's similar to slack we use slack as well so this will just give you like a quick message in teams that someone entered in creds one thing we made a kind of stress is that while we did go with kind of freer options for all this stuff it's not because we were trying to be cheap with it it's just that a last alert offered the offerings it provides are good enough for our use case that we didn't need to seek out something like expect but if it didn't

we probably would have gone down the route of that commercial option but having in not having to deal with commercial licenses is really helpful if you want reproducible quick automated deployments so another thing we wanted to stress with this is that while the Blue team can do so many different things on there and there's only a limited subset of that thing you can see as it pertains to yearn for struck so while blue team might be chasing down like a fishing lead on the back end clearing all these like you know threat intelligence sources looking at virustotal looking at some historical DNS stuff you might not be able to see a lot of that so the the alerting and the

the logging we're focusing on here focuses on what exactly you can see on your infrastructure so breaking that down into fidelity related areas you have really high fidelity alerts so a new c2 is being established that's really high fidelity because it's either really good in that one of your campaigns paid off and you know I will put the holder on a system or it's really high fidelity because you're can't campaign just got caught by the blue team and you know they're analyzing something like a Samba on the other end of that though you have something like really low fidelity alerts so just something a DNS lookup of one of your fishing domains every bot on the

Internet is scan is the Internet's being scanned continuously so if they're doing a web request that's DNS right there so these are not really actionable so for us we wanted to focus on actionable information that we could respond to immediately the common breakdown that everyone faces sort of you know in in the blue team space hey we have assumed how do we get good alerts out of that we sort of have the same problem just trying to figure out hey what do we actually wanna learn on that there's das so so of the activity that you'll see from a blue team there's certain things you want to break down into what you can profile out

of it so a few broad categories are one the source location where they're coming from so are they coming from a certain geographic region a certain range owned by a known company how they're react how they're interacting with your infrastructure so is it being loaded on a browser versus command-line options things like that and then it's going to be very based on who's doing it so if you have just a target user that you sent a phishing link to they're probably coming from you know any old IP possibly in a something geographically close to where their headquarters are they're loading in and a browser they're loading and all the associated resources if you're loading it on a browser so the

JavaScript to CSS stuff like that however a blue teamer there they could be coming from you know certain security vendors or loading it with security tool so with that you can kind of develop these indicators of analysis so if you're a blue team you're kind of familiar with the idea of an indicator of compromise this is like the red team side of it so what we can look at in terms of what the blue team is doing so if you're learning it from a command-line tool like curl or W get and you don't bother the change is user agent those are pretty like high to medium fidelity alerts that we can detect on our infrastructure if you're

loading it so say you have a ticket open in ServiceNow and that our site is being loaded from ServiceNow it's likely coming from a tor exit node or if you just have anything coming from a Touareg - no really for that matter because the normal user isn't likely not coming from that another thing you can look at our sandbox characteristics so when you load a situ in a sandbox the thing loading it isn't going to look like a normal user usually you know have like no domain associated with it or I'll have a wonky name like apiary or Bob or it'll be you know local admin which is uncommon for c2 so these are all

things you can look at when you get information in your sin so here we have some example alerts so we simulated the activity and then got the alert sent to our team's webhook so this first one is what it would look like if someone entered in credentials into your phishing site and the bottom one is if you get a new C - we call it beacon because we're using cobalt strike which is the name of its C - these are for potential blue team activity so like I said curl user agent those kind of command line tools detecting those user agents in your logs that bottom one again sandbox so once cobalt strike pushes the C to log you can look at you

know what user is loading that C to answer that middle one there are well a SAP list of security vendor IPs things like what virus comes from classes things like that unlikely for a user to be sourcing from so what we've done is on our redirect errs have an HT access file that redirects those away from our origin C to server so what we can do is then take that redirect which is in the Apache era log and alert on it so when you get that specific redirect we now know that it's security source and it's preferable than having just that raw source we're using in the HT access file in the rules because it's

much more manageable and this last one is kind of those informational alerts you're interested in seeing so in cobalt strike we use an aggressor script profile that'll limit the commands you're allowed to run so certain commands are pretty dangerous and are likely to get you caught pretty instantly so we want to block those but we also want to know when our people are using those commands that they shouldn't be using so we have an alert setup so that we can see that there they're running those commands and reacts appropriately these would be really beneficial to sort of like the red team manager or whoever is providing oversight for the engagement they can go in see hey you

know you know maybe we're trying to let's say PS exact to something and PS exact a common command that's going to get flagged by a lot of different security tool sets that are out there that is a non safe OPSEC command that we are that we have and so this blocks it but by alerting on it it gives the Engagement Manager the ability to go and say hey who is the one who ran this look into it and then just give it a teaching moment for the team so and this second one is kind of a more generalized version of that so not related to OPSEC but just general commands alerting so you can have kind of a really quick

audit ability of the commands you're running so if you want to maybe alert every time you run mimikatz logon passwords you can have that right there in your web hook rather than having to sift through all of your c2 logs we have a couple demos prep of just kind of these situations and Dan if you know sure all right so this is just a real quick one of us we're gonna go ahead and curl one of the pages that are redirect or hosted or that one of our readers has so we go ahead hit the curl command it goes ahead and attempts to pull down the page and then we see here in teams that

we go ahead and get a new curl user agent alert scene that someone used curl to go ahead and pull this I pull down the page so something like that so potentially something an IR person is doing a Blue team person is doing just to alert us on that this next one this is just a generic phishing site that we put up because we definitely use this on red team's so this is just simulating someone that clicks on a phishing link we have a portal set up they're going to go ahead and enter in their credentials they log in they get a nice fluorophore and we go ahead and see here that we now have a credential hid in our team so

there's a little bit of a delay I'd say probably what 30 seconds I think we've seen which you know if you guys use him you know there's always going to be some type of delay with getting the logs in but 30 seconds isn't bad for us for this so that one is our credential harvest and then this next one is using our command and control tool cobalt strike and so here we're just going to go ahead you can see we don't have any beacons and cobalt strike right now so we're gonna go ahead and get just a powershell one-liner just for testing purposes we'll go ahead and put it in in a victim system we go ahead we see we have a new

c2 Channel established it's designated by that and then we go ahead and there's a new beacon hid in the team's Channel so this is just sort of a quick demo of sort of how this works what it looks like when you know an alert comes in so very similar and you know like Evan was saying if you use something like slack you know this could easily be ported into that you're just replacing that so with the work we've done for this we've also come up with some example rules that you can use for a less alert if you want to kind of go down this route these are all included and I get Hubley released today for this topic I will

probably the link at the end of the talk and I'll also make the slides available as well so I can just run through these the first one which is getting a new c2 beacon the second one is that up sack block the command block though there's another one for that that just generalized command 1 it's not shown here but it's it'll be available the third is just that Karl user-agent the fourth is the credential harvest hit and the one after that is a variation of that so if you know the domain that target is going to be on you can set an alert so that when credentials are entered that are of a different domain

you get an alert so potentially someone entering in creds outside your target the one below that is looking for common you know users in sandboxes with a news beacon being established following that is just a general URI hit so if you have a specific path on your web server that gets a hit you can alert on that so this is pretty useful if you have something like a payload server and you want to know all the times that payload is pulled down you can get an alert on that using this and the last one is that vendor IP looking for the Apache redirect so this is sort of our just generic lifecycle we're gonna walk you through we called

it from forest a fireplace to stick with the whole logging theme so sort of the first phase of this is going to be the attacker coming up with a phishing campaign hosting some type of payload and then sending that phishing email and so with this the next sort of phase with that would be some type of mail gateway email security appliance something gets that and potentially has the ability to start checking the link it can send a link or if they you send in sort of a macro enabled Word document that has sort of some weaponized code in there it has the ability to potentially you know you know check that out run it in a

sandbox so sort of that mail gateway is sort of the next step and then assuming that your stuff is good it just continues it on to the end user who then executes your payload and you establish your initial situ into the environment from there you can start to move well I really do whatever you want go after your trophies escalate to whatever you need to you know get the flags that are required for the engagement let's say along the way you're dropping some harder facts for persistence or you know maybe something finally fired on you know the month of situ that you've been doing and the the blue team starts to check things up like oh let's let's

start to correlate this back and find the initial email so maybe they'll take that and detonate the initial email into a sandbox and so you know you might have things that you can see that is being random sandbox and then sort of the last step of this is the blue team going through and performing manual inspection on your your infrastructure maybe they're checking out other payloads starting to sort of mess with the situ things like that it's sort of where the sim that we that we put together comes into play is after you send the initial email maybe you'll get an alert in the sim that something's being done from a known vendor source IP maybe it's trying to

scan you know the link that you sent in or it's trying to detonate in a sandbox that's coming from one of their IPs the next one is we can get an alert when a new payload is delivered and a situ is established so that's something that the red team wants to know we can then start to you know continue out throughout the internal phase of the testing and then additionally we also want to know anytime something's potentially ran in the sandbox that the blue team's doing so there's potential indicators that we can alert on that we have sort of that rule for that we'll get the alert and then that last one is just sort of

manual investigation from the blue team so some of those you know indicators of analysis that we outlined earlier those are things that we could potentially trigger on and then you know let us know that hey you know our stuffs you know being inspected we need to adapt you know move over to a different C to spin something else in so this is sort of the life cycle that we put together just for you know quick demo purposes so sort of the key takeaways of this what we wanted to put together why we did it the big one was getting log aggregation so obviously it's extremely difficult and taxing on the red team if they have to

go in and manage each system separately go in you know SSH in grep out logs you know a tail things trying to identify if something you know happened things like that so having that all pushed to a central place that you could do searches on and then eventually you know create alerts for but also that the ability to export everything out is nice and to be able to search all from you know a single console that provides us with with a lot of good things the other part of this was the counter our perspective so we want to know when instant responders are doing something so we can sort of counter that you know we have the alerts created now we can go

in and once one of those fires we know hey we need to change tactics or you know the the fake thing that we stood there you don't work they went to it and now we can you know continue with the other route other things like that and then give you an early warning signs too of you know hey you know this is getting ran in a sandbox maybe they're on to us and give you the idea that you need to start to change and adapt to it and then the last part of it was reduce mean time to respond for the red team to read team events so like I was saying earlier you don't have to worry about your situ

being ran you know a beacon being ran for several hours before you respond to it you can get the alert right away that you have a new one have your team go in and start to do whatever they need to sort of get you some persistence or whatever you whatever you have for your methodology there and what would it be without some DevOps so along with this work we wanted to create or produce something that would allow us to do this continually so we don't have to spend that a long amount of time throwing up a new infrastructure for every engagement we can kind of just press a button and we have our elk stack up and ready so since

we're going for kind of software installs and configuration one with ansible for configuration management we found this great resource that is for ansible elk snacks we kind of took it made some modifications to our needs added some coal strike support to it and so now we can have you know all our servers stood up and provisioned properly we can have the Elks a stack available and the clients pushing logs to them and going beyond that if you want to start provisioning your servers automatically you can use something like terraform which will allow you to kind of define the infrastructure you want and then deploy it so this kind of helped us you know speed it up for when

we need this in production yeah so the big win with this one was hey we have a standard you know Red Team infrastructure that we're probably going to use this the majority of them at least gets us a skeleton that we can use by automating this whole thing you don't have to have a red team member now taking a week to stand up and configure all your infrastructure using something like terraform you just tell it hey I want these types of servers go ahead and stand them up in these regions or you know configure these security firewall settings on them things like that and then having ansible go in and you know configure hey this needs to be an Apache

web server and configure you know mod rewrite for it this is a cobalt strike server so we need these specific things configured on it having ansible going in you know doing that setup and then the biggest part is honestly pushing beats to all of them and then configuring the logging so that that's all being pushed right back into your ELQ stack and you know you can do this so it saves you you know a week you can do this so we sort of get the that skeleton put together this gives you the ability to stand this up really quick it makes it repeatable which was sort of the whole goal of this because if it's not repeatable and it's

extremely you know cumbersome to put together and it takes up a lot of time we're just not going to be able to do it so we can't add additional time to the prep for the engagement we just want to reduce that into this so we put together github repo of kind of all the resources we developed during our time looking into this so it includes some sample configurations for the out stuff it includes sample configuration for elastic as well as all the alerts we talked about it includes the ansible roles for deploying this the base ones that we referenced as well as the ones we added and then some shell scripts so if you don't want to go the ansible

route and you just want to have a script to have your client start pushing logs to ELQ you can do it that way and not have to worry about ansible so this will be will make these slides available so you don't have to worry about kind of taking a picture of these this but these are some resources we thought were useful kind of going down this route the first one is the red team infrastructure key if you're not familiar with it and you do read teams this is invaluable the second is how to provision infrastructure using terraform with a focus on red teaming the third is red team logging so kind of a similar take to what we've done though

this one used gray log the one after that is that htaccess file we referenced with all the known security vendor IPS and that last one is kind of the reference elk & elk stuff for ansible so these are all great resources that if you're going down this route you should check out yep so real quick well so this is the git repo that we put up this morning this is Evan was talking about we have everything up here we have a quick directory structure of sort of where things are outlined and then sort of a quick road map and then sort of inside of each of these folders are the different things that you need so if

you're looking into the Alaska lured stuff here's the rules that we have for that quick description of sort of what everything does and you know the one caveat I will say is you know these these rules that you're putting together you'll you'll probably have to modify to sort of whatever use case you want so you might not necessarily want to alert on let's say like WMI usage from your red team or you want to have additional one so you can sort of go in and modify these to add in whatever you want to alert on but it at least gives you a simple configuration to start with

so with that I know we finished a little bit early but we have a we have some time for questions if anyone has any

yeah so we actually for this one and sort of the red team operators are sort of the blue team operators in this engagement so you know we don't have to worry about going in and Sara Lee patching things a lot of this has stood up right before the engagement starts so the majority of the updates and things we're gonna be using updated systems things like that the one thing that we would do is you know test against sort of your firewall configs making sure that your c2 servers are only accessible from let's say you know your source which is you know your VPN or something like that and your redirector is ensuring that you know no other traffic

can get to it without going through sort of the approved means and you know it goes without saying making sure you're walking down anything doing everything you know sending things over securely doing the whole standard thing because you know the one thing that is within with red teams is you know there's a lot of sensitive data so ensuring that all of that is protected the entire way through so yes that is something that we do have to worry about

yeah yeah so he was just saying that with a lot of you know Sims our elk in particular if you go ahead and do the five minutes of extra work that it takes to get all of your logs formatted into a JSON format and when it ships it over it gives you a lot of additional sort of capabilities with the sim itself so you get different values you can perform statistics on things like that so good point

sure so a good question do we have any plans to implement alerts for things like honey tokens being you know pushed in that was one thing that we actually did not think about but it's definitely a move that I think a lot of instant responders and you know it's something that sans has sort of started pushing too so it's definitely a good idea to just sort to include so yeah good point any other questions all right well we'll be around after I'm just hanging out for a little bit so this [Applause]