BSides PDX 2023 - Essential Logs Pyramid SIEM (Eric Goldstrom)

[Music] all right if you're here to learn about remedies for arthritis or ways to remove that pesky wart that's now what this is okay so today I'm going to be talking about the importance of log management which which is you know always interesting right after lunchtime and building a detection and response capabilities and how to sort of mature that and a bit of a preamble here so when I say log management what I'm referring to here is the collection of all the Telemetry logs data from endpoints within your organization centralizing that into a single place and then being able to query that when you need it now for the security information and event management um that's really layering on the analysis component of uh that platform and then includ including alerts correlations dashboards and those sorts of things so the thing is the the use of a traditional Sim as a log management platform is a direct conflict of interest right what I mean by that is as you're scaling out your use cases as you're scaling out the data sources that you have the cost becomes exponential right so that's why it's important to um differentiate between log management and Sim so that's challenge One Challenge two um so most of my day-to-day is uh mostly uh incident response work among other things and often times I'll talk to organizations about their visibility right what does their EDR uh look like what log management do they have what security tools do they have in place today and that happens from the preparation phase all the way through Lessons Learned before and after an incident and so when I speak with organizations especially small medium-sized businesses one of the reactions typically is hey I want to deploy a Sim I want to deploy a sore uh security orchestration automated response system in my environment and almost always that has to be uh repositioned in such a such a way that instead of deploying the Sim instead of deploying the sore uh the focus needs to be on log management and building out those use cases first and then Challenge three uh ultimately is back when Sims were originally built DEC decades ago depending on the I guess Sim historian that you ask um they were all on Prem so these were systems that were put on Prem not necessarily scalable and as times evolved as use cases started to be built out data sources started to be added to these Solutions it became sort of a nuisance because some vendors would rearchitecturing a cloud infrastructure be Cloud first use cloud services while other vendors would instead kind of lift and shift their Appliance from on Prem into like a ec2 instance in AWS so that doesn't really scale instead what needs to be the focus here is ensuring that it's a cloud first product so given the challenges organizations often face I'd like to share this detection and response Tech maturity model so when you're building out your sock your DF teams your threat hunt teams it's all about people processes technology right so people ultimately the most important component of that and you'll have your L1 L2 L3 analysts you'll have Engineers analysts um for threat hunting and like I said the most important part of those teams you also have the processes the drive the strategy of why you have those programs to begin with and then what I want to focus on here is the technical comp component right the tech maturity model starting at observability so the idea here is akin to building an application um with multiple microservices so by decoupling each of these uh Technologies you can then integrate them later on and this helps to prevent vendor lock so that's a that's a huge part of this model preventing vendor lock as well as being scalable over a long period of time not running into the same issues that you may have you know 10 years ago when you've had a a traditional Sim and as I mentioned we start here at observability that's really the foundation this is log management this is um EDR getting all that data into one place but it's also um building out that data Lake um this is becoming a a huge thing um lots of organizations lots of vendors are starting to build this out and this is really reflective of the times with uh Sim and how they're being built out today so I think organizations are starting to take this into their own hands and taking ownership of their own data as they should again this is uh really the the lifeblood of uh a sock um ensuring that you have all of that data that you need for those use cases um so I'm going to recall back to um 2020 a few years back how many of you had the chance to uh respond to the solar winds incident pretty much the whole room okay I suspected that was the case so again work in incident response the Saving Grace in many cases were those DNS logs right if you recall back to the solar winds cases if you had DNS logs retained in your environment you could then decode those understand the C names and if that Sunburst dll um those was deployed to those Orion servers was activated or not so that was very telling very early on with those Python scripts that were developed by a lot of organizations and so having those DNS logs on hand were critically important for the response actions now of course you would obviously uh go about the forensic image analysis and understand the uh the the configuration file and the Sunburst D itself so tying all those data points together was critically important um but going back to why logs are so important I would never recommend to anybody to to store DNS logs into SIM right they're just too dense this is is the perfect use case to have a high volume log with very few use cases into a log management platform instead of a Sim so this is um exactly what I'm talking about and then when it comes to analytics um I mentioned the Sim product right we're all familiar with that um one thing I do want to call out here is if you have other areas of the infos team for example The Insider threat program you may want to layer on a ubaa user entity Behavior analytics platform which sometimes is integrated with a Sim sometimes it's not um but a lot of organizations again um may want to start with a Sim and then layer that on at a later date and then we get to level three so this is the enrichment um process so integrating that Sim that you've then deployed over several months and looping in all these other data points adding fields to your Sims so that you can create High Fidelity alerts if you can create High Fidelity alerts you can then automate that later on and what I mean by enrichment is just simple things like integrating your sim or uaba with active directory so you can understand users and their roles their job descriptions what they do exactly in the organization Andor things uh to the you know further extreme integrating with things like a thread intelligence platform this is how you gain value in the the products that you have in your organization right A lot of times what I'll see with these tips uh thir intelligence platform or provider is that they're used ad hoc you know look up IP addresses look up hashes look up specific um ioc's look up specific um thread actors and what you can do is add value by integrating via V uh API uh into the product so that you have an opportunity to respond uh quicker to incidents and then last but not least uh the the reason automation really is at the top of the stack is because once you have those High Fidelity correlated alerts to reiterate you you get more value out of the product we always refer back to that you know one or two use cases for swore products which is automated fishing response or automated IP address blocking at the firewall if you have two or three use cases why not just set up a script server and automate those or use a you know Lambda in AWS you may not be ready at that point if you just have two or three use cases but if you've gotten to level four and walked through these first three layers you then have a multitude of various use cases to then integrate with a sore so at that point you're sort of ready for that so one of the positive trends that I'm starting to see is kind of the differentiation again between the log management platform and the Sim right you can see here this line right in between this is vendor Integrations and a lot of these log Management Solutions that are up and coming over the past couple years are Cloud first but on top of that they're building apps or they're building uh bilateral apis which allow you to query data from your sim but store the data itself in the log management solution obviously when you're doing that you're saving a ton of money um but you're still able to have those uh Sim analytics alerts Dash boards so case study I'm going to uh emphasize the importance of log management one more time and then I'll stop I promise so um a couple years back um we got pulled into an incident it was in AWS with a client and they had a thirdparty AWS monitoring company uh company that got compromised and their keys were exposed in that breach access Keys secret keys and the actor leveraged those to gain access to an S3 bucket so they wanted us to understand exactly what happened so naturally our first question was well what logs do you have in your environment are they in your sim where are they stored so this organization um didn't have them in the Sim product mostly because Cloud Char logs if you've ever tried to put them in a in a Sim they're very volumous just like DNS logs are right very expensive but they had the fores site to centrally manage those logs in their their security account so they didn't have the ability to query we had to go in use a AWS clue schema build that out for them we could leverage AWS Athena to then query those logs but they had the foresight again to store those logs and then from there we able to Pivot on that access key understand the I am accounts that were compromised understand the IP addresses that were associated go back and forth between the two and discover that additional accounts are com compromised as well not only that we were able to detect the enumeration it was a user agent string Scout Suite it's a very common um capability and enumeration tool that thread actors will use but that was what a user agent string that we were able to Pivot on as well so once we had all of those data points we could pivot to the S3 access logs look at all the get requests from that S3 buckets to see exactly what data was stolen so Sim wasn't touched at all but there's a very specific use case that this company understood which was log management all right so let's talk about use cases um so I've spoken to uh organizations um about this several times and sometimes you'll see organizations who will build these out in sort of a catalog style in Knowledge Management platforms and sometimes you'll see organizations just rely heavily on the you know the built-in out of the box detection within their Sim and that's okay that's a starting point right but depending on who you ask you can either have a top down approach when you're building out custom use cases or a Bottoms Up approach what I mean by top down is the Strategic initiatives that align with those use cases bottoms up being the more tactical approach which is the more engineering and detection uh sort of use cases I think it should be both I think all stakeholders that are involved with the SIM with log management should be in this discussion and all those things should be grouped together in one centralized location so this is also an opportunity to map these to a framework time and time again we hear about the miter attack framework that is a fantastic uh framework to map these two um and you'll also see some of the specific tactical use cases I have listed here for example suspicious Powershell execution um there's sometimes multiple uh tactics in the miter attack framework mapping to different ways that that tactic uh is is done that TTP is done um this is an example of that right if it's a Powershell encoded command the response action may be to decode it um and then escalate if it is a download string you might want to detonate that specific URL or understand what that ioc does with thread intelligence so this is um to that point also an opportunity to add another column and know what those response actions are right so if this alert triggers in the platform now I can take this response action and that also while coinciding with the automate column feeds into that sore level four level so if you build this out early on you can kind of work these up and down that maturity model and the um the idea here is that again you can have this in something like SharePoint Confluence whatever it might might be but when you're building out the detections of the cell don't put them in in a Knowledge Management build them out as detection as code right use GitHub use gitlab um build those out and then you can link them into the use cases but those should correlate um one for one in most cases um and then finally the one thing I want to talk about here is if you're leveraging both a Sim and the log management solution to fulfill your use cases um then you can uh instead of eliminating use cases when you're Bill gets too high for the Sim you can promote or demote these going up that stack right some of these might get automated at some point and some of these might get demoted at some point depending on the threat landscape ttps and threat actors change over time so your use cases and how they're deployed to change over time as well so here's some of the approaches for building use cases going back to that I keep on mentioning use cases so how do you actually build these out in your environment I'll say it again miter attack framework great way to map these out you're not going to get a a guaranteed 100% coverage of these you're also going to have your own uh internal um capabilities and your own detections that are unique um and maybe not spoton with miter attack but this is a great starting point you can go in um see what uh thread actors are doing um look at thread intelligence Prov uh providers and see what Tas are doing and map those use cases one of my favorite usea uh one of my favorite products rather is uh Sigma this is really the uh the deao standard and the poster child for detection engineering at this point if you haven't heard of it and you want to get into uh detection engineering and building out detections uh custom to your environment this is a great thing to look at I have a link here go check it out on GitHub um when you click in there you can see the the high level um I'll say it again use case you can click in there they have yo files that um provide the specific detections and then further or if you have different Sim products they also have a conversion tool so that yaml file can get converted into the query language so that you can then load that into your sim tool as an alert so really really cool stuff um love Sigma um and I think it's uh it's going to get really popular here with with detection engineering additionally I said this before but continue to work with internal stakeholders right um it's not just the the sizzo and a couple engineers who should be driving log management and should be driving um The Sim product it can be it leaders it can be uh application um and developers uh who need these error logs who need these uh application or user logs need to understand that need to uh need to um respond to troubleshooting issues those sorts of things so find those use cases and then double down on those security use cases that way again you can get more value out of the product and then last but not least um when I'm what I'm talking about with log management and deploying that product I'm not saying don't talk to to the Sim vendors early on you'll definitely want to understand um their primary uh data sources they'll want to ingest to to have the most bang for the buck for the product right they'll typically say firewall Windows event logs maybe proxy logs um cismon is a good example as well um but understand their built-in use cases so that if you need to deploy it and when you need to deploy it you already have that positioned in your log management platform and then the big point that I want to make here is as you are demoting promoting all these detections make sure that you're testing them right testing them time and time again purple teaming red teaming there's the atomic uh red teaming from Red Canary that you can use for unit testing ensure that when you're deploying these detections you're able to detect them and that you're doing that on a continual basis so um huge proponent of testing and also if you're able to automating that process so here's some of the the key takeaways um so I'm not going to read every single one here um but what I want to talk about uh specifically on this slide is is log retention I didn't mention this before but um one of the considerations when you're deploying these types of tools is regulatory concerns you might have a requirement to retain logs for a certain amount of time if you're doing uh open source deployment of your log management solution you may be able to leverage sort of the built-in pools I mentioned AWS built-in Services you can do frequent access infrequent access use Glacier and then set up that uh retention life cycle to ensure that it aligns with um you know those requirements internally and then additionally because some of these data sources are such high volume you may want to keep them for instant response use cases for you know two weeks 30 days but it might make sense to allow those fall off over time one of the other uh key areas here is centralized versus decentralized I think depending on on who you ask um a lot of organizations kind of have to take a decentralized approach for various reasons right I understand that that c caveat but I think in a in a perfect world which it's not um all these logs will be essentially located in one area so that you can uh bring all of that data together when you're doing those correlations and then the last Point here understand which logs are required to fit use cases I'm going to pound that into the into the ground with use cases right log with a purpose build detections based on those use cases right build with a purpose and then continually test those detections test with a purpose and with that I'm going to stick around for a little bit if you have any questions um feel free to to say hello and thank you [Applause] much [Music] he