From Hours To Minutes: Automating Incident Response Triage With Open-Source Tools

Good morning. It's an honor to be here finally. I had this in my plan for many years, but finally I'm here. Raise your hand if you have used any of these tools before. Ah, I love to see that. And I can hopefully show you some more tricks and how to tie this together. Okay, I am Marcus. I work for a Swedish company called Sectra. We provide medical imaging solutions for healthcare customers all over the world. And I work in the cloud operations security team where we host our cloud hosted environments. I like all things on the blue team side of security, trying to focus on incident response and forensics as much as I can. I

also had the possibility to take a lot of CELS courses, which I like, and that brought me to Munich a few times, but it was many years ago. And the picture is me in Bolivia when I took care of a baby puma in the rainforest. That's maybe where I should go back when security thing goes too rough. Okay, the agenda, I will set the stage with a use case, show you some challenges with forensics, present my favorite tools and add some automation to that. Then we'll look at the case example and then show you how you can get started with these tools in a quick way. At work, as I said, we provide these systems for medical diagnostic imaging. The most common is radiology images like you see

on the screen here. And we have customers all over the world. And they are under attack, unfortunately. Healthcare networks are worse than the internet, I think, sometimes. So imagine that this is a doctor at your local hospital. He's working in the radiology department, analyzing medical images to recommend further treatment. This is a really crucial function at the hospital. One day when it gets back from its coffee break, it looks like this. And that's a bad day for everyone, of course, especially for the customer, because they may have to close the emergency room. They may have to redirect ambulances. And they will spend a lot of time fighting to get the systems back up. And my

company will try to help them, of course, to get the radiology system back up. That's my way too common use case for me. But you may have other use cases. The common one would be to just investigate a general security alert from your CM or your EDR or your MDR provider. This triage process that I will show you can also be used for operational troubleshooting. Maybe you have a server which behaves odd-list late at night, but nobody knows why. Then you can use this process to see exactly what is going on when this happens. And if you're starting out in security, you can also use this for forensic CTFs. Luckily, there are quite a few

CTFs these days where forensic artifacts are provided. And that's how I started out as a forensic script kitty. So almost everything I show today has its roots in my days as a script kitty. I'm just a bit more organized these days. Some challenges with forensics. Well, the time span from initial compromise to the endgame is shrinking, so we don't have much time to investigate if we want to stop the attacks in time. And the attacks also involve many endpoints. It's not just one endpoint, but there is normally a lateral movement, so you have to cast a wide net to find all the affected endpoints. Disks and memory are pretty large these days, so the traditional way of grabbing a full disk image is not really working. It's just

too much data. So we need quick data access to just the forensically relevant data. This is what the triage process can look like. You have one or several endpoints with suspicious behavior that you want to investigate. You grab the forensically relevant data with a triage acquisition. We need to process that to transform the raw artifacts to information that we can analyze. And of course, we want to reach this analysis step as quickly as possible. And here, automation is key, of course, not just for speed, but also for consistency. Triage images, as I said, we don't want to pull this full disk image because that just a lot of data that we don't need. What we

need, the forensically relevant artifacts, is just a really small portion of the full disk contents. So we want to pull, of course, event logs, registry hives, browser history, file system metadata, and also more live artifacts such as the list of running processes, network connections, DNS cache. And that's what we primarily need for the investigation. And if you do that, you may have a 200 gig disk on a server. The triage image will end up being a two gig file or a collection of files and compressed. It's just two or three hundred megabytes, which is so much easier to transfer, process and handle in general. How do we get this triage image then? Well, enter my favorite tool, Velociraptor. Velociraptor is normally used traditionally in a

client server setup where you pre-deploy Velociraptor clients on all your endpoints, and then you have them talk back to Velociraptor server where it can issue hunts, search for files, contents in memory, and all kind of cool stuff. But when there is an incident, you normally don't have any pre-deployed Velociraptor. But the bonus feature in this tool is that you can create an offline collector. And in Windows, that is basically an exe file, which is 50, 60 megabytes large, and it contains config and functionality to pull all the forensically relevant files, just like we want. That 3-H image will end up in a zip file, and that file can be manually copied, or you can push

it to a file share, or even better, to a cloud storage. And we want this to be easy to use because while we may have expertise on the receiving end, the user running this offline collector may not be a tech-savvy person. So this should just be an exo file that you can double-click and run, and it will do its job. That's the goal. How do we know what to pull from the system? Well, some of you may have used CAPE by Eric Zimmerman. It's a great tool to create these 3-H images and also to process the data. CAPE is not open source. It's free, but not for commercial use. But something good with CAPE is

these CAPE files targets. And those are definitions about where to find these forensically relevant artifacts. This is an example for the Windows event logs, like the path where those files are found, and the file search globes that you can use. And there is a big library of lots of such definitions that you can use when you config CAPE, but also when you config Velociraptor. So this is how you decide which files should we pull and put in this triage image. And luckily you don't have to know much about which artifacts to include because there are these compound targets like the Cape Triage and Sun Triage. If you select that one, it includes lots of good

definitions. So you get almost anything you want. These definition files are open source, handled in GitHub, so you can push your own when there is a new artifact in Windows 11, for example. It won't take much time until a new definition shows up here, and then it will be included. Those files will be included in your future collections. This is really an active community around this. But we don't just want to collect static files. We also want to have these more live artifacts that I mentioned, the list of running processes and network connections and DNS cache, things that will disappear when you reboot the machine. And that is something that Velociraptor can also include apart from this more static file collection. You can also include third-party

tools. Say that you want to use Sysinternals Autoruns. That will give you good information about everything that is auto-starting in the system. And that is normally where you find malware persistence. Sysinternals autoruns is a separate exe file, but if you select that artifact when you create the Velociraptor offline collector, that autoruns binary will be included in the offline collector. So it's still just one exe file for the user to double click. And it will also include your third-party tools. You can use it even for memory acquisitions, including the WinPMEM. But then, of course, your triage image will be much larger. So you could use a separate offline collector for that case. Something new in Velociraptor

is adaptive collection of files. Let's say that you have a malware on the system and it has persistence using scheduled tasks. Then the normal 3H image will include lots of metadata about that malware that has been run when it showed up on the file system. But you normally wouldn't get the malware binary itself. But using this adaptive collection, Velociraptor will enumerate scheduled tasks and link files and and include those files in the 3D collection as well. And it will exclude signed Microsoft binaries of course, because you don't want to end up with everything in Windows. So how do we build this offline collector? There are various ways. One way is to just download the Velociraptor

binary and run it with the GUI command. it will bring up a web interface where you can quite easily create your flying collector click by click the problem with that is like after a year you may want to update your flying collector to get to use all these new artifact definitions or include something else and then it's hard to remember exactly what you included last time so a better way is to use this on the command line you put all the config in this spec yaml file And that makes it easier to just tweak one setting or add one new artifact and regenerate a new offline collector. You can handle this as config as code

in your repo. If you just want to get started without feeling too much with building it yourself, you can use this 3H ZIP collector. That is also an open source project by Eric Capuano and Whitney Champion. It's a standard Velociraptor or Flying Collector using a default config, which is really good as a start. Of course, you want to tweak that and adjust it for your local needs, but it's a collector you can just download and run, and it will produce a ZIP file in the same directory. And a good thing with this repo is that it includes a build script that you can use for your own build and tweak for your own needs. Running this offline collector, we know we have a few endpoints now with suspicious activities,

we want this to be run. You can do that manually, you can walk up to the machine with a USB stick, run the offline collector and pull the file, the resulting zip file manually. You could also RDP to the system and run the offline collector, but then you would expose your admin credentials. And if there is an ongoing intrusion, you really don't want to do that. So in that case, PowerShell remoting would be a much better way. A special case is that you can run the offline collector against a static disk image. Let's say that you received a 500 gigabyte full disk image, but you don't want to push that to your processing pipeline. Then

you can run the triage collector against a disk image and get a small small size 3-H image instead, which still contains what you need. And if you have an ADR in your environment, you can always use that, the live response functionality, to run the offline collector. That is a really quick way without having to disturb the user. We run this offline collector, and as I said, my ideal way is to push the 3-H images to cloud storage. So I have, this is an example from Azure, which is a storage account. And then the offline collector includes config with access keys so it can write the collection to this file drop. And of course, you have

to make sure that they cannot read anything, any data there, because since this access keys are included, they can be extracted and misused. So you really don't want to expose your triage collections because they contain a lot of user stuff. And you can also do nice things as when there is a new upload to this IR file drop, you can get a notification in Teams or Slack or whatever that there is a new triage image to analyze. If we stop here and if you would walk out or just stop listening, I think just preparing a triage collector and having that ready, publish it so it's easy to find for anyone, that's really a huge step

forward because you don't want to fill with this when something strange is going on. You just want to grab it, run it, and then you can analyze it. Yeah, the system slowly. So make sure you do that or even use this pre-baked 3H zip one. That's a great thing. We now have a lot of triage images, but we want to have those processed, ideally automatically. Then OpenRelic is a great platform. It's an open source platform maintained by Google people, but it's not an official Google product. And that's used to create these workflows. So you can treat it as your central repository for forensic files, disk images, triage images, or single artifacts. And then you create workflows where you decide which tools do you

want to process the data with and And where do you want to send that output? And of course, it has some large language model support as well. So you can generate summaries and it can guide you in these result files from the different tools, which we'll look at soon. OpenRelic is using workers and each worker is a docker container which is a really elegant thing because you can run all these on a single VM as a proof of concept setup but you can also scale out and have many servers acting for different tools if you need more power and you can use Kubernetes clusters or anything as long as you have shared storage between these nodes. There are really good workers available, some easy ones that just

extract files from an archive, some more complex ones that will run tools like Hayabusa and Plazo and Timesketch that we will look at soon. And there's also a good template, so if your favorite tool is not available as a worker, it's quite easy to create a new worker for that tool and to be able to include that in your pipeline. And you can, if you want, push that to the Open Relic Marketplace on GitHub, and have it so we can all benefit from that. This is an example workflow. We have a disk image file to the left, a full disk image, and it gets parsed. And you can have both sequential flows and things happening

in parallel. So we have one sequence where event logs are extracted, sent to Hayabusa, and then uploaded to TimeSketch. So the output from one step is, of course, the input to the next step. And in parallel, this case, Plazo will process the full disk image and also generate a CSV file that will be uploaded to TimeSketch. So this way you can create really complex workflows or really simple ones as well that you can reuse. Here's an example. We have received a zip file with various security logs. Among those are Windows event logs that we want to look at. So we log into OpenRelic. We create the folder. We upload the zip file that we received.

And then we want to create a workflow using that zip file. We don't really know what it contains, but let's focus primarily on the Windows event logs. So we use the task to extract files, and then we config that task to just care about the EVTX files, which are the Windows event logs. And when those are extracted, we want them sent to Hayabusa to produce a summary and HTML report. And in parallel we want to send the full zip file contents because it could contain something else valuable. And that will be parsed by plazo which we will return to and it will also create the csv file. Then we kick off and start this workflow

and when it's running you can save this as a template that makes it possible to use it in a consistent way the next time you need to do something similar and you can share this within your team of course. The extraction step has finished. We can see that it found the security EBTX log 34 megs, which has also been parsed by Hayabusa. And PLASO, as always, needs a bit more time for its processing. And everything that is extracted or the result files are added to this repository, which you can see down below here. Hayabusa is my second favorite tool. It's great to quickly parse Windows event logs. It will apply Sigma rules and its own Hayabusa rules. And currently it's more than 4,000 Sigma

rules that can be applied. and also good Haibusa rules. And if there is some malicious activity on the system, it will always show up here somehow. So the output from this would be like all the alerts from the applied rules, and that will give you a hint about where to go next. This is also, since it's so quick, this is also what finishes first. So this is a good start to just get a view of what's going on in the system. It's created by Yamato Security in Japan. And I think I said it's open source, like anything here. Let's go back to this security log example we looked at before. This has now finished, and

we're going to look at the Hayabusa HTML report. That's a summary of all the rules that triggered alerts. And here we see the AI summary that will give us short information about what has been found. We see that it is PowerShell Empire-related activity, which is bad. And we can also see that, yeah, the system seems to be compromised and it's an active intrusion. Then we can scroll down and see the more specific details about all the rules that triggered alerts, because there is definitely more going on in this data set. And this takes just a few, yeah, like a minute or so. Plazo is the other favorite tool that I had here. And that is also maintained by Google, even if it's not a

Google product. It will parse almost anything with it. timestamp so you can throw disk images or single artifacts against it, memory images, and it will create a super timeline with all activity that it can find, like forensic activity. And it just includes a lot of different parsers, so you don't have to really know much about where you find different artifacts. It will just parse it and give you a long CSV file or similar with all activity. This is an example from a normal Windows workstation. It has almost 14 million events. So as you imagine, the output will be quite large. Opening up that CSV file in Excel might not be the best thing. Maybe not

even Timeline Explorer, but we will look at another tool for that. That is TimeSketch. TimeSketch is also an open source tool maintained by Google. for forensic timeline analysis. Here you can upload the output from PLASO or Hayabusa, or other timelines, and then you can work collaboratively on this and annotate and tag and flag suspicious things that you have found. And there are also various analyzers that can enrich your data such as do hash lookups or GUIP location or apply sigma rules here as well. It is using OpenSearch as a backend to make it quick. So even if you have 14 million events, it's really quick to search in this. Going back to this IR file drop with all the triage images waiting to be processed.

And we now have the processing platform. So we need to bridge that gap somehow to have the automation, have it fully automated. This is an example from Azure. There are similar constructs in other cloud platforms, of course, and you can do this on-prem as well, but it will be a bit different. So we have the triage image to the left. It's uploaded to the IR file drop automatically. Then there are events sent to a logic app that will send a Teams notification that you have a new triage image uploaded. And it will also send an event to an event hub. And those events you can subscribe to with a consumer VM. And in this case,

it's a Linux VM listening to events about newly uploaded files. When there is a new uploaded file, it will download it and push it to Open Relic. And this last step, we're using a helper tool called Open Relic Pipeline. That is also an open source project by Eric Capuano and Whitney Champion. It will give you API endpoints and you can push almost any forensic file against that API endpoint and then it will upload the file to OpenRelic and start the relevant workflow automatically. And that's just what we want. A good thing about Open Relic Pipeline is that the install script also includes the installation of Open Relic, TimeSketch and Velociraptor. So you get almost DFIR

in a box. So that's a good way to get started with a proof of concept setup. Of course, you want to have this on several VMs in a production environment. But just to get started, it's great. Taking a step back, we now have a good compromise endpoint. We have run the offline collector to create a triage image. It's uploaded to cloud storage. It's been downloaded by the consumer VM using Open Relic pipeline to push it to Open Relic. And there automatically we start processing it with Hayagusa and Plazo and the result will be sent to TimeSketch. We want to see this in action. Here is a good simulation or a forensic data set that was released during the pandemic. It includes disk

images, memory dumps, network captures. But I focused on two disk images, one from a domain controller and one from a desktop system. And then I have created three arch images from those. And we start with the domain controller. We have uploaded that three arch image, created from the default disk image. It has been processed by Hayabusa and Plaso. And we will start with the Hayabusa HTML report as I said, because we don't really know. Something weird is going on. And luckily, I think that Hayabusa will tell us that. The AI summary shows that, okay, there is Cobalt Strike related activity. That's quite bad in a domain controller. And we also have public RDP connections from

a public IP. And that's a bad combination. And scrolling down as we did before, we will see even more alerts being raised from the various Sigma rules. So pivoting on that public IP connection, because that could be where things started. Let's go to TimeSketch and look at the full Hayabusa results. And here, when you search for public, we see that, OK, we have various RDP connections from the public IP. To the right, we can even see the public IP where the connections originate from. and they are using the administrator account. Quite bad. Let's do a context search to see what else is going on around this time. Well, there is obviously brute force attempts against these servers, so it's exposed on the internet, obviously. Those brute

force attempts succeed. We have this RDP connection that we have already seen. And further down, if you follow the stars here, we have the COBOL strike. service being started, there's directly after another service with something core updater and further down we can also see that there is lateral movement because there is an RDP connection to another system in our environment, the desktop system, which we have also analyzed with Hayabusa and Blossom. So let's look at the desktop system instead to see what happened after that lateral movement. We have 3H image, it has been processed, and this time we're going to look at the full data from Plazo instead, not from Hayabusa, but Plazo, that will show information from

more than Windows event logs. We search for this core updater that we can pivot on to see is that present on the desktop system as well? Oh, yes, it is. And Plazo will also show this, that information from many different sources. So we have the NTFS information when the file was created. We will see... when it has been executed at least once. That's coming from the Windows prefetch information on the system. And we can also see that there is a service created for this core updater just as for the domain controller. And Further down we can also see information from the system resource usage monitor that there has been network communication going on. We can see exactly the number of bytes being sent and received

from this binary. Not the network traffic contents, but there is signs of network communication. And this is how you can really pivot between these data sources in a really quick way. How can you get started with these tools? Well, it's quite easy, luckily, because the Velociraptor binary, you can run it, as I showed before, with a GUI command. It will bring up a local Velociraptor server that you can use to create your offline collector. OpenRelic also includes an install script, so all you need is a Docker environment, and then you will be up and running in a few minutes. The same for TimeSketch. It needs Docker, and then it will spin up all it needs. And the Open Relic pipeline is even better. It includes the Open

Relic TimeSketch and Velociraptor installers on the same system. So great for a proof-of-concept install. Everything I've shown is available on GitHub, so you can search there and you will find. And some closing remarks. It might be a bit overwhelming, and it doesn't have to be complete, fully automated, and perfect. Even if you're just focusing on this creating an offline collector, then you have something you can pull the 3H image manually. And you could pre-install this Open Relic server with a good set of default workers. And then you can upload your 3H image manually there and create your workflow manually. But even that will save you a lot of time and make your investigations more consistent. So focus on that. And if you feel that

it's worth it, you can try to automate this fully with these events and the consumers and stuff. But that's the tricky part. The tools themselves are really easy to work with. Thank you very much for your time and attention. I think we have actually a few minutes for questions. And otherwise, you are free to find me during the conference. Look for my Swedish flag. I will be around all day. And I love to talk about this. Okay, we do have time for maybe two quick questions. Why is the runner awesome? Here's a question.

Does this work? Maybe a pretty technical question, but how are you dealing with the offline collector being in a wrong signature binary on Windows systems? - Can you repeat the question? - So the way you build your Velo offline collector invalidates the signature, how are you dealing with that on-- - Exactly, because yeah, it will be signed, but since the offline collector includes the config, it will invalidate the signature, as you said. So you want, ideally you want to sign this with your company's signing certificate to make it, yeah, 'cause otherwise it might be blocked just because it's a failing signature. That's a good idea. - One additional very quick question. Is the internet necessary for the AI capabilities in Open Relic? Is internet necessary for

the AI capabilities in Open Relic? No, not really. I mean, the only thing requiring the internet is for the cloud to upload the triage image, but you don't need that either if you don't want to. So everything else can really be run in a disconnected environment. Okay, perfect. Thank you. Quick question? I just wanted to check, does it also pass sysmon logs, event logs? Oh yes, I mean sysmon logs is like, that's, if you don't have sysmon logs or process tracking logs, then a lot of the sigma rules will not work. So if you have sysmon, that's perfect, because that's where the user stuff lives. Unfortunately, you don't have sysmon by default. But that's another thing you can remember. Deploy sysmon everywhere.

Cool. Thank you very much. Very interesting talk. A round of applause, please.