Wes Lambert - Endpoint Excavation: Digging Through Host Artifacts with Velociraptor

[Applause] all right everybody thanks for coming out to the presentation hopefully you had some good lunch enjoyed it and won't be too tired while i present to you today i'm going to be talking about how you can leverage velociraptor in your enterprise if you're if you're a traditional enterprise you know with an infosec department or maybe an mssp type organization or just kind of an incident responder just basically what velociraptor is and how you can apply it in your usage in your daily life now velociraptor is an open source completely free enterprise forensics and monitoring platform and also a little bit of response in there as well it's completely written in go it's compatible with

linux windows and mac os so you can deploy it on pretty much you know any type of endpoint that you have in your environment it's very lightweight very scalable it's really flexible because of the query language behind it and it's super fast uh you know being written in go a lot of that you know lends itself to being a little bit more a little bit quicker from that perspective so continuing the architecture for velociraptor is going to be something that a lot of folks are probably familiar with the typical kind of client server architecture the client will check in with the server and see if there's anything that it needs to do any tasks that it needs to perform or any data

that it needs to collect then we have this nice administrative interface that connects to the server and we can review all of that data that's collected from the server and then you know review what we've collected from the client and then go on from there now there are various deployment modes in which you can set up velociraptor one of them being the standalone option where if you're inside of an enterprise and you just want to stand up a single velociraptor server in your internal environment you can do that and then point clients to it within your environment you can also have multiple front ends to velociraptors so if there's something where you're having a high number of

clients or maybe there's a different way in which they're going to have to access the manager maybe if there's some segmentation going on or whatnot there's some load balancing you can do there with multiple front ends and as far as having road warrior or you know a lot of us work from home today having endpoints that aren't inside of the office or inside the office network we can deploy velociraptor in the cloud so we can set an instance up in aws or gcp and then we can have clients connect back to that additionally you can build a triage package a single binary that you can deploy if you're going into an incident response scenario or if

you have a host that is not able to reach that server for example maybe it's an air gap environment you can drop velociraptor on that host you can have it packaged with some other tools and other dependencies that you want to deploy on that host then you can go off and collect that data and perform those actions doing doing so in that fashion and then also we have a mode where uh it's very simple uh velociraptor gui command just a single liner where you can stand up a velociraptor server and a client on the same box and this is really useful for performing development of artifacts and we'll get more into artifacts in a little bit and really

just a general evaluation

mode and behind velociraptor is a core language i know some of you may be thinking oh another language but this language is very flexible and very powerful this language is used from you know use for everything in velociraptor basically for collecting data from endpoints performing monitoring and response and really just managing the server in general you'll see that as you go through velociraptor and you start collecting some of these artifacts this vehicle or vql is going to be used for those operations for example listing files or you know performing you know managing disk space and that sort of thing as well as collecting data from endpoints so you'll you'll see that as you go through it

now what artifacts are i mentioned artifacts and these are essentially queries packaged into a neat little container that encapsulate that expert knowledge of d for practitioners for example if you have this individual that knows how to collect this data from the end point and knows how to do so from a set of queries inside a velociraptor these can be packaged into a reusable form right and we can use these artifacts so you know joe joe can go and use these artifacts he can go take this artifact that this other person wrote and go and immediately use it in his environment and collect that data he doesn't need to know anything about how it's retrieved he doesn't need to know anything like

that he can just go run the artifact collect the data and go from there and you'll see there are different few different types of artifacts that i'll discuss in just a second but i want to kind of get to some of the most common things that we might do whenever we're performing incident response or investigating a host right we may want to search for certain files of interest or we may just want to see what files are on the box the core plugin that velociraptor uses for this is going to be the glob plugin it allows us to search by file name and search by file size or some other properties of files but it's a really

basic construct and really kind of one of those core plugins that we use throughout the artifacts you'll see it used in conjunction with other plugins and other functions but this is one of the most popular plugins that you will see in addition to searching for content some of you might be familiar with yara rules anybody used the rules before all right awesome so typically when we're looking for a or trying to get a content match uh based on the content of a file or even a memory we can use yara right and the yara plugin allows us to do that we can search for urls and process memory maybe you know some files malware or something

we can search for binaries right we can look for different malware signatures based on the content of those files and there's no need to parse these files unless they actually match the content if our yara rule if you know for searching and it doesn't match our rules and there's no need to you know parse all the data and go through that so it saves us from those expensive operations of going through and parsing every file and then trying to find out what we want so it's very efficient now for ntfs analysis there is a lot of good stuff that velociraptor has to offer for example mft you know one of the things that we might check on a box is

mft and check you know when files might have dropped on disk uh see if a file dropped on disk i30 right we want to carve that slack space in the index stream so i mean there's a lot of good stuff like that uh usn right again kind of tracking to see what files have been dropped to disk uh usm will typically rotate right so it's something that we would constantly be watching as opposed to something that's static and then vss right we can even search volume shadow service copies to see if that file was present or to see if something we're looking for is present so it's very powerful when we're performing that ntfs analysis with

velociraptor drink of water all right continuing another thing we might look for is evidence of execution things like prefetch background activity monitor shim cache am cache and srum i know you know sram yes sir so so yeah i mean all of these things these are you know things that we would look for commonly and these are things that we can you know artifacts that we can leverage in velociraptor and i'll give some examples of those in just a minute another one is windows event logs anybody ship windows event logs look at windows event logs when performing incident response okay right so we can do parsing on the endpoint we can pre-filter before we forward logs or before we forward

relevant events this prevents us from consuming a ton of data that ultimately breaks our log management solution and then causes problems for us instead of being able to get at the thing that we're looking for in the first place sometimes and then even etw event tracing for windows we can hook into the etw providers the things that are ultimately feeding these windows event logs we can hook into that and see if there's any foul play going on right maybe windows event logs aren't always as they seem

another thing that we tend to look at is volatile state so wmi you know things like mutants mapped memory we can look at all this stuff with velociraptor there are default artifacts for us to be able to access in here and i can't i don't have the time to go through every single one but just trying to give you an idea of what's possible

now there are different ways that we can collect this data and perform these actions one of them is going to be targeted collection so when we're performing targeted collection we're typically focusing on you know maybe a single host single client we can specify multiple artifacts that we want to collect when we do this and then if we want to we can select for a lot of these artifacts to have these files uploaded so we can perform further analysis later maybe we want to take those files and and you know do our own static analysis also hunting right this is a big one we can take these artifacts and we can apply them to a whole fleet

of endpoints clients will enroll in a hunt when we initiate it as they come online so you may have some clients that are disconnected at first maybe their the laptop is offline maybe they're just you know not able to connect to the server at that time but whenever they do have a connectivity back to the server they will connect back and engage in that hunt they'll go start processing those artifacts that you told them to collect and then we can stack results from those hunts to identify outliers or commonalities in that data

another concept that velociraptor leverages is notebooks if you're familiar with jupiter notebooks and being able to post-process data from that perspective it's very similar to that a notebook is a way where you can take your queries that you're developing try them out before you try to run the artifact against a client or run the artifact against a whole fleet of clients it's a great way to be able to test that stuff and to post process even you can take the results from a hunt and post process those results in a notebook right there in the hunt results it's very powerful way to be able to filter stuff out and again stack that data and see what's not quite so normal

or see what's common across these hosts now i mentioned the different types of artifacts that we have with velociraptor from the point of detection there are client event artifacts and these are going to detect when something happens on a client right for example you know i saw this executable or i saw this file or i saw some user log in maybe sometime that they weren't supposed to or they should never log in it's a honey right it's honey creds or something like that it's a fake user so we can detect when things happen on the client and then feed those results back to the server and because of vql we can actually act on that data once it

gets back to the server through a server monitoring artifact and this basically turns velociraptor into and really you should be thinking of it as an engine because of this vehicle we can act on those results that are sent back from the client so we got client hits we can then act on that and then if we want to send an alert to something like slack we can do that if we want to perform additional collections or additional hunts we can do that there's i mean the possibilities are essentially endless right so you can it's just a way for you to perform that detection and then respond to that another thing that we can do is perform

remediation right here we have a couple examples of a quarantine right here so if we want to quarantine a host we can do that if we want to remove certain malicious scheduled tasks we can do that obviously in some organizations or in some instances you will not want to do certain things like this but the capability is there if you would like to do that and again you can apply somewhat of a workflow to have something that's automated from the vql perspective i mentioned that single binary that we can take velociraptor and we can package it up and deploy it as a single binary on a host and perform that targeted collection so we can use external tools if we want

to deploy sysmon on a host we can package it with sysmon and have it you know all out to all of our endpoints we want to deploy bloodhound and you know test some attack paths we can do that if we want to deploy it with you know cape files targets or you know other binaries that help with a triage collection we can do that so we can take all of these external tools all these external dependencies and wrap them in there and then velociraptor will manage those and then we can go off and get that data back and you know it's it's much more quick you know it's much quicker than having to go off and use this individual

tool go get go get the results at you have that all available to you then in one single package as far as automation there is even though vehicle can do a lot of the automation that we're referring to there is a grpc based api that's available so you can hook into velociraptor and you know start a workflow essentially start a hunt or a collection from that api you can also get results back you can do anything that you can do in a vehicle query so it's anonymous to just sitting there and doing it from the box that's another way that we can help to automate some of these actions or you know really gain some

efficiencies mentioned automation um we can also send the results to elasticstack or splunk there's an artifact for each of those i do this with security onion you might have seen maybe a few attended a sock 21 then you might have seen where i was demonstrated sore lab and that essentially is going to hook in with n8n and velociraptor from security onion but there's a lot of different ways in which you can integrate this stuff so it's amazing to see exactly what we can do with it because the possibilities again are pretty much endless now if we want to start digging we can grab the latest binary from this url right here it should always have the

latest one there and if you want to test it out you can run velociraptor gui and what that's going to do is it's going to set up that server it's going to set up a client and then from there you can go off and test your queries you can test those artifacts and play around with it get comfortable with it get your feet wet and then maybe consider standing up the deployment from there all right and so just going to walk through a couple things here um just talking through some ways in which we can identify if a file existed right we may find ourselves asking did this file ever exist on the system and i mentioned before there are some

different ways that we can do that now the windows ntfs mft artifact can parse the mft and search for file names if you'd like to do that the yara ntfs artifact can utilize yara and we can search the mft essentially so it's much quicker than parsing the mft right we're searching with it we're searching with the aura first and then we're extracting what we want and then i mentioned earlier the usn can also be used the artifact can be used to parse the usn journal the usn journal is constantly tracking that you know constantly rotating and we can watch that and store those hashes essentially into a sqlite database and then we can search for those later that is

maintained and we can then have a record of all the hashes that ever exist on this host and that's what this local hash is usn right here will do then i mentioned the i30 and the vss as well but aside from that i want to go into a little bit of a demo about scanning process memory i mentioned that we can do this with yara and that this is quite common right with fileless malware for them to reside in memory and not draw files to disk and this is a very trivial demonstration but just to get the point across i want to pop over here and the resolution will probably change a bit okay so what we have here this is just a

windows 10 vm and i've run the velociraptor gui command on this box and now it's set up the server and clients i can see the server here and again i apologize for the resolution some things may be cut off but we can see the server here and what we'll have here is just some some status information and you know currently connected clients some disk space information and then we can also go over here to this little icon it's going to play nicely okay oh that's a hunt okay and this is where we would kick off a hunt if we wanted to do so we're not going to do so do this right at the moment but just

wanted to kind of navigate towards that here's where we have all of the artifacts for example i mentioned the yara there's a detection yara glob artifact and it's going to essentially return a list of files right and then run yara over that list i mean as you can see we can scroll down you'll see a ton of different artifacts for linux mac os windows all that kind of stuff but i don't want to derail from this other discussion too much i'm going to go over to a notebook and we're going to work with this process memory real quick we're going to do this demonstration okay so i've got you can't really see anything right here there aren't any

results right now so you won't see anything any data available but i'm going to click to edit the cell and try to drag this up so you can see it but essentially what we're going to do here is we're going to do a ps list and we're going to look for any processes with the name of notepad right and then this is just really to narrow it down this is probably not a realistic demo case because you wouldn't necessarily be looking for notepad obviously but uh to demonstrate the point we're going to look for any processes with the name of notepad or with notepad in the name and then we're going to apply the process accessor to those

and we're going to apply a yara rule and this yara rule is typically is you can't see me a very simple rule you know just text so we're going to go over here and we're actually going to launch

notepad we just type you can't see me in there we're not going to save it right so this is all about seeing if we can find something that's in memory right and while this may be benign there are lots of great use cases and reasons why you want to do this in the real world so we're going to go off and run our query here when we run that we're applying again that process accessor so it's taking the process memory it's treating it as if it were a file and then it's applying the yara rule to that and so we can see in this somewhat academic example we have the droid we were looking for right

we can see you so we can do this with real malware right we don't have to do it with notepad but this is a capability that we have at our disposal just one way that we can leverage velociraptor for scanning that process memory now let's go with another example let's see what do we have here play from current slide

all right and this is just an example of the query here and please do try this out at home if you get a chance to and we can see here that we were able to scan that process memory and if we were to use the windows detection process memory artifact that's already built into velociraptor we can do this from a detection perspective we can be looking in non-standard locations or in certain locations for these binaries or for these executions and identify that in memory now another thing that's pretty tricky and and a lot of love tools can't get this right windows can't get this right natively process spoofing parent process moving right a lot of platforms rely on process

creation logs from things like sysmon and how they track those parent-child relationships now a lot of times i'm not going to say a lot of times but these relationships can be spoofed right this means that no matter what you're sending to your sim you're collecting all of the sysmon data and you're sending it to your sim or your log management platform and you're killing it with logs you're never going to see this because you're going to see what they want you to see right and this is all because of the windows create process api also it doesn't really help because it allows non-admin users to do this so as a non-admin user you can spoof the parent

process of your process we're going to demonstrate that select my parent was written by dda stevens who's very active in the community and it allows you to spoof the parent process id or the parent process of a process right so we're going to try this out with onedrive.exe i'm going to flip over here

that's not what i want okay so let me get over here into a blank terminal and make sure everybody can see that yep okay so what i'm going to do first is go to task manager i'm just going to get the pid process id of onedrive right here i've just opened task manager click to the details here i'm going to get 4760 is the one that i want to emulate so i want it to look like onedrive.exe spawn this process as the attacker so i'm going to use selectmyparent.exe and the process or the executable that i want to execute is notepad and i want to make it look like it was spawned by one so let me double check once before i

screw up this demo 47.60

all right so that quickly uh we've launched notepad and its process id is 5356. okay now let's go over to the sysmon logs here and again i know this might be painful so please bear with me trying to adjust this here and let me actually just find notepad okay so we're gonna move this up okay we'll just get the details make it a little easier okay so now if we go here it is notepad.exe sysmon reports this right the image we were working with notepad we were spawning notepad but if we look at the parent process id it's 4760 and it shows the parent process as one drive which we know is not the case

that's pretty scary pretty scary that even cismond that we all rely on to have you know that ground truth and to ship to our sims and our log management platforms even sysmon thinks the parent process is one drive so how do we get at the ground truth

what we can do here is leverage etw i mentioned etw before event tracing for windows and we can use this query right here to actually watch etw and this is the specific grid right here that we're looking for and we can look for anything where notepad is in those event details and we can get an idea of where there's some outliers there so we're going to do that real quick pop back over here

all right and so i'm just going to refresh this real quick okay so let me stop this i've got this notebook here this is for parent process id spoofing this is the query that i mentioned just a second ago and we're watching that etw provider and we're looking for anything with notepad in the event data so we're going to save that and we're going to execute select my parent again so we can drag this up a little bit or scroll down okay so we're just going to do it for a different process let me actually kill this other one so there's no confusion wait let me get this out of the way cover that okay

wait that one's open i'm jumping all around somebody stop me all right yes clear that okay so i just want to demonstrate here how again with select my parent we're going to run it with notepad and assume the apparent process of onedrive right so sysmon knows no better and that process 8208 or 8208 looks like we just got a result here and look down here just the one no we will see a result here right okay so here is where we see the where is it sorry i'm jumping around again process id 8208 right notepad.exe and the event data and then the onedrive parent process id of 4760. but if we actually look in the

system data it's going to be 1696. so it's completely different right but how would we identify that uh you know we can identify the process that actually executed that in a different way we can use the there's an artifact the etw detect process spoofing artifact and we can run this as a client monitoring artifact and this is going to watch for inconsistencies between those parent process ids and what's reported in the event data and the system data okay so i've actually already got this running if i go over here and i click on the results here i'll scroll down a little bit and it does take some time that'll take a minute to buffer on the side until you

get the result back but we can see here essentially this is a different result set but essentially what we will find is that the real parent and again i apologize because of the resolution the real parent right here and the suspicious process right here we have select my parent as the real parent the suspicious process of notepad and then if we see if i can scroll over take it down the bottom let me scroll come on yeah let me control minus maybe oh it's too small oh things down here okay all right so let me go in a little bit okay so anyway we have the real parent over here selectmyparent.exe then we have the purported parent right

here onedrive.exe right

so again we can see the actual parent process id and the system data and in the event data we're going to notice the spoofed parent process id and then we took that query and implemented that client monitoring artifact and now we can see the suspicious process of notepad.exe the real parent of selectmyparent.exe and then the claimed parent the one that was purported to be the parent process which is one drive now there's another thing that you could do i'm not going to necessarily demonstrate it today but uh there was adam chester from xp insect did have a blog that he wrote about uh subverting etw so there's some very good research about that and essentially the i believe it was the com

plus atw enabled variable if you set that then you can actually even subvert etw zone hoax right or the process is hooks in atw it won't be registered so even in this case we might not even see it from the etw perspective i'm not going to go into all that but just something to consider that you know even with sysmon and even with etw you always have to be careful about you know how you're looking at things and you know and have those different layers and and be able to look at that in different ways now that is all i have at the moment if you would like to hear more about velociraptor you can

follow velocidex on twitter i'm also up there and mike cohen uh there's a github repository for velociraptor right here also these articles and these docs over here medium.com velociraptor ir it's a really great resource to be able to test out velociraptor and apply some of these use cases similar to what i've discussed there's also a discord if you have issues or questions or feature requests mike and others are very active on there answering questions uh that is about all i have unless you guys have any questions and i would be glad to answer them unless would you rather me do the uh the other first which way yeah yep okay so again what is we're gonna do the

giveaways first uh what is gonna be that container for the velociraptor query language there's various queries and that expert knowledge to be encapsulated in the first one yep yes all right and i didn't even tell you what you won because that's how awesome i am it's a mystery but you won this alpha networks usb wireless adapter

yes sir [Applause] all right now let me do this right so linux basics for hackers up for grabs all right so i don't think a good question i can never think of these um what is one way that we can how do we test our queries or uh okay yep notebooks okay very good thanks sir [Applause] yes sir all right any other questions about the presentation yes

right yeah so so in that instance you would see uh the system would report the same as the event data right so um yeah because we saw that that difference between like the actual system data and the event data um it's you know it's kind of where the the difference comes in and and where etw is seeing it and then sysmond is seeing it because it's subscribing to that event data right awesome anybody else have any questions concerns or snide remarks yep

yeah so that um it will it's right so it it takes an argument right like the path right or a path so you can filter it down right and you can modify the artifact itself so it's essentially what you want to be looking for or watching for right so you may not you may watch everything you may not watch everything you may watch specific things right specific um named processes or or whatnot that are likely to have that happen right yep yes ma'am

yep

right so um a little bit of feedback but uh yeah so so essentially you know there's the kind of the sysmon level subversion right where you're it's actually it's only hooking into the event data from etw right and then there's the etw subversion where uh you're basically telling the process not to have its net assemblies inspected and there's no hooking into etw for that process from that perspective right

yep so like what evidence of velociraptor's usage itself right so there there's certain you know like directory structure and everything um so you do have that and obviously you have the agent and server communication but but that's that's essentially you know it's kind of a trade-off right so um as far as the evidence itself um anything that you collect right is not necessarily stored on the the server the results of the query are on the endpoint the results of the queries are sometimes right some and then like they're also sent back to the server right so there's certain data that's kept i won't say all the queries are stored on the on the endpoint or

anything like that or all the data that you collect is it just you know depends on if you're sending that data up to the server and then the results of the queries are sent up to the server essentially but there could be things like um you know if you're maintaining a client-side file hash database if you're watching things like usn journal and you're tracking all the hashes that are on there then that will reside on the client because that's how we keep track of that but you know just it's those particular cases yep

yeah so actually that's something i've actually started to look into is you know tampering right so as far as with velociraptor and kind of disablement so um right so i think with anything like any edr or any any kind of platform like that um it's important to have some some ways to identify that so i mean you can absolutely write an artifact right now for it right the power of equal is that you can do anything you want with it right now so anybody else have any questions yes

i can't speak to that i can say that i know mike is very adamant about you know open source and has developed you know quite a few and worked on a few open source projects so um i don't foresee it going to any licensed version but you know obviously i have no control over that but um but yeah i foresee a bright future as far as maintaining an os or an open source you know long term initiative yeah oh okay awesome yeah

right

right right i've heard maybe like inside rdr yep so okay all right well cool anyone else have any questions all right well thank you guys