Josh Bower - Enriching Osquery with Actionable Context

hope you guys are having a good day so far it's been good good alright alright my name is Josh Brauer been in AI T 15 plus years and InfoSec 10 plus years really been focused on network and end point detection the last few years you can catch me on twitter at defensive depth and email Josh defensive DEP comm so today we're going to be talking very quickly about enriching OS query with actual context anybody here familiar with OS query a few maybe yeah maybe a half ok anybody using OS query on a regular basis ok great none that's great good so we will definitely be looking at some things that I think will be helpful

for you ok anybody here in my talk last year about OS query all right maybe half the room all right so this is a new talk so no worries ok it's not a duplicate alright so OS query for not familiar with it is a open source project that was first published by Facebook back in 2014 as of this summer it now lives under the Linux Foundation which is great for the longevity of the project the key thing about those queries that allows you to envision your system as if it's a relational database okay so you install it on your computer you say select star from users semicolon and you get back the output it looks like it's

from a database like you've just run a sequel query ok and so excuse me and so one of the key aspects of those queries that allows you to use the query language that we already know which was which is sequel ok there's no proprietary language that you have to learn it's also has wide support among lots of different platforms including the BSD s Mac OS Linux and Windows so those are really the distinctives of OS query now there are two pieces to Oh square you can run OS query I the interactive shell and what that means is you install OS query directly on your box you open it up and you can run your queries

in your querying information about your local system okay there's also au Square D which is the daemon or the agent mode this is where you push out o Square to your endpoints you have a management server which there are open source management servers available and you schedule queries to run every so often so you say select star from users I want to check that every five minutes and if there's a difference if there's a change send those logs back to me so I schedule that queries so that's every five minutes on my servers it checks to see if there's any new admin admin users if there is I get that log sent back to my

servers or excuse me my my log management service and so that's what the daemon version or the agent version of those query is is that making sense so far we're gonna be we're gonna be moving pretty fast since this is lightening talk okay so OS Cori has 220 or excuse me 230 plus tables if you go to iOS query that IO slash schema you can see all the different tables that are available for your you to query these are the the tables are what you can query to get all that relevant information we're talking about there is a user's table a certificates table all sorts of different tables that you can query to get information from last year

we looked at the Chrome extensions table chrome underscore extensions this is actually a screenshot from the website this shows you the different platforms that this extent or this table is available on here are the different columns you can query so you ID name identifier and then here's what a query would look like select the user ID name identifier column from Chrome extensions and this is what is returned okay nicely formatted text here of user ID all the chrome extensions on the system the version and their unique identifier okay now we talked about really using OS query and more an individual or interactive mode last year today we're going to briefly talk about we've got OS

query installed on our systems across our organization and we went will talk about extensions from that perspective now if you remember we talked last year this is September of last year the mega Chrome extension was hacked version three three nine four and so we talked about how we can use those query to find that specific version of the extension we simply just say select the user ID name identifier from the Chrome extensions table where the identify er is the mega chrome extension and the version number is 339 and then we would get back the result if we had a compromised Chrome extension on the system but what happens and this is more likely the case what

happens when you've got OS query deployed on a hundred different systems on your on your network and you don't really have an indicator of compromise you don't really have this information that hey I'm gonna go write a query that tells me this you know this is what I'm looking for that's malicious or compromised what if I'm just getting back a bunch of data about Chrome extensions how do I know that something is malicious or has been compromised how can I find evil even in that context and that's what we're going to be talking about today if we just get back this data what can we do with this type of data that's where I'm gonna try to do a

live demo and we will see how this goes I think it'll work first of all for the first prize how many tables did I say are currently available on a man you all right all right all right your your choice right here pick one and then leave the other one on there first hand up thanks so much and that was really good I'm gonna have to find a more difficult question all right so the the scenario or the context is we have OS query installed on our on our endpoints let's say 100 plus endpoints we're sending data back we've got a scheduled query every five minutes select star from Chrome extensions we're getting differential results back this is kibana

which is part of the elastic stack elastic search cabana logstash that's what I'm using in my environment okay and if you can see this this is what we're currently getting back we're getting extension date if I scroll down here this is what we're already seeing we're getting the extension description the identifier the name so we're getting this information back but the question that I have is out of the box this is good information but how do we know if we're just getting hundreds of you know this kind of log resort results back how do we know that any of this is evil is there any other context that we can add so glad you asked so if we go over to

let's grab the unique identifier which is right here of the Chrome extension we come over to the Chrome extension webstore okay chrome.google.com/webstore we're just going to copy and paste that unique identifier in there that brings up the extension we were just looking at swag button and there is some other extra metadata that query doesn't currently bring in we have the the rating we have the amount of users that's right in it we have the total number of users we also have some metadata down here on the right-hand side when it was last updated the size languages now what happened is um this is late last year I got a log back from my environment that said someone had

installed the signal private messaging Chrome extension okay if you're not familiar with signal private messenger it's a messaging solution that is supposed to be very very secure private encrypted and the encryption right they do have a legitimate Chrome extension but coming from this particular system and this user I thought it was a little odd so I looked into it more I went actually and looked on the Chrome Web Store for this it was a brand new extension but it was labeled you know a signal private messenger and but it only had I think I had like three ratings and it had less than 50 users okay and so when I went out and googled for the

actual signal private messenger Chrome extension that one had a ton of ratings and I had over a hundred thousand active users so just showing the difference just in the amount of users versus you know less than 50 users I knew that there was something off on this particular extension there come to find out it was removed from the chrome store within the day it was a malicious extension so my goal at that point was well that's kind of interesting can we get some of that extra metadata into my pipeline for logging so that maybe I can add that to my data and maybe get a bigger picture is that making sense all right so that's

what I started doing excuse me I started looking for a way to integrate that information around that time duo security came out with a new app called CR excavator okay anybody here from duo security okay too bad I was going to give you kudos either way excuse me either way this is a great new app it's also got a free API and so if we paste that identifier in there that's gonna come up with a port has metadata that we were looking let me make this bigger here all that metadata we were looking for but it also has a specific total risk score that they have they have they have their own way of identifying risk for extensions so

that's there they have risk over time they have permissions they also have all the way down potential external communication from the extension which is kind of interesting so this is pulled from the source of the extension so lots of really interesting data that we can get for a Chrome extension so it has an API and I'm using log stash so I said well let's just go ahead and build into the log stash pipeline let's hit the API and let's get the extension metadata for each extension that we're seeing in the environment and so that's what I started doing if you're not familiar with log stash pipelines I'm not gonna go into a whole lot of detail on this the key

aspect is you have an input that's where your logs are coming from you have a filter clause which is what I have right here that allows you to add and delete and youtaite your data and then finally you have an output and that gets output to in my case elasticsearch which then you can view with bonham so this is just a very simple filter clause what we say is that if the extension or if the name the log is chrome - extension - filtered so that we know we're only gonna do this on Chrome extension logs then run HTTP this tells us that this URL go do a get request on the API see our excavator i/o for the

specific extension puts the response data into a new field called C E or Chrome extension - raw target headers put into C - headers then the next one is a mutate clause all we're saying here is copy out certain fields from that API data we're copying out the total risk rating the webstore rating the rating users and the users then we're adding a new field called report we'll see what that looks like in just a second and then just remove all the raw data okay and then output that to elasticsearch so every time a Chrome extension log comes through this is gonna run and it's going to grab the metadata for that from the duo security API and then put that into

the log event so that we will see something like this so these are all the extensions that I had come through we see the name and then if we drill down here are the I mean here's the old stuff down here you guys should be able to see that yep so here's all the old information we had write the description the author the name but then the new stuff we added the new metadata is up here so we have a rating so let's add some of this to the view let's have total rating the rating users and the risk rating and the users of the extension so we have an nice little data table now of the Chrome extensions in

our environments and you can see that we have in this scenario we have the name we have the people or the actual rating for the extension the amount of people who've rated it the risk rating from duo security and the total users of the extension now this is okay it's a data table you know but if you're actually looking at this day in and day out you're gonna want to create some sort of a dashboard right you don't want to look at it in this sort of format and so I went ahead and also generated a dashboard that took all that same information but made it a little bit easier to read so the first

part here is we have non unique extensions across one end point five high-risk extensions in our environment and that means anything with a total risk score of greater than I think I said 400 is a high risk rating and then we have those high risk extensions over here to the far right with a little bit more information about them if you as an analyst are wanting to drill down into one of these extensions you think this one's kind of interesting we can click the pivot link and that's going to go ahead and bring us to the full reports from do security so that we can get more information about that and then we also have the the middle portion here is just

a data table that allows us to slice and dice the extensions okay so we can say show me everything with a certain risk rating how many total users we can see that there's an outlier here right so we have one extension called ice doto2 with total users of 455 users no rating because nobody has rated it so far right and so that would be an interesting one to look into further okay so that's just that's about all I have for the demo we brought you all the way from we have kind of this raw extension data that we get from OS query and we bring that to finding out what other context can I add to OS query to

help us understand if an extension if I don't know that an extension is malicious what other information can I add to that to get us to a point where we can say I need to follow up on this or not we looked at the duo security API and then we looked at the pipeline how to bring that in and then finally we looked at what that might look like from a dashboard perspective that making sense I went really fast but it was lightning right all right so the question is you ready question I talked about logstash pipelines what are the three sections in a typical logstash pipeline config yes man good job yeah I have a book for you right

here Chris Sanders and Jason Jason can you sign this one of the author's is here Jason we'll have him sign this for you all right any questions or comments at this point it's gonna pull this up real quick so last thing I do have we talked about OS query there is an OS query class that I run learn OS query calm and we talk about this in a lot more depth I actually do a demo exactly on what this looks like and bringing this data in from logs - okay and but I'd be happy at this point to go in depth all of this is available on the public repo the the logstash pipeline I

will tweet out this deck as well as the github code and everything that I talked about in today's session I'll tweet that out later today and you can find that you can find that on Twitter all right any last questions yes sir what has been the most difficult it's got this pretty broad question right so the question is what has been the most difficult to detect and investigate with like OS query I think some of the difficulties I've had with OS query is what exactly am I trying to look for right like you have access to 230 plus tables I mean just massive amounts of information that I can live query or I can schedule and

so what exactly am I looking for and then actually that actually went into my talk yesterday as secure Union conference where I talked about writing a playbook and that came that playbook came out of my struggles with structuring detection strategies around things like OS query and some of the other things I was working on really what iced what I started doing is I look at I mean some people try to look at it from the perspective of you have the mitre attack framework you have all these techniques let's start building detection strategies from an o squared perspective around all these techniques and you could certainly do that and that would be one place to start but I also

just start with some really basic stuff that I see in an everyday environment like I see you know domain admins setting or people in our environment that have domain admin privileges that are using their accounts as service accounts you know so I just start writing detections based on things like that the known stuff that really shouldn't be happening so I'd say my struggle has been knowing how to use honestly the power that that all these all the visibility that you have how do you know how to use that efficiently is that making sense and I think as a project I'm very involved in no square as a project I think we could do a lot better at kind of especially

new people to the project at helping people off on the right foot of starting some of those detection strategies so I don't know if that's where you're coming from but any other comments thoughts or snide remarks histories yes yes

yep yep sure yeah that's a great question so Jason's asking and I think it's even a broader question is that you're you're using a free tool free API duo duo's I'm CR excavator putting this into production what happens when that falls over or what happens when the person who's maintaining it leaves right so I had talked with the guy who created it a little bit we chatted I don't know the long term support a couple thoughts I actually before and I read this up a while ago so it's been about a year and a half I actually wrote something like a miniature duo excavator for me personally because I wanted to do something like this already but it

doesn't have I didn't have quite the level of detail that they have and so I just once there's came out I just got rid of my internal stuff and started using theirs so if that went down I would probably just pick up mine again and finish it up and maybe publish it for me I typically just look at that kind of stuff and say let's let's run with what we got right now and enbe yeah yeah exactly exactly I don't know how much time I have am I done I'm good two o'clock okay I mean I did go lightning-fast on that that's great okay any other thoughts or comments yes sure so more process auditing right so excuse

me so from a query perspective OS query has a process auditing support built in for Linux and Mac OS not yet for Windows but I hear that's coming and so we can check we can check running processes and parent processes and things like that from a Windows perspective but it's not it's not fully integrated into the system yet and so I'm not doing a ton of that at this point but what I am doing let me give an example like network connections would be another one so all the concerns about public RDP and are the people and abilities in the past year we have a query that we run on a regular basis every 60 minutes that

looks for port you know so RDP traffic inbound from an external IP address and the idea there is that we don't have to run network auditing you know process auditing from that perspective connection auditing because we don't need to check every two seconds for network connections because we know if there's an external RDP port open you're just going to be hit constantly and so we just assume we'll see connection like every 60 minutes or something like that so from a the the shorter answer to your question is I personally am NOT doing a whole ton a process auditing from a system on excuse me a query perspective that's where I would rely more on a

system on from a Windows perspective this Monda has a lot better capabilities especially for filtering from a process auditing perspective I would love to see more more system on type stuff built into O's queries so that I can just run O's query across the entire infrastructure and not have to have OS query Plus this mahn and things like that and there is a an extension out there that people are using that duplicate 400 square that duplicates a lot of system on type things but it's an extension at this point it's not in query core it's a long answer to your question any other thoughts or questions yes sir

yeah yeah so like fleet management what fleet manager is specifically for OS query or yeah yeah so there are four O's query so if you're running the you've got query as an agent or daemon installed to your to your organization's endpoints you need a management server right because it checks in every so often is there a new configuration change what queries should I be running there are a bunch of commercial and open source fleet managers out there I've used both commercial and open source from the open source perspective we are I have typically used collide fleet from an open source perspective and that's what I would recommend that is also the one that we have really deeply

integrated into security onion the new version of security onion called hybrid hunter you install it you run it you hit one command and it generates it installs collide sleep it generates query packages you need specifically for that install of Clyde fleet and you can deploy that very simply and quickly and so that would be my plug for for security onion but Clyde fleet would be the one that I would recommend unless you are unless you're completely cloud-based there is an AWS fleet manager I cannot remember it off the top of my head it's horrible but there is an AWS a fleet manager that runs completely in a the AWS and so that would be one to

check out as well okay Wes you got anything for me ok great ok thank you all so much for your time appreciate it [Applause]