Hunting Before Day Zero

okay I'll get started hey uh I'm Ryan labov that's labov rhymes with the Cove The Grove the stove uh and today we're going to talk about hunting before Day Zero so a little bit about me I'm a founder of startup security and uh in my free time I like to do Disaster Recovery storms not hard drives um two weeks ago when I should have been prepping for my slides I was out helping a bunch of farmers with some tornado damage which kept me up late finishing my slides which I did uh mercifully early because I'm a responsible adult um so what do I do at startup security we secure the tools Technologies and companies that are

building the future we want in practice that means helping series A B and C startups go from uh nobody does security to somebody does security um and uh also in practice we do quite a bit of security observability and research um so my background I started as a software engineer uh I was then tricked into becoming an SRE which is like a mix of software engineering devops and CIS admin it's like keep the software running even though you don't know how which is a lot of fun uh and eventually that led into doing more and more security work into what I call security observability and doing uh making tools for that um so how I operate and this is going to come

through with the with the presentation uh I prefer to go first principles up not big tools down meaning let's understand the problem not just throw tools at the problem um I love open source tools so I love looking at other people's projects and reading their code and using their tools even though it can be painful to do and set up at times uh it's great to be part of that community and then I am powered by curi curiosity and Shameless iteration so we'll see lots of mistakes and there's probably some in the slide deck but that's okay uh also all the slides the code and a range that I put together so you can play with some of the things we're going

to talk about so available on my GitHub um which I'll link to um also about me there will definitely be spelling errors here I'm sorry I did my best uh so today's agenda um my main argument is understanding the fundamental signals from Linux gives us an insight to compromises on our systems and networks um we're going to start with a story about how I learned about signals uh before I knew what signals of compromise were um and then uh we're going to talk about um some philos philosophy of tactics of using signals of compromise to um hunt for vulnerabilities and then uh a bit of a conclusion so what are you going to get out of this uh if you're a

beginner or intermediate I hope that you'll get a better understanding of the signal signals available to you on Linux and how you can increase your situational awareness with what's happening on your systems um and if you're more advanced uh I want to convince you that having the ability to ask specific questions in your systems in your network without relying on big tools is a superpower and I'll show you a open source tool stack that we use to do that um and you can use either tools that you find or um you can swap any one of them out really so also um this is a picture of my hotel room as I Was preparing this

talk there's probably four or 5,000 Pages worth of material reference material that I study for this topic I'm not going to cram it all in the slides so my goal here is to Peak your interest and to give you some next steps like pointers to other information so you can understand what you might be interested in not comprehensive coverage um also uh big thanks to the organizers I appreciate the opportunity to speak about this um okay let's get into it so how to accidentally discover a zero day uh and for the definition a zero day is a vulnerability that has not been disclosed publicly some important parts about this there's usually no patch available because nobody knows

about it and there's no way that you can track this like see if my systems affected because none of our tools are able to detect it yet um so I made up a client to redact any sensitive information here let's say we're talking about Hyperion Corp thank you logo AI for this lovely icon um so the the goal for this client there was a critical service that had no firewall uh and we because it was so critical we wanted to start securing it and so so the the key is it's critical and it cannot go down if this service goes down the whole infrastructure crashes um so how are we going to bring up a firewall well we need to understand

the service we need to understand the expected traffic and then we need tools to observe the traffic um and so being at startup security I'll tell you a little bit about startups specifically in hypergrowth this is after you have users investors give you buckets of money and say cool your only problem now is growth don't worry about anything else hire as many people as you want spend as much money as you want and um the thing about that is growth means more than everything including security um and when people go down this route microservices are a popular choice I'm not here arguing that they're the right choice they're just a popular thing in in architecture you'll find and I'll

give you some bad like leadership napkin math on this if you're like okay we've got these big problems and if I can break them down into a thousand small problems well articulated that comprise my problem I can hire one team to solve each problem and go really fast um there's obviously some practical uh differences to how that ends up working out but that's that's our environment for for context and so again microservices we expect them to do one small thing we expect that some traffic is going to go to our users and back to our services and there's going to be some traffic that's North and South traffic and some traffic is going to go

east to west meaning our services are talking to each other and it's not uncommon for large companies to have hundreds of microservices so our story here's our infrastructure that we can use to kind of understand the situation there's a load balancer there's a few um front-end services like the website the order portal the admin tool we expect users to be talking to these services and then there's uh a few back of the house services like the service that's sending our email the service that's doing our user and authentication flows which we'll call arbac role based authentication control I I'll say arbac almost uncontrollably in I mean users and authentication Service and then metrics so um again let's understand the

the traffic that we should expect to see when we're bringing up this firewall we should expect to see servico service traffic meaning uh our order portal says Hey can user one delete order 7 uh we should see maintenance traffic meaning of a system administrator comes in to troubleshoot maybe they're using teleport to Shell in we should see deployments we should see system level updates okay and then to understand what the service does like I said earlier it is used to figure out which actions people can perform this is why it's so critical um and then behind the scenes there's a a go service that operates some business logic and the data store is a key value store this was reddis and

so think about the key being your user ID and the value being what actions you can perform and then there's some database source of Truth underneath it and so um again we're expecting to see network calls that look like this uh other services trying to authorize actions u in in reality instead of returning this we might return a token but okay so also let's think about what could happen if this service is compromised uh so if you poison the redis cache we can authorize users to take actions they shouldn't be able to take um depending on how we've compromised the service we might be able to xfill some of the data either from redus or from the database and depending

on the network we might be able to move laterally within the network um anyways okay so we know what the service is we know what our goal is of bring up this firewall so what are we going to do um and we have this uh this idea that we can look at TCP traffic we don't we can ignore UDP and icmp and we can count some very simple details about this traffic to get a really good understanding of who should be talking to this host right now we can use that to drive our firewall configuration turn it on without breaking anything and um again just quick primer on TCP um in order to establish a connection it's a three-way

handshake uh the the host sends the sin packet to your destination your destination sends a sinac or synchronized acknowledgement and then your server uh acknowledges that and that's how you create a connection so by counting these really small details about in bulk about um these connections we should be able to understand the traffic that's coming to the server and so a common thing if you're doing this on a physical Network you might use wire shark um since this is a service running in the cloud uh and we have access TCP dumps maybe a little bit easier um there's also this world where we could use something like lib pcap which is is what powers TCP dump to um

to have a program we write kind of Reason about and track what uh what the data is coming instead of having to track it and then process it and then uh one of the things we discovered in this project and one of the things you'll find that I'm a very big fan of is ebpf which is extended Berkeley packet filters um this is really cool it is a way that you can run user code in your kernel and um we'll explore what that means in a minute so we built this agent at track syx and we had a configuration that looked pretty good but we noticed something weird there was no reason this service should be talking to users as much as it

was originally that would would have made sense because people could be authorizing directly to the service but as as we continue to refactor this infrastructure that didn't make sense anymore especially at the levels we were seeing and then also we noticed that this service was talking to to other services that it shouldn't be talking to because it shouldn't be talking to the email service um at this point the network allowed service to Service uh communication cart blanch because it's a service they should be able to talk to each other they didn't have any granularity but um sus traffic plus forensics we found something very wrong uh which is why I think signals of compromise are so interesting um so

zooming out what we ran into was a signal that we observed that ended up being a signal of compromise so and the small signal revealed quite a bit about a system and so um one of the some of the class classical axes for signals of compromise is signature based detection which is I see a file name that matches something it shouldn't uh threat intelligence databases which is uh just a compendium of these type of things Behavior Analysis looking at how a system operates or some type of complicated databased learning that's based on behavioral analysis um but the more I thought about this it really reminded me of one of the conversations that I know from Sr land which is the difference

between monitoring and observability um and so monitoring and it's a nuanced difference monitoring is I can I can collect exhaust from my system but observability is I can understand the system based on certain events that I see happening maybe as part of the that exhaust maybe is something else and uh so one of our goals has been to have this idea of nimble security observability and one of the core things that we always talk about is the ability to use the smallest signal possible it's hard because I don't know the right word for this like we've used big signal versus small signal complex versus fundamental molecular versus Atomic um but I'll I'll give you some some

examples here let's look at what would it mean to use a big signal of compromise um so a big signal of compromise is not so much an event it's a series of it's a complex situation it's a circumstance that you find your system in so um some examples of this let's say we're using like normal signals of compromise to detect something like Cobalt strike C2 pretty popular uh tool for doing crime and stuff so something that we we often would look for right so um pulling from Red Canary and just looking at some of the popular ways to to detect this um patterns and named pipes um finding specific patterns of process execution uh finding specific patterns

in things that we find in command line processes and the problem with this and again this is a tool it's not saying it's a bad tool but a problem with this is it's a game of cat and mouse because like ah your Rex C catches naughty process. exe like rename it it's like boom uh so that's where I like to think about this kind of inverted is what is the smallest signal we can look for and detect a problem and so let's think about the same thing looking at command and control systems what are they trying to do on our system and so and one of the things uh and so what I'm wanting to figure out is

what can I what what events can I use to describe the things that it's doing and so what is it doing it's doing some amount of data exfiltration Maybe encrypting your data for ransoming it maybe moving laterally within your network privilege escalation all kinds of bad things but let's focus on um looking at trying to find something that's doing data exfiltration which is just moving your files and it's worth noting people are going to be very sneaky about how they move your files here's an example somebody uh sending out data through DNS tunneling traffic like you have to look for that's why it's so hard to use these big signals but if we Zoom way in what small signals

do we have from a system level to help us detect that I mean that our files are moving and we have the file system and the network how do we interact with these system calls and the system call is a API that user space can communicate with the kernel to do things on the metal and so your your files don't move if you don't engage with these calls is how your system works and so it is a very small signal and it's a very it it is a single source of Truth we can look at and so how can we Snoop around at system calls um and so going back to ebpf we can use

this to probe certain calls and get um we can collect data on them with uh I'm using Prometheus here visualize them with grafana and query and compose them with Prometheus again which I'll show you so um again epbf for snooping system calls um there's different probes you can set up in uhu well I'll just show you one in a second it'll be easier to understand uh so how do how do we use this though um we're going to write a small C program going to compile the program into bite code um we're going to load it into whatever tool chain we usually write programs in maybe python or go um we're going to process the data and

we're going to clean up and um again here in this I'm using Prometheus for observability into these things it's what a lot of our companies already use for observability in general so it's just a tool chain we have um you can swap it out with whatever type of observability tool you're using um and grafana which you'll hear me mentioned several times has a direct integration to promethus so we can graph and alert on things that we find um and another important thing to notice to mention is anything you use really needs the ability to filter Aggregate and calculate rates and quantities across your data um in ideally in multiple dimensions um and so let's talk more about file

access so how can we detect that somebody is trying to xfill files from very small signals um worth noting uh this is focused mostly on Linux or exclusively on this talks exclusively focusing on Linux um and this is a little bit hard because everything's a file on Linux uh your directories are files devices are files processes are files everything's a file uh so um we have to take that into account but if somebody wants to move your files with data in them what are they going to have to do so here is all the system level calls which I'm going to read through an excruciating D just kidding uh but so there's lots of system calls involved in

this um filtering it down and and looking at the ones that um are used to access data with a tool like cat we see five suspects um and filtering down even more similar to how we use sin andac every time you read data from a file you have to use read and so very easily we can set a probe up on read and say I want to count every time a PID reads from a file and so we can figure out okay our web server is reading a lot of files that's is normal um but not malware dobin is also reading a lot of files this is not normal um so you can start to see a lot

of really interesting things from our system in in a a very small way because this storing this doesn't take up a lot of data either um and so again we can go through similar exercise to figure out um weird file access patterns uh unexpected file modifications file creation and deletion these are all really easy to understand with um with system calls um so and same thing in the network um with ebpf uh so again as you might imagine uh networking is very complicated uh but and the number of system calls used to do it is also extremely long you're creating sockets you're connecting to them buying them listening them you're doing all kinds of stuff um but going

into this common the most common way that traffic's coming in and out of our servers there's actually one to two uh Network calls that we really care about specifically this connect one so anytime our um this sin and act dance happens we can spy on happening with connect to get really interesting data about um what's happening on our on our systems so let's see this in action and uh I'm going to show a very high level demo of this I also have it in GitHub so you can run it on your computer and uh complain when something in the clan compiler breaks because it's very hard to do but I tried to make this so it's

it's running in uh ansible with vagrant and I try to extract a lot this out so you can actually play with this on your own um but let's say our goal is to this is like the hello world example for um for ebpf stuff I'm calling it pastropastro1

from the kernel to our program um through ebpf map in our program all we have to do is load this uh probe the call we want which in this case Cyclone is the one that's responsible for creating new pids um and then every time that call uh every time we detect that getting called we just increment our counter uh which we do here and we could we could collect more data too but um we run it on our server and we can start seeing new pits getting created in Prometheus um we can query them and also to know that it's working here's a program I just wrote that should create a thousand pits right and so with maybe 30 lines of

code we have a really cool way to track and see um that there's large spikes of things happening on our system um and you could optionally set up alerts on this and so this is kind of a contrived example but I want to give you the idea of how easy it is to to probe these different calls and we can ask some really complex questions in our system um so and then a few questions I would have uh if I was sitting and listening to this is how would I know which system calls I could observe in each system there is something um and this is from Ubuntu we can look at all the system calls and we can also

do use something called estrace which you can op you can execute any program just to see what it does and look at the system calls involved in uh in that and so you can use that to kind of Base your your research on anyways like I said um all this code with luck will run in uh on Virtual box with vagrant I set up with anible Prometheus grafana so you should be able to go play around with it if you want um if you're interested in the nuts and bolts of this please reach out to me it's something I spend way too much time on and uh find it extremely fascinating even though uh I know it's not like uh

how would you say it's not going to be like YouTube top five videos snooping on CIS calls but I think you can do some really cool things with it um cool so let's let's go through a few conclusions here um the bigger and more abstract our tools get the more we lose touch with our systems um this is especially true in the world of containerization virtualization I mean your codes running on in a hypervisor on a VM in kubernetes on a container it just it's really hard I remember when I started I was moving moving a HTML file from with FTP to a another computer that was across the room right and so I think the bigger and

more extract our systems get it's more it's more and more important that we understand how to observe them um also the tools that we that I use here are not big tools that cost thousands and thousands of dollars a year they're all open source and so we can use tools to get insights we don't have to we we can we don't have to say like I can't get this cool information from my system because I can't afford XY or Z these are all tools that are available to us if we're willing to deal with a little bit of pain and setting them up and learning how to use them um and then observing small signals

can help us solve big problems um again we happen to discover some vulnerabilities with this but also helping us objectively stand up a firewall and know what data is coming in and out of our system and being able to communicate to our stakeholders that we're not going to break production based on this data that we've observed as a superpower too so not just hunting ghosts but also for doing normal things um and uh um my observability security observability stack of choice is ebpf Prometheus grafana um and then I really believe uh ebpf is a really neat technology a lot of the big tools that you know and love or hate um use it under the hood to to get this

type of observability data but ebpf is also evolving at a rapid rate both in terms of what it can do and the tool chains available it solves really interesting problems in the networking space for things like kubernetes um and you can do things with not just observability but doing things at a really low level um earlier I said uh big signals is a cat and mouse game small signals is a ghost hunt so please be careful because you can get nerd snipers really easily looking at looking for ghosts and finally Happy Hunting