BG - An Introduction of the Kobra, a Client for the Badger Version 2.0, Providing Tactical Situation

If uh you're not in looking for the secret breaking ground track, you're in the wrong spot. Uh folks, uh we're not supposed to. You know what? We're all like old friends at this point. I do I even need to introduce you guys? I mean, you do you've been like 5t away from everybody? Yeah, I I personally shook everybody's hand. Um I'm going to be the target for the tomatoes later. Well, anyway, ple please welcome the brain trust of the University of Illinois of Champlne of Champlain. Uh, Amed Fawaz, Edund Rogers, and William Rogers. Uh, they have stuff to tell you about Badger and Cobra. Yep. And let's get away from the eye chart. And like uh

our lovely MC said, I'm Edmund Rogers. I'm a researcher at the University of Illinois. And thank you, Ahmed. I'm loud. Um, so, uh, they decided to put the lapel mic on me. Um, I do smart grid security research primarily during the day and then at night. We do I like to call it wire research and I've been doing this for a long time. I've been a critical infrastructure defender for almost 30 years and worked at a lot of different places. Uh, my bio's online. This is my son, William. He's the primary maintainer of a lot of the code that we work on at night to get a good and nice separation of duty between the things I

do at the university and the things I do outside of the university. And Ahmed is a PhD student at the University of Illinois. If this goes really bad, he may not be graduating in 18 months. We don't know. So, we're all here at Bsides and uh it's not surprising to me we didn't get a lot of turnout because the really sexy talks are right down the hall where you can learn how to tamper with medical devices and kill people. So, um, as defenders, we're always the the also rans or the Charlie Browns of security conferences. And we just hope that we can meet or push the expectation of somebody who's sitting here is really interested in defense. And what we want

to do with our talk is talk about our research and uh try to motivate you to join the project with us. We love people who come in and tell us that we're batshit crazy or we don't get it right because things don't get better or stronger unless we all work together and um because security has many shortcomings. We don't have a language to reason about security. We don't have any guarantee of data integrity especially when cars are driving off the road and you know my MacBook is going to be owned on Thursday when they give the firmware talk and there's no organization and defense cannot be perfect and I think when we discuss defensive ideas with people um

and fellow researchers they often fall into the trap of um trying to be perfect and we can't be perfect. We need to understand that the attacker can succeed one time in 10,000 and be a black hat, but if we fail one time in a million, we're out of a job. So, we need to be able to talk to the people that we work with and let them understand what the risk is regarding certain things. And if that doesn't work, then maybe we can talk about today things that you can do to help people make the right decision. But as Sergey, my friend says, if anybody knows Sergey brought us um he can't be here today. Um and um I just

put a shout out to him and a talk about everybody hear about weird machines. One of the problems that we have is that the software that we use does a lot of unexpected things and this is where we get a lot of the problems that we see with malware and a lot of our current solutions try

Hello, I'm Geek here. Unfortunately, we had a little video problem, so we don't have any audio up until just about the 11 minute mark. Sorry about that.

kind of the formula um came up with a formula because um people always like formulas and the idea is like so we're going to add these three scores together divide by three uh these summed weights come up and then if everything comes below 33 things should be good and then we get into yellow and in red. And then these are numbers that you can change inside the algorithm to make the scores come out like you might want them to. So, like I said, we got more data than just a million packets. And um what I did was at my house, I don't know if anybody else is like this, I've been recording my internet connection since I

had one. And um so I use physical taps both inside and outside my wireless access point. So I can look at data outside of my wireless access point and inside it. I keep a running score of all that data certain number of gigs and have for years. And um so we were doing some research on some of the algorithms in the badger and then I let some processes run and I got some pretty giant files in TCP dump. I actually had a file with 998 million packets on the inside and 1.3 billion packets observed on the outside of my network. And I found out when my drive filled up that um I had a problem

because I had these 400 gig files. So when you're a researcher and you have a lot of data and um you know after a lot of processing figured out it was 4.5 months of data there were 28 hosts and it's 400 gigs of data about um you get to see a lot of interesting trends that come up. So, like I said, from TCP dump, we went in and we wrote some packet rates to files. What we wanted to do is just count in andout data and kind of like just sanitize uh the raw tcp dump packet into something that we're interested in, things like IP address, source port, destination port, timestamp, that kind of thing. so that we can get things like

packets per second, packets per minute, and just run that all the way through the entire however long the the capture might be. And then we wanted to make unique destination files. So we wanted to be able to capture files to individual destinations so that we can do joins against how many file how many people internally are connecting to a destination. And we could do these cool things with graphs if you could get the data to be a little more manageable. So looking at the data um we observed uh 54,928 destinations from the outside big bad internet. Now 38,28 of those uh IP addresses actually resolved via reverse lookup. So it's about a 70% I think it

came out to 695. I rounded it to 70 70% success rate. But the interesting thing was on the inside there are only 29,829 destinations. So there's a little bit of a difference there. Um a few thousand hosts on the outside that um don't ever make it to the inside for some reason. And um I don't want to really get involved in the why of that because I could probably just do an hour talk on the data. But the the thing that was more interesting to me as far as the d this piece of the data goes is that 99% I was say 98.5% of all the 29,800 domains resolved via reverse lookup. So this really tells

me something that when when I see a new IP address, if I can't resolve that, um there is a succinct chance that this might be something a little fishy. And then this is interesting from the perspective of an observer or a defender that's been doing this for a long time is like way back in the 90s when we were doing this, you wouldn't connect to people if they didn't resol if you didn't have a reverse lookup for them. You couldn't connect to web pages if you didn't have that. Um so simple things like this can really give you an indication if you just are able to look at your data and organize it better. Uh, and then maybe somebody's doing math

back there. I don't know if their phone has been hacked yet uh because of the SMS messages, but uh there's a little bit of a discrepancy between uh destinations that were found outside, which is 29,74, and how many IP addresses I saw as destinations inside, which is 29,829. That comes out to 125 domains. So, I kind of looked at the domains that were um on my inside trace but not outside. And there were RFC1 1918 addresses that either go through other parts of my network that never make it to the internet. Uh things I knew about the inevitable Microsoft DHCP address range, uh which is an IP address you get if you don't get a DHCP response. Um and then

18 of the IP addresses were on my inside LAN. So there's just inside LAN traffic and then uh likely all of these were to wireless networks which is why it went through the AP and then uh multiccast 7 other 13 which would just be like random IP addresses that probably happened while the internet connection was down but the interesting thing was 47 of the IP addresses were owned by Apple. Um, we took a little look at this and basically the short story is when the internet connection is not working right, Apple tries to phone home to see if there's something wrong with the internet connection and depending on what kind of Apple you have depend and

they think they change the IP addresses cuz I've got a list of them. Very interesting. But from there, we we created a directory structure where we took each of the 28 internal hosts that had IP addresses on my network. And then we we put together all of the destinations for every packet that we observed in the 4 and 1/2 months with the idea of how far can we reduce down this data and still get a running snapshot of all the things that have happened in the last four and a half months on this um 400 gigs of data. So at the end of the day after we've done a lot of pruning and um we can get it down to 291

megs. So this is uh source sorted unique list of destinations. I have average packet rates calculated down to the second and I have DNS resolution for all of the destinations for 28 hosts inside my network for 4 and a half months. I can keep it inside a modern laptop in memory. So, this really started to peique our interest in research because we're like, well, if we can get a 4mon footprint down this small, then if we built a client, we could correlate processes with network flows, maybe provide some automatic mitigation based on red or bad things, and provide an independent decision- making that's at the client. So the client's not going out to the server and saying, "Hey, what should I

do about this?" So we want the client to make his own decisions about what they do. And then

um Okay. So you're This is where we're supposed to start talking together. Remember rehearsal? Yep. Well, we thought more people would show up. Yeah. Yeah. Well, you know, the crowd is really rowdy today. This is what happens when you get the spot after lunch. So, hope you had a good lunch. If somebody walks in now, we like, "Oh, you missed most of the talk." Sorry. Yeah. Oh, that's the wrong talk. That's down the hall. I'm sorry. Um, sorry, man. Let's get out a meter or something and we'll So, besides it's canceled this year. Oh, sorry. Anyone have any medical devices? So basically, well, this didn't even need to be really close to it. So is there a button on

here or just turn it on? Yeah. Yeah, that's why Will was having the problems. Let's let's get back in there. So we had this 291 megs of data. I think that I can keep it in memory, which is really cool to me because then I can we can build this client. And my friend Ahmed here is like a student and he's studying these types of things. And we're like, oo, maybe we can work together to build a client that has this automatic mitigation. And now Ahmed is going to kind of So the screw this go ahead. It's really loud. So basically what we were thinking that instead of having just one server work around making the decisions

of what gets to connect to whom, we can just have a on every host in your network. There's a thing living in in your host like it's like a rootkit basically that does the same kind of data aggregation in the kernel and then basically make decisions. But then what you would need is that you would like to to get the scores from Badger into that client to make decisions if a certain IP address is okay to communicate with or not. And for this basically we we built Cobra and Cobra is just a a kernel that lives in the kernel and it could work with Windows 7 and we're looking for more versions in other operating

systems. And the thing it does is it collects network traffic and it correlates the network traffic with the processes that started it. And the reason we we care about that so oh so instead of that works that's better go ahead keep going. So basically instead of just blocking all flows coming out of an an a machine you can check what process is sending the traffic. So if it's a critical process that shouldn't be communicating with the outside, you block that that flow. Or if if the whole flow shouldn't be communicating to begin with, then you also block the flow. And basically, you can even stop processes if you don't like the processes or if you don't like

the actions they're doing on the network. And in order to do this, you have to generate the network process baseline and you have to communicate with the server to get all the updates. And so basically this is how it would look like. Uh you you have your process correlator and this is the piece of software you build and this piece of software basically looks at what processes are ex are are in the kernel running and what network traffic they're doing. It updates the baseline and basically starts block actions if needs to. And if we see any kind of traffic that we didn't expect, we communicate with this badger server asking if this is an okay IP address or not and update

the baseline and take action. So when I started this work, I haven't I I didn't do any I haven't it was my first time doing any kind of kernel programming or reverse engineering or anything. So it was pretty like a brand new work. It's like u so Ahmed's little project before this was we reverse engineered the modulation scheme for a certain vendor's meter right so after he got finished with that I'm like oh now start looking at kernels so I'm just going to explain some of the to to the depth we look at the kernel so this is just snippets of things we had to do and so when we built our thing instead of going the route of using

system API in order to get processes and network graphics and these things. We wanted to actually learn the internals of the kernel and get the data directly from memory instead of using the calls. And that way if somebody's hooking the calls, we're not dependent on them. We're we try to depend on the minimum possible. And that I mean it's it was 6 months of work just to get to learn all the everything from the beginning to even start a debugger was a hassle for me since it was this was my starting point basically of looking at things. So all I knew was MEIPS from undergrad courses. So the first snippet of code here, I'm trying to to iterate

the list of processes in the kernel and this is not groundbreaking. If anybody's done any kind of fruit kit programming, they would know it, but it was new for me. But the next stuff are more interesting. So in this in this case, you get one. So you use the uh the system call or the API call to get the current process and we you we basically well I basically grabbed that pointer and started looking I looked at the the structure of that it's called e-rocess and to get the structure you we basically I use the debugger and it gives you all the structure of the internally and I iterated the list and I grabbed the image name and it's at 2 E0

basically that's where it lives in memory with respect to that object and so that's the first way and then this line of code basically is iterating through it. So in order for me to grab the network traffic I could I used windows the window filtering platform but I wanted to also get more assurance that I'm getting the right piece of information. So I went ahead and started looking at the actual if this process has opened the socket or not because if the process has opened the socket and I see the same traffic from the windows filtering platform then I have better assurance that this traffic is coming from this process and not some from somewhere else and in order to do that I

had to reverse engineer the handle table in Windows and it is a so basically a handle so there are objects in Windows and each object when you request when the process wants to use it. It requests a handle and then the handle is just a an index to a handle table which belongs to every process and in order to get to the objects you have to basically index the table the right way. So that's how it is. And so and after I I looked up how the table looks like I found a function called handle lookup in the kernel. So I copied that assembly code and added to added it to my project and then I can just basically use that look

up handles based on the index and then the second step is after you get that actual pointer to the object we have to parse the object and this is where I found something surprising is that well the mouse doesn't work okay so basically here the object I had to do object minus one and it was kind of confusing why would you put a pointer with an extra bit in it and then decrement the B. So I was a on the video you're not going to be able to see it but we pointed at object minus one. Yeah. On the screen and those people in the back can't see anyway. So that's what they get for

sitting in the back. So I wanted to make sure that I have the right assumption there in terms of getting rid of that bit. So I went to the debugger and I looked at one of the handle one of the entries in the table and it had the the the address on top. And when I re when I looked at the actual memory, there was an extra bit at the end. So it was 6631. And my hypothesis is that I have to get rid of the one to be able to get access to the data. So I used the handle extension in Windows debugger. And if you can see here, they they don't have the one. So they actually got rid of the

one in order to get access to the object. But that wasn't I mean that was still confusing for me. So I had to make to be sure that that's what's going on. So I basically what I did is that I added a break point into one of the helper functions in the kernel and I stepped through it to find that the at the moment they get the pointer to that object do they decrement or not and I found out that they actually did decrement. So for some weird reason that's how they did it but I had to be sure and that's how I did it. So all in all, I'm able to get an object for a

socket and compare it to the traffic and make sure basically that a process that's generating traffic has a socket associated with the traffic to get higher assurance and then we can b we can correlate process traffic with network traffic and then we move into responses. So are there any reverse engineers in the audience that maybe know why minus one? No. So we're right they they learned something new today. So that was good. So in terms of responses once you see that new that weird kind of flow we would either want to block the process or block the connection or drop the packet. So sometimes you don't want to stop a critical process but you don't

want it to do that weird connection. Other times if it's a browser maybe you just kill the process and the connection. And all these can be done using normal uh IP filters and net filters and these things. And we have some ideas here on how do you stop a process? You either use the system API, you could remove it from the scheduleuler which is kind of cool because I mean it's still running but they it's not getting any time or you could crash the whole machine if you're that aggressive. And when Ahmed was doing the research he really got adept at crashing the machine. So yeah, that was my like 10th one or 50th one or something. I don't remember.

That's how you learn. Yep. Exactly. 20 to 30 blue screens per day. Um so and Yep. you know, we're um you know, we're uh you know, if you have any questions, you know, we kind of anticipated maybe somebody might have a question by this point, but um you know, everybody's all actually so so much process going on perspective. How are you guys sort of controlling that or thinking about that? Obviously you're injecting Chrome or whatever you would expect those right. So

there are so maybe you should paraphrase the question. Yeah. So he's asking if for example somebody is doing process injection into Chrome and Chrome is a process that does actually go to different places how do you make sure that you're do how do you basically work with this case right that does that capture it okay so we are trying to make sure that control the first of all control the process injection as much as possible so sometimes when somebody tries to fork you can see those fork events and you can block them. So you wouldn't allow Chrome to fork anything but itself for example and this is one way you can work with this case and the

other ways is that you keep a tight control of what IP addresses are allowed anyway. I mean these kinds of things are the things that you have to restrict the the actions in the machine and you you can't there's no silver bullet. It's just it goes back to your Yeah. Perfect versus good enough versus right now we have attackers running wild and as defenders we just have to politely say thank you sir. May I have another? So the whole idea about it is that maybe we can do some things to um maintain a better idea of connectivity because one of the things we really learned as we were building towards making this client and doing

this research on all of the things on the wire is how incredibly small in a modern computer you can get a very wide footprint of all the data that passes through your house. We have very varied users at my house between teenagers watching YouTube videos all day to stuff that I do. So there was all over the place and it's always interesting to see things that were in common, things that were not in common and then egregious users of bandwidth which ended up in videos of course um with one user particularly using a total of 85% of all the bandwidth that was at 400 gigs to u YouTube among other places. But the whole idea being we want to be

able to maintain that down to try and get inside the curve and reduce the amount of process and then um we're not quite ready to speak about the rest of the answer just yet. Y yeah um it's a work in progress. Uh so um and then this whole idea from a we wanted to move out from the kernel space into userland some decisions about whether or not um a connection is good or bad. We didn't want the the badger server to be able to send a secret packet or a well-formed packet into the the the Cobra and say, "Oh, by the way, I need you to shut these connections down." uh we want the client to make

that type of connection because the the the uh the device should know or have a really good idea about what is normal traffic and those things that are abnormal. And then from a an analyst perspective, you need to be able to make decisions about how aggressively you want to mitigate things that are out of the ordinary. There perhaps are servers on your network. I know on my day job I do SCA security stuff and I can tell you a very deterministic network would be a good candidate for this type of client. Um and um so in terms of the data exchange, the Cobra can calculate baseline data for destinations, right? and and the basically the

weadger and can ask for that data and Cobra can ask for the same kind of data back and yeah so basically it's like a cooperative of some sort and then um so like everybody else we're going to have a GitHub and there's like the the link of the GitHub for the project um at some point in the future there'll probably be us Cobra will be a part underneath when it gets released uh very soon. And um what's going to be there now if I can make it back to the internet safely in Las Vegas during this week without my MacBook being owned by firmware viruses and worms is tutorial for the Badger server. An overview of

the Badger Ontology. We really shot right through a lot of the CPTL stuff which was what we did at the black hat talk last year and Gabe Weaver uh one of the fellow researchers at the university is not really here to speak about his piece. She's really driving that project but building ontologies for a lot of things so we can provide uh cyber physical um crossover to where Badgers also wants to go out and I'm on another funded project at the university where um providing this state estimation into a cyber physical system and then getting operational data back from that system so that I see something's going on but I don't know necessarily whether or not

it's something important. or if it's operating out of its normal operating conditions because I'm a security guy. I'm not and so a way to bridge data in an operational world and then also being able to look at things inside the kernel to come up with the best decision about what we should do and then let the client make some of those lowhanging fruit type of decisions. Um, and um, we got a lot of work to do in the future and um, invariably this is always the one slide that you get to last. So maybe we wanted to keep the list short, but um, so we think it's kind of cool if we can ask the actual user on

the machine to verify things we're saying. So for even if we just want to verify that there's an actual user on the machine, we can we we hope eventually we can pop up questions, dim the screen, wait for a reaction. So stimulate the user to do something for us in order to resolve some uncertainties. And the second thing and the second thing is I kind of like I wore the bides shirt from Berlin so I could remember as well. I saw FX do a talk on his his um he has this idea about low or zero configuration identification. Uh he calls it trinity. Um I'm working with him I hope on uh integrating possibly some of this

technology where we want to be able to set up things in a network so that we can have uh a very low or zero configuration way of identifying things so that I know you're the client. I think you are and you know I'm the badger server. You think I am because one of the things we really struggle with is this idea of trust when we're bouncing information back and forth about state. Uh and we're both making decisions on that information. Knowing that you are who you say you are is going to be very important to us when we want to scale this. If I want to do this for a million clients or something like

that, I want to be able to have something that's low overhead because those million clients may not necessarily be very processor intensive and um yeah. Yeah, perhaps we we talked a little bit about that earlier. So the the the person asked a question about uh using this authentication to prevent a DOSS condition because I can impersonate the Badger server and say this guy's bad. You don't really want to go there anymore. He's been compromised. And then uh I get a score, oh maybe I shouldn't go there anymore. And um and that's exactly right. And then you know this is a work in progress. I wouldn't put this on the Defcon network this weekend obviously. Uh and um it's part of the

reason why it's not released yet. Uh but it will be released soon. We didn't make the deadline for the conference and we apologize for that. Yeah. What is Cobra pushing up to the server? Cobra will send um state data back up to the server. Application name. Um basically um at this time we're get we're just getting u red green yellow so we have a destination and uh through either internal calculation the client makes a determination about this is something that's out of the ordinary I think this is marginal and then through aggregation from multiple clients you make a the badger makes a decision about oo a lot of my guys are seeing this and then the badger is going to be able to

say hey we're seeing this in other places. Are you seeing this too? Because we think this might be something that's emerging. This idea about trying to detect things that are moving horizontally in your network. Perhaps they're not trying to communicate yet. But one of the things I found that was interesting in I have a wired land and a wireless land. And being able to look at the intercommunication on the land was very interesting because not all mach whenever you have a machine that moves horizontally in your network like I could get in and run end mapap and see that how do you programmatically or algorithmically say this guy might be doing this kind of thing moving

horizontally in the network and trying to find those things that maybe not don't get into the perimeter and all the nice gizmos that you have protecting your wire. So, great question. I'm glad we checked those two guys hands because, you know, they're like the guys coming in after the medical device talk or coming in like, "Oh, man. What's going on?" Looks like these guys might be Oh, yeah. Well, I think we might actually be done uh enough to answer any questions or see you after the talk um to have a wider conversation about things. And then that's the GitHub where you can look us up. Look up and then that's my Twitter handle. So, um, if you want to contact

us or meet us at the conference, we we'll be here all week. Thank you. All right. Thank you, gentlemen. Does anybody else have some further questions? We do have some time left. These guys in the back are like, "What the [ __ ] just happened? You guys are finished." Sorry, man. They told us to do 40 minutes. Shouldn't have gone to the medical device talk. Now's the time to rip my shirt off and make my 100. Yeah. Yeah. Yeah. Go. Go. What? We'll we'll edit out and post.