Meatspace Indicators and Incident Response

response and security architecture. I formerly was a civilian contractor at Coast Guard CERT and now I work at the Software Engineering Institute CERT CC, the coordination center. And I have a bunch of kids, a wife and a bunch of dogs and stuff like that. So what's this talk about really? I think one thing is I believe these are my two main points. Maybe at the end people can tell me if I was right. The first is a lot of times people, security practitioners, get caught up in sort of the hustle and bustle of doing security and forget the bottom line. You're doing it for a business. In this case, I'll probably use business to really mean government, military,

private company, nonprofit, whatever, anywhere you do security and can get paid to do it, whereas doing it as opposed to doing it as a hobby. And sometimes people forget the actual goals of what they're doing. And also another one is that technical and non-technical indicators, meaning things in meat space, are important.

So I'm just going to jump right in. I forgot to check this slide. It did. Oh, it shows up fine. So this is an indicator that set off an incident response. Anyone want to guess what it indicates? That's a good one. He said right here, somebody is outside smoking. So that's not a bad guess. Like I said, I do have kids. So that's a pretty good guess. Any others?

Lost my keys? No, that wasn't it. I'll give you a little more context and then people will probably get it. So when I found that, I took it out of the door first thing. And when the screen door closed the rest of the way, that's our side door, ground floor, I realized that screen door's really creaky. It's really loud. And so when I took it out, I immediately thought, well, I have a teenager in high school. So that's bad news.

And then some more context, we had recently remodeled the house and any real estate agent I think who knows what they're doing will tell you don't do this, don't put a teenager in the ground floor bedroom, but we did after we remodeled because we had let the other kids all upstairs share in a bedroom spread out upstairs. So one thing with that, we definitely did not weigh the risk properly.

And some more context, I had an inexpensive home lab running a bunch of things like full packet capture, session monitoring, snort, just all that kind of stuff that some of us like to do in our spare time at home, even though we also do it at work. So my immediate response when I found that was to look for more indicators. So I went in my daughter's room. And like I said, she was in high school. And I found two more good meat space indicators. One was her screen was from her window also on the ground floor. It was under the bed. And the other one was the window was unlocked. So I knew right then, obviously, if any of you who either have teenagers or

maybe once were teenagers, you know, someone was either sneaking in or sneaking out. So that's what the indicators told me. So the rest of the time here, I'm going to walk through and show you how I investigated it. So the first thing I did after finding the additional indicators, Our daughter wasn't home. She was at like a pizza party or something for a friend. My wife went to pick her up from the party. It was like, your party's over. Come home. We confronted her. And at first she said she didn't sneak out. Her denial was pretty weak. It was really bad. Like, I don't care how bad a parent anyone was, they would not have believed her. So then she admitted to sneaking out. and

she told a story like, oh, I met my friend out front, she was having problems, she came over, we just sat out front on the curb talking, blah, blah, blah. I was like, yeah, that's still suspicious. Don't buy it for one second. So just stepping back for a bit, I hope all through this to draw some parallels between this and doing incident response for business. So I want to point out there are real there are meatspace indicators you'll see in business I think a lot of time maybe only some of the time depending what kind of business you're in who you're doing security for but just some common examples I it's not really hard and fast but I sort of think of them two

ways technical indicators found by users example that you get phone calls saying we got phishing email or something's funny is going on with this website they may describe what essentially when you translate it from user speak to security speak, it'll be cross-site scripting. And then non-technical indicators like for instance,

country A sees that country B has a jet that looks exactly like theirs even though they've never sold any jets to them or any plans or something that might tell you, hey, someone stole our plans for this jet. Or same thing with a company. You're about to release a product, and suddenly you see someone has a competing product that's basically exactly the same. Maybe they just had a similar idea and came out with a product. Maybe not. Maybe you need to look at all your monitoring and figure out, did someone actually get in and steal our plans, our engineering plans, what have you? Other obvious ones, I mean, say you're a small business, front door's broken down. That's

a really obvious one, but it's still an indicator something happened. So back to my incident response. Well before this happened, I'm not sure if she was setting up for the sneaking out or not, but once upon a time before this, so sort of a flashback within a flashback, she said, you can read my email, can't you? And I frankly was a little bit insulted. I said, of course I can. Why wouldn't I be able to read your email? You know what I do for a living. So, yeah. On the other hand, I had always told her, look, I'll respect your privacy. I'm not going to look at your emails, IMs, whatever, unless you give me a reason. I'm not, you know, I...

I won't do that. So when this happened, I told her, you know what? You've broken our terms of service. And I'm going to look at everything now. So actually, when I did a dry run of this presentation for my family, because it's very pertinent for them, I wanted to make sure they were good with me sharing all this information. And my younger daughter, who is now a freshman in high school, saw this, which I just wrote it up as a joke. Obviously, I never did a real terms of service with them. I did warn her, say I would respect her privacy and not look at her comms unless she gave me a reason. But my younger daughter saw this while I'm doing a dry run in the

presentation. And she's like this. I'm like, yes. Miss, what is your question? And she says, do we still have that?

So I'm a little worried now about that. Just one serious thought on the terms of service. Obviously, in businesses, you're going to need one and a real one. Even at home with families, It's probably not a bad idea to make clear to your kids or whoever what you will look at and what you won't. I mean, previous to this, I had only looked at actual interesting activity, like if my kids went to a malicious website or things like that. So to kick off the actual IR I did with tools, the main tools I used were Squeal. A lot of people are probably familiar with it. Now part of Security Onion has been for a while, so basically sessions, or some people call by flow,

packet capture, IDS. The IDS didn't really play a part in this. And then I also used some command line tools and shell scripts, which I'll get into in just a few minutes. Now one thing to note about is I could have also used IDS, like for instance, if my kids used IM or things like that, you could, you know, with policy alerts, you could have alerts just letting them know when you use, when they were letting you know when they were using IM or things like that, so you could look at it if you, and my opinion as a parent is you should make it clear with your kids if you're gonna look at something, tell them, warn them about it, be sneaky about it. I

think it's better to just tell them, look, I'm gonna be this way as a parent and try to stick to it as much as possible. So the initial investigation, there are some definite parallels for this and business. So you need context. I knew she's in high school, she's sneaking out, she's probably sneaking out on weekend nights Friday night or Saturday night, which means she gets home Friday, she IMs with her friends to plan sneaking out, and then the next morning maybe they're IMing again to talk about how awesome it was. Oh, they didn't catch us, it was so fun, that party ruled, all that kind of thing. So I would just do a query. Now this isn't an actual query.

Unfortunately I've lost the hard drive that had all these packet captures from when it happened. I just mocked this one up. This is just like on a query on Friday, October 2nd for IRC actually.

And so having that context in a business is just important to know what you're looking for. It helps a lot if you know what's normal on the network, what people should and should not be doing, hours you know if you have regular business hours versus 24 by 7 all those things matter when you're trying to do incident response so this is then just an example and squeal you find some I so I find the irc in this case the example for her she was actually using I am's and look at it this is just a screenshot of me doing send in a help to Nick serve on an irc network but in her case I

found some actual IMs showing not only was she sneaking out, but she was doing many things you don't want your child doing in high school, which probably most of us also did. I would say if you could, maybe I should have put a Venn diagram up, like all the things teenagers do when they sneak out, and the things my teenager did was definitely all inside there, but it wasn't the whole thing. So she was doing some, some of the things but not all the things. So then after that we obviously talked to her, she's punished, all this sort of thing. And at that point it was a decision point for me because I had to decide,

well, so now what do I do? I want to investigate and do more, but if I manually look at all these transcripts and squeal or what have you, it's going to take a long time and frankly, just by talking to her and scaring the crap out of her that I know everything, I had mitigated any further incidents, in the immediate future at least. I knew that just by talking to her, she wasn't gonna sneak out at least for the next few weeks, probably a lot longer, but I knew then I could sort of relax and do a full investigation. And that's another context thing. If you're doing response for a business, you have to know whether or not you need to take immediate action, you have

time to respond, how deeply you really need to get into things to know what's going on. It's very important not to work every incident the same way. You have to use experience and knowledge of the business context to figure it out. So I made an immediate adjustment. I said, you know what, this is dumb. Let me just whip up a really ugly shell script to take

DougSong's message snarf and cycle through every packet capture I have. This one is just an example. I think this will work on like the default security onion directory layout for packet captures. The real one I used also had some other things like I had to do some UCEDD to replace some stuff for formatting. All I did was whip up the script, dump everything into an HTML file, leave it running overnight and that way I could review it at my leisure and figure out the full extent of how long she had been doing it, what was going on. And I think this is important also for business. You have to be somewhat flexible and understand maybe when your tools aren't good enough or are not efficient enough.

Like I wasn't going to waste hours and hours manually doing queries and pulling out all this stuff. I said, well, it's no rush. So I can write the script, leave it running, it's done whenever it's done, and I'll look at it later. And the other thing to note, I'm an analyst, I'm not a developer, so any scripts I write are gonna be ugly. All they have to do is work. Hopefully in business, you have someone you can fall back on if you have something like this that actually works, you can pass it off to a real developer to clean it up, make it available to all the analysts, and things like that in my house. I guess one of my sons started learning Python two years ago,

so he would be my dedicated developer, I guess.

That's right. Yes. Well, I'll get to some of that in lessons learned, hopefully.

So the results from this now So a lot of this I had to recreate after, but this, I still have the actual chat logs HTML file. When I do an LS, it's 2.1 megs, and word count, it's got 15,000 lines, more than 15,000 lines. So it was a lot of chats I had to go through. The other thing to note, though, is it showed one problem with my script. I actually did not differentiate between anyone's IMs in the household. So all my IRC chat is in there too, and anyone else who's IMing. So obviously, when you pass something like that to a developer, that's something they would want to fix. So you could specify, like say, port, IP, what have

you. So that then, if you want to run something, it's no longer all screwed up like this, basically. So next, I'll get to a snippet of the actual IMs. And I'll just let everybody read that for a minute. So the blue is my daughter, and the other color is,

I like to call him accomplice number one.

So I don't know if you can tell, I've told parts of this story before. And whenever I, one of my coworkers always says, whenever he's reminded of this, that last line is the best Father's Day gift I could ever hope for.

So then there were a, couple more, you know, there was much more than this obviously, but then the last few lines were literally these. I mean, I copied and pasted this. I did not edit it at all.

So, I think this drives home a good point too, which is that sometimes attackers, even when they know you're watching them or know what's going on, they do stuff like this. They will... I mean, she knew I was reading her IMs at this point because we had talked to her several times over the course of days, yet she still was talking to her friend about what they had done and putting up something like this, which now is hilarious, but back then I was pretty mad.

So what did this teach me? One thing, businesses too, you want to learn what attackers, adversaries, whatever you want to call them are doing. So just as an example, I sort of learned her TTP for this. So that then if she wanted to do it in the future, if I had wanted to, I could make defensive adjustments to automate catching it. So basically, she would coordinate everything and the techniques would change. B, she'd use IM in the exact procedures. She'd do something like, you know, say they're going to sneak out one night and after our lights are out for half an hour, let's go out. I actually, she went through the window. It would have been a better story if,

yeah, the tennis ball was another time and she just forgot to remove it. So, You know, also another procedure would be put the tennis ball in there to keep the screen door from creaking and sneak out. So by learning this, just like learning TTPs in sort of the real world where you're doing this for money, you can figure out how to help prevent it from happening or at least detect it earlier and not go through all this manual investigation, hopefully, to save some time.

So then I had to think... of defensive adjustments and you know this is just kinda spitballing like some may not even actually work like broscript to detect when her phone is can I have a mail server in my basement so to detect when her phone checks in with the mail server from during you know overnight hours from outside or home Wi-Fi except actually that wouldn't work cuz the mail connection is encrypted, so I couldn't actually do that. But sometimes you just got to brainstorm these things. You could do a script to look at the mail logs, so I could also detect the same thing. It would tell me at least her phone was out of the house overnight. And for that, you could even use something like

OS Sec. Another one with a house alarm, you could see if it was disabled at night, so she could sneak out. And the last one, and it's highlighted because that's the one I actually did, and we'll sort of get into why, that gets back to one of my earlier points, just talk to her and tell her to cut it out. Because, I mean, like to keep her from sneaking out, I could also board her up in the bedroom and only let her out when, you know, like lock her in her bedroom. But that's not what parenting is about, just like, when you do certain things for business, you have to understand what your business is about. Some of the defensive adjustments, if they break your business, like my

business is raising my daughter as a father, that's my business. So I would be breaking the business if I just physically restrained her from sneaking out. I'm trying to teach her to figure out things on her own.

So a little bit about attacker countermeasures. It's interesting when I was thinking of this, One thing she definitely learned was use her phone for texting from that point forward because as long as she's not connected to our Wi-Fi, it's much harder to monitor. Now, these other things aren't even attacker countermeasures, really. They're just things that have happened over the years. So sort of the same thing that makes us... Let me back up.

You know, she was actually using a dedicated IAM application at this time. Now nobody does that. Everybody goes over web ports, right? Like you have Facebook IAM, whatever. It's all web applications. And then of course, particularly recently, everybody's moving to encryption. So what makes our communication safer also makes attackers' communication safer. And that definitely applies to business. It can make it harder. to monitor for this. I do have to plug flow a little, because where I work, we do write a lot of flow monitoring tools, our developers, which are pretty cool. But it's definitely harder than being able to just pull up a packet capture and look at the plain text. So at the same time everything's

collapsing to the web and encryption, everybody's using all these different apps and things to communicate. So you don't actually necessarily know what to look for when you're in there. You don't know what their communications methods are. Like with MessageSnarf, it was easy The IM she was using was just basically HTML anyway, so I could just dump it to a file with a few regular expressions to tweak the formatting, and it was easy. Now, with all this stuff, I mean, I think I've listed up there Snapchat, Facebook, Instagram, Twitter, WhatsApp, Skype, Kick, Whisper, whatever. You don't know what people will be using to communicate, and I think that doesn't just apply... Yeah, right. But you don't know which

one they're going to use for what is the big problem. Yeah, exactly. Yes, in real world, in business, like malware or actual individual actors will switch things up a lot to make it harder. So, you know, given that, what are some other things I could do or you could actually do in business with problems like that? There are a lot. Like I said, profiling, sort of knowing your network and the systems on it, understand the business environment, like knowing when... People should be

at work when they aren't. Like for my business of being a parent, it's knowing she's on the ground floor, that was stupid of me, her normal hours, and things like that.

Tool and Architecture improvements you have to think about essentially what kind of architecture could I have used to make the job easier and make it easier to monitor while at the same time there are obviously in business or many privacy implications so you have to be careful about all those things it's a you have to weigh it carefully so the summary of that incident response, I used that retrospective analysis to confirm what happened with pretty high confidence. I'm pretty satisfied that I know what happened. I don't think all those IMs talking about her and going out to parties and stuff was just to fool me because she was doing something worse, you know? So in business, it's a lot harder. Sometimes you never really know what happened.

You think you know. but not necessarily. Sometimes you're very sure, sometimes you're not. Most of the time it's somewhere in between and you also have to weigh how much is your time worth to get a higher degree of confidence than you already have. Because sometimes it's not worth the time, sometimes it is. I made some adjustments to prevent similar incidents. The main one was just saying, don't do this crap. which and punishing her you know she was cut off from basically everything for quite a while and that worked pretty well the tech not technological adult adjustments I wasn't really down with that I'm as I said I don't I'm not going to be there her whole life to keep her

from doing stupid stuff so I want her to learn to some degree not to do it on her own basically Now I have this script. If you're in a real security environment, hopefully when you do stuff like this, it's worth keeping around. Like I said, you have the developers to help keep it functioning, tweak it, make it better. I mean, in the real world, I definitely, in past incident response environments, many times an analyst script to try to make something faster ended up being something pretty permanent that people would use every once in a while to help automate. The script then can also be easily modified to do other things. Basically, any time I want to go through a packet capture, if I have another tool that'll look at

a packet capture, I can just stick it, plug it into the script. And the last one, which is really important for me, my first daughter got it out of her system. I have three more to go. And unfortunately, they have learned a lot through my constant retelling of this story. It's actually what made me decide to do this, submit this for B-Sides. One, it's a little off the beaten path. You won't... I mean, a talk like this, you know, it's funny, it's got some neat little things, but it's not going to be a bigger conference. It doesn't need to be. The other thing is it's always a favorite over a beer at the bar after a conference. Like when I go to... whatever,

ShmooCon, whatever other conferences, when I talk to people sitting there afterwards, I tell the short version, just in case you were wondering. I don't make people sit there for 50 minutes. But it's always really popular, so I decided to figure out what are the actual parallels between doing this at home and doing it for business. So now I'm actually going a little faster than I expected. I still have a few more slides though. Oh, I know

one more thing to mention here in the results is if you did want to do something besides using technical, I mean if you wanted to use technical means to figure out sort of how to stop this type of thing. Maybe there's an application for business. Like I've heard some businesses on their phones or even some parents will prevent their kids' phones from doing, like, say, image uploads. So you can't post questionable content like sexting, whatever, things like that.

So lessons learned. Part one.

So hopefully I showed that sometimes indicators really are in meat space. Obviously at home they are. Like for example, my dogs, all the indicators of problems with them are meat space. I don't get any technical warnings with my dogs. But I've had many experiences, like I mentioned before, where phone calls from users or people in the field are actually what kicks off an incident response. I think it's important to remember that. A lot of people like to badmouth users or what have you, which I totally agree with, by the way. But sometimes they can be helpful. And of course, this is not really that unusual to think offspring are insiders. My kids are the ultimate insiders. They're more

trusted than outsiders. And most of the technical monitoring I do at home is of no help to catch them sort of threatening household security, I guess. Sure, they browse to, like my sons are big gamers, and they browse to some sites, forums, and stuff that definitely will sometimes be serving malware. But that's pretty tame compared to most things people actually care about. And I will look at that sort of thing, but really for What I'm trying to do as a parent, it's all going to be meatspace indicators. So one more slide on lessons learned. Be flexible. That's just going back to the script, but also just in your thinking generally. You want to be ready to change the way you're doing investigation and response. Particularly,

sometimes the upfront cost is a lot, but in the long run it makes things a lot easier. I think that's one definite problem in business is we get so caught up putting out fires that we don't get allowed, sometimes this is actually a management problem more than the responders, but we don't get allowed the time to really do that development we need to mature our incident response. The other thing is understanding the media and long-term impact. This goes back to my point of once I had talked to my daughter, There was not really any more need to do more monitoring, increase my monitoring, or change anything else because I knew as a parent she was scared to death of getting caught again. She knew

she was in so much trouble. She was immediately on her best behavior for a while, as I think we all would have been at that age. So that's just a context thing. Know what an incident means in the immediate future and the longer term impact because if you don't, you will end up wasting time doing things that may not matter actually.

Okay, so now sort of to circle back around to my original points. Why does this matter? So like I said, I'm not sure I showed it perfectly well here. I gave a few examples, but security exists to protect businesses. It's a means to an end. So I know

the keynote this morning, he had some good points about sort of, you know, if you ask a developer, there'll be no security. If you ask a security person, you won't be able to do anything. And I think that's really important because a lot of times in security, we can lose sight of that fact. I know I have

In the past, I've had that happen where I suddenly realized, wait a minute, as this incident response, I'm asking the business to do something that actually breaks more than just say, leave in the box own for a little while or something like that. Just as an example, I have a few examples of this actually. In the past, when VPNing into a business I worked for, the host-based security policy would not let me VPN from Wi-Fi. So why do you give someone a laptop if you want to be so secure that they can't VPN from a Wi-Fi network. I was working from home, in fact. So that's a case of security breaking the business, essentially, because I'm supposed to be able to VPN in. If you

don't want me on Wi-Fi with a business laptop and connecting to the network, don't give me a laptop to connect to the network from somewhere else, basically. Another example...

Sort of this one is made up. I'm not sure how accurate it could be but if say you have a business that has a factory and they have some systems there that are owned in some way and Shutting them down would impact the factory actually working well if it costs More to shut down the whole plant then just leave them own till off hours and fix it then then maybe you should just leave them own for a little while and fix it later I mean say you know a You could be spending hundreds of thousands of dollars if you have to shut down some systems, whereas if they're owned in some way, depending how they're owned, it may not actually be worth that

much money to do an immediate response. You may have to do things, I mean, you have to think about other things like resiliency, just maybe you could do some things without actually completely taking the box out of commission that we'll just have to do and then at some point when it's better for the business is when you actually clean it up like when the you know if it's a factory say when the factory is not rolling like overnight or something I guess my point is with that don't just follow a script to pull the plug you have to consider what it means when you pull the plug you have to Think about what else you can do and how it'll be impact the business both leaving it plugged

in and unplugging it. And the second one is sort of another example of that. My goals for my daughter as I said are for her to grow up be happy fulfilled person basically and I want her to be able to navigate the world without me. I'm not going to be around forever. So if I just use technical means It's sort of a parallel. She wouldn't learn it herself. Sure, she got punished, but at the same time, I want her to learn how to make good decisions in the future, basically.

It's funny. Yeah, that's the last bullet of why this matters. So the other one, going back to

meat space indicators, sort of combining technical and non-technical stuff, you really, I think, need to understand your business's objectives. I mean, a lot of businesses call it a mission statement. It might be a combination of a mission statement and many other things. But if you don't understand that, you aren't going to be able to properly understand incidents, you aren't going to be able to properly respond to them, at least not in every case. You need to really understand all that stuff. Like when you do something, when you make people pull boxes offline, when you go to someone and say, oh my god, we're totally owned and this is how, you have to understand what it means for the business. I think, to really

be doing a good job at incident response, because they're gonna ask you that too sometimes. You can't just say, oh, you know, our database, they popped it with SQL injection, and we have to do this, this, and this to clean up, and when they say, oh, well, what does that mean for our production environment, you can't just say, oh, I don't know, ask the server administrators, you have to deal with them and figure that out. And then the other thing is there are many ways for incident response should be tied into things like disaster recovery. I mean, flood, fire, earthquakes, physical attacks, what have you, depending on your business, even just the sprinklers going off accidentally is another

aspect of tying the technical with the meat space. So that's basically the end. Future related work. I have two sons. They play a lot of Steam and Xbox. I do have a story that is a bar story. It's not a story for here because it's not very technical, but it's pretty funny. The one thing with this, I will say they are Just as policy in our household, because they do a lot better in school, they're barred from playing video games during the week. So Friday after they get home, they're allowed to play. A lot of this stuff has policy alerts and snort that make it really easy for me to see if they were doing anything. I think it's Steam actually not only Do

you see IRC? Because some of the in-game chats, I don't know if it's, I can't remember if it's a specific game or Steam in general. In-game chats are on IRC port. It's detected as IRC. And downloads are torrents. So if you have torrent alerts on, you'll see them.

So a couple more slides, shameless plugs. My company is hiring CERT, also known as the CERT Coordination Center. If anyone's interested, we have offices in Pittsburgh and DC. We also have a network security conference called FlowCon, which originally was all about using flow for monitoring, but is now more of a general network security conference this year. It's in Portland, Oregon. If you don't like Portland, it's in a new spot every year. So finally, I'd like to thank my family. Obviously, this would not be possible without them. And are there any questions?

Yeah, I mean, if you have comments too, that's fine. It doesn't have to be a question. OK, thank you very much. I guess you'll get out a little early.