BSides Las Vegas 2019 Day Two - IATC

I see everyone made it to day two. Welcome back. This is the ground floor session for day two of B-Sides Las Vegas. A few announcements before we get started. You guys are old hat with this, but we say it. We say them every time anyway. First of all, streaming has started, so if you could please remember to turn off your cell phones. And if you have questions at the end of the talk, please use this microphone so that everyone on the internet can hear you. We'd like to thank our sponsors, especially our Inner Circle sponsors, Critical Stack and Valinmail, and our stellar sponsors, Amazon, Robinhood, and Secure Code Wire. It's their support and all of the sponsors that make this event possible. So this

talk is CTS for Fun and Profit. Please welcome David Tomaszewski. Thank you.

So I want to thank everyone that made it to the first talk of the day. I know that can be a little bit of a challenge, especially after maybe getting back into the Las Vegas party scene for the first time in a year. So a little obligatory disclaimer, the views and positions in this presentation are my own and don't necessarily reflect those of any of my employers, past, present, or future. The obligatory bio, since this talks about CTF, I've been playing CTF for at least 10 years. I don't even want to try to count how many that I've actually played in, but for the last several years I've been on the staff of two what

I consider significant CTF competitions. One of them being at B-Sides San Francisco and the other one actually being the Pros vs. Joes CTF here at B-Sides Las Vegas, which is literally in the room next door. So I've seen both sides of the table and I wanted to share some of my views from that point. In terms of employment, I'm currently a security engineer at Google. I work on our red team doing offensive security exercises and adversaries. I'm going to document this for questions. Oh, sure. Thank you. You ready? I'm ready. You ready? Are you all ready? Who out there is hung over this morning? You don't count. Excellent. So you are the people that are actually paying

attention. Okay. Well, good morning everyone. This is B-Sides Las Vegas, I am the Cavalry. Good morning everyone. Here, hang on. Good morning everyone. This is B-Sides Las Vegas, I am the Cavalry. This talk is how to treat your hacker and responsible vulnerability disclosure. Oh, very good. Yeah. And it's by the award-winning. The award-winning. Monta Elkins. Monta Elkins. I also occasionally blog and I sometimes post things on Twitter, although not so much these days. And of course the obligatory outline. First I want to give everyone a quick primer about CTF. Maybe you've never played, maybe you've never had the opportunity before. But then I want to get into really the core of this talk which is talking about the skills that you

use and that you learn in playing capture the flag competitions. It's really interesting to study which of those skills actually apply to a career in security and to being a security practitioner versus which skills are more just about gaming and having a good time. Certainly they're both value in both of those but a lot of times people are looking at CTFs as a way to build their skill set for their job and so then I'll talk about how players can improve the overlap and get more out of CTFs but I'll also talk about how organizers can improve CTFs and make them push more towards the educational side for the players participating in that CTF So what are CTFs? There's a bunch of different styles of CTF. Generally CTF capture

the flag. I imagine everyone has heard the term. There's a few different ways that they can be run. One of the most common styles, probably because it's the easiest to set up, is what is called a Jeopardy style CTF. And in a Jeopardy style CTF... We'd really like to thank our sponsors here at Bayside Las Vegas, specifically our Inner Circle sponsors, Critical Stack and Valley Mail. Some of our stellar sponsors, there's just so many, Amazon, Blackberry, and the NSA to name a few. So thank you, NSA and Amazon and Blackberry. Your support is really appreciated. Our talks are going to be streamed live, and as a courtesy to everybody, please silence your cell phones now. So go ahead and set them to silent if you haven't

already. If you have any questions, I'll run over with an audience microphone, which I can't find right now, but we might have him go ahead and, Monta, go ahead and repeat the question if not. So just to make sure everybody on YouTube can hear you. That's it. I'm going to read a quick bio of Monta now. Monta. Monta. Monta. Monta. Long A. Long A. Yeah. Or not. It just reloaded. Of course. It's okay. Okay. Nobody's watching. Nobody's watching. As the name implies, players get to pick and choose from a panel of problems that they want to attempt to solve next. So it's not linear. Usually the challenges may not even be related to each other. They're each standalone. So you can

look at it and you can go, "Hmm, I think I'm going to play web today," and just go straight for web challenges. and ignore everything else. And for the most part those allow you to solve anything that you want in any order. Again, just like the TV show Jeopardy, right? Alex, I'll take web for 500 please. On the other end of the spectrum you have what are called attack defense CTFs. And in those cases you have actual networks that players involved in the CTF are either attacking or defending as the case may be or sometimes both. And they're running networked services and often you have to find vulnerabilities in those networked services and exploit those vulnerabilities in the networked services. And sometimes you even have to -- are responsible

for patching your own service

So Monte Elkins is the hacker in chief of FoxGuard Solutions, which is an ICS patch information provider, a security researcher and consultant, and U.S. patent grantee. He is considered by many of his friends to be the Chuck Norris of the ICS cybersecurity industry. Somebody likes it. Monte has been a speaker at more security conferences than his enormous ego can remember, including DEF CON, B-Sides, CS3. Skip that one. GE Digital Energy, ICS JWC, Toseba ICS. It's so many. Yeah. He's also at the SANS ICS summit and was named Cybersecurity Professional of the Year by EnergySec. In his spare time, Monte creates a totally safe for work Coke for Strippers electronic... Coke and Strippers. Coke and Strippers electronic projects YouTube channel. Yeah.

Yeah, it's one you have to see. Yeah, all right. Thank you. Yeah, yeah, though. It's token strippers You can search for it. I wouldn't do it at work, but token strippers Monte if it says for death Preventing attacks on your system and then finally you have there there's some miscellaneous ones that Don't really fit into the other categories right CCDC for anyone who? had the opportunity to do that in college or may still be in colleges the collegiate cyber defense competition and CCDC is essentially a defense only CTF. They bring in professionals to attack the network and the college students are responsible for defending the network. So it puts very much an emphasis on blue team skills, on forensics, on network traffic analysis, on firewall management, on

host management, and all the other aspects that come in the day to day operation of CCDC. information security practice in a network. There's also a few out there. Afterwards, I have, that was so good, I have a thousand in cash for you featuring Nikola Tesla right there. All right, welcome, I'm glad you all are awake this morning. We're gonna talk about how to treat your hacker, in quotes, and responsible vulnerability disclosure. So that starts something like this. Hey, somebody just called you up, your company up, and says, "We, there's some..." CTFs, there's one pretty well known called Pwn Adventure 3, which is actually you play through like a video game client style thing, and you actually go through a

storyline with a little bit of a plot and everything. There's also one brought by SANS, Counter Hack Every Year, called Holiday Hack Challenge, that's online, free to play. and Holiday Hack Challenge, again, has a storyline and a game client and everything like that. So there's some different ways that you can approach these things. So it kind of is a spectrum though when you think about it, right? Like trying to categorize into precise categories is actually fairly difficult. There's an entire spectrum of the challenges here from the realistic to the contrived. So on one end you have real services, a business-like environment, probably using real world vulnerabilities, CVEs seen in the wild, everything like that. And at the other end of the spectrum you have really contrived CTF such

as fictional architectures or services that serve no purpose except to have built some of these challenges myself. I've built challenges whose entire service is you send me shellcode and I'll run it for you, right? There's that's a pretty rare occurrence in the real world. I mean, I know some of us may feel like some vendor services are designed to just run shellcode, but that's not usually the reason for their existence. So in terms of contrived CTFs, right, DEF CON CTF is probably the best known CTF in the world, right? It's top tier, the players who are in it are absolutely wonderful, but it's a contrived set of challenges. It's a game, it's for the purpose of finding out who's best at playing CTF, not necessarily who's best at

being an information security practitioner. And if you ask the organizers, they're totally upfront about it, right? It's not intended to be a test of practitioner skills. A couple years ago, they in fact invented their own architecture for DEF CON CTF and handed players the ISA, the Instruction Set Architecture Manual, 24 or 48 hours before the beginning of the game. And this architecture was middle Indian. So the order in which the bytes were being considered was from middle first and it used nine bit bytes and a number of other weird architectural features. And that's really cool to see who can wrap their head around a new architecture and wrap their head around porting their existing skills and their existing toolkit to this new architecture. The teams were building their own

custom disassemblers, their own custom decompilers, their own custom exploit chains, ROP chains, everything had to be built from scratch for it. But at the same time, having to deal with a middle Indian architecture is not something that security practitioners find themselves needing to do because as far as I know, this is the first time anyone has ever built a middle Indian architecture anywhere. I do have to give mad props to the Defcon CTF organizers though because I guess they had to build their own virtualization system to run this entire thing and design a functional CPU architecture. So I have to believe that years of work went into that. I also give DEF CON a lot

of credit. This visualization is actually a visualization that they use during DEF CON CTF these days. It looks way more interesting than actually watching a CTF. If you've ever stopped by any of the CTF areas at a conference or something like that, you know it's just a bunch of people hunched over their laptops staring at a screen. It is not what movies are made of, but DEF CON CTF has managed to add a little visual interest for the audience. Pros versus Joes CTF, as I mentioned, that's a CTF located here. I wanted to compare and contrast that to DEF CON CTF, especially because they run in the same week. They're only a couple of days

apart. It's defense focused but not exclusively defense. So our teams here, Joe's teams start off as blue teams and for the first day their goal is to defend their networks. And there's four teams, they each have their own network and they each have to defend it. And then there's a red team of pros attacking that network for the players to defend. However, today, the second day of the CTF, we let the teams get a taste of blood themselves, and they get to go purple and attack each other's networks. Some people think that pros versus joes sounds like it's just going to be a bloodbath, but it's not about, from a scoring perspective, it's not about are you beating the pros or are you not, it's about which of the

teams is doing the best at defending their network and at attacking the other networks. So it's not about out-swimming the shark, it's about out-swimming the other guy. It uses entirely a real-world software and environment. It's just maybe like a corporate environment where that software hasn't been patched in a while or is configured poorly. I'll go ahead and tell you the root password for many of the machines is ABCD1234. Hopefully the teams know to change that pretty early on, like rotating credentials is an important step. If they didn't and any of you guys know anyone else out there, you may want to tell them that they should consider changing their passwords. And so this actually, real services, like I said, FTP, HTTP, SSH, real world services that

are actually used, Samba, SMB on Windows, Windows domains, real world, very real world. I also want to touch base on war games. War games are sets of challenges to be solved. They're not a real-time CTF. They are more of a do it at your own pace kind of situation so they teach a lot of the same skills and can build a lot of the same skills but you don't have to dedicate an entire weekend to it right most ctfs tend to run for like 48 hours they tend to be on a weekend and i mean some of us we do like sitting there staring at our screen for 48 hours straight but i understand that that's not everyone's cup of tea so war games let you come in sit

down for an hour two hours give it a try uh and then you know, put it off till next weekend. So now getting to the educational value of CTFs, right? How can you best use CTFs to build your skill set and really grow your career path through these games? Well, can you learn from CTFs, right? Like just as a general question, is there a learning opportunity here? Well, the short answer I give you is yes, you can learn from CTFs. The question is how much will you learn and what will you learn? So in terms of what will you... I was not able to gain access to all of the systems to look through them myself. Unfortunately, the attacker still had access to

the email system, even though passwords were changed, everything like that. And more emails went out, and the amount started going up. The diplomatic corps for the Netherlands was trying to be helpful even though they were not asked to intervene whatsoever by the ambassador. And they were trying to be proactive and unfortunately they made matters extremely bad. They're very good people. I've still got friends with them. But sometimes when you intervene in certain matters and you don't know the entire picture, you can make matters worse. And that's just life, right? So the second extortion attempt went out to GCC member states, so that's basically part of the Middle East. Qatar and Oman were very nice. They actually gathered evidence from their embassy to hand off to

me, which was quite unusual because, again, I'm not a national of those countries. But they were trying to figure out what was going on, keep things contained, didn't want more damage to the reputation of Saudi Arabia or themselves. So they were extremely cooperative. The attacker, using the embassy email, went ahead and, I wonder if this works, signed it with "Embassy, ISIS." Right? So you know exactly who it is. And that's nice of them, right? You know, save many lives. Give us 25 grand from each of the different embassies that they had sent this email to, still using the Saudi Arabian embassy business email account. So, you know, it's a friendly email, right? So what had happened was the diplomatic police or diplomatic corps,

they had sent out a warning to the back end, other embassies all around The Hague saying, hey, we've heard about these extortion attempts. and if you happen to get any of these emails, go ahead and contact us and we'll try to help you out. But the problem was, because the attacker was still in the email and they got copied in, and the diplomatic corps did-- - Right, some people think of this as a technical skill, I really don't. Thinking like an attacker does, thinking how they would come after a system, but also thinking what their motivation is, what their end goal is, and there's different types of attackers with different end goals and different motivations, and getting inside that mindset is really important as a security practitioner, and it's

not something that you will typically see happening in a CTF, right? Like in a CTF, your goal is exploit X, get the flag, and get the points. Attackers are not looking for flags. Attackers are looking for particular access to data, particular access to systems, and you have to understand their motivation as to why they're going after something and what it is that they're going after. Also being able to take multiple approaches to something. In a CTF, there's usually one way to get the flag, at least if the challenge author has done a good job, there is one defined path to the flag. In the real world, a single service may have multiple vulnerabilities that you

can attempt to exploit. A single system may have multiple services running on it, and a single data store that you're trying to get to may have multiple systems capable of accessing it. So from the security practitioner point of view, you have to look at it and think about whether or not the path that you are currently pursuing is the right path to pursue. When you play CTF, usually there's only one path that you're pursuing at a time. Another skill that's absolutely critical to the security practitioner is communication skill. Most security practitioners, I'm sorry, we have to write reports. I know most of us don't love it. I don't love it. but it's an absolutely important part because it's really important to communicate, particularly to non-technical individuals or non-security

aware individuals, what the impact of a vulnerability may be, what the impact of their business decision to not use TLS may be, what the impact of any number of other choices that are made could be, as well as communicating to them the steps that they need to take to remediate issues, right? I've seen people file bugs And if you've ever participated in a bug bounty, you'll see some of these where they come in and they just say, there's a cross site request forgery here. Well, it doesn't tell you anything about what the impact of that cross-site request forgery is. Like, is it a cross-site request forgery that results in the user being logged out, or

is it a cross-site request forgery that results in account takeover, right? Those are dramatically different circumstances. And being able to communicate this is an absolutely critical skill. And again, it's not something that's traditionally exercised in the capture the flag space. Also, teamwork. A lot of CTFs are team-based. But at the same time, what you'll see happening is it'll basically be just a division of labor. The teams will sit down and one person will take this challenge and another person will take the second challenge and a third person takes a third challenge. And I hate to break it to you, that's teamwork only at the most rudimentary level of the skill. True teamwork involves being able

to work on a single problem together, being able to Bounce ideas off of each other being able to mentor people who don't have as big a depth of skill set as you do in that particular area just divvying up the labor It's not a great approach because what happens when your web guy runs out of web challenges to solve or in the real world what happens when you're tasked with performing an application review of a monolithic application and you're told you have three days to do it, but it's okay you have four people and You can't just cleanly divide the application in most cases. You need collaboration. You need cooperation in that space. So there's some overlap between the skills as I discussed. I largely think of the

skills that are involved in CTF for the most part as a subset of security practitioner skills. There's very little that I would say is outside of the practitioner realm, even things like steganography and obscure technology. Then every step you want to take as a security practitioner becomes an argument again, right? You're arguing with management and, you know, and with finances and that kind of thing. So picking a standard in itself has addition, and it's not just even in vulnerability disclosure, but picking a standard has that benefit of cutting down on the number of arguments you have. I don't have to argue about each step of this thing. We've already picked the standard that we wanted to do. I think industrial control systems are

kind of late to this party. A lot of other things like popular commercial operating systems have had a lot of attention in the past because of the, you know, maybe not quite accurate stereotype of a hacker in the basement who has access and some free time or curiosity or ability and explores things like Windows, looking for vulnerabilities, and notifies Microsoft or the manufacturer and they develop a process for dealing with those vulnerabilities. Industrial control systems tended to be rare and expensive and so they didn't get this. They occasionally be useful in the practitioner space, but a lot of the things genuinely are useful to a practitioner. But then there's a lot that's not covered by the CTF space, right? Being a security professional

is so much more than being a successful CTF player. The flip side to this is not every security practitioner is going to make an amazing CTF player. Their skill set may more closely align with the areas that are outside of the overlap here. But in terms of getting educational value from a CTF, what our goal is to do is to maximize this area of overlap, right? Either as a player or a CTF organizer, if you want to be able to use it to build your industry skill set, then maximizing that area of overlap is going to be the best way to get the most bang for your buck. So let's talk about communications. I said

that's a really critical skill and I genuinely believe that that is a critical skill, especially if you're in consulting, but even in an internal security team, I think it's really important that you are effective at communication. One way to do this is doing what are called write-ups for CTF challenges. Some of the bigger teams do this quite a bit. And what a write-up is, is a description of what the problem was, the approach you took, any dead ends that you may have encountered along the way, and what your ultimate solution was. And being able to explain... - The same amount of scrutiny over the past 20 years or so. But now, thanks to lowering prices

and things like eBay, you can buy old equipment for cheap and start doing testing in your basement again. So industrial control systems are now starting to get more scrutiny than some of the other operating systems, some of the other more commercial, popular type of applications have already gotten. So they're a little late to the party. Maybe there's a little catching up to do because of that. Maybe not quite as mature. All right, so if you get this call, by the way, who in here are electricians? Anybody? Electoral? Okay, well, so, by the way, you're supposed to ground these devices. You run a green wire to ground to keep people from being electrocuted. That's ground. Okay, all right,

so things not to do. If you get this call, don't hang up. Don't ignore them. Your vulnerability is not going to go away because you hang up. A matter of fact, ignore them and it will go away. chances are good they're not the only person that knows about this vulnerability. These things in a term that someone else can understand actually builds your communication skills a lot. I find that when I have to explain something to someone else, it forces me to stop and think about areas of the problem that I didn't really think about. There may be things that just happen to work, but I don't actually know why they worked, right? And in the heat of the CTF, I don't

often care why something worked Got the flag. I'm moving on but when I go back into a write-up I'm like why did I come across that as the base address that I was using for this or I did happen to work But I don't really know why so going back and doing it also reinforces the technical skills that I used in there But being able to explain it to someone and break it down step by step really builds your communication skills some of the higher level CTFs actually require the winning teams to be able to prepare write-ups for their challenges. This is also done as a way to ensure that the work was original for

those teams and that they're not just copying off of someone else or sharing solutions. I also think it's important to include how the vulnerability would translate into real world impact. If this is a service running in a real enterprise environment, what would the impact of a vulnerability like this be? Because I think it's a useful learning lesson to people coming into the space to be able to see these write-ups, understand vulnerabilities, understand vulnerability classes, and understand the impact of the vulnerability. So you can say, oh, if this had been a real world service, this would allow full account takeover on this service. They're just the only person who has bothered to let you know they

know about a vulnerability, right? Other people may have, and particularly more well-funded people, probably already know about this vulnerability. So it's definitely not going to go away just because you don't listen to them. It's still there. It still exists. Don't start out by claiming there's no problem. Oh, our product is wonderful. What are you talking about? Our website is great. Our customer lists are safe. Listen, pay a little attention. They're going out of their way. Don't threaten or begin legal action. We'll talk more about that in a second. Don't assume the worst of this caller without any data. Again, just because of it, you know, isn't a guarantee that they're benevolent, but it's a pretty good indicator, right? They brought

you something. They brought you a gift that they potentially could have sold or used elsewhere. So... Don't necessarily assume that your system is configured optimally without any data, right? You might have not set it up according to the best best practices and so forth. Yeah, don't claim there's no problem immediately. All right legal action Other vulnerabilities allow remote code execution on the machine, which of course gives you access to any of the accounts in there, right? Which is a different scope, but for purposes of a CTF, both of them may earn you the flag that you're looking for. In terms of teamwork and leadership, like I said, it's not just about dividing up the effort. I mean, first off, play as part of a team. Quite honestly, CTF is

a lot more fun if you play as part of a team. sitting there even if you're not physically co-located with the other members of the team having someone else to communicate with having someone else that you know is just as frustrated as you are having someone else who's beating their head against the same challenges you know maybe it's a little bit of schadenfreude but I do love having that support system there in place And you can work on challenges together. It's really good when you have someone else that can be a sounding board, maybe you can explain things, even what's called rubber ducky debugging, if anyone's familiar with that. Sometimes you start explaining something to

someone else, and halfway through the explanation, you suddenly realize what you were doing wrong, because you're forcing your brain to slow down and think about the problem step by step. So even just being able to sit there and go, okay, so I was doing A, B, and C, Whoa, really helps. But other times you can explain to someone and maybe they will just say, you know, have you tried looking at this? And it'll be something you hadn't considered, but that'll be the breakthrough you need for the solution. It's also on the coast in California. Anybody knows whose house this is? Farber Streisand's. This is her house. And how do we know this? Because back in the day, there was... a photographer that had taken pictures of all

of the California coast, what, 112,000 coastline photographs, and her house showed up in one of them, and she was concerned about her privacy, and she wanted this taken down. And so she initiated a lawsuit. Here's her lawsuit. So, before she initiated the lawsuit, this picture of her house from the air had been downloaded six times, two of them by her lawyers. One month after she began this legal battle, her house picture had been downloaded 420,000 times because of the publicity of such a... I include the picture three times here just for reference, okay? So... And we give it a name, right? The Barbra Streisand effect. How do we know a real opportunity to get experience working with other personalities or other backgrounds or things

like that? Some people on a team may be really assertive and going after certain types of challenges. and other people are not so much and are sitting towards the back. And you get this sort of thing that you have to get accustomed to in the work environment, working on a team with people of finding out what their strengths and weaknesses are and how to bring them into the fold and get them more motivated to work on the challenges you're working on. It's also a good opportunity for mentorship. Many of us have built our careers through mentorship in one form or another, right? A lot of us may have gotten our first security gig by knowing

someone else in the security field. A lot of us may have met someone at a con or something like that. Playing on a CTF team, there's a really large CTF team called Open to All. And as its name implies, it's open to anyone who wants to play on their team. And I know at least it might make you your company, your organization look like the bad guy. No, we're just going to sue people instead of fix things. Maybe as importantly or more importantly, researchers might not contact you in the future with vulnerabilities and you've lost valuable resource. They might do something else with them. They may not bother. They may sell them. They may keep

them. They may hide them. Somebody else can find them and take advantage of them. So just that chilling effect of keeping people from bringing you these gifts, reporting this to you, has some cost. What day is it? All right, so we're a security crowd. What's a no day? Okay. Come on, you're a way that team who have gotten their current security roles by other people that they knew through this CTF team. So even if you're not getting value out of the CTF, you may get value out of the CTF team and out of getting to know the people who are there. Conversely, if you're looking to build your. So what are some other ways that you can get more out of

it? Right. Stepping outside of your comfort zone is important. This may actually reduce scoring at the time that you're playing. It's very easy to fall into the trap of saying, "Well, I know web really, really well, so I'm only going to focus on web challenges." Or, "I know exploitation really, really well, so I'm only going to do exploitation." And that's great if you want to win the CTF. That's probably the best strategy for winning the CTF. The flip side is, if your strategy, if your goal is to build your skill set and to grow yourself, that's not a good way to do it. Continuing to do the same things that you have been doing and

that you're already excelling at is not a way to expand your skill set. Stepping outside of your comfort zone is how you learn things. As I mentioned earlier, professionally I do offensive security, adversary simulation, So I love doing forensics challenges and CTFs. It's not in this particular position. I started talking to Davey on Twitter, and I proposed another perspective on the same story, at least another headline, another summary. The summary could have said, "Windows 10 O-Day info released by Google according to responsible disclosure guidelines." Allows you to consider possible risk and mitigations to the security of your systems now. Well, that sounds a little different, right? I mean, it's about the same thing But it's a different perspective the perspective that

a letting you know about a vulnerability gives you the opportunity Perhaps to mitigate some of the risk, right? There is an upside. That's why we do this thing. I If there were no upside, you call us all jerks, we all go home. But now you have the possibility. You might shut the service down, or maybe you can change the config, or change the firewall rule, or maybe move to another vendor, whatever it is. But if you know about the vulnerability, you have some possibilities. So I asked him about this. He's like, well, that was his headline, and that's fine. And he said I could use this with permission. Just from a kind of perspective, right? Are we, was it really an old Windows 10

zero day security bomb? Or was it advanced notification? Just different perspective. All right, so you work at a company somewhere, perhaps. What do you do? We talked about a lot of things not to do. What can we do? Make it easy for people to contact you, right? Have like security at your domain.com. it can be hard to find the right people to contact, right? Who usually gets this call? Who's the most exposed people in a company? Tech support, maybe, what's some? Yeah, yeah, yeah. Yeah, customer support, sometimes sales, right? So they're the most likely ones to get the call because they're easiest to find. They're the easiest to locate. Unless you've talked to them, unless you've trained

them, unless you've sort of set the system up, they don't know what to do with it. They're like, "Hackers, we don't want to talk to you." And there, you just took this gift and you threw it away. So, maybe it's not going to happen. But having enough challenges at different skill levels can be interesting to them. You also have various interests and various learning objectives in your audience, right? Some people really want it to be hardcore classic CTF problems, exploitation. They just want to get shells and cat flag, submit the points and be done with it. And then other people are coming there because they do want to build their skill set because maybe they're

in IT help desk and they're trying to break into security and this is their first opportunity to play around with some of these security things. Maybe they really want to learn about web exploitation. Maybe they really want to learn about binary exploitation. Maybe they want to learn about exploitation on weird architectures. I've run challenges on ARM and MIPS. I've never run one on Middle Indian though. I just can't bring myself to do that. Alternatively, you might be running a private CTF, and I've run some of those at my employer and other venues as well, and these are more targeted to particular skill sets and very often to people with a similar background, right? So I've run CTFs, for example, that are security awareness CTFs, where we present

a CTF to developers, but they're not security professionals, but we specifically present web security challenges, for example, because they are web developers and we want them to have a better understanding of these web vulnerability types whose names we throw at them when we say, "Hey, you have an XSS in your service." We want them to understand the implications of that and I think a great way to help them understand that is to show it to them in the form of a CTF. These tend to be more focused tend to have a more similar background so you can really narrow what you're doing in terms of building the challenges out, right? And this is again a

spectrum, some of them will be more or less one way or the other. For example, there's a great CTF called So Hopelessly Broken running the IoT Village at DEF CON and some other conferences. But it's the IoT Village, so surprise, surprise, all of their challenges have to do with IoT devices. and things like that. So they have the sort of narrower focus, but still it's a public CTF, different backgrounds, different skill sets. So gamification, right? I've been trying to avoid this word because it's been a buzzword for a while, but it turns out that CTFs are what education professionals call gamification of education. Gamification is basically where you motivate students to learn by adding game aspects to the learning process. And

it actually turns out that there have been many studies that have been conducted in psychology and education and other fields that have shown that gamification improves learning performance and skill progression. It also increases retention and has shown that players, sorry, player students feel like they have put less effort into learning the same skill as they would from just listening to a lecture, reading a book, something like that. So it turns out that if you're having fun when you're learning something, it's easier to learn. It sounds super intuitive, but apparently So I'll come back to the practitioner skills we try to teach in Pros vs. Joes. Pros vs. Joes explicitly has education as one of our core goals. That's why

it's the Pros vs. Joes model. When we look for Joes in that game, we're looking for people who are students, who are new to their career, who are making a career transition. That is what we mean by Joe, is we mean someone who doesn't have the 10 years of experience that some of you may have. in the security industry. The environment is reasonably realistic. I put quotes around it because it's hard to simulate a full enterprise environment, but there's a Windows domain with domain attached servers and clients. There's Linux servers, it's varying versions. You'll have Windows 7, 8, 10. We finally killed Windows XP from the game environment. Even though it was a lot of

fun, it was actually getting really hard to keep running Windows XP VMs on modern virtualization. And it has real services in the environment. We have mail servers, we have DNS servers, we have file servers, web servers, you name it, anything that you might see in a real world business environment, SQL Server, MySQL, is running in this environment that the players have to deal with. And one of the key things is we have a score bot that checks these services and actually interacts with them. So one way you can make a totally secure network is you can just turn the network off. But if you have the scoring based on whether or not these services are

available and responding, it makes that a lot harder for the players. They can't just put deny all in the firewall rule and then grab a beer. So how do we help the players to build these skills? Well, we have the Red Cell Pros as a sparring partner for the Blue Joes, right? So the initial start in Pros vs. Joes, the players are trying to defend their networks. They are Hardening network firewalls, hardening host firewalls, changing passwords, disabling unnecessary services, patching services, all of the things that you do as a security practitioner or that you ask your IT department. Because we don't want to just throw Joes in the deep end and give them no guidance,

every blue cell comes with two pros, which are security industry professionals, most of whom have also played our pros versus Joes games before. and they serve as mentors and leaders. The role of the Blue Pro is not to do it for the players, but to help the players do it and help them understand what to do, but why they are doing it. Help them understand the value of the firewall rules that they're implementing. Help them understand the value of the GPO policies that they're pushing to the Windows domain. Not just blindly, hey, do this one magic PowerShell command and all your security problems will be solved, but here's what it actually does. But even as

a pro, you learn new things. I've been a blue pro before I moved to the gold cell, which is what we call our game administrator team. And I was a blue pro despite having a job that is very much on the offensive side. attributed us and asked if we'd test their fix, they were such a joy to work with, right? And I don't think that they're a bad company because they've listed a vulnerability and a fix for it, right? I think they have a mature security program because of that. And I know some of those guys, they do really care about their security. Anybody remember the Kaminsky bug, right? DNS. So... Dan has this, I was asking him, he was trying to coordinate one of the biggest sort

of responsible disclosures probably in history of tons of companies and internet service providers to fix this DNS bug. You can look up the Kaminsky bug if you don't know about it. His advice was if you are a researcher or a company, just don't be a jerk, right? Try to be nice, try to be reasonable, try to be responsible, try to listen to folks. even created this flow chart. Learned a ton of new things, even from my Joes in the environment, because some of them are expert Windows domain admins, but they don't know anything about Linux security. Or they come in and, you know, they can look at a Palo Alto firewall and it's magically configured,

but they don't know anything about host security. So they're still learning new things, but I'm learning things back from them as well. So it's really great for an educational mission. But one of the best things about it is at the end of each day of the... Which you all can read all of, you know, and memorize. We have a quiz after the... I will point out this one thing. This flowchart's a little older, but it says, you know, if you go through all the right stuff and notifying them, the company should be okay, right? This should all work out because it's 2012 and everyone has a security team now. So I just want to remind

you all it's 2012. and all the security stuff is worked out. Other people do bug bounties. They will pay you if you will let them know about vulnerabilities, which can make a lot of sense. We have what we call the hot wash, which is where the red cell divulges what it is that they have done to the blue players. The end of the first day, Red maybe holds back some of their information. It is a game after all. But at the end of the second day, Red is fully open. It's an open book. How many of you as security practitioners get to compare notes with your opposing force, with your adversary, right? You get to

find out not only what it is you saw, how they did that, but you also get to find out what you did to complexity as you go through and you build up skills rather than requiring a giant leap. So very often you want to build these with challenges that have some kind of real-world applicability so based on real CVs based on real forensic situations that you've seen If you are building CTFs and you do forensics though, don't just grab a PCAP from your live network and throw it into the CTF That's probably not a great option But you can build a clone network, you know use a few virtual machines and do the same sort

of thing. So based on some real vulnerabilities, for example, an Android application with SQL injection via Android Intents so that another malicious application can start reading data out of the private data store from the first one. Or just using realistic environments, right? of a real system. Build up a VM image, use it for a little while, include browsing to a few pages that contain the flag or something like that in there, and then take the disk image of that and give that to the user. If you're afraid of the company or for some reason you don't want to be exposed, they will maintain that for you. They will help you contact companies. They do this on a regular basis. So they generally know

the right people to talk to and they join the right voice to do this in so They are your friend if if you want in this place in this space So a little more thought about perspective, right? I'm I'm a big believer in understanding other people's perspectives now. It doesn't mean I I agree with their perspectives, but it's useful It's useful to know them So, if you are a security researcher and you're discovering some vulnerability and you're trying to talk to a vendor, some of the things that may be going through their head is, "Aren't all hackers evil?" or "I don't understand this, you have to be evil." Right? "You were talking about something wrong with my system, you must be evil." Right? If you might be approaching them

with a gift of hard effort. "Are you trying to harass me?" I'll look bad if I mitigate, acknowledge a bug and-- Or fully functional apps, right? Rather than building an application that is just the vulnerability wrapped in a thin layer of an application, you can actually take an application that is a real world application and introduce a vulnerability into it as a CTF challenge. So that way the skills of hunting for the vulnerability and of figuring out where the vulnerability is, which are much more applicable real-world practitioner skills, get exercised by your players. So like I said, the progressive challenges, I like to do approximately three steps. Introduce the concept, add a little bit of a complexity to the second step, and then force an edge case or

something that really steps it up at the end. So I expect everyone to be able to get the first challenge. There's enough information there that you can definitely get it and definitely get introduced to the concept. The second challenge should be achievable by most players if they're willing to put in a little more effort. The third challenge will be a stretch goal for players that have never seen that concept before, but your more advanced players may actually still find themselves challenged but accomplish the third one. And so this gives you not only challenges for your range of players, also skills. Not everyone may get to the third level, but they will still get something out

of that experience. So as an example, I've built before an encrypted file system where the first one is an obvious encrypted file system like a Luke's volume with known partial password, right? We know that this guy was always putting his car name, like the fact that he has a Porsche in his passwords. So it's very easy to brute force. So then the second one is maybe a file system without a known password.

In your work with Aramco, how much consideration is given to cybersecurity from a transportation side of actually shipping oil and the navigation systems on board those ships and other capabilities, if I select on the production side in terms of refineries and other cyber capabilities there? So is there a focus on cybersecurity when dealing with shipping oil versus production? Now at the time that I joined, Saudi Aramco had just sold what was the largest fleet of oil tankers in the world called Vela. However, it was also my job to monitor that particular shipping fleet in my office, and I was with a crisis team for any physical threats. We did have, shortly after I joined, An insider over in Saudi. About password lists and where do I get good password

lists and if you download RockYou you'll probably be happy. And then the last one is like an encrypted file system with a harder password that requires password mangling. And then inside it, there's deleted files that need recovery. You know, I'm really throwing some curveballs in there, right? Normally, you open encrypted file system, you see the files there. Well, this was an empty encrypted file system unless you started doing data recovery from the encrypted volume. Arabia, who was very peeved that he did not get the job he wanted to, so he put on Scribd some of our private communications channels, which was then used by a group to try to take over our boats, one of

the boats, and we had to call out a team to physically take everything back. So I also did crisis management for that with a person who was from the Royal Navy who we had hired, who was Scottish, with the thickest Scottish accent I had ever heard in my entire life. Throwaway noise when doing an encrypted volume is incredibly important because you get a bunch of junk back when you decrypt empty sectors. An alternative is a SQL injection. I actually did a four stage one with obvious SQL injection that would just spit back the entire query it was trying to run in the error message if you had a syntax error. So that players should be

able to just fix their syntax slightly by looking at that error message. And then on the next step, I just tell you, yeah, it's, you know, bad request 500. We had a large thing like that. Don't give you the error message, but still give you obvious feedback when your SQL is not working. And then move on to blind SQL injection and introduce that as a concept. And then finally, a blind SQL injection that was hidden in like a base 64 encoded cookie. So they have to start thinking about how web applications work and the fact that there's more than just form inputs as inputs to a web application. So this builds up the skill set

going from the obvious to what should be the less obvious. So what motivates players in these things? The competitive making as they go through the series of related challenges. It's also nice to have categories or tags for your challenges if you're doing Jeopardy style because then players can see that they're checking off in a particular area and growing their skills. Any questions about approaches to CTFs as a player or as an organizer? Do you need to speak to

any experience with applications like boot to root type challenges or doing-- - Yeah, absolutely. So the question was about these boot to root challenges which are virtual machines that are pre-configured with vulnerabilities in them and as the name implies, you boot the VM and then your goal is to get all the way usually to a root shell or something like that on the, I think that they generally fall into the same category as what I refer to as war games. They're not usually time sensitive. And one of the aspects that they lack in terms of player motivation is... Another one is I was asked to evaluate a security vendor one time for my company. I

went to a conference. I'm talking to them. They tell me about the features. It's an appliance. And, you know, we do a lot of patching work. And I'm like, oh, well, how do you patch this? They're like, we're running security enhanced Linux on this appliance. We don't have to patch it. I'm like, what? I'm like, look, I like security enhanced Linux. It's a great product. It's a great tool. I'm a Linux bigot. I'm in, but you still have to patch it. So then the problem becomes, how do you say, thank you very much. I'm busy at the moment. All right, so that's us talking about vulnerability disclosure. Hopefully you get some perspective, whether you're a

researcher, whether you're going to notify companies, or if you are working in a company or business that might get this call one of these days. Hopefully you will get the call. Somebody will let you know and not keep that from you or keep it private. And so I have a scoreboard to compare yourself against other players because a lot of people are motivated by that. I think they can be useful. The problem with some of the boot to root challenges is for whatever reason people short change themselves sometimes on them by cheating. Since you have access to the whole VM and it's locally hosted, there's usually ways around them to just get straight to the

flag. I don't understand why someone would do that with a... Mr. Lee, a friend of mine, that this should be his next book, Hackers in Me by Rob Lee and Monte. Instead he did like threat intelligence in me whatever all right This is how to treat your head just to solve to learn for themselves But I've actually seen people do that and then like post a write-up on like oh Here's how I got the flag by like instead of booting the VM I mounted the file system attached it to like my Cali box and then just went in and got the flag And I feel like they're kind of cheating themselves, but I do think

they're interesting particularly if you're trying to learn a particular skill to find relevant boot to root challenges and use those as a skill builder. Hi, my university's cybersecurity club would really like to start its own CTF team. Some challenges we face are that students get-- I was thinking if there were none, that was the perfect talk because I answered them. Military inventory. It's so big it opens from the front and the back. The hydraulics will take it down completely, do this way and that way, and you can load other airplanes, tanks, and it's the only aircraft that can load the portable U.S. Army bridge that can go in there. I once loaded up semi-trucks, several of them, inside my... Since you're in a university environment, being able to play

in person, I find as a lot of people, I don't know if you've tried that, like playing some CTFs, And then you can also just look at some of the CTFs. There's some out there that are easier CTFs in general than other ones. So if you try that and organize it. For better or for worse, I'm sure you already know this, not 100% of people are going to turn out to be wanting to play CTF even as security practitioner. That particular duty, you're also the ground commander. Full security of the aircraft as well. We used to like to joke that we were typically assigned three guns on the aircraft, one for the flight engineer, two

for each loadmaster, and there were by default three parachutes, and guess who's getting the parachute? Me. So that was a lot of fun and it gave me a lot of good experience in avionics, airframe, and aeronautical engineering and a lot of math. So I'm very, very good at math. And I can pack for 19 days with no check-in bags. Right? Bonus, right? Who said the military didn't give you any teachable skills as a-- - And not 100% of people who think security is awesome are gonna stay in the industry. Burnout is a problem industry-wide, and I think Cybersecurity Club is no different than that. - Early on you said there should be one path to the flag. I was wondering why. - So I didn't, if I said that,

that was not what I was trying to imply. I was saying that CTF writers usually try to put one path to a flag into their challenge because they usually are building it around this one cool vulnerability in their head when they're writing the challenge, right? So I'll sit there and I'll be like, All right, I have this great idea like an off by one memory corruption that lets you overwrite some metadata on the heap or something like that. Making the difficulty correct, the score will come out to be correct to the difficulty of the challenge is our hope. Essentially grading on a curve, yeah. We think actually, so that way the challenge that gets solved 100 times and the challenge that

gets solved once are automatically appropriately scored. So we're interested, it's an experiment, we're interested to see how it works out and play in that CTF, it'll be available online and you can give us feedback on whether or not that's a good scoring mechanism. I've received the stop sign here, so thanks everyone for coming and enjoy the rest of your peace sides. Well, so the difference would be, though, in the model we're looking at, everyone who solves the challenge would get the same number of points. So it's not based on, like, if you solve first, you get the full value. If you solve second, you get lesser value. If it's solved 100 times, that challenge becomes

20 points or something like that. Can I bounce an idea off you? Yeah, absolutely. I may need to get out of the way for the next speaker, but I'm trying to push it out of the way. That's a great idea, though. I like that weighted scores. We're going to try it. We're going to see how it works. Doing it correctly is hard, because if you just divide by the number of solves or something like that, it's a... We've done some simulations. Yes. Yes.

Do you have a clicker or something? Oh. But I have one question, I think. Was this like an attack of opportunity or was it more like a targeting, targeted? It was a bit more. It was, yeah. So you're just waiting for the right opportunity and then... Yeah. Yeah. It's rather highly personal. Yeah. So something had obviously sparked this thing and then it became like... Alright. Yeah, something. Yeah. Yeah. No, yeah, yeah. So is ISIS a threat? Cyber? I need to get rid of this. There are no chasers here, man. Not normal. Really? I saw it, yes? I'm sorry, but I can't see it on the screen.

Test, test, hey hey. Can you hear me? It's very low. Maybe I'll... I'll go to the center and see if I can turn it on. Hold it, hold it. It should be. See it there? Okay, just want to be sure. Start with something with sugar. Yeah. It's a vodka? Nice. Can you hear me now? Hey, hey, hey. It does mics. Not so much. How many chairs do you need? Just two? What is it? Two chairs for you guys? No, we need the microphone. We need another mic. We need another chair. Do you guys need water? test test test test test test okay so don't curse or something yeah a a a

The mic is on. The mic is on. One, two. The mic. Mine's on. Hello. How's that? Can you hear me? Test. Test. Test one. Test one. Good. One, two, three, four.

Ladies and gentlemen, we can't do it without them. Again, critical staff will be found out. until the end of the talk. We'll go ahead and pass the microphone around to you and make sure that you are able to be heard online when you ask your question. So just raise your hand and we'll come to you. Please go ahead and silence your cell phones now. Again, we are streaming online, so we really appreciate if you go ahead and silence your phones now so they're not interrupting the speaker. We'd like to thank our sponsors, especially our Inner Circle sponsors, which are Critical Stack and Valley Mail, and our Stellar sponsors. That includes Cylance, Microsoft, and Robinhood, just to

name a few. Their support, along with our other sponsors, donors, and the volunteers, they make this event possible, and we really appreciate all of them. And if you have any questions, again, please hold them until the end. And without further ado, thank you very much. Great. Good morning, everyone. So I'm Brett Goldstein. I'm the director of the... How we doing? Everybody good? Here we go. Love it. We've never seen technical difficulties, so... Vegas. So thanks for having me. This is my friend Harlan. He's an engineer on our team. And we're going to tell you a whole bunch of stuff over the next hour. So it's a story that will essentially have three parts. I'm going to

give you my background. It's kind of weird. And sort of like hopefully that will be sort of morning fun. And then I'm going to talk about DDS and why we aren't what you think we are and how it's completely weird that we are in the middle of... Just one more minute and we will get the show on the road. All right, who's got the best B-Sides joke? Who wants to give a joke today? What did you guys do last night? Drink too much. Drink too much? That seems to be a problem. What else did you do? What kind of alcohol did you drink last night? What kind of alcohol did you drink? - Yes, how much, here we go. - Yes.

- You got it? - Change the player from the player. - Is it okay? - Switch. - Never does. - You can't go. - All right. - You should be ready to go. - Okay, here we go. - Thank you. - All right, all right. It's gonna be, they promise you. - All right, all right. Okay. Hello, hello everybody. Welcome to this B-Sides Las Vegas session titled "Prisoner #6". We're really excited to be here. My name is Nimrod. This over there is Lavi, and we'll introduce ourselves formally in a minute. In this session, we will talk about the prisoner. In 1968, television series in which a former British intelligence agent is imprisoned on an island, also known as the village,

with other former spies, all because they know too much. Now on this island, also known as the village, prisoners are only referred to by their numbers, with our prisoner being number six. While escaping from this village, as our number 6 soon discovers, is an extremely difficult task. Nevertheless, our industrious number 6 never ceases to try. A quick question: How many of you heard of the prisoner or actually seen it? Oh wow! This is fantastic. So we're going to have a lot of Pumatch. And our village is going to be a Docker container that imprisons us. During this session, we attempt to escape from our village, from our Docker container with the goal of taking over the underlying host and even

eventually running code on that host. Do we succeed? Well, stick around to find out. All right, so let's jump into the agenda. Now, in the meanwhile, make sure that the mic is up and running. All right, so just before I'll cover the agenda, how many of you know how the series, The Prisoner, ends? All right. So you have another reason to wait for the end of this session. And if you already know, don't spoil our spoiler. All right. So-- Sorry. All right, we'll start with the short introductory section. In this section, we'll introduce prisoner number six, for those of you who are not familiar with it. We'll also take the opportunity to introduce ourselves, and one important

thing that we're gonna do is try to understand what the hell prisoner number six has to do with containers, and Docker containers in particular. So this is gonna be the introductory section. We'll then move on to the section called Welcome to the Village. This is a container introductory section. We'll try to define what are containers, the missions that we like, and what is a privilege container in that context. We'll see how privilege containers can be deployed, and we'll also check if and where those privilege containers are used out there in the wild. So this is the Welcome to the Village. We'll then move on to a chapter called "Village may be Determinal" as the escape of an attacker from a container to

the house may be. So this is the "Knowing too much" section. We then move on to a section that's my favorite called "I'm not a number, I'm a free man." In this section, Rod will be introducing or demonstrating several... - Okay, when the cache expires, hold on, wait for the command. When the cache expires, you can see that the local DNS cache is empty. Now, we will send a new DNS request by sending a ping. privilege container escapes. Those escapes are based on our Linux security research at CyberArk Labs and are also based on two patent pendings we have on the concept of privilege containers. In this section Nimrod will go in a step-by-step walkthrough

inspired by the prisoner who yelled at the beginning of each episode and I think you'll see some of it very very soon. I'm not a number, I'm a free man. All right. --will trigger, of course, a new DNS request. And the DNS response will resolve the new IP and map the host name to the 1337 IP. Now, let's see how it works. The victim computer sends a request to evil.com which goes to our DNS server. The DNS server will... One more thing that I should say about this. In this section, the demonstration will be based on a Docker container playground called PlayWithDocker. And you will see that in a moment. It's a wonderful way to showcase how a web-based, live web-based containerized environment looks like.

And at last, we have the mitigation and closing section where we'll be discussing the attack vectors along with a couple of mitigations and final thoughts. So with that, Yimra. - Malicious server IP. Next, the browser will issue a GET request to the resolved server IP and execute the following malicious JavaScript code. The JavaScript code will call itself until the evil.com HTML title is changed. Once the title is changed, the attacker knows that the rebinding process was successful. Alright, so this is the... Can you hear me? Alright. This is the... We're going to have to find something to do with that. Okay, can you hear me now? Okay, so this is the opening scene from the prisoner

and this is our intelligence agent who is now entering his superior's offices at the British intelligence with the intent of submitting his resignation. He then heads back to his London home. The nation has its price and our former agent now entering his London home is then sedated and kidnapped to the village. Delivering a political and social statement And then it's very, like, people walk very appropriately and it's, you know, it's that whole bit. And I'm going there and I was catching up with my friend Chris Lynch, who I knew from, I was on the board of Code for America for a number of years. And so Chris runs this defense digital service thing and I'm like, what is this? Because I'm in the Pentagon and

I'm like, I'm giving people advice, but I'm like, we need to raise the bar. We need the best in technology. How do we solve problems? Because at the end of the day, what do I care about in this space? That technology should never get in the way of the mission of national defense. And how do we get, like, I found it completely enlightening. I'm like, how is it that I have the very best that I'm seeing out in Silicon Valley, but I'm not seeing the very best when I'm at DoD. And that just irks me because at the end of the day... Life in a technological society, a society which may be regarded as a

collective prison, The prisoner is a war of attrition between the faceless forces behind the village, a community reminiscent of Alcatraz and its most strong-willed prisoner, our number six, who ceaselessly struggles to assert his individuality while plotting to escape from his capturers, those who are after the knowledge he possesses. We have been deeply influenced by the prisoner, but we are not the only ones. The prisoner has been presented in several other media. For example, Homer played number five in an episode of The Simpsons where he was... So, you know, Chris comes, I meet him, and then we go into this office and there's this sign that says Rebel Alliance. And I'm like... "Huh? What unit is

Rebel Alliance?" And I walk in and it was like walking into Silicon Valley. And it turns out that we have this bastion of talent at the Defense Digital Service that we bring in some of the very best. And we're going to be talking about that. But what's fascinating-- Where he was taken to the village and was chased there by a big balloon. which references... which references... sorry about that... which references... the village's rover. We'll see more of the rover shortly. Iron Maiden, The Clash, an 80s influential rock band, and other music bands have all directly referenced The Prisoner in their songs with Iron Maiden even incorporating an entire song with dialogues of the prisoner in there the number of the beat and also lost

and also has been featured in the movie Shrek, which is quite interesting. We believe the prisoner is still relevant today with the rise of tech giants and their control over our everyday life. Multinational organizations track our every move both in the cyber realm and the physical one brainwashing us with ads and feeding us with information they want us to digest we are all living in a technological prison all right so um just before we're moving on to continue telling you the story of uh

high as you might hope. Now we came across this diagram which is part of a textbook people. So how can we have excellence in technology if this is a type of supporting material that you have? The bar needs to be raised. So the Defense Digital Service, so let's talk about what it is. So I have about 70 folks. Harlan's one of them. I'm another one of them. About 40 of them are civilians. About 30 of them are active duty folks. And the civilians come from like all the places you'd know. And they're people who say, I have expertise. I am like Harlan's an engineer or I'm a product person. I'm a designer. I'm a security engineer. And I want to

serve. And what do we do? We bring people in on a two-year appointment. It's a two-year tour. Can you hear me well now? Yes. All right. Let's move on with this. All right. So who are we? We didn't introduce ourselves properly. So this is Nimrod Stoller, a security researcher at CyberArk Labs. And my name is-- And you come in and you work on things that you can't even imagine. They're all over the spectrum. And then what we do, because I think it is critical that we also have active duty folks in. So those are folks in uniform, people who are out in whatever service it is. And they bring a couple things to the table. There are, there is a love village of

it. I'm a security research group manager at cyber labs and we are both based in Tel Aviv, Israel. And um, so uh, what is it that we do? Um, we both, a part of a group called Charlie Group, which is focused on the security research of emerging technologies. Like, for example, things that we do is security research for containers, for example, and we do other hardware and software stuff as well. Our offensive security research is the ground on which we work to develop new innovative ways to mitigate the attack vectors and attack surfaces that we find. But at the core, we have to work on a mission. And the core sort of types of projects that we do. So one, We're firefighters,

okay? At the end of the day, the Defense Digital Service, if there is a technical heater which is just absolutely horrific, we are going to be the people you call. Because I will say all day long that my team has the very best in technical talent anywhere in the DoD. My folks are amazing. and they can solve problems and they will go wheels up in hours to anywhere in the world to fix a problem. And that is amazing. Two, we will advise on projects. So say there is a big enterprise initiative. We're the best in technical advisors. There are really stupid ways to do procurements and we could pick the worst in tech. So if you don't put the nerds in the room,

how are we ever going to do this right? Like for those of you that sort of have some history here, you know, you go back to when did we like the idea of having security in the boardroom? There was a day long before that, I remember in the early days of Open Table having to sit down with the board and justify cybersecurity expenditures. And they're like, why would we spend on that? I'd be like, oh, because this nonsense is going to be hacked, yo. But the world has changed. So how can we do to the village? And the village is the fictional setting where everything is happening here. But who runs the village? Apparently, the village is run by a democratically elected council with

a popularly elected executive officer known as Number Two. But everybody in the village knows that this process is rigged and the operators of the village are simply using it as a means of control, of their control over the inmates. Tools and alcohol are strictly forbidden in the village, but there are no walls or physical barriers to prevent escape. And apparently, there are no visible guards. a control room monitors control lives. That's the type of stuff that helps me sleep and that's the type of stuff that the team works on. Now what architecturally makes us super unique is who's my boss? I work directly for the Secretary of Defense. So I am a direct report. My team is all like the

70 folks go to me. I go to SecDef. Now for those of you who are less familiar with the military, that's really weird. We have all sorts of deputy secretaries and then chiefs of staff of the services and the joint chiefs and all those things. The Secretary of Defense has determined that having a team like this is critical to our national defense. and we have that ability. So when someone gets in our way for doing the right thing, that is who I escalate to. And it's a pretty remarkable type of team. Now, we... - Circuit-tabbing cameras located inside the village and observers spy on every move of the villagers in order to foil escape attempts with the

aid of rover. a large white balloon-like device that chases would-be escapees and carries them back to the village. Let's see now such an attempted escape with... How do we grow more? But this is my team and we have folks all over the place and there are certain areas we're growing in right now. Security engineers, feels kind of important, right? Real focus, note, blue band, security engineers. The rover.

Alright! Much like a container escape, you'll see in a moment, I promise. So just before we're getting into the more-- We're in Afghanistan together. This is me saying, oh my god, I have gone from open table to a Black Hawk in Afghanistan. And actually, it's really comfortable. You just sort of settle in. You just make sure those straps are on really tightly. You'll be hearing from Harlan in a minute, who's really doing something interesting there, which I'll defer to him to talk about. But we also try and bring a lot of the spirit to the Pentagon. I'm not sure how many stormtroopers you've actually seen walking in the E-ring of the Pentagon, but we have quite

a few pretty wild pictures there. Now, one of the reasons we're here is part of our portfolio, which I didn't mention before, is what we call the Hacktha portfolio. Now, I strongly adhere to the concept of check your work. Now, I can build or write the most amazing software in the world. I still need it to be checked. I can tell everyone in this room I did my best to secure a system. but it still needs to be checked. Now, historically, in government, we haven't done the best job of security. That's a bummer. So during my tenure as director, it is absolutely critical that every day we raise the bar on this. And this is from

Actually, we'll demonstrate that in a few moments. So those are containers of privileged containers. And now, what are the things that make a container a privileged one? So here are a couple of things that we covered in our research. So first of all, extra capabilities. Adding extra capabilities to a container will make it a privileged container. And I have a couple of examples for you. Just before that, capabilities are distinct permissions broken out of root. Some of them are HTTP ports. So you need this information in order to get the right response from the right web application. I did not write it, but to talk with many assets in the same web application. We created iframe. It's really

ordered to... Those permissions can be either disabled or enabled, allowing the container to do something connected to the host in some way. So we have here three examples of such capabilities. For example, Capsys Admin. which allows a process or a container in our case to run administrative operations like mount, unmount, set namespace and so on. We also have CapSys module which is a capability that allows us or the container to load new kernel modules or uninstall existing ones. web application as we can and to do it asynchronously so we have iframes and we connect to those iframes with web messaging so now we have the mechanism of how we can send requests but let's see the real request so we send the request to the

malicious web application how we can get the information we need from the web. - Or capc's boot is another example of capability that will transform a container into a privileged one, which allows a container to modify the kernel. So those are a couple of capabilities that we should consider when talking about privileged containers. Turning security controls to off might also transform a container into a privileged one. So we have seccomp, for example, seccomp, which in simple words, and without insulting it, is a Cisco filter. - Application in the victim browser network from the malicious web application. So the malicious web application has a responses dictionary, and it will save the response object we're using Node.js. It is possible

to create a global response object that will have a key value paired responses and also we can just create a timeout mechanism to set a timeout when the response is... So by turning it off, we allow the container to run any syscalls that is permitted to run and eventually again transforming it into a privilege container. We also have AppArmor, AppArmor which is used as a mandatory access control which confines the container to a set of limited resources. Just to give you an example, IpArmor can block certain read or write, memory read or write, it can block certain syscalls from running and so on. IpArmor is a tool or a security control that is part of the security in depth that Docker container allows. Allowing a

container access to sensitive files and folders might also create a privilege container. And we have here one very straightforward example, which is the Docker SOC. Docker SOC, which actually allows direct interaction with the Docker daemon, which runs as root on the host. So all of these are ways to create privilege container, not only the --privilege flag. All right, so enough of definitions. Let's talk about how privilege container-- --already rebinded using the retinal and the DNS rebinding. Now after the victim browser responded with another push of a message using WebSocket to the malicious web application, you will see the same identifier, but now you will see the headers of the response and also the data, the body of the response.

So I'm starting here with a simple, maybe a bit naive example of Kubernetes, the system named Namespace. one of the containers in the system namespace and surprisingly or not containers running within the system namespace are privileged containers and what you see here is that will come back to the user and now the attacker malicious web application finds the right response and just respond using send function and returning the response with all the headers even where basic authentication headers and with all the information and with all the body of the response to the attacker - F, effective capabilities. We have the three FFFF, which means the container has all capabilities. And seccomp0, which means seccomp2off. And as I mentioned before, that means we have here

a privileged container. And this is how privileged container looks like in Kubernetes. We have another example here of a QProxy. under the Kubernetes infrastructure, QProxy, which is responsible for... And if it's binary files, the attacker will still be able to download those binary files using the same technique. So, as I said, you can find the retunnel code, source code, here in GitHub, github.com.

And now I will talk about the easy part. Why is it so easy? It seems so hard to set up all these things, like you need a DNS server, and you need the core application. - In the Kubernetes environment, it has access to all devices on the host. You see here, the SDA is here. So actually any process or an attacker on that container have full access to the host. Now those are probably naive. You might say that those very sensitive containers are running within the system namespace and I can agree with you of course. But let me share with you another example that I think resembles another type of privilege containers. And this example is of Datadog. Datadog is a Monitoring service

allows monitoring of cloud. When you deploy it, you actually mount the Docker SOC into the container. Obviously creating a privileged container because any process, any attacker on that container will be able to talk to the Docker daemon, which runs its root on the host. and we have a compromised host. And I think that Datadog is just an example of very sensitive services that are running as privileged and are privileged containers. All right. The past, right? So put in a name, get a result. Cool. name, email, and then if you click it, it pre-fills their information and it just reuses the last investigation that they had again, right, and refreshes whatever the data was in between. Perfect, makes total

sense. Saves me time, saves them time, everything's great. But something weird happened when I entered names in there. I noticed that it was returning names I didn't recognize. like just random people I'd never heard of. So I asked around the team like, "Hey, does anyone recognize Billy Bob?" And they were like, "No, no idea who that is." Huh, that's weird. So I tried a couple of things and it very quickly became apparent. Okay, alright, very good. So, Lavi just defined what privilege containers are and we've also seen some examples of privileged containers out there in the wild as we define them. But here during our live escape, we're going to use a different platform for our live escape demo. And the platform is called Docker Playground

Websites. Now those websites are actually two websites. One is Play with Docker and the other is Play with Kubernetes. And this platform, those containers, start out as privileged containers, as the V just defined, but they are then, by using various Linux defenses, such as AppArmor and SELinux, those containers-- That what this box was actually doing was not listing all guests that I previously invited, but rather was listing all guests that anyone had previously invited. Okay, somebody forgot a where clause. Whatever. So I start looking people up. Who's visited the Pentagon that might be interesting? Well, you know, Eric Schmidt. He does work. Is his email in there? Oh, yep, it is. Environments are hardened and they are transformed from privileged ones into non-privileged

ones. So what are Play with Docker and Play with Kubernetes? So Play with Docker and Play with Kubernetes are a wonderful initiative. They allow users to load and run Linux containers in a matter of seconds. It gives the experience of having a free Alpine Linux virtual machine in browser where you can load and run Docker containers and experience the Docker platform first hand without the hassle of first having to load and configure it a few months ago we noticed a vulnerability with the PlayWithDocker container and after we continued to research that we discovered that we can load Linux kernel modules from the container to the underlying kernel And once we wrote an exploit for that, we managed to even escape using this route all

the way to the host and eventually run, remotely run code on... Three kilobytes for a name and an email address. That doesn't seem right. That's really weird. What's going on here? Before I advance to the next slide, I should say for the record that the data that you are about to see is all synthetic and I just generated it yesterday from scratch because it looks like this. You'll see some interesting things in here, right? So first of all, you've got name, okay. You've got a UUID, good, UUIDs are great. You've got update time UTC which is a string of slash date. - Generally speaking, Linux kernel modules are dynamically loaded code which runs in the context of the kernel. So it is

an extremely privileged code. We notified the PlayWithDocker maintainers and their vulnerability was fixed a few months ago. So back to our story. We managed to help our prisoner number six escape once before from the village and return safely to his home in London. But now, according to our story, number six is back to the village and we have to find another vulnerability, another way to exploit the Docker playground again because our initial vulnerability has been fixed. Huh, that's not good. Investigation, oh great, investigator4979 did this investigation and this person was approved. Okay, so now instead of scraping this to get the records of all the people who visited the Pentagon, I get... So what does knowing too much mean

in our story? Well, we know that prisoner number six, along with the other prisoners in the village, all know too much. This makes them a threat to the organization behind the village. And therefore, because of their specific knowledge, they can never be released back to society. It's on the host. And that means that our attacker has access to secrets and credentials injected or loaded in all containers running on that host. For example, by using Docker Inspect, our attacker can inspect the inner definitions of all of the containers running on the host and extract secrets and credentials. If our attacker has privileged access to the host, it can also run code on each and every one of those

containers. For example, by using docker exec and docker attach. Another important thing that our attacker can now achieve is our attacker can stop and even completely remove containers from the host, any container our attacker may want to. Well, go to the back of the computer and pull all the cords out of it? That should do the trick. The thing that I want to emphasize here though is that the people who built this system, the government people who built this system, I'm not so sure about the quality of the engineers who built it, they did everything right. They did everything by the books. In fact, when I talked to them later, I found out that this

system had actually been supposed to have been launched about six months earlier, but it had been delayed because they needed more time to run security audits on it. In fact, this application had gone through about a year of security audits. It generated tens and hundreds of pages of audit data, of security processes where they had to document how there were no non-US citizens who had access to commit code to the code base. And our attacker can also create new containers or replace containers that were deleted by using Docker create and Docker run. Well, this is another interesting point. By having access, privileged access to the host, our attacker will have access to the Docker Hub credentials if the host is

a development machine. The Docker Hub credentials are used to-- They could put something into the database that would get you to give the records out to that person, and that would be really bad, as opposed to giving it to anyone who used the website at all. This process is how the government does security right now, today. So this visitor management system for a major DoD installation, when I say the stakes aren't that high... ...to send or to push images to the Docker Hub. Now, with these credentials, our attacker can infect all the Docker images or other repositories of the organization, and perhaps even demand ransom afterwards. And last but not least, the network. There's probably

an inner network that our host is connected to. This may be, for example, a container orchestration network. And by leveraging lateral movement within this network, our attacker can move from machine to machine and perhaps eventually take over the entire organization. There's recently a lack of opposition in the matter of free elections. This is not good for our community and reflects an acceptance of things as they are. We know what we must do. What must we do? Thank you. But people, it is my pleasure to present to you the fondly number six. I am not a number. I am a person. In some place, at some time, all of you held positions of a secret nature

and had knowledge that was invaluable to an enemy. Like me, you are here to have that protection. ...work, to give your time to serve, but also to help us engage the community at large. Right now, the DOD did a thing that no one else in the federal government had done a couple of years ago thanks to the work of our team. which is the DoD created a vulnerability disclosure program. So anyone, anywhere in the world, if they find something in any DoD system, they can report it to us. And as long as they abide by what the industry considers to be responsible disclosure practices, the DoD will not press charges. In fact, they will thank you. And that project is

incredible. We get reports. Many of you have accepted the situation of your imprisonment. Keep going. I shall be running for office in this election. Good people, let us applaud a citizen of character. May the best win, or number six. Okay, so now we've gotten to the-- Whose own governments would pay them very dearly for that information. And instead they come to us so that we can fix it. That is an awesome amount of responsibility on us And that is an awesome amount of power that we are putting out into the community. Stage where we speak about our live escape and here we detail step by step what we intend to do during the live escape.

We already discussed the point where we managed to exploit the Linux kernel module injection issue, but this is now fixed and we want to show a live escape. So we had to find another exploit. After some research on the PlayWithDocker container or platform, we discovered that we can exploit the system console of that platform. Now the system console is a device, is a Linux device, feature. Maybe this is why it tends to be forgotten. So what we're going to do is if we have a valid user in the system, on the host, then that user may use that feature which enables users to log in and type in their username and password and after that the system console would open a shell on the host and

allow that user to run code on the host. So this is our target, but before that we have another step because we need to add a valid user in the host's etc password file. So we have two steps to perform here. One would be to add a new user or to change an existing user on the whole CTC password file so we could log in. - yet to yet another unsuspecting website visitor. Thank you. Do folks have questions? Wait for the microphone. - I'm gonna take you one second.

So do you have any plans or is there anything in the works right now to work with the reservists? I know that you have active duty folks working on it, but it seems like a reservist would have a very good skill set to help with this. We do. We have, I think, one or two reservists on staff right now. It can be... Because the reservists aren't in full time, getting them activated to do work can be a little dicey sometimes. And the second step would be to exploit the system console, to inject keystrokes into the system console, and we do that by using a small utility we found online, which is called TTY echo, and

we use it during the live escape. Well, here it starts quite small. But on top, you can see that these are all the messages from the-- --around reservists. But yes, it is definitely something that we're interested in. And if you are or you know somebody who is a reservist and you might be interested, please reach out. We can talk to your commands and see if we can come to an arrangement. So in general, we like to meet amazing people who want to do things. So I like to say, if you're amazing, Reservist, whatever you are, then make it our problem. Like there's nothing that gets the Ubuntu system console on my machine. And below, you can see a user

trying to log in. So it's user one, and he's typing in his username and password. And after that, the system console opens a shell on the host for him. So this is what we're going to exploit during the live escape. And now to the live escape.

okay we'll try to use the i'll try to use this i don't think this is working i'll try to shout inner here on the right side and on the left side is our attack machine which we will later on run a listening netcat on which will accept the incoming reverse shell connection from the host. So we start with a brief reconnaissance phase where we try to chart the borders of the container. And so I am not going to question anyone for doing. I think we should have a few different approaches going and then always be willing to pull back and say, how can we do it better? But wherever it is, we're going to help foster innovation and try and

make it better, but always, it always can be made better. How does the DDS address DOD IT and IA policy deviations in a manner that can be replicated outside of the DDS like other DOD agencies? So that is, that's a really great question. So I'm going to give you two answers, right? I'm going to answer the question you didn't ask, which is how do we deal with it? And then I'm going to answer the question, how do we spread that out? So one of the unique powers that DDS has is we have this giant trainer that we are located in and see if it is privileged or not. So we're going to start with UNAME and I'm

going to do this and here UNAME shows us the version of the kernel 4.4.0.1.54 generic and the underlying host runtime libraries which is running Ubuntu but we may have other runtime libraries inside the container we're going to check that by looking into etc.release okay Okay, it is released. Now this tells us that inside the container we are running with runtime libraries from Alpine Linux. So we're going to first use Alpine Linux's... ...defense has authorized DDS to waive any DoD policy or regulation that we see fit for the purpose of bringing technology to the Department of Defense. So all the IAEA paperwork, all the like bullshit 300 page RMF that does such great work, we can just toss that stuff out. And

we can and do exercise that power for other agencies inside the department. Package manager to add libcap, which is a library that will help us decode the capabilities that Lavi was talking about earlier and we're also going to load our TTY echo and other helpful apps from our tech machine and untar it with automatic tools like SQL map so We'll see the DVWA application. It's a vulnerable web application with lots of vulnerabilities. We'll check the SQL injection vulnerability here in DVWA. We'll just set user ID to 1. We'll take the response because we still need the cookie. This is very good for us as attackers. and the capabilities inside the container, which is this long hex number. And we

can use CapSage to decode them. Let's go over them really quickly and see what we have inside the container. So we have the following capabilities. We have CapNet Admin, which allows us to the container to administer the network. CapSysMoney, the retina cookie, we'll save it aside.

You can see the whole response here with the cookie. We'll open CMD, or here's PowerShell. Although I had PowerShell, but still, it was the faster option. Module, which we've seen before, which allows us to load and unload kernel system modules. Ptrace, Capsys Ptrace, allows us to debug other processes. Capsys Admin is the catch-all capability that L'Aviv was talking about. that allows the container to perform administrative operations and capsys boot is a capability that allows the container to completely replace the underlying kernel so from now we just execute the sql map command with the request file and it will find that we have mysql database here of course we'll tell it yes it's mysql don't try another database and now

it will find the attack vector generic union. So still yes. Back of perspective it seems like we are pretty much privileged here in this container. So our next step would be to look into proc command line command line. So this file contains the parameters that are transferred to the kernel when the kernel is first executed. So we see the first parameter is boot image. That's the image the kernel is running from. Don't waste your time. We know the attack vector. And the second parameter is the root device that the kernel and also the host is located on. As you can see, it is designated as a UUID. And we're going to use findFS somehow. FindFS. to

find out the underlying device that the host is located on. So we see the device is Dev SDA 1. Let's see if we have access from inside the container to Dev SDA 1. So it seems like that inside the container we have access to Dev SDA 1. So as I said, I think as a community, we all want to do better and get smarter and find the right relationships. Thank you as well for this great presentation. I'm curious, you're competing with everybody for the shortage of skilled IT and security workers. So I'm curious if you are looking at any nontraditional type job positions. I'm thinking part-time, job sharing, very short-term, maybe 90-day contract projects, you know,

and those are just some examples. If you have other things that you're doing, I'd be curious, you know, how you're dealing with the shortage of skilled workers in the competitive environment. So, I'm not sure I completely agree. So I find that when I'm like, I believe in the I'm going to go out to some hack night or meet up or something like that and go talk to people. And, you know, I did one in D.C. a couple of weeks ago. And last night is like the best thing before you go. You go to bed. I get this LinkedIn message. And dude was like, your talk inspired me. I want to come work. And I'm like, okay, I can sleep tonight, that's

money. And so I find that when we, so look, when I was on the outside, there's no way I could know about this amazing group. What are you gonna do, go to USA jobs and sort of poke around? It's just when- Password, good. So this looks like our first target is achieved. Now, let's go back. and take a closer look at pro command line so we've seen boot image and root but here we have a very interesting argument and this argument actually tells the kernel that the system comes with the queue that chrome has and you don't have this problem We tried to bypass this problem using web workers and iframes and stuff. It didn't work, so it's still using this library we created

called JSpool. You can just check it out. It's in the same project. and its console is to be connected to the TTYS0 device to a serial TTY0 which is actually COM1 and that means that if we can inject keystrokes into COM1 or TTYS0, serial TTY0 then those keystrokes will eventually end up at the system console so let's see if we have access from inside the container to TTYS0 It's something like one minute, the whole DNS rebinding process for an entire internal network. Between one to two minutes, it's the faster you will get in the internet, believe me. I saw many tools. Okay, another question? This was difficult to see from the back. Were any of the services you exploited on the internal

network SSL-enabled? No, no, of course it's one of the ways you can protect from DNS rebinding. And again, we have access to TTYS0. So the next thing we're going to do is we're going to inject keystrokes through TTYS0 all the way to the system console by using our TTY echo utility. So we're going to inject to TTYS0. What do we inject here? exploitation so if you want to protect against dns rebinding ssl if you don't have a bot that has ssl termination off or something like you can just check that this spot will not verify like selenium do it by default for most of the web browsers because they don't want to check this cell because you know when you create an internal web application

The SSL is just... Everybody has an idea? So we saw earlier that the system console is waiting for a user to type in their username. And we just inserted a new user into the whole CTC password file. So we just need to do that. And by pressing enter here, since we don't have a password, the system console is supposed to at least open up a new shell on the host and give us access to that shell. So let's press enter here. Now on the other side, on our attack machine, it's time to run a listening netcat. We are now running full fledged in a shell. So we can do whatever we want. We can run bin busybox netcat and we want

the netcat to do a reverse shell. to our attack machine port 1990 and run bin sh now if everything works okay i hope it does then when i press enter here opportunity if you want to do something completely different which will be one of the hardest things you ever do for two years but you will get like i go back to the the picture like not these pictures um like These are the things you'll do. And it'll be super hard, but it'll be super awesome. And that's just, and you should only do it for two years. And I can tell you, I will only do it for two years. And then I will go back to zero homicide land

and have a timeout. But any more or are we? We have time for just one or two more questions. Hey, how's it going? So I feel personally like I filled out a lot of DoD forms and I put my social security number on a lot of DoD forms and forms that really didn't warrant it. But there's a spot where and you can't leave it blank. So anyway, we should be seeing a connection on the top left hand side of the screen on our attack machine. Let's try that. Okay. Yes. We this is great. because we just helped number 6 escape from the village and return safely back to his home. But let's see what we have here. Let's first take a look at the file system root. So

these are all, we've seen them already, and you remember the dash at the beginning. We can also look at our ID. And if you recall, we defined our user as UID 0 and JID 0. But we can also see who are we inside the host. So here you can see that we are... Seeing the sort of exposure that someone can... and the damage that can personally be done to an individual? Or do you think it needs to be taken a step further or integrated more? Or what do you think is the solution to that? Sure. So the DoD ID number... theoretically on paper the DoD has ordered that social security numbers aren't supposed to be used anymore using the DoD ID number. -

Logged in as user number six and we are connected through TTYS0, the serial TTY that we meant to connect through. So this all seems very good for us as attackers. But let me show you another thing. You see, we can also, if we have access to the host, As we said earlier, we have complete control over the Docker daemon. So we can, for example, run Docker PS and see all the Docker containers running on the host. And as you can see, there are multiple containers. We are only one of those. there are multiple containers running on the host. So those may be... That was the live escape. And now, as I promised earlier, we need to discuss what can we do when

mount is not allowed. Because mount is a very privileged command and it is probably the first command that defense teams would block if they get the chance. So, we had to find other solutions if we can't mount it. So, here we show a couple of other methods or paths that we have. And the first one is use-- - Clearance you might need? - No. - We have time for one last question. All right, thank you guys very much. - Thank you. - So what's the best way to get in touch with you guys? - Let me give you my card. Oh, cool. Great. But we do, we have this like weird cyber, people working on cyber policy. Okay. And it would be nice to do stuff

We do shit. So cool. Yeah, reach out. I'll reach out. Hi. Great talk. Thanks. I don't do cards. Harlan does cards. So Harlan's a good contact. So Harlan will be, or where is my, where is Roro and Claire? So Claire in the back is going to be the person who's going to, did you see Claire? Claire is your best contact. Okay, great. Cool. Hi. Hi, Erica. Okay. Great. Cool. Great. This session. First one is I hope you learned to know Prison Room Number 6. I know that I did. Second thing is, privileged containers are a thing. So if you're a red teamer, in your next engagement, try to look down this privileged container path. If

you're a blue teamer, it's something to look for. You should have the kind of visibility to what kind of containers you're running, and you should have control over what kind of control, who can deploy such privileged containers. And I know that we promised to show you how this how the series ends. So I'll play, but feel free to move on. I know we should end the session. And we'll be here for Q&As. Let's give a round of applause. What kind of blogs are you-- do you have a blog? Maybe you can--

Thank you.

three times at the end if they want to ask definitely i've allowed a bit of time so that's the only way to have them wait or just don't say anything yeah yeah yeah all right let's go ahead and close the door thank you all right good afternoon everyone this is the b-sides i'm the cavalry track if you are still talking please exit the room unless You are this gentleman here, Richard Manning. This is the certification and labeling of IoT. I'd like to go ahead and do a few quick announcements here. One, our sponsors. We would like to thank our sponsors, especially our Inner Circle sponsors. That's Critical Stack and Valley Mail. And a couple of our stellar sponsors, Secure Code Warrior,

Paranoids, and the NSA. Thank you all very much. Without their sponsorship and so many others, including the help of the volunteers, this would not be possible. So thank you. Another reminder, cell phones. Please turn off your cell phones during this talk. It is streamed, so we do need you to go ahead and mute those right now. Thank you. Without further ado, Richard. Thank you. So, hi. I'm Richard Manning from the NCSC in the UK. Thanks for having me today. We're going to be talking about red teams, IRL threats, incident response, and we're going to wrap up talking about how defenders can prepare. So, does anybody recognize this book? Feel free, pass this around, flip through it. Go ahead, grab it, go for it. Flip through, that's a

volume one copy of Hacking Exposed. It was published in 1999. There is no digital version of that book. I used to see it all the time walking around bookstores. Do you guys remember bookstores? Pretty well locked down. We just don't have any other means of getting information out there. And you've also got the time. to get that information out there. So kind of what the malware does, for example, if it's got time, low and slow as the talk goes, get the data out there and try and exfiltrate what can be done. Of course, ways to get around it. So currently, if we look at ways that data filtration can take place, It was published in

1999 and it really captures the hacking zeitgeist of the era. And right on the cover is a quote from Aleph1. Amongst other things, he was known for smashing the stack for fun and profit. Basically, it was the first step-by-step guide to exploiting stack-based buffer overflows. So this was published in 1996 in Frack. And I know, 1996 sounds like eons ago. but it was actually really contemporary to this book. And around that time, early 90s, sorry, late 90s, early 2000s, there were not a ton. DNS, there's the most common one, which is your DNS tunnels, as I mentioned before. There are many tools out there. If you just search for them, you'll find them. They tend

to be easily picked up by most things out there these days that are looking for that kind of attack and that kind of exfiltration of data. Other ways of doing it, of course, is if you've got binary data, you could base64 that. use it in subdomains and then pick them up on the other side. ... resources out there to learn hacking and security online. It mostly looked like this, like shady e-zines, super shady forums, like really sketchy looking places. There weren't, you know, threat reports to read or like bug bounty write-ups to read. It was basically this. And these were sort of the elite like underground hacking groups at that time. Late 90s, early 2000s,

I feel like really represented the end of an era. But at that time, inside books like Hacking Exposed, this was the typical... ...are being performed and convert them back into the data that you want. And the text records also very much a well-known attack as well. So what systems know, and by systems I mean anything out there from a product point of view that's trying to prevent DNS exfiltration, what they're doing pretty well is looking, for example, for abnormally long requests. Okay, so if they see a DNS request, they look like they're look like their first line of a corporate network. You got a firewall, a perimeter, a bastion host, the internet, and you'd always see these phases of hacking listed out. Recon, scanning, gaining

access, and depending on who published it, there would be some steps added or removed, maybe vulnerability identification or priv ask, but you typically see phases like footprinting, enumeration, password cracking, buffer overflows, clearing logs and stuff like that. So 20 years later, we're literally still using the same language to describe hacking and intrusions. So again, some of the steps are added, some of the steps are removed, but you can see that they all sort of follow this similar formula, this similar model. So fast forward up to 2011 and Lockheed Martin introduced the cyber kill chain. And you can see it's really not all that different. So let's pivot for a second. Shout this out if you

think you know it. Blank is the enemy of security. There is only one wrong answer. The wrong answer is users. Anyone have any guesses? The answer I'm looking for is complexity. Complexity is the enemy of security. Unless your company is really, really tiny, it's probably really complex, like wildly complex. Hundreds of users, thousands of auth attempts, thousands of servers, millions or even billions of logs. What have we done? So last year we published the UK Code of Practice on IT security, which I'll come on to in the next slide. Now that basically had 13 points. and three that would make up what we call the baseline. These are effectively no default passwords, a vulnerability reporting methodology, and a defined end of life for a given product. Now

that was developed with extensive input from the wider community. Josh and Beau were involved with that earlier on as well. And hopefully there's a general theme that matches lots of other standards around the UK will become apparent. Now, David Rogers presented on this here last year and has been doing the same thing around the world. My colleague Peter presented on this at RSA, an event at RSA earlier this year. So we're trying to continue that engagement with the wider community to try and make sure that what we're doing isn't completely out of sorts with the rest of the world. And we know from Mikko and Bruce Schneier are all saying the same thing. It's time

we listen. So if you're to make a map for a modern tech company, it would probably look something more like this. maybe something like this, it definitely does not look like this. So what do tech companies look like on the inside? Well, they're filled with SaaS. They are filled with SaaS for just about everything you can possibly think of. So the whole idea of a perimeter, you know, that's been gone for more than a decade, but it is completely gone at this point. So in modern tech companies, Windows is really on the downtrend. You know, there's always Karen in accounting that needs her spreadsheet macros, but we know companies like Google and Cloudflare give Chromebooks

to their employees. Some devs prefer Linux laptops, and it's really not uncommon to see large fleets of Macs in tech companies. So this is the typical view of a university, basically Macs everywhere, but it also reflects the broader trend of Silicon Valley. So being able to detect and respond to threats on Mac We produced the code of practice. It was delivered by a department called DCMS, Department for Digital Culture, Media and Sport, also known as the Ministry of Fun in the UK. That's available online. Please feel free to download it. It's been translated into numerous languages as well, so you can read it in any language you really feel like it. We published that last

year. So modern companies may be built on hundreds or even thousands of microservices. So if they're not built with security in mind, they can be just as vulnerable as standard servers, but I do find that they tend to reduce the overall attack surface while still letting developers rapidly deploy code and services. So modern companies, everyone's a remote worker at least sometimes. So it's really normal to have your critical apps and critical services available over the VPN for those users. That in turn means you are opening up your network to all sorts of users' home machines, their infected Androids, and you're getting logins from unknown places, unknown devices all the time. In particular, have really shifted

in the last 20 years. Let's take a look and see how red teams have been taking advantage of these changes. So I call these the not so new phases of hacking. The first thing an attacker might do is phish a user's credentials, phish their second factor, and attempt to add their own two factor device. They'll then gain some form of persistence that's not gonna get wiped out with a password reset. and they abuse the trust and the access they have to gain more credentials and expand access. Now in the case of a red team, they're gonna keep doing this, more creds, more access, until they raise the noise level until they're detected. So let's zoom

in on the first step, phishing. Anything that can potentially send a notification to a user is potentially a phishing vector. So there's a lot more out there other than just spoofed mail. Things like public calendars, public Slack boards or Trello boards, public code repos. Basically, anything that lets you at mention a user. 84% of favor is focusing on this baseline. Those are the highlighted points in the middle there, which is quite handy because that's where we want to start really, so they're broadly in line with what we want to do. 96%, which I think is quite a significant score, are in favour of government regulation in this space. What it doesn't show you is that there's lots and lots of disparity in how we regulate. So whilst they're happy

with that they want some sort of regulation, how we do it, where we do it, that's very much up for debate and different retailers, different manufacturers all got a different opinion in that there. So other highlights that came out of that consultation, which will be published later on. Lots of people looking for further clarification and definition on the technical terms of IoT. You look at any IoT standard, any website now, nearly everybody defines IoT slightly differently. And if we're going to regulate stuff, we need to try and rally around a-- LeCroy, you know me. You know me. Thank you, sir. And your badge actually will get you into the buffet for something a little broader

selection of vegetarian food yeah i hope you have some carrots thank you cheers so anything like public calendars anything that lets you mention a user and send them a hyperlink is potentially a valid phishing vector so these services tend to let you set your own display names so when you set your display name you can use it to fit your social engineering definition They want clarification, definition of how we're going to apply legislation. Would it be UK specific? Would it try and sort of go further and overseas? And then there's more detail required on things like the technical implementation of such IoT. Are we talking about devices that can connect directly to the internet? Devices

that connect through some kind of hub? Or devices that only become smart or IoT-like via an app? Where's the line being drawn on what we can regulate against? Pre-checks, maybe that's expressing urgency or using an emotional appeal. But the really dangerous part of this technique is that it bypasses tons of email gateway security tools. So it's super dangerous. So we know 2FA phishing has been happening in the wild for quite some time. I encourage everyone to check out this report from Amnesty International. It was co-authored by Nex, who is a really awesome Italian hacker. First of several that I'm gonna shout out in this section here. So we know there's tools out there for pen

testers as well. So Modliska's reverse proxy that claims universal 2FA bypass. It can easily intercept OTP tokens and pens to steal users' credentials. There's Evil Engine X2, it's a real heavy hitter in this space. It's another reverse proxy, but the Fogatic-looking phishes. And you can see from this diagram, there's a lot more going on than just grabbing browser form data. These near-transparent proxies are sending real data to and from the SAML endpoints. So this tool, Mirena, was introduced at Hack in the Box Amsterdam this summer, and it was developed by two really awesome Italian hackers. The first guy's name is Ope. He's the lead author of BetterCap, which is the man-in-the-middle framework to be using

right now. The other co-author is a guy named-- - Is that right? We know that's not probably the right answer, but toaster manufacturers, for example, aren't in that head space yet. Other questions that were raised up were about implementation. If we implemented any sort of rules and regulations, how would that work? Would there be a voluntary period? Would it be a couple of years of this is what's going to happen and then regulate after two years? Lots of questions around supply chain, development chains. A product that you buy in the shops today has obviously had probably a year, maybe two years worth of development and delivery time to get to your store. Where do we

regulate? Do we regulate at the retailer's point of sale or do we regulate at manufacturer? And how does that affect any standards that we put in place? The actual label we're talking about as well. If we start labeling, how is that going to look? How will the consumer understand what the label means? Do we need to use some-- - Author of the web application Hacker's Handbook, and he's also a core developer of Beef for 10 years. By a show of hands, raise your hand if you've ever popped a shell with Beef. Anybody? I love that tool. So when these two guys got together, they made a brutally effective phishing tool. I encourage everyone, whether you're

attacking or defending networks with 2FA, check out this talk. It is super, super scary. So, they released a companion app with Marina called NecroBrowser, and it sort of handles a lot of things that you'd expect a headless browser like Selenium to do. The focus here is on automated post exploitation of hijacked accounts, things like automatically backdooring accounts with SSH keys, automated password resets. So, I cannot talk about 2FA without mentioning this blog post from Trail of Bits. I think it is an absolute must read. There's so many standards changing in the 2FA space. We have FIDO, U2F, WebAuthn, and there's a lot of subtleties in between them. So give this a read for sure. Does

that work in practice? Another idea was proposed was negative labeling. Do we just promote or do we just talk about the products which don't pass the certification? Arguments to and from for that as well. And then there was also lots of questions about regulatory actual sort of assessment. Some companies or some organizations wanted us to push for independence of lab-based assessment of products. Some companies were much more pro favor of self assertion. I probably don't need to tell this crowd this, but if you're doing 2FA over SMS, you should stop doing that as soon as possible. I think a lot of us saw the story this summer with this guy lost over $100,000 from his

Coinbase account due to a simporting attack. So Fortnite did a really awesome campaign this year where they gave users a free emote for turning on 2FA and I think that's brilliant. If you're a defender and you can incentivize 2FA adoption, you should consider it. So another common tactic of red teams is hacking without exploits. This is something HD and Val Smith talked about more than a decade ago. Look, I love root shells, but ultimately hacking is a means to an end. It's not about those sweet shells. It's about the data. This is effectively where things could lead to. This is not defined. This isn't necessarily going to happen. But what it shows is that you've got lots of different government agencies in Europe, BSI in Germany and DIN, and

different industry groups, different manufacturers, all working together in different groups to try and get a grip of IoT security and how it could affect them. Now, I was at a meeting a few weeks ago, we were trying to merge certain standards together, and whilst, yeah, there was lots of discussion about actual specific wording or implementations, the general feeling was, yeah, let's all try and do the same things, try and make it easier for manufacturers and consumers to sort of value around one standard if possible. Otherwise, it gets too confusing, and I'll show you some confusion later on about why it needs to be fixed. But the underlying feeling is, yeah, consensus is there, it's just

a matter of trying to sort of get it all into one place. So as I just mentioned, We've done some mapping of the various standards. Now, it's against the UK code of practice, but there's lots of stuff in there which is non-UK specific. And it's a really good example-- this paper, linked to the bottom there, is a really good example of how IT standards exist and how they interact. David did this for his company. He's done a blog to talk about it. And it was updated this week. It incorporates various changes to the policies and standards we've talked about. and how they interact with each other. - Is individually sandboxed, or more likely they're running something like a Chromebook, which if you have exploits for Chromebook, you can easily

parlay those into cache. So in this scenario, a user's running a reasonably secure operating system, and they're on a network using SSO. When they've been phished, attackers have massive amounts of access. And once an attacker has 2FA hijacked, the first place they're probably headed is to the user's inbox. And once they're there, it's all about grabbing more credentials. They are going after plain text credentials everywhere they can, in the mail messages, in connected drive accounts, in documents. And again, if they have access to the inbox, they can control users' password resets. So when you have 2FA hijacked on a network with SS-- But we're confident we can get something done next year, depending on who's

in power. We're also going to continue to work with the Europeans to, again, make that a European standard as well. However, as we get closer to certification legislation, we've still got some questions we need to answer. based on the left-hand side there, we need to understand the manufacturers are delivering what they say they're delivering. They're not just playing lip service to these sort of requirements. We need to understand how to gain confidence in such certification, how to sort of build that into the process so that we know what we're doing. And importantly, we want to make sure that the floor doesn't become the ceiling. This is a phrase that Peter refers to. So we know

the three baseline reports, baseline requirements, they're quite low level. They're quite basic. We knew we couldn't go in there with the full 13 requirements of secure comms and stuff like that. manufacturers just wouldn't work straight away. - So, the VPN is your C2. You don't need to have a beacon on a server that's just gonna get detected. So a topic really important to both attackers and defenders is persistence. But how do you gain persistence when you haven't owned a computer or a server? Well, all of these apps have varying levels of access to the data in my Google account. Now, Google's really clear about warning users before you give these apps access. Many of them

have almost full access to the data in your account. And you can create OAuth clients, service keys. There's a number of ways to gain persistence. Probably a lot of people remember this incident from a couple years ago where a Google app named itself Google Docs and it requested access to manage email and this was spammed to tens of thousands of users. Now Google has gotten a lot better about locking these services down. There's still quite a number of ways to gain persistence on a cloud account that's going to survive a password reset. So you make it better for the manufacturers to meet those needs. And to do that, we need to push these two things

on the axes here. So we need to make vendors, consumers, different stakeholders, we need to educate them more about the cybersecurity of their devices, different ways to do that. And down the bottom there, we need to make them care more about cybersecurity. And to do that, there are various methods we can do, carrot or stick, really, and different types of approaches there to be put in place. Using G Suite, if you're using something else, there's other similar options in other services. Maybe it's an API key or an integration or a connected app. But once an attacker has access to a user's inbox, they have tons of ways of keeping that access. They can enable mail

forwarding. Malicious mail filters can be deadly. They can catch those password resets. They can also catch alerts and warnings from IT and security. So when you combine all of these techniques together, you make detecting compromises extremely difficult for incident responders. So at this point, I absolutely have to clarify something. I think exploits and implants are awesome. Attackers use them because they work. Infrastructure hacking isn't going away. But if you're a defender, you need to be able to detect these other types of someone that pops in excess on your company blog or does a subdomain takeover. So unlike red teams, IRL threats don't care what fiscal year it is, they don't care who's on vacation, they

show up totally unannounced. Although sometimes technically they do announce themselves. They're probably highly skilled and have access to some Oday. There's not much you can do about Zero Day. You can try to sandbox your apps, you can focus on detecting post exploitation, but if you do get hit with Zero Day, try to make the best of it. If they Consider that these attackers might be financially motivated. Think about the data that you're protecting. It might be worth $100,000, $200,000, $300,000 or more. Likewise, IRL threats are often well-funded. They can afford code signing certificates. They can afford botnet access. They can afford VPSs. And if they're using implants, they're probably using advanced implants. Okay, well, what

do I mean by advanced? I mean staged, packed, crypted, novel form of C2, novel form of persistence. Yeah. So yeah, that's one of the key points you need to understand how best to do that. The difference stands out there for gaining confidence in products, but are they the right thing for this kind of sector? - This might be jumping into this afternoon's discussion, but how do you anticipate sort of working with Anissa as they are looking to sort of implement this new cybersecurity act, and my understanding is consumer IoT, is going to be one of the early areas that they're looking for for certifications there. So how does the NCSC plan to continue? So we're working with them already, very much so. And we know there's

different time scales in play. We also, you might have heard of Brexit. We're not quite sure how that's going to impact all this. APTs are still out there using modified versions of China Chopper and modified versions of Poison Ivy. Shout out to APT10. That's not really what I'm talking about. IRL threats are highly organized. They're not putzing around on your network enumerating SNMP and trying to crack your Etsy shadow password. They go in, they get the data they're after, and they quietly leave. They are by far the most difficult threats to detect, and therefore they are the most important threats to detect. Doing so requires-- - Lots of options there, but we are very much

tied into that piece of work and support it. It's just they're working at different time scales. Anissa might take a couple of years. So this is sort of the vibe and the feel when you know that you have a confirmed incident on your network. Everything is burning. Everything hurts. But as an incident responder, you have to be ready to switch roles. As an incident responder, you've got to be ready to ride into battle. And I feel like doing IR is a bit like being a firefighter. Not that we're doing anything heroic or dangerous, we're not. But firefighters focus a lot of time on prevention and education. But they all know that eventually, sooner or later,

that alarm bell is going to ring. And when it does, you must be ready. If you're an analyst, you might be seeing dozens or maybe hundreds of alerts of a day. You have to take every alert seriously. Pulling the smallest thread can lead to something much more. All the training you've done, all the millions of dollars of security tooling, all of it comes down to this, responding to a real incident. And look, I think I'm good at catching hackers. If you think you're good at catching hackers, an incident is the time to prove it. It's a lot more than a cat and mouse game, but I think it's okay for defenders to have a little

fun hunting hackers. We know they're having fun as well. It's okay for us to have fun too. And it's really important whoever's leading the incident. I'm perceiving the correct information on that side. The other thing to look out for as well of course is try and mix the domains. So we're building that into this framework as well. So you can register a pool of domains that your bind server will listen to on the other side. And it can randomly just change those or it can use them for various things. So for example, welcome4.xyz.com if xyz.com is being used. and your bind server picks that up, that means it's the stock. Whoever's drafting the communications, whoever's

making the high level decisions, it's important that they focus on doing that and delegating work. Ideally, whoever's leading the incident is not doing technical work. So let's shift again. I want to spend a minute to talk about truth with an uppercase T. So this is a really weird image. If you look, you can see that there's a ship on the water, but it almost looks like there's another ship upside down floating on top of it. This is what's called a Fata Morgana. It's a type of superior mirage. And then you can send the MyDOM domain or any other domain and end it with something else, .xyz.com, and that'll mark the end of the binary data.

So that way you can build some kind of headerism so that when you're building the data on the other side, the system can make sense of what exactly it is receiving. Again, this is going to take a while depending how fast you can push data out. So if you're doing this directly against the DNS server, yes, you can do it quite fast. And it has been confusing sailors for centuries. The exact same thing can happen when you're looking at heaps of log data. So when an incident cracks off and you're looking through all these mountains of logs, you need to be really open-minded about what it is that you're even looking for. Try to drop

your assumptions. Try to drop your biases. The nature of an attack is going to be really, really, really complex. Your logs are really simple. They're just black and white. Try not to lose sight of the fact that your logs only tell you part of the story. That's going to take a bit longer. You can generally fire off the request quite fast against the SMTP server. But again, you want to try and play under the radar of where detection tools are going to start figuring out that you're sending too many mails and too much data out of the company. The other thing to keep in mind as well is obviously use a dictionary. So the system just takes any word list as a dictionary, but it's got to be stuff

that makes sense, right? So because statistical tools that are statistically analyzing DNS queries to look for DNS exfiltration are quite good. and with machine learning and things like that, they can be pretty good at understanding not just what's English. - They tell you the part that you have data for. And you know, when an incident cracks off, you know, everyone has their own pet theories, like, oh, I think it's this, I think it's that, but you need to embrace skepticism and change. The more you investigate, your theories are going to evolve. And I know when an incident starts off, like I'm like full of adrenaline and it's really exhilarating and exciting, but you have to

pace yourself because it's likely about to turn into a marathon. So logs are the lifeblood of both analysts and incident responders. It's crucially important you have the logs you need when things go bad. And there are countless ways logging pipelines can break down. Try to ensure you're getting all of your logs all the time because you are desperately going to need data. Every single OS command executed in your company needs to be logged. This is something OS query can help with. every DNS request, every email link click. And the more log retention you have, the better. If you have a month of logs, that's good. If you have three months of logs, that's great. If

you have six months of logs, that's even better. The more log retention you have, the more of a baseline you can build. So let's talk about standardization for a second. This is what will really bring your detection and your response to the next level. There is a really good reason why militaries push conformity and push discipline. It keeps people in sync and it helps minimize errors. What am I talking about standardizing? Just about every part of the incident process. The acronyms you're using, you want everyone on your team obviously to be using the same time zone, the same terminologies, even down to the detections and the alerts that you're writing. So I had a couple of coworkers

give a talk at Circle CityCon this year. It was all about standardized alerting pipelines. And really it's just about how we use tools like Confluence, Jira, Git, and Splunk to standardize the detections written by disparate analysts. So that way we can all read and understand each other's detections and we're all speaking a common language. Check that talk out. So when you're documenting, it's really important to document how you found something as well as what you found. And always, always, always include context in your investigation notes. You don't know what assumptions you have or what assumptions the next reader is going to have. And I think it's really important to create a timeline when you're investigating an incident.

Not only is it going to give you a holistic view of the incident, but you know executives are going to ask for it. And I think it's really important to maintain a list of unanswered questions. When you have a group of people working on an incident, different people are going to know different things. So I cannot talk about standardization without talking about the MITRE ATT&CK framework. This has been one of the best resources in recent years to standardize ATT&CK terminology. Now, it's more of a knowledge base than it is a descriptive set of phases or steps, but I find it incredibly helpful. So let's talk briefly about containment. If you look through Hacking Exposed and old books from the 2000s, they describe containment as the

second step of the IR process or the third step of the IR process. I find that this API token that's customer facing and in production, how quickly can you get that credential reset if you need to? A lot of these things require working with other teams. And again, there is a lot more beyond password resets. If a skilled adversary compromises a user's account, they have a huge number of ways to persist. They can download 2FA backup codes. They can change password reset questions. They can modify source code. So what can you actually do to prepare for this stuff? Consider implementing bug bounty programs. A lot of people find value out of bug bounty programs. A lot of

people find they have a really high return on investment. Atlassian won the bug crowd program of the year this year. In total, we have given out more than $400,000 to hackers and security researchers, and we have gotten tons of bugs remediated. So zero trust, it's not the easiest thing to implement on a big network. It can be pretty complex. But the idea here is that it's an IT security model that requires strict identity verification for every user and every device. And the emphasis here is on least privilege, restricting users to only the data that they need. You should be writing detections, lots of detections, not just detections for low level hacking tools, but you should be writing detections for anomalous behavior and

detections specifically customized to your organization. And if you're ever having to do incident response, you should definitely be threat hunting. You need to be threat hunting for the compromises on your network that you're just not yet aware of. So talking about hacking, Talking about incident response, it's really easy to get lost in the technical bits, but doing security is about working with humans. It's about working with other teams. So this is something Werner Herzog says to all of his film students. Read, read, read, read, read, read, read. This applies equally to analysts as well. Read threat reports, read bug bounty write-ups, read the New York Review of Books. It doesn't matter. Read as much as possible. It will only help you as an

analyst. And look, you need to understand how your company makes money and how it works if you want to effectively protect it. I think it's important that you interview users that are, you have to be comfortable working with teams outside of security. I'm talking about legal, privacy, compliance, all of these different teams are going to help you when an incident does arise. you should consider QTRA, quantitative risk analysis, can help you get an understanding of what your most significant risks are and also what your accepted level of risk is. And there's a lot of established frameworks out there that can help with this. And so important, as a defender, you never ever waste an incident.

And by that, I mean you get the most out of it. Maybe it means asking for more budget. Maybe it means asking for more headcount. Maybe it just means asking to get those old backlog tickets worked on. But as a defender, you make... every incident and you make the full advantage, take full advantage of it as possible. So a few takeaways I want you guys to remember from this talk. If you're a defender, you should feel comfortable thinking outside the phases of hacking paradigm. Exploits and implants are awesome, but attackers don't need them when they can abuse...

Trust and least privilege. Defenders must be able to detect threats that do not use exploits or implants. You need to have a reliable logging pipeline and a focus on standardization is what's going to bring your detection and your response to the next level. In conclusion, it's been 20 years since Hacking Exposed has been published. Guys, protect your networks like it's 2019, not like it's 1999. That is my time. If we have time for questions, let's take them.

Or, you know, lunch. I have a mic if anybody has any questions.

Hey, great talk man. Thank you. Yeah, so I don't focus on security. I'm more of a software engineer. Of course I should be, but what's the recommendation that you would give to software engineers? You know, security wise. First thing I would say is yeah, read the bug bounty write-ups. Like whatever your software stack is that you're developing for, there's surely bug bounty write-ups for people that have exploited that software. Any other questions? Alright, thank you so much. Not quite HD Morse speed, but... Glad you could make it, dude. Got invited to the Paranoids, and I think some co-workers are going to be linking up. Yeah, definitely. Thanks for talking to me. Thanks, man. Timing is perfect.

Awesome. I'm trying to keep a pretty quick pace. Yeah no really good job. Hey. Sure. It's really tricky, especially when it comes to phishing 2FA. Educating your users helps. I mean, we know that training isn't perfect. No matter how much you train people, someone is still going to click something. It's a really hard problem right now. A lot of the effort should probably be focused on detecting post-exploitations, like the ones that already have access to the account. It's really not easy, especially if you're using Google accounts like G Suite. It's not easy to get those logs into a SIEM right now.

But implementing zero trust, I don't know if Google created it or if they just popularized it, but... Look up zero trust. The whole idea of zero trust is you authenticate not only the user, but the device that they're using. So you might end up installing certificates on people's computers. That way, even if someone not only gets credentials, even if they get a second factor, they need to have a device that has a certificate on it in order to authenticate to the network. And then still, those users should be restricted to only the specific apps that they need. If you don't have zero trust right now, it's just like...

Yolo, hey, if I can connect to the VPN, you can get it. I don't necessarily have strong opinions one way or the other. I'm sure they're all good, but we've had really good luck with BugCrowd. Yeah, thanks for coming to talk. I really like your blog. Oh, awesome, man. Thank you. Yeah. I try to make something a little bit different. I don't know. I like making talks, like the talks that I want to see, like more quick pace and just, yeah. I'm secretly taking my company. Yeah. I think you guys probably don't meet those over. Yeah, yeah. It's a lot of stuff I like. I feel like some people already know this stuff, but I

have so many new employees, like literally 21-year-old interns, never heard the word frack once in their life. Yeah. They don't know what CDC is or anything like that. So I think it kind of helps to give them a little bit of the history. Awesome. I will ignore every other comment and only hear that. Thank you. I super appreciate it. So I wanted to check and see, do you have any contact information? I'm going to get a copy of it.

I'm pretty sure it's okay for me to give out the slides. There's nothing too sketchy or shady on them. Yeah, they have some contact info on like the last slide, but I just flashed it up real quick. Not many business cards or anything, but I can definitely send you my email and my info. I've definitely considered it in the past. I...

You've got to fix this right. You should value your freelance so much. I like that. I like that. That's a good pitch. That's a good pitch. But I have definitely done freelance work for my friend startups. Like, oh, I have this photo app or like, oh, I have this new recipe app. And then like, I kind of step in, you know, before they do like an MVP or before they get uploaded to the Play Store or whatever. I'm like, can I just make sure you're not... I'll email them to you. I'm also interested. Great talk, by the way. Thank you. Thanks so much. It's so obvious. Verses are like, oh, I'm going to have slides. I'm

going to read my slides. Thank you. I'll just jot down what it is. It's probably OK if I take this. I'll give my work one, and then I will also give the first one. If you can read my encrypted handwriting, then feel free. Oh, just encode it. Oh, good idea. I was going to say, I could also type it in. That's probably more legible. I'm streaming live right now. I am the happiest employee. I'm the opposite of this rumble. I think it's pretty normal for people. Actually, my team is really cool about they want us in there where our value is in the industry. So we're pretty open and pretty chill about chatting around and stuff like that. We're

using gear for the same thing of course we got a drink our own champagne but So for our vulnerability management specifically, so we are in Expos, we have a pretty big install of Expos, and I gotta say, it's tough out there for vulnerability management. Like, before, last time we worked for HP, and at HP we had the biggest implementation of Expos ever. It's tough. It's really hard. Like, servers go up and down in an hour. And, like, to scan, you know, whatever we have, like, 10,000 servers, like, it takes more than 24 hours. So by the time the report's generated and things scored and then it gets emailed to the service owner, you know, it could...

You know, hey, that AWS server's been offline for 12 hours already. We don't have a big dashboard, but we have been working with Nexpos to get tighter integrations with Jira. For us, the actual struggle is... mapping, like, we know what our IP ranges are, but it's mapping some random AWS EC2 server to an actual owner that's going to respond to it and see, you know, hey, this has RCE and it's exposed to the internet. Yes, it's only staging, but no. It takes this right now. Like, put an SLA on it. But there's nothing really super special. I don't know if my coworkers are giving talks about it, but we have spent a lot of time trying

to get NextOS to work really nicely with our systems. We're a big Splunk shop, so we use Splunk a lot, but it's mostly Jira and NextOS. We have them married together. Jira, no one is like, Jira's my favorite tool ever. But I will say, it is super. And so we've tried to just do as much with the APIs as possible. If you have less than 10,000 servers, it's probably a lot easier. And if you can solve the problem of how do I map this random EC2 server to an actual human being that's going to respond to a Jira ticket, that's the hard part, I feel like. And then just routing the data around is kind of easier.

But yeah, if you could get all the responsible parties to own up to it, then you've really solved the hard problem of bold scanning. It's tough. One of my favorite phrases, and I wanted to include it in this talk, there just wasn't really a spot for it, is bold scanning is not patch management. So I have been crying... You build your own scanners for just detection with MNUT, and you have to have a... No, dude. And a... I get it. It was bad at HP. We were literally getting tens of thousands of reports and it was wrong operating system. They were showing us IIS bulbs on Nginx servers. We were like, how is this even possible?

That was built on Nixbox, but I don't fault any one of their products. InMap is trying its best.

Best practice Yeah. Yeah. Check one two three four five six seven eight Check test Testing Test Check check one two. Test one two. Yeah, but why am I not... I see myself. Check one two. And I sound like poo poo. I'm just... Ooh, what the front... Check one two, three four, five six, seven eight, nine ten, eleven twelve. Check one two, three four, five six, 7, 8, 9, 10, 11, 12. Ladies and gentlemen, please welcome to the event Bert Kreisler. Check 1, 2, 3, 4, 5, 6, 9, 10. Check. You're welcome. Check 1, 2, 3, 4, 5, 6, 7, 8. The one and only. Okay. So come just walk past me like this and keep talking. So,

hey, today... Bologna sandwiches. Bologna sandwiches. No cheese. Hold the cheese. No cheese. Extra mayo. Bologna sandwich. Extra mayo. Don't lean into it. Yeah. Hello? Hello? Hello? Hello? Test one, two. Check. One, two. Test one, two. Hey, hey. Hey. Hey hey one two three four five six seven eight nine. Hey, hey, one, two. Dude, thanks. I appreciate it. Cheers.

I was talking to a talk that had a really silly title. It was called, Excuse Me, Your Sword Is In My Eye, Responding to Red Teams and IRL Threats. So it was kind of about modern incident response, and I was like, I can expose a volume one copy from 99. Just sort of talking about how we kind of need to get away from the faces of hacking paradigm. You know, recon, scanning, gaining access, maintaining access. Like, guys, do you not realize, like, you can get destroyed without following any of those steps. A lot of it was about 2FA fishing and networks with SSO. Check, check. One, two, hey, hey. Test, one, two, test, test. But what if I... Let

me hear. I think that's okay. Yeah, you think that's okay? Well, you're good on that side, but I mean... Hey, hey. attacks that do not use exploits and do not use implants. So that's the most fun part of hacking though. I want to use my root exploit and drop my root kit and do all that fun shit, but attackers don't need to do that. A lot of times they get that, they fish the 2FA credentials, they have access to your VPN, just get access on the data. Like the entire network. What do you say to customers when you focus on their secrets? Yeah, like people that are using whatever.elassian.com. But I mean, even still, people that are using server, the stuff that we find when

we find vulns and we do these awesome bugbashes and find all these sweet criticals, that'll make it down to a patch will be made and that'll go out to Jira server and all of our other server products. But if you don't want to wait...

Be hanging out in the breeze for three weeks while that patch is coming down and your admin is trying to find downtime to install it. Like cloud software kind of makes a lot more sense. If you're actually using it as cloud. Right. So are you running on, on VMs or containers? Oh no, we're, we're Amazon shop. But like, you see two or three days? I think, so here's where I can start getting a little further away. A lot of things are built on a set of microservices, like auto-scaling microservices. So I try, I know about... I'm sorry? Like Lambda and that kind of thing. I don't think we're using lambdas. All I know is that we're using microservices

in conjunction with auto-scaling groups. And it's sharded out really nicely to where like if all of a sudden someone starts getting hammered with traffic, we can scale pretty easily. But yeah. So your idea of patching is just build those microservices, right? Oh yeah, no, it's totally continuous. The whole thing is like fully continuous. And that is one of the really cool things about microservices. Like I think microservices low-key are one of the biggest security improvements on enterprise networks in recent years, and nobody's talking about it because it's not all that interesting. I think how many enterprises are even doing that? Big ones. There's plenty that are throwing a service in a container because it's easier

than getting it, right? Yeah. In terms of ephemeral and CI and stuff, I think most companies that are even using containers are just having long-lived campaigns. Yeah, and it's probably, you know, and the focus is kind of more on like slightly bigger companies, maybe someone with slightly more maturity. Like I have a friend that works at Uber and I'm pretty sure he told me there's more than 2000 microservices there. So like people are really adopting them. Like it's, it's people are realizing or like spin up a Docker container and you can do like my one or two cool things and then it can connect to everything else. I don't know much about the container side of things because my background has all been sort

of big in the east coast enterprises not sort of the day stuff where everything is much more active in the architecture but what are the Yeah, no, I can hang out for a minute. There's definitely a lot more to it. We have a whole team that does our microservices and on top of that we have other like meta services that scan every individual container for like common misconfigs and common vulnerabilities because Like I said in my, something I said in my talk is like, oh, like, hey, if they're built with security in mind, microservices can be, if they're not, they can be just as vulnerable. We just wrapped up. We're still hanging out. Yeah. But

let's see. What were we just talking about? Like securing microservices? I mean, it takes a long ways to get to that point where you're actually able to implement tools to scan your own containers. But like, I think even a not great better than having developers. I mean, think about the alternative. Those 2000 microservices at Uber. Imagine if those were 2000 EC2 servers, all that need patching. all that have outdated libraries, all of them that can get popped with root kits and web exploits. It's like when you take all that tax surface away, like yeah, you can still hack microservices and maybe still get access to the data at the end of the day. But like you're not

leaving RCE hanging out there. That's for sure. Like maybe you expose one or two API endpoints. It's not like you have like, whoa, I have I've seen you're on Shodan now. Like let's go on a Shodan Safari and find all your servers. So at least from that point of view, just not having ports exposed and dealing with that, it's a big improvement. Yeah, I think they make a pretty huge improvement, but you know, My red teamer who was sitting in the front row there, if she wanted access to something in a Microsoft, she will get it for sure. But for the average everyday amateur hacker, I think it... really can help protect, especially an internal network. A

lot of our microservices are strictly internal services only. But there are things in the past that would have been like, we need a database server, and we need an application server, and we need to somehow connect these. And with our microservices, we make one change, and it affects every microservice. So we implement encryption at rest, bam, we just got 500 services all using encryption at rest by changing one big thing. Same thing for scanning for vulns. It's like whatever we do in our microservice architecture, we can then do it for every microservice. So it makes fixing things like way easier. How's the screening going? It's been all right. It's amazing. Yeah, for sure. Yeah, we

have one. Yes. Okay. Sure. Yeah. I actually, in the four years that I've done it. Yeah. No, I'm sure. But also, like, the thing is, because, like, the safety option, usually in the road, we have to be inside to make sure. So we don't really get to. I mean, I always watch the recording. Smooth, yeah. I mean, typically every year. It's only, like, every year is always, like, fairly quiet. It's more so, like, at midnight and half. We're on, like, the graveyard shift. Because that created, and sometimes. I don't think I need it. Or is that concerning? I think for the most, I'm more concerned about the view.

Oh, okay.

Yeah.

Okay, so this goes on my shirt? Yeah, you can just pin it to your shirt. Why don't I, like, know how to do literally anything? No, it's like the more technical stuff we do, the harder basic shit gets. That's a generous interpretation of my overall incompetence. Alright, let's see how that works. Oh, okay, hot. Alright, so I'm supposed to ask you...

you would like to be introduced with my name so you are katie how is your last name oh it's ladue but i accept any pronunciation okay i don't want to just be like here's katie ledoux you know what i'd be chill with it but okay so i'm going to introduce you and do my like sponsor announcements that i'm required to do And you can say from Rapid 7 too. From Rapid 7, okay. Okay, yeah. That was nice of them. And... Okay, they're not on our sponsor list. Oh my god, we're not? No. I think they're rude. Right? I'll drag them when I get home. Yeah, you should. You should be like... I love to drag.

You guys even listed in sponsor list at the beginning of this talk. It was embarrassing. Yeah. This makes me just look bad in front of all the other hackers. Exactly. Alright, so you have a 25 minute talk, right? Yes. Okay, so...

If and when you start running close to time, I will have this sheet. I will be flipping it. Hopefully I will be paying enough attention. So I'll give you warnings at 10 minutes, 5 minutes, and 2 minutes. And then I'll just hold this big thing up and kind of wave it around. And then at the end, we will do audience questions where I will hand the mic around to people. Do I have time for questions on top of the 25? I don't think so, because I think we have a 14:30 talk. Yeah. So maybe budget in like five minutes for questions. Yeah, I can do that. We can probably stretch it for like two or

three if we run really close. Sure. Yeah, that sounds good. Oh yeah, and then we have this. Oh wait, I have my backpack. Okay, I'm just going to put this in my bag before the mic turns off. Yeah, now we have like seven minutes. Cool. Oh my god, maybe I don't have a hanky in here. It's so off-brand for me. That should have been my unreasonable speaker request. Honestly, I stand a handkerchief. It's true. It would be very practical. Breaking news, I have one. Thanks.

It should switch in a second. It just takes so long to switch man. Nice.

How do I pronounce this? You can just say DFIR. It's Digital Forensics and Central Spotted. DFIR? Yeah, that's right. DFIR? I'll be at the back. Should I sign up 10 minutes before the end? Okay. I'll leave the questions. Like 10 minutes is fine. 10 minutes. Okay.

- Test, test. - Live, except for underground. As a courtesy to our speakers and audience, we ask that you check to make sure your cell phones are set to silent. And if you have a question, we will have you use the audience microphone, which I am holding, So YouTube can hear you. Please raise your hand and I'll bring it over. For this talk we'll do... Do you want to... Oh, there you are. Do you want to do questions at the end or questions in line? Oh, questions during it are great too. Okay, so questions during it are fine. If you have one, raise your hand and I'll... ...of looking at a few security topics that I happen to feel are pretty critical and

maybe we lose sight of some of these things because they seem too simple. Going from there to talk a little bit about IoT security, some of the common problems, examples of that. I've been researching for about 30 years in security and I'm going to go into that in just a second, but really what I'm going to be presenting is sort of a culmination of a two-year long research project where we were through both surveying and active testing. kind of assessing various aspects of hospital security. So not just med device, and by the way, this is not a med device security talk. You're probably all relieved of that, but we're going to talk about that a little bit, but it really is looking at a hospital

facility as just overall a basic system and finding ways to exploit the system. And with that, let's get started. Welcome Katie. Is it happening? Can you hear me? Yeah, good? Yeah. Okay. Let me know because I can yell a lot too. So I am Katie Ledoux. I am the Senior Manager of Trust and Security Governance at Rapid7. And we are all here today because of a fun thing called information security questionnaires. We know them. We love them. And for anyone who is not familiar with what I'm talking about at all, third-party security risk management is requires collecting a ton of information from vendors. So if, for example, you want to buy a Rapid7 solution, you very reasonably might want to

know how we might be protecting any of your data that we are storing. But how do you or... As they said, ask away. And if I'm going too fast, just let me know. As far as learning objectives, I'm not going to read through all of these. I imagine you guys will be able to download this slide deck some way through the event, so you can read through these. Your risk management team collect enough information to make that call, to really know, can I give this vendor the green light or are they, you know, are we taking on too much risk by working with them? Today the way a lot of these companies go about collecting that information is they send a giant infosec

questionnaire to the security team at the vendor company and like everything we do at rapid7 we want to address these questionnaires in the way that we think is best for our customers and intuitively we might think Doing what's best for our customers is the same as doing exactly what our customers whether that approach actually is what is best for our customers because there are some consequences to taking this approach and So one of those consequences is slowed security maturity. These questionnaires are very time consuming. I actually am just wondering who has done a security questionnaire? Who's worked on it? Okay, so you know the problem statement. It was everyone, YouTube. So these are very, very time consuming, as you know, and while you are working on them,

there is a ton of work in your backlog that you are not touching. So is deprioritizing meaningful security work doing what's best for customers? Methinks not. Problem number two, our team is not engaged by this work. This is a team of security professionals. Filling out spreadsheets is really not fulfilling to them. So is letting some of our most talented security professionals leave our company doing what's best for our customers? Again, I do not think it is. Risks. But, you know, I could take these games that I wrote in BASIC to school and sell them for $5 and they have to pay me to get the little code and all that kind of stuff. So, you

know, it was really cool to get into that and have an opportunity at that age to start actually writing software and even trying to sell it. But I want to talk a little bit about that and how that led me into security. Just some sort of screenshots. It's difficult to get screenshots off of a Tandy 1000 monitor. I started getting a bit paranoid about security with respect to software when one of the kids in school said, "Hey, my dad cracked your game and found your code. I want my money back." How did you do that? I kind of knew, but fell into it because of that. And thirdly, there's some quality control concerns here. So human error happens and I know from

experience it's very likely to happen when you're three quarters of the way into an 800 question questionnaire. So in addition to the fact that it's just kind of, it's grueling, you can make mistakes, some of these questions are really ambiguous and vague. You know, a lot of these companies are sending the same questionnaire to every company. So if there's just a generic... I started building my idea of copy protection and warnings into software. And, you know, warnings like this, you will lose all files at your own cost. Then when we actually look at some of the code, I'm doing things like just plain text values. Obviously, this is tangential in a way to the overall concepts

and concerns that we have today. - About what are your password complexity requirements? One analyst might read that to be our corporate environment password complexity requirements. Another analyst might think that means what are the password complexity requirements for a user who's logging into our solutions? And there's just, it's too subjective, there's too much opportunity for misinformation to get in there. So again, I It's another way that I don't think this approach is really what's doing best for our customers. We don't want to be giving them misleading or confusing information. So with all those things in mind, we thought that there might be a better way to handle these inquiries. Something that was both better for

us and also, most importantly, better for our customers. So we started down a new path. This is also creating documents that address our security FAQs, the stuff that you see over and over and over in all of these questionnaires. And we had a plan. When a customer asked us to fill out their questionnaire, we would send them our documentation and we would ask them to review that first and then let us know if they had any remaining questions. The documentation that we included in this sort of, you know, like security package on, you know, not necessarily hacking a specific system, finding an exploit in a specific process or system, but rather holistically looking at things kind of within line with the ideas of threat modeling. And we're going to

be getting into that in just a couple of minutes. And of course, the hospital security part that I mentioned is really where we're going to be focusing on as far as some of the latest projects that I've worked on and some of the interesting findings that stemmed from that. So I just wanted to throw a few key points out, primarily just now that we're getting notices of breaches and compromise pretty much every week now. The media has shaped things in a very interesting way that lead a lot of people to think that hackers are magical in some way and that maybe they sit down at a particular system and within 30 seconds, like on Swordfish, First, an overview of the solution, some information about the product. You

have to remember that the team that is doing the risk assessment is probably not the same team that is buying your solution. So giving them some context is great. And also remembering you don't need to sell them on the business value. The risk management team is not the one probably deciding which vendor to use, but they can stop you from using a vendor that's very risky. information that is relevant to them is not really the business value, but more what data types are being collected, what's the data flow, if you can do data flow diagrams, those are huge. Required integrations and permissions, also as someone who does vendor assessments, these are always impossible to find.

It's like my third email to some random person there where I finally find out what permissions this solution needs. And any security features built into the product, does it integrate with the single sign-on solution provider that you're... So, um... We also have the idea that a lot of tools that we utilize today, threat modeling, other processes for planning security strategy, those work for attackers too, in analyzing systems and finding weaknesses. Far-fetched is no longer relevant or acceptable as something that I've kind of grown up believing because if you guys have had any experience with red team exercise, - Features in there, that sort of stuff. Of course, secondly, overview of the InfoSec program and controls. So, white papers are my favorite way to just give

a big picture overview of the program. It gives you information in a much more digestible content than like a eight bajillion line SIG questionnaire. We have a white paper for our corporate environment and for our product environment, which is a helpful breakdown for customers. Standardized questionnaires, if you've had to fill one of these out for a customer and you actually feel really good about the content in there, like it's been reviewed adequately. I've had a word pad and then you saw all of that going back throughout the whole class.

Another thing to be aware of is the background activity monitor. So this has been in Windows 10 since the fall 2017 release. This is within the registry, so in the system hive under the current control set. It is an actual Windows service that was created by Microsoft. So the name of the service In the in the registry here is BAM. This is going to also record all the applications that executed. It's going to have the time that that program last executed and what's nice about it is it's also going to tell you the user or the SID in particular responsible for that program running. So this is a screenshot of that out of the registry.

You can see here we are under the services key. This is the BAM service key that we saw before. And then under user settings, these are the SIDs for all the users on this particular machine. And if you click on one, you just get this full listing of all the programs that ran. And like I said, you can also get the timestamp as well. There's also recent apps. So this is another one where we get all the programs that executed on the machine. We get the time of the last execution. We also get the number of times that that program ran. So especially when we're doing like those insider threat cases and someone has like CCleaner on their machine and they say, I just downloaded it

once and didn't really know what it did, didn't use it again. But then they ran it 500 times since they downloaded it and continually wiped all the artifacts. It helps in those cases to prove them wrong. It's And what's interesting about it is it's meant for monitoring system resource usage by applications. So for every application that does any type of network connectivity, you'll have that logged, how much data that particular application sent and received across the network. If applications use significant resource like CPU resources, that will get logged. All of the push notifications that pop up at the bottom get logged as well as energy usage. So there is this tool on GitHub called srumdump.

Really different than, you know, some of our Fortune 50 companies. So I do feel their pain in that respect. But we still got this 15. We're filling them out from scratch. So they're getting the accurate information they need to do their assessments while we are getting so much time back to actually do security work. And another plus for everyone is the process of doing the security assessment is actually going much, much faster now. So this is 2017. This Green line here is the number of tickets. So you can see as we're getting into the end of the quarter, not surprising, we're getting more and more tickets, more and more deals. And in Q3, when we start pushing back with documents, we are, despite the

fact that we're getting more tickets, we are still cutting the amount of time it takes us to close out those tickets by more than 50%. So that's great for customers. I know from experience doing vendor assessments, I don't want to be a blocker. I want to get the-- I have the reference at the end of the slides. You can pull that database from the machine, run it through this Python tool, and it's going to produce an actual Excel file. So this is not like a comma separated file. It's one of those XLSX files that you need Office 2007 or greater to view. But it breaks all of the data out for you. So these are

the tabs at the bottom, the network usage, application usage like we saw on the last slide. And then this is an example of network tracking. So you get the full path of every application that made it as I can to enable the business. And obviously sales loves this because cutting five days out of the sales cycle can be the difference between a deal getting done in Q4 or getting pushed into Q1. So everyone is very pleased with this. Amazing progress, but there is still more that can be done. We are riding the high of the success and we're still wanting to make improvements on it. So, Up next, we make a little change in our...

This is the SID it was running as at the time, and then the number of bytes sent and received by that particular application. And it's very similar, I didn't want five slides of Excel sheets, but it's the same thing like for this, it'll tie the data back, network connections and so on. Again, really telling you everything that happened on the machine, and it's always nice when we can tie it back to a particular user. There's also the AM cache. This started with Windows 8 and is still there through Windows 10. How many people have used this in investigations before? Some people look very happy that there's a slide. So this is one of my favorite artifacts now. This is something that every investigation

we do. Initially, these questionnaires were coming to us through sales and through customer success, whoever's customer facing, and going directly to InfoSec. We are adding a new team into this workflow. The content strategy team at Rapid7 does a lot of sales documentation, so they fill out RFPs, scope of work documents. They are documentation professionals. So questionnaire comes in through sales, goes to content strategy. They push back with the documents. 50% of the time, we never deal with it again. Love it. The other 50% of the time, if they have a few follow-up questions or needs to do the whole questionnaire, the content... You know, tend to just not even know where to get started when it comes to threat modeling. When you have Microsoft's models

and all these different nomenclatures, it becomes overwhelming. So I want to talk through a few principles that I've kind of adhered to that I think make that exercise a lot more straightforward. Before we jump into that, I made reference earlier to quality. I know everybody is familiar with this, but it's always fun to just remind everyone how much issues cost to fix in production. We know that that's really where all of our cost is going in terms of fixing bugs that made it past a certain phase in the SDLC. we kind of think about that from a quality perspective, it kind of goes without saying that software, quality software, can be secure software. Not necessarily, but the two definitely play a very,

they have a very cohesive relationship. I'm gonna jump past that. And so when we talk about just step-by-step threat modeling. There's lots of examples. I just kind of pulled this one. There's plenty of variations and permutations of these steps, but the main ideas kind of remain and those main ideas are the ones that I found typically get sort of lost in the rush when trying to plan and really develop solid security strategies that make sense for what we have and what we're... hey, the security team says they won't fill this out, but I'm not sending over this documentation. They told us to fill it out, someone needs to fill this out. And that gets escalated too. And also, I totally get where they're coming from. I'm not

the person who has to tell the customer we didn't fill it out. That's what would entail. And that can be an uncomfortable experience for them when they don't have this big picture context. So once we got that leadership, that buy-in from sales leadership, everyone was on the same page by like, oh, no, no, no. We know why you send the documentation. Like, actually, customers love the documentation. You should do that first. That's the process. We know it works. But we did that pretty reactively, whereas if we had thought upfront about what teams need to have buy-in in this new workflow for it to actually work, we probably would have done that a lot earlier on.

Obviously, partner with marketing. You're making documents. Make the documents look nice. Make the content digestible. The point here is to... Somewhere that it could actually be used. We're achieving this, certainly. Things are getting better. But if we don't have that overall understanding, we really can't possibly think that we could effectively secure that system if we don't really know ourselves how it works. It sounds very simple here, but when you sort of extrapolate that to a multinational project where you have teams upon teams upon teams working on this stuff, we see a lot of that happening. The collective knowledge is strewn throughout different organizations, different document systems, learning systems, and all that stuff. And again, it gets lost. So, not that I'm trying to preach to you guys, but

just something that I've found along the way plays a very big role in understanding how to secure a system. And then decomposing the application or the system. Classic problem. Make our customers feel confident working with us, so having nice looking documents helps. Obviously, automate, we do this as security professionals wherever we can because we're very busy. We had a nice macro in our ticketing system that would just automatically attach all the docs. We could customize it based on what type of a customer they are. Lots of opportunities to automate this workflow. you're going to want to assign owners to every single piece of content. Imposing the problem as much as we can, identifying components and the blocks of the system, how they work together. Threats, I don't want to

really go do a big talk on threats and all the different ways that we interpret what a threat is or threat agents and this stuff, but it's definitely a part of the system. It's probably one of the more difficult elements of a threat model to really sit down and affect. Whether it is in your knowledge base or it's a white paper or something else, your security program is changing all the time and you want to keep this information up to date and it's really a lot of work to do that so it really needs to be someone's actual job, like an actual responsibility. And lastly, make the content self-serve wherever you can. Why not? If

someone can complete their entire vendor assessment just by looking at stuff on your website and you never engage with them, great. Like, that's better for everyone. You can do security work. They can get their whole vendor assessment done without having to, you know, go through sales and then security and then, like, that's wonderful. So... We have trust.rapid7.com, everything that is from that security documentation that is available with what's next. Most important thing that we're going to be focusing on next, there's actually two enablement sessions. The content strategy team, the sales documentation team is like very, you know, we work at a security company so this might be unique, but they are really into learning more about security and active projects

we have going on. So we're doing enablement sessions with them so they can answer questions in a really thoughtful way and reduce the amount of questions that are trickling through to the security team. And I'm very excited about... But we also know that Batman has threats. The Joker, etc. The part that I haven't really spoken to yet, and this is really fundamental to taking the output of a threat modeling exercise and applying it to your strategy, applying it to budgeting for security spend, but the protection, the controls and other security mechanisms that make sense for this system with respect to the assets and the threats that are targeting them. Very simple, but I think quite meaningful in terms of how all of these things fit

together and how we could apply it to something a lot larger and of greater consequence. And before we do that, I wanted to talk about just... Sessions with sales. So what we're working... They are totally bought in on the documentation, by the way. We just had a quarterly business review with the content strategy team and they said that when sales comes to them now, they're not saying, "Hey, will you fill out this questionnaire?" They're saying, "Hey, can you send over the docs?" So they get it. But what we want to do now is work with them to move the vendor assessment process earlier in the deal cycle. So instead of waiting for the deal is about to close and the customer just found out that this is

part of their procurement process and there's a rush to do it right at the end of, um, like right before the deal is supposed to close, bring it up really, really early on in the deal cycle. And, uh, and also, you know, if your security posture is really good, if your documentation is really good, use that to your advantage, use, use that as a competitive thing that you have that your customers don't. So, um, Also, I don't have a bullet for this, but... ...and teach classes, etc. But I've always found that classic video games taught me a lot of really interesting security lessons. Probably didn't realize it back then, but looking back, I think it's

pretty cool. So I wanted to show you a few of those. But if you guys remember Shinobi... Mega Man, these are awesome games. So let's talk about Shinobi. If you're not familiar with Shinobi... There has been a lot of chatter about, and also just as I've been talking to other people about this at other companies, whether it might make sense to come up with a... cutoff at which, a deal size cutoff, at which you would say, hey, if you're not spending a certain amount of money with us, we are not going to fill out your questionnaire. This is not something we have in place right now. I know it's very controversial because the amount of risk you are introducing to a company has nothing to do

with how much money they're spending on your solutions. My thought on that is, I think that it is okay as long as you have a really thoughtful exception problem. Does that make sense? So this is the other view of credential isolation. Microsoft usually refers to it as credential guard. So again, the idea is you have this separate guest virtual machine running. That separate virtual machine has the actual credentials. And when you do something like log into the system, that password you type in gets checked in the virtual machine, the second one. The real credentials are never exposed, or at least a copy of the credentials are not exposed, where you can access it directly. So it has proper documentation on that? it's really not fair to

say, okay, well, then you just have to trust me because I'm just not going to tell you about our security controls. But if you do have the documentation, and you know, you have to scale, right? Like if you have a very, very, very small services engagement or something, and a customer wants to spend 40 hours talking through security controls with you that you have already written down somewhere, it might make sense to have some sort of a process. So things like live credential harvesting, where Mimikatz will inject code into lsas.exe. It can still try to do that if it wants, but the credentials aren't going to be there. Even if Mimikatz gets to inject its kernel driver and it scans the memory of lsas from kernel mode, it's

still not there. So that way of directly harvesting cached credentials really doesn't work anymore. And that was a huge pain from an IR perspective, because all of those cached credentials otherwise were readily available. Again, not something we have in place today, but I know it's a lot of people who, something a lot of people in this space have been talking about. So it is what I would call a spicy meatball. So add me on Twitter and let's talk more about this. But also I would love questions. Yes. Oh wait, I wish I could give you a microphone.

I did, I saved five minutes for online. People were kind of making fun of them on Twitter too, where they said like, oh, key logging's dead and this credential harvesting is dead on Windows now. That is not true. Don't believe that hype. The cache credentials are hidden in a place where the tool won't be able to access them, but you can certainly still steal live credentials as people log in. Any user land key logger is still going to work. You can still call set Windows hook ex and get your DLL loaded into memory And Mimi cats also has a cool module called MIM SSP questions But have you done any mapping to different? regulations standards

controls and Questionnaires so that if they feel like it wasn't answered in your documents you say yes, actually it maps directly to this one That's a good question we So, I mean, we do have a full SIG filled out for our bigger products, and we haven't mapped our SIG questions to... ...is register as basically a package to monitor passwords. So as you log in, and then it can just steal it that way. This is very similar to what... Yeah, we haven't mapped our SIG questions to like a, what is it, like a CAIQ or whatever that one is, Cloud Security Alliance one. We haven't done that so much as like if we have it completed,

we will give it to you. But, you know, it is weird. I feel like if they're being specific enough to say you didn't, that rarely happens to me. Like if they're being specific enough to say, oh, you didn't answer that question and I can say, oh, I did. Like if they're actually reading the documentation, the follow-up questions are generally like, Oh, wow, we didn't cover that. Like, yeah, I could definitely talk to you about that specific thing. I tend to get either, I want you to fill out the whole thing from scratch or the follow-up question. Like, I don't, yeah, we haven't done that because I don't think we have that internal security program. Yeah.

So when you're answering questionnaires, which I get bombarded with questionnaires from major companies, how do you draw the line with what is too much information to tell the vendor that sent you the questionnaire? Like I can tell them what all my OSs are, how many servers I've got, real specific data, but I feel that they don't need to know this information. And do you decide that information? If we look at just a generic system architecture, this is, I think, maybe the various systems involved with hotel guests checking in, the front desk people doing the cards, all that kind of thing, all the databases, all the web servers. If we just generically use this as our understanding of how the system works at

a very rudimentary level, we can start to focus in. Build paths around and this is further in the the threat modeling concepts But the idea of trust zones trust boundaries areas where we see communication between systems and There's effectively an inherent trust at least the way the the architecture is designed when We start to understand those things. That's when we can actually start to kind of like what we did in shinobi see petition yourself or do some other security team decide what is actually a proper to be shared out versus not? Yeah, I mean I would say that you know we have our sweet sweet NDAs in place and we are we err on the side of sharing more information than less. We definitely have

situations where we get questions where our response is to explain why that is that information and again, same questionnaire to every single vendor. If we can explain, hey, the answer to that question actually does not help you understand the risk of working with our company. That is generally how we would go about not, we don't want to share more information than we need to. We want them to have every single piece of information that they need to actually calculate the risk, but know more than that. And we do all the time say like, "Hey, it just seems like based on the scope of work, this isn't relevant." - Once we've done that exercise and maybe we take this concept, we run tests against the system to confirm what we

think we found. And we're gonna come up with a list of not necessarily vulnerabilities, but suspect vulnerabilities, potential... - If you want to talk more about that, and we don't get pushback on that very much. - Yesterday at the B-Side CISO event, this issue came up, and what one of the CISOs recommended is that the companies charge a fee for non-standard questionnaires. Was that my CISO? It might have been. No, it wasn't your CISO, but this is a problem that's pandemic. Yes. That actually, so that's another thing I've heard discussed a lot is will we, say we won't do it under a certain like deal size or will we charge to complete them? I would say

we will actually say anything beyond this many engineering hours we will start charging for, whether that is like support beyond the regular support amount or something else like that. It is a framework that companies use. We don't use it yet. Okay, I think we have time for just one more question, and everybody can feel free to catch up with Katie afterwards. hey um just curious if you considered um as you have to scale more any of the uh like maturity models that are kind of in the works for highly regulated industries like uh dod is kind of trying to push one through this year uh it's just rather than you know do this hand-to-hand combat which it seems like you've sliced and diced very well um getting that

that cert and you can just show your merit badge and and satisfy the-- - You know, it is actually, I am talking to customers all the time about if we do this cert, then what? Like what do we unlock? - Down to actual devices and systems. We weren't testing anything production or hooked to a patient. So the primary objectives of the exercise disabled patient vitals monitoring mechanism in a specific patient room. So think about all of what that means. We have to identify the patient, we have to identify the room, we have to find some way to influence and disable the patient vital monitoring system. And you guys all know, I'm sure, what I'm describing, but All of the things connect to a patient in the

bed. It goes to kind of like a hub. And then you have the displays in the nurse's station. You have the displays in the room. And all of those things, of course, are capable of alarming if any of those vital measurements change at a certain rate, go within out certain thresholds, et cetera. So there's that and then we have to prevent re-enabling of that mechanism until notified. With this cert, it is interesting to me, you know, we have a SOC 2. I think next on our roadmap would probably be ISO 27001. It is really interesting to me that we're talking to peers how for for how many people that does not seem to reduce especially

that 15 who says okay great that you have that certification but there's only one way for me to import information into my risk management system and it's with this questionnaire um but i do think honestly like a company that i think with being notified is we're kind of playing out a hacker for hire scenario so you know we don't actually want to kill somebody in a hospital but if someone wanted to pay us to disable some systems so they can kill them, well, that's kind of a different story. So we were kind of playing that scenario out. Ensuring no direct or indirect alarms go to staff members. Of course, we... - So really well is

Atlassian. They will not answer your questionnaire. They do not care how much money you have. And they also have like every certification. There's like, okay, yeah. What control hasn't been tested by a third party? Like I have certs on certs on certs. So I think they can really get away with that. If you can't accept those and you really need like a junior security analyst to write down the answer to believe it, like I think that's a you problem and Atlassian is still thriving. I would say maturity wise for our internal security team, we aren't like at their level yet. Like we don't have certs on certs on certs. So I'm still going to keep

answering these until we have those. But like when our trust page looks like Atlassian's trust page. Internet access. There were hospital production systems that were at least pingable from that wireless network. We let them know they weren't pingable anymore. We don't know what they were, but that's interesting. Yeah? When you say hospital production, is that internal IP or is it just routable from the internet? Internal IP, yeah. Good question. Repeat the question, yes, please. Oh, yeah. Sorry. The question was, was that internal or IP route or internet routable? Yes. Yeah. All internal systems. Obviously, there was some access control within the ingress for the wireless network into the Starbucks wireless network. I don't think necessarily that could have been used

Anyway, I'm confusing myself. But yeah, they were internal systems. As I said, we didn't really work to identify them. We wanted to just let the staff know. But that was really interesting and I think could be, had we not notified them and just played that out, that could have also led to some really interesting things. The patient and vendor kiosks that were placed throughout the hospital were also really important. There's interesting things with these kiosks and various attempts at kiosk mode and constraining the user interface. - To like your collar or something, it doesn't have to be in front of your mouth. And then, there's a switch on the top, we'll just turn it to

on. You think this one's working now too? It seems, I tapped it and it seems to be working, but if it starts falling over, I'll just give you the handheld. Those waters are all untouched. Currently we have a custom question. We have to have a custom question. We get turned. Can whoever's near the door just shut it for me real quick? Thank you. Okay, good afternoon. Welcome to B-Sides Las Vegas Common Ground. This talk is, I don't have the title ready, Getting CVSS, NVD, and CVEs to Work for You, Standardizing and Scaling Your Vulnerability Risk Assessment with Matthew Hahn and Luke Zutowski. I tried. I'm sorry. No worries. Thank you. Luke works great. Thank you. Real quick, we'd like to thank our sponsors, especially our inner circle sponsors, Critical

Stack and Bell email and our stellar sponsors, Amazon, Microsoft and Robinhood. It is their support along with our other sponsors, donors and volunteers that make this event possible. These talks are being streamed live and as a courtesy to our speakers and audience, we ask that you check to make sure your cell phones are set to silent. And if you have a question, use the audience microphone, which I'm holding. Please raise your hand and I'll bring it over. And with that, let's get started. Welcome to Luke and Matthew. Cool. I guess it's working in some sense. All right. So thank you for coming. Luke and I want to talk about CVSS scoring, how we can help

you with it. Luke and I have worked together for several years doing vulnerability management, vulnerability analysis. As you can see, we have some different interests. And Luke has moved on to a new job, so he has to give a quick disclaimer. I currently work for Microsoft as a security analyst. I'm here not representing Microsoft. These are my own views and opinions and don't necessarily reflect that of Microsoft. Just want to get that out of the way. So hopefully a lot of you have seen these acronyms. You don't have to go through them in too much detail. CVEs, those are the common vulnerabilities and exposures. NVD has a database of all the CVEs. including CVSS scores

and CVSS vectors, and they have an API that anybody can use, so you can access that data. CVSS is what we're going to talk about. They're handled, managed by First, and I want to point out I work for First Information Technology. Kind of what you would expect from IoT, the types of IoT devices you're going to see in hospitals. Lots of interesting physical input/output ports, running Linux, hasn't been patched since 2009, You guys remember the Shellshock family of vulnerabilities. This was vulnerable to all of them. Not hard to get root on this box. They have this lightweight web server. A lot of the things that I'm speaking to that I think we all agree are pervasive throughout a lot of the IoT, connected health, connected auto

stuff that everybody's getting into. Through a variety of research, I'm sorry, reverse engineering, getting around some interesting file integrity monitoring like watchdog processes, we were able to tamper with... ... services which is fits and I am not working for first. Back confusion out of the way. So what we want to help you guys do today is we want to talk about how to do C++ 3.1 scoring. Also how to categorize your assets, look at your inventory, understand your environment, and do some scalable integration of that data, which turns out doesn't have to actually be that hard and can help you with your security and your compliance folks both. So these are some stakeholders you might recognize. And basically disabling it

outright was difficult because it's always sending data to these other servers. So instead we found a way to just take a cache of the data and replay it. so that we kind of cut the input off Kind of like if you want to rob a bank, you just loop the camera feed. We basically did that with the vitals. And so at that point, we actually had somebody connected to the monitor. Folks that CVSS can be useful for. We're going to show you things today that internal security teams can use to drive remediation. It can meet compliance requirements with auditors. It can help management make informed decisions about what they want to do with the business, where they want to make investments, and can help relate your

security posture to clients. There are many other uses for this, but these are just some of the stakeholders. One thing before we get started too we want to make very clear, I think this happens in our industry a lot, that words get used to mean things that are not as specific as they should be used to be. So, CVSS is specifically designed to score severity of vulnerabilities and not risk. Once you start using CSS environmental scores, you can start getting closer to risk, but there's a lot of factors that come into play that you should be considering when you're talking about actual risk. You need to think about the threats, the exposures, your business, which

wasn't really many formulaic updates. It's more language and interpretation. And they do address that somewhat there, but it still gets misused in our industry a lot. Particularly, for example, FedRAMP and PCI both call their risk ratings They call them risk ratings and they're CVSS scores and they're really just severities. So, you know, it's pervasive throughout what we see out there. So just want to be very clear we're talking about severity and we're trying to get closer to risk by doing environmental scoring. So why are we using CVSS? There's a lot of different ways to score vulnerabilities. Penetration testers will come up with what they think the risk is just qualitatively. They might also use CVSS

but At first blush, they might think of, is it high risk or low risk? We know there's those tables with NIST where it's impact versus likelihood. You can get a chart. But that's fairly qualitative as well. There are quantitative values here. The scanners, they spit out tool risk ratings. They also spit out CVS ratings. The tool risk ratings, their numbers-- --truly coordinated, planned, funded attack against a health care facility. If we kind of go back to just the general idea of IoT devices I wanted to talk about something that's kind of unique to them, but BOR attacks, break once, run everywhere. I'm sure you guys are familiar with these. But those and other pervasive mistakes make hacking IoT really interesting because any of the

exploits that we might be able to carry out, say to find cryptographic keys, other secrets, What we see with a lot of these devices is that those cryptographic keys are the same on every single device. So if we've cracked one device, we've cracked, but we don't have access to them. So what we're looking for here is we want to make something that we can look at all of your assets for all the vulnerabilities that we're finding and understand and be able to manipulate those efficiently, right? Also, because of that, we need to have something that's publicly available too. We can't have something that's obfuscated. We can't have just qualitative analysis. We need numbers and availability. CMS offers both of those. CMS has vectors that have variables in them.

You can manipulate those variables and it gives you numbers from zero point manufacturing and just an overall production perspective. It can be challenging to automate that sort of thing so that there's an actual per device level of some sort of security. As we kind of continued researching, if you guys are familiar with the IoT Hacking Village at DEF CON. And if you're not doing your penetration testing in VR, you might be doing it wrong. So how are we going to get there? You can run a raw scan. You can get a scan of your system, right? You can scan all