Lessons from Black Swan Events and Building Anti-Fragile Cybersecurity Systems

Okay, we'd like to welcome everyone to our next uh presentation. I'm I'm Dimmitri aka Rura Pente. I'm running Passwords Con for PUR this year and we'd like to welcome everybody. Uh talks have been very good so far, so we sure you're going to enjoy this one as well. Just by way of reminder, we'd like to thank our diamond sponsors, Adobe and iikido, gold sponsors, RunZero and Profit. A reminder about uh taking photos. So, you can take photos of the speaker and his slides. He's okay with that. But the besides uh policy against photography is that you cannot take photos or pictures of any of your fellow attendees without their consent. So, just keep that in mind, especially if

you want to take a slightly wider wider shot. that applies to the whole conference as well. Awesome. So, we're now going to enjoy lessons from Black Swan events and building anti-fragile cyber security systems from somebody that's uh been in the field for quite long, very experienced. I've heard many of his talks and I'm sure you're going to enjoy this one just as much as all the others. So, let's hand over to Dave Lewis. >> Thank you very much. That was the nicest way to be called old I've ever heard. I appreciate that. >> Well, thank you everyone for being here. And this is really an interesting moment for me because I realized I've actually never spoken at Bides. Uh oh, no, that's

not true. I spoke at the first Bides Las Vegas. That is actually that is true. And this is my second time. I just remember that as I started talking, but it's one of those amazing things is like to see this event grow into what it has become is really cool and that you you have taken the time to be here is I do appreciate it. So I've been doing security in one form or another for far too long. Uh I've been doing this for 31 years now and along the way I've done all sorts of different things and during the pandemic I got into you know whiskey distillery ownership as well as a soccer club in Vancouver, Canada and uh it's

kind of interesting along the way. So I do work at one password. This is not a sales pitch by any stretch of the imagination, but I absolutely love working there. Great culture. I also do supporting for sighteline security, uh, BIOS as well as come on slides. There we go. Gnostic AI and a few other companies as well. So it really keeps me hopping to say the least. Now one password. The one thing the only reason I'm putting this up here is because people think we're just a BTOC company. So we spent 19 years doing B TOC. So it's an understandable thing. In the last year, we've really gotten to the enterprise space, and that is as far as I'm going

to go there. If you ever want to talk about that sort of thing, feel free to look me up. Now, I am a hacker in Canada. That means a little bit of a different thing. We tend to approach things a little bit differently. And yes, I am Canadian. So, this is one of those things which is All right, we got another one. All right. Shh, they're going to get you. Um, it's one of those amazing things like I did the keynote at Biz Knoxville this year, and it was really amazing. I came out on stage and I said, just to be clear, we love all you. we just don't like one person in particular. Um, and

that was very wellreceived. So, that's just it is like as part of a community, we all are from different walks of life and it's amazing to be a able to be part of this and I do appreciate it. Now, I do feel like this particular character far too often after having done this for 31 years and realizing that this keeps happening and when I say this, I mean all the security issues that pop up. What I do like to talk about is blast black swans as a cyber security event. Now this is derived from a book by Nasm Taleb a fantastic book and it was applied to markets and market conditions and things like that. But if you take

these lessons learned and apply them against cyber security it is always a incident that has massive impact that in hindsight was easily solved and sound like something that we've heard far too many times and this is just it. We have to get better at shoring up our defenses. Shocking statement I know but this is one of those really amazing things that like we keep having these same conversations now black swan events like I said there's a level of unpredictability you know oh we never could have imagined that that particular company could have been breached massive impact millions of records in these days billions of records being exposed and always the rationalization it's like oh if they had

only patched how many times we heard that and it's never that easy not patch you was a great example and then We look at other I'm doing a little bit of a sprint here because I didn't realize I was only had 20 minutes. So that was on me. So we have all of these different types of breaches we've heard about. And this is not to throw mud at any of these. This is just to really illustrate this could happen to any one of us. And you know I've actually gone through a data breach myself and that was a very unpleasant experience in the company I was at. I went in in the morning there was a stack

of notes there saying you know from my admins saying this public Wall Street Journal called, this journal called CBC called. like, "Oh, this is going to be a bad day." And that's just it. It's like that was how we measure things. And now we have to deal with this on a consistent basis. But was the incident that happened a black swan event? Was it really unpredictable? Was this something that we could have addressed? A lot of times, yes. How many people remember the Marai botnet? Show hands. Yeah, there's an interactive portion. Okay, that's better than I was hoping for. So, that's good. So the really amazing thing is the Ry botnet was using default credentials on

62 different devices that were attached to the internet that made a massive botnet. And then I heard, oh yeah, well Dave, that's an old story. Well, this is really interesting. So here's an article about the largest DDoS attack to date by the Mariah botnet. And oh wait, what's that date there? Let me just blow that up. Right there. That was this year. Mariah hasn't gone away. It has gotten bigger. And these are the kind of things that in hindsight could have been fixed programmatically. There's no reason to have a device that when it attaches to the internet, it doesn't change the default credentials as soon as you log in. Having default credentials hardcoded in devices and you know my former

employer vendor name, vendor name as username and password that is inexcusable at this point in time. There we go. And this is how I usually approach it when people ask about cyber security. I say this is generally how I approach things overall. and passwords. They have limited utility. This is a control. This is not a security control. It's like your house key. Yes, you can lock your door. You can protect your stuff, but when you leave for the day, if you lose your key somewhere along the way, somebody finds them and they have picture of you and your kids. Well, it's not that difficult to figure it out. You know, Google lens like, "Oh, this is

this person. Oh, okay. There you go." Get into the house. House doesn't know any better. And large groups of uh try that in English. large groups of criminals out there are making bank on this like this particular Vietnamese group here. Uh whoops back $71 million can The crazy thing is this group of hackers that was making all this money based on stolen credentials. Four people one group out of many and this has been going on for years. When I used to work at Akamite back when the world was flat we used to follow these groups all the time. There's so many of them and they're still around today and bad things keep happening. Like back

in the day when I was working at a power company, we had this really interesting thing where we found an ARP storm. We couldn't figure out where it was coming from. We traveled all across, you know, all the different systems saying, "Is it this, is it, is it this?" None of it made sense. So finally, I just broke down and I did, you know, AAMS Razer. I grabbed a tile lifter, walked out into the data center and started lifting tiles because I'm like, "I'm out. I'm out of options." The third tile I picked up, there was a Cisco 1750 blinking back at me that nobody knew about. It was under the floor. So that company used to be part of a

larger company that had been split into five smaller companies. It was a direct connect to another company that was ostensibly our competitor at that point. That was inexcusable. The best part was they didn't know about it either because it was under the floor in their data center as well. We never did figure out who put this in, but I guarantee you it was somebody I was working with. So, we really do tend to follow our own rules. And we have to make sure that we are looking at better ways to improve things overall because we can't make the same mistakes over and over again and hope for a better adventure. And having that mentality is

something we have to address because when we're dealing with, you know, finance, HR, all the rest of the organization, they're very good at what they're good at. But expecting them to be well verssed in security is not going to work unless we are working with them because if we're not getting the message across and they make a mistake, we can't vilify the user. It's not going to help. We have to make sure that we're doing a better job because we can't be that flaming sword of justice where the answer is always no because data breaches are going to continue to happen. This is one of my favorite sites. Anybody familiar with information? Yeah, informationis.net. All right. If you for those that don't,

look it up. It's amazing. They just do all these really u great data visualizations for data sets. One of them is data breaches. So this is a relatively recent one where you see orders of records of 1.3 billion records, half a a billion records there, half a billion there. The stakes have changed. We keep talking about data breaches and yet we keep having the same problem, but now the breaches are getting bigger. So we have to look at better ways to approach things. We have our ways that we triage events and deal with things. But this is something that was brought to my attention by Rich Mogul. If you're familiar with his work, he does great work with Cloud Slaw and

various things like that. This is something from God, I forgetting. Well, he does physical incident response. So, if there's a hurricane or something like that, he would go through and address these. And part of this is how they would approach matters. And here is another slide. Come on. Where it goes. Oh, yes. the disaster response smart book from the incident command system. So it's not ICS and control systems, but you know, I can see where that collision can happen. And this is how they go through and they triage events. These events, yes, they're going to have massive impact. They can be unpredictable, but they do have a playbook on how they can address it and

then fundamentally reduce the risk to those affected. This is the kind of thing that we have to change our thinking as security practitioners because these events keep happening. the orders of magnitude of breaches is so large now that it's really unconscionable and flip it on its head and look at ransomware attacks back in 2017 they were all the attackers were going after the large companies like you know Bank of China NHS so on and so forth because this was the big thing at the time and then we saw it drop off in 2018 they weren't going after the size was based on the size of the company but as we transition over here to 2010 all of a

sudden the ransomware operators are back in force. Can anybody tell me what happened in 2020? >> Bingo. COVID 19. All of a sudden, the attackers were sending emails saying, "If you don't click this link, you're going to lose your healthcare coverage, all the rest of it." Nobody knew what was going on. I didn't know what was going on. Anytime a package came to our house, my wife and I would get on full hazmat gear and do sanitize the whole thing because we had no idea how it was coming in the house. This is why ransomware operators started being successful. And as they progress across to 2023, which was the most recent data, the [clears throat] bubbles are getting

smaller because the attackers are going after the smaller companies because a lot of these small shops have contracts with larger companies. So the attackers are then pivoting through there. So we have to make sure that we're doing a better job of approaching these things because we have to learn lessons from these data breaches and we have to look at what went wrong. And this is not a case of pointing fingers. We have to say, okay, we have to have frank and open discussions. How many times we talked about better communication. If we look at the financial systems, we look at juristprudence, we look at medical profession, they're really good at sharing lessons learned. We have a lot

of work to do in our field. Capital One, this was a shadow IT moment where misconfigured W led to 100 million credit cards exposed. Uber, 57 million users exposed for a GitHub repository that got breached. Come on. Oh, yeah. Dow Jones. Amazon S3 bucket. This is my favorite one to pick on because it's so very simple. Anybody can set up an S3 bucket with a credit card and a web browser and off they go. Unfortunately, they make these things public and that it's a data repository. You can put whatever you want in it, but they are discoverable and most people don't understand that even though when you log in in the console in bold print says do

not make this public unless you know what you're doing. Paraphrasing of course, but that was essentially the message. Now, I used to work at a high-tech manufacturer. Within the first couple weeks on the job, they said, "Okay, let's go through and look at all of the accounts that had super user status that belong to users and no longer with the company." All right, we did that and we found there were 10 user accounts that had root access that the users were no longer at the company anymore. And I tell this story over and over again, so if you've heard it before, don't give it away. One of these users, their account had been used in

the last two years, but they were no longer at the company. Can anybody guess what the wrinkle was with this particular account? >> Was cooked into a system where it was running as a human instead of a service account. >> That's a good answer. I hadn't heard that one before. But the actual reason was this person had actually been dead for five years. >> So, we had a ghost in the system. What happened was it was somebody that was using it on a post-it note. Cliche, I know, but they were using on a post-it note to go in and check a crown job. So, basically close to what you were hitting on there. But this is the kind of thing

that is absolutely maddening because this type of thing happens far too often and we're like, "Oh, that's okay. We're in the cloud." And same problems persist. It keeps happening. This is normally where I'd have my coffee break, but I just realized I have no idea what I did with my cup of coffee. All right, moving on. We have to look at things like the psychology of passwords. We always like to pick on people like, "Oh, your password was querty." Okay, well, why is that? We didn't teach them well. But people try to do things that are going to be as easy for them to understand. And we have to make sure that we're looking at ways to give them

better ways to approach things. How are they going to do better on their security approach? How can they pick a better password? If you've never seen this XKCD comic, go look it up. I love these love this strip in in all of its glory. But there are better ways to approach that. We have better technologies that are out there. All sorts of different ways that we can approach better password security. Multiffactor authentication, biometrics, phto, pass keys, all of that will help reduce the risk. And fundamentally, that's what we're trying to do. We're trying to reduce the risk overall. Are we going to solve 100% of the problems? Hell no. That is not part of the deal. I

wish it was, but it's just not, you know, realistic. We have to look at proactive measures. We have to feel figure out how we're going to create resiliency. You have to have systems that can take a hit. When I was at my previous company um that I mentioned earlier, one of the things was we did not believe in nine uh what was it 59 that was not part of the equation. It was 100% uptime and we designed our systems exactly like that. We had one nation state actor that took a poke at us. They told us they were going to because they were testing our systems. Thankfully, we stayed up 100% of the time, but that's just it. These days,

the technolog is available for you to have resilient systems that can take a punch. And when you're looking at it from the perspective of a black swan event, not only taking a punch, but being able to learn from it and do better, you want to have systems that can take that hit and be antifragile. And this is really about the concept of exactly what I was just talking about of having a system that can take a punch. So I realize I'm getting ahead of myself in the slides because I did this talk in Athens, Greece a couple weeks back and the slide projector didn't work. So I had to ad liib the whole thing and I'm

actually kind of happy I'm on point. So, shadow it is another piece of the puzzle that keeps coming around to bite us, especially when it comes to unmanaged devices. And the really fun thing was is I had this or had a very similar talk to this in Toronto, Canada, but just over a year ago, I said, "All right, everybody in the room. Who here has um controls in place from a technological perspective for shadow IT?" The whole room put their hands up. There we go. How many people have policies in place to deal with shadow IT? The whole room put their hands up. I like great. I was like, "Okay, dramatic pause turn. How many

people in here have shadow IT?" Everybody put their hand up. It It's just a thing. And that's just it. When I'm talking to customers and saying, "Look, how are you managing this?" They often say, "Oh, it's not a problem. We have a policy, but it yet it exists." It's sort of like Schroinger's IT security problem. So you have to have good preparation because in order to protect your organization, you have to do your work to look at where the problems are uh within your organization and be able to address them in an expeditious fashion. And when you're dealing with data breaches, for example, you have to have a good incident response plan. And when I say a good incident response plan,

practice it. I worked for a power company where we had an incident response plan and I found out we hadn't touched it in 10 years. And I looked at it, it was completely out of date. As a tangent to that, we went through and looked at all the policies that were available and there was one that made no sense what whatsoever. It was a particular certification we had to have and I couldn't find this for lover money. I searched online, nothing. Finally, the person who wrote it was still with the company and had been there for 20 years. And this person admitted to me that what they had done was they had found a certification

online did an a search and replace for cyber security. And so that was part of our policy set that senior management has signed off on this particular set of policy. Well, this particular policy document was originally a certification for swimming. Unfortunately, this happens a little more often than I thought because the first time I gave a talk where I brought this up, I had two people come up to me afterwards and they had similar type stories. It wasn't swimming but it was very very similar where somebody had done a search and replace just to tick the box and say yeah we're all secure now. Now shadow it is usually done out of a case of speed and convenience. It's

never out of malice. People are trying to get their projects done to satisfy you know the aspirations of the business and other times it's you know perceived inadequacy. Yeah pretty neat perceived inadequacies with the solutions that are available. One company I worked at, we had seven different logging and monitoring solutions because people didn't think that the logging monitoring solution that was in place could help them with their project. So they added in for every project that was being delivered, they had yet another logging monitoring solution. There was no sunset provision for any of these. So they limped on in perpetuity. The other piece too is we're always told to innovate and experiment, but usually that happens in production.

And I wish it didn't, but you need to have an environment in order to address these things because you have a risk to data security. You have to look at the risk of data loss because I'm sure everybody's using sanitized data in their shadow IT. I'll stop risk and compliance. There are real fines that you have to be aware of. This is a problem that can have material impact for an organization if that data gets leaked. Risk the bottom line, that financial impact. The beige desktop's one of my favorite stories. I'll be really quick on this. This was How many people have ever seen a beige desktop running a mission critical app on? I'll stop there. I got enough answer.

This is maddening because this happens so often and people don't know how to get it off because the code was written by a summer student, not properly documented, it works and suddenly it has become priority within the organization. We have to get better at addressing these sort of things. And if you have a summer student writing code for you, get them to document stuff so that you can deal with it because otherwise bad things are going to happen. This one here was another system that was causing problem in the network and we had all of the contact sheets for all the different uh servers set up. This one had no name associated with it. So after trying

exhaustively to go through the entire company of how who owns this, how it was being run, all the rest of it, no answer. Finally, I did what I don't recommend people do, but I did it anyway. I pulled the Ethernet cable and I tied a note to it and I walked away. >> Scream test. >> Exactly. The scream test. Sure enough, in came Nick. Love this guy. Came in beat red. Absolutely angry as all get out. He's like, "This is mission critical system." I'm like, "Come with me." [clears throat] We walk over to the rack and I said, "This yours." And he goes, "Why is it unplugged?" Said, "We had a problem with it. This notes for

you." So I took it off and it was sealed and I had signed it across there. So he looked at it, he goes, "Huh?" Opens it up, reads the letter, goes, "Hm, thank you for your time." And he left. It had been nine months since I wrote that note. So that system got taken offline really quickly. But that's just it. It's like a lot of times it's like these missionritical systems aren't necessarily mission critical. So we have all sorts of different case studies. I'll skip through that on the for the sake of time. But when we're looking at the future of cyber security, spit on the floor if you must, but zero trust architecture is a really good way to go

forward. This is not a product anybody's trying to sell you. If somebody says it's a zero trust anything, they're lying. Zero trust is a strategy. You want to be reducing risk in your organization. Continuous authentication, verification, micro segmentation, extended access management, and the rise of cyber insurance as a safety net. The problem here is too many companies are relying on this as their get out of jail free card. This will help you cover off the investigation dealing with, you know, critical messaging to stakeholders, but it will not make you whole again. So, if you're relying on this as your get out of jail free card, you're going to have a bad day. There is the government uh in Hamilton, Ontario

recently went through a data breach where they have spent about $18.3 million uh with people doing an investigation restoring the whole. insurance company turned down their claim because they had no multiffactor authentication which was in paragraph one of their agreement. Let's skip by that. If you are looking at problems within your environment, we talked about it earlier where you have an environment where you know something is going to be it couldn't happen here. It's unpredictable. If you have 10 people looking at the issue, the job of the 10th person is to say yeah the bad thing can happen and here's why. Because if you have everybody agreeing, you become a Greek chorus. You're just going

through the motions. You want to have somebody that's on staff that can literally say this is the alternative theory that we have to look at. We've checked all the boxes. So the road to hell is paved with the good intentions and we have to make sure that as we move into artificial intelligence, especially from an agentic AI perspective, we are approaching this in a right way. AI is not here to replace people in their jobs. I've seen companies make this mistake. CLA, to their credit, fell on their sword. They let go of 700 people and they said, "This was a mistake." AI can help 1x staff and we want to make sure that we're approaching it from that

perspective. And that brings a whole host of security issues with it as well. And I am at time so I'm going to cut myself off there, Emma. Yep. I got a full stop. All right. Thank you very much for your time. Um, the entirety of the slides will be available if you want to email me. I'll just do put that up real quick because I have a lot of stuff here. >> Just put a QR code. >> A QR code? Yeah. You get out. [laughter] Oh, yes. I have a podcast if anybody's interested. The Chasing After Podcast. There you go. There's a QR code. And I guarantee you there is no uh never going to get you up there. All

right. Camera's down. Okay. And yeah, feel free to email me gatka@1epass.com. I can give you a set set of the slides or we can chat or whatever. And I really appreciate everyone for your time today and thank you so much for being here.