Data Breaches: Patterns, Costs, and Lessons

I hadn't exactly planned to do this I volunteered they just seemed like something fun to do because I've worked on this thing for so many years and now that I'm basically just a talking head and everybody else does the work I'm like it's okay I feel I feel like I cross that line now so I'm going to do is talk about data breaches and I've been talking about in a way that is a little less than normal most people are like hah look at them they got pantsed and that's never a good thing and I've been through this for quite a long time so quick introduction for those you don't know me thank you the resolution

doesn't work there I my name is Dave Lewis I have been doing this sort of thing for about 25 odd years give or take if we go even further back I was duping video games and selling with kids in school in 1983 so I self-identify as a hacker I'd like to use that word as our word because I like to use the term attacker for the criminal event so if you look on Google you'll find not everybody likes us all that much it's a little bit disconcerting at the best of times but one of the interesting things is as a part of my role I get to travel the world and give talks all over the

place and it's really really rewarding so I get to you know trumpet the flag so much so that I actually got a tattoo on my arm of Maple Leaf because of something that happened in India a couple years ago I was in a hotel there and I was just getting ready to check out to go onto my next destination and this wall just woman comes up to me she goes excuse me are you an American I said no actually I'm from Canada she goes good ever been so scared in my life what would have happened if I said yes I'm still trying to figure that one out so generally speaking when I'm talking to folks around the world I say were

usually dressed up as homicidal maniacs going across the she did a she device it along a high rate of speed my tongue just tripped I do work for duo and my I do talking and stuff I give talks around the world about 40 times a year we got bought by Cisco a year ago and the one thing that really got me to talk about this in the first place was data breaches so data breaches are something that have been near and dear to my heart for a very very long time if we go back it's around 2012 I'll talk to them about that a little bit I really started tracking them and one thing that I

noticed is that nobody really goes through and has a set nomenclature that they stick to they don't have a conversation where they say okay these are the these are how what happened is how it happened these are the lessons learned and then sharing it beyond that and it really bothers me because they're like oh yeah it's okay we lost that Windows laptop but it's okay because it had a password not going to work so I last for the last three years I have gone through every possible publicly available data breach notification I could find to read them if you ever have trouble sleeping easy easy way to fix that there are so much of this to go

through and the one thing that I was like hoping to do is have some trending analysis some sort of you know statistics that could go through but it was such a dog's breakfast that I couldn't do that and it's really amazing as we're going through this and then people are saying oh we do it this way we do it this way to do it this way and it's like there's no commonality and people really look at things from a very wrong perspective in so many different ways so one of the things that really gets me is when we're talking about security we like to talk about all we have a firewall everything's fine oh we have a perimeter

everything's fine so before I ever got into this racket I did a degree in classical studies in archaeology and one of the stories that came up along the line was the sack of Rome in 410 AD so this is when the Visigoths showed up they took over the city simply by not letting about anybody in or out of the city they use their own perimeter defense against them as a weapon now we flash-forward to 2019 we still have organizations that are going it's okay we have a firewall and they have a completely flat interior I know I've worked at some of these companies and this is a really vexing thing because we can do a far better job now

when we look back in time we look at different types of data breaches we look here in 2005 where 40 million credit cards were dumped and there's Heartland processing where there was a untold amount there we're going through were like why did this start keep happening why were all of these credit cards being dumped and a lot of it was because of the law of unintended consequences so XP command shell was a tool that was built into a lot of database databases rather that would allow the database administrator to remotely administer a database it was put there with the best intentions it was the road to hell is paved with those and unfortunately a lot of times you

would see databases being productive promoted into production with is still enabled I was at one company where we had penetration testers come in and start doing testing on us and one of the first things they tried was this thankfully that was disabled and I was really funny because I texted them at the same time they're like that's not gonna work and they're like who told you it's like I'm actually looking at the intrusion detection system now data breaches are a really frustrating thing for me because they keep happening how many people in here are familiar with Amazon's s3 buckets show of hands awesome that's probably the best turnout I've ever had when I've said that the

really frustrating thing here is that anybody with a credit card and a web browser can spend one of these up and by and large the folks that are setting out best three buckets so those of you don't know it's a data repository you can put whatever you want in it images flat files whatever it happens to be the problem here is that we as security professionals haven't done a very good job of telling the folks that don't know why this is a problem when they spin these up to make them not publicly available because there have been so many data breaches that have happened where an s3 bucket was exposed but it didn't have to happen that way now the

really funny thing is it throughout my career I do I've learned a lot of lessons and I've made a lot of mistakes and along the way I've collected all sorts of stories which I write for various publications one story that I did with Forbes excuse me I went through and I set up an s3 bucket took screenshots put it through and the steps there's a huge banner across the page saying do not mark this public because then people can find it there are scanning tools that are freely available on github that will iterate through s3 buckets looking for these sorts of data repositories and people will keep getting hit now when I put these sort of

names up here this is not to guilt anybody this is a show this could literally happen to anyone and they keep happening now it really doesn't get really personal for folks until it happens to them and I myself as suffered through a data breach years ago and I look for those who don't know I've been in the Toronto area for about 25 years working on in various companies and we had at one organization with that we had a data breach and it was really really frustrating and I know how painful those things could be but it doesn't really seem to resonate with people until they have their own data breach until that hits the home because they don't

understand the value of the data they're protecting be its source code customer information whatever it happens to be and that is really where it gets kind of vexing now no coño conversation would be complete without your mandatory you know machine learning AI sort of nonsense but this is just it the attackers have the ability to do use these sort of tools the year there are tools out there where you can go through like Amazon sage maker and other things like that where you can create your own you can create your own applications this is one of those things you can go through now the thing is most of the times the attackers will never ever go to this extent why

because they don't have to we ask defenders leave too many low-hanging fruit and it allows the attackers to get in but it is possible that this can happen now one of the roles that I have is I do speaker operations at DEFCON every year and then one year we had this it was the DARPA Grand Challenge where they had all these supercomputers across the stage were attacking each other and it was really an interesting event behind the stage there were huge tubes or hoses full of water just to cool these things but the really interesting thing is at one point one of the machines created an attack that nobody had seen before it created its own

attack against another machine so the possibility is there but I wouldn't even begin to worry about that yet because we still make it far too easy for the attackers and flash warning attackers will try anything sorry about that so when we look back and we look at the expo facts breach that happened not too long ago actually a while ago now but this is a great lesson learned because everybody's like oh they got popped they go through this they got through that it's like the problem here is Brian Krebs pointed out one really interesting thing and also Brian Krebs is not your IDs system he pointed out that there was a web interface that was

available with it'll easily guest credentials that would get you to the exact same data and this interface showed up in 26 different countries I think it was this was a very easily guessed username and password this is just it the attackers don't have to drop any sort of zero-day or any sort of exploit when they can just log in and get it and the in the media and I've been guilty of this as well talk about things but we don't always talk about the right things so you hear about unauthorized access the insider threat everybody talks about that web breaches but they usually don't talk about the missing patches because nobody wants to talk about that but this is just it this

is the sort of things that we have to make sure that we're spending time to focus on so we are securing our environments now I absolutely love Star Wars so I'm gonna liberally rip off rogue one because rogue one really hit me as one of the best data breach movies that was ever made so first you have your firewall at Manny's working away there he's like okay I'm doing my job I'm gonna take my coffee break in a little bit oh wait somebody use the wrong password wait why are you here you're not supposed to be here you're not scheduled to be here and they're like oh don't worry about it we got redirected it's no problem you

know is everything's cool he's like okay send me the code there's your sequel injection it's like oh yeah no problem in you go so off they go into their destination now if this was an attacker getting into a site if you believe any of the reports out there like Pokemon or any of the other ones you'll see that it's like 200 days on average before anybody notices they've been breached if this statistic is actually true this would take quite a long time now once the attackers are in they try to escalate their privilege obviously they want to get around and to get more information by this point there's so much damage done that's kind of hard to

recover but eventually the indicator to compromise show up when everything starts blowing up but by that point it's kinda too the attackers are in they've got their beachhead and the egress filtering has already begun and then all of a sudden it's like oh wait the data there goes but the best part about this is as we go through this is we realize that at no point was that data encrypted so even in the future or the past depending on how this movie timeline goes things were really bad now back in 2012 I started tracking data breaches and this was one of those things where I went through and I was just on my website look at matrix

org I was tracking all these this was one here LinkedIn six and a half million records this was headline news in 2012 everybody was talking about this now here we are in 2019 talking about orders of magnitude of billions of records how is it this is keeps getting worse we have to be able to do a better job of this and I know what it feels like because I've been there this is a webpage that was for a company that I was working at that got compromised came into work one morning the phone was ringing off the hook and there was several different media companies CBC Global Mail all the rest of it they're like oh yeah you got breached we want to

talk to you and I'm like what this was a server that was part of the marketing team that was supposed to have been taken offline I contacted the marketing team they said it's okay it's gonna be taken offline two weeks later this happens was I mad at them no why because I at no point and my team at no point did we follow up with them to make sure this had been taken offline this was using a deprecated version of WordPress well you had me at WordPress and they got in and they were able to mess around but thankfully sheer luck this was no longer connected to any back-end systems so it was really dumb

luck but the reputational damage had already been done and when we look at the attackers we go through like this is data that I had from my previous job when I worked at Akamai we go through and quarter-over-quarter this was the attack patterns that we would see hands down sequel injection local file inclusion cross-site scripting we're always the top three attacks why because they work and that's a really frustrating thing because you go through and you look at the data breaches you say okay this was a couple years ago and it's like wow that is something else and then six months ago it's like okay this is getting a whole lot worse so - a

couple years ago six months ago and roughly a couple weeks ago so this is a site called information is beautiful dotnet they do data visualization for all sorts of datasets one of them being data breaches this is really cool but it's been so hard for them that they go back they had to redesign their interface because it's been getting so crazy so it feels like everybody wants a data breach and a lot of the problem is when we look at it is like are we using the right security controls not always and then we have problems that come back at us like security debt like this when this happened for those you don't know that's wanna cry when these when this

happened that frustrated the hell out of me because this was a vulnerability we had known collectively for 10 years and yet it happened so when you have a data breach you have to look at the various types of cost that you have to worry about the cost of investigating the incident remediating the findings ever been in an organization where you get popped and you get a list of things to do and the manager goes oh no that's ok we're good we will just accept the risk yeah frustrating communications that you have to do so when we had our data breach we had to communicate with external folks internal folks shareholders it was an absolute nightmare and there was no actual damage

done beyond reputational that was really really frustrating yeah look at the legal fees and goes on from there potential loss of revenue one of the really interesting things is one of the side gigs that I do is I work with a group called secure OSIS where we went through and we looked at data breaches over time and it was amazing that over time each one of these companies that got popped their stock went up afterwards so maybe they do really want a data breach I don't know but there are costs that we'd have to look at like for example when Verizon was looking to buy Yahoo and Yahoo failed to mention that they had gotten breached and that caused

a 350 million dollar haircut on the deal price so that could have actually gone very very much worse than that and then we look at things like this where a company said oh it's okay we don't need to do income because nobody told us we had to that was an actual rough actual quote from a party at that company this is amazing to me so we look we talked about those things we talked about earlier we mentioned the Equifax thing now if you and your organization had a breach and suddenly you'd lost staff at your breach as a result of that how would that go for you so let's look at Equifax who got a pink slip the seaso

retired incidentally made eleven million dollars in four years I don't know how I get that gig the CIO retired CEO I'm sure the CEO is fine nope CEO retired with a ninety million dollar parachute but in one fell swoop the entire senior management that company was lopped off ostensibly all of it without exception of the CFO but imagine if that happened to you and that's not to beat up on Equifax that's just to use them as a lesson to look at what would happen in your organization if that went down how would you recover and we look at this here's their competitor the next day we're suddenly serving up malicious is gonna show you malicious flash this

was the next day so when you're looking at these types of things you're looking at the cost involved don't just go flying by okay the cost of all do you have to look at things like compliance often compliance is one of the adults in the room nobody likes to talk to them I know I've done that gig in the past but without compliance you are not looking at the economic incentives for breaches they can help you go through and understand the fines that could be incurred as well as you know making them at an ally to get your job done to help you build your business case in order to get your organization secure so failures

happen all the time mistakes happen all the time as long as we're learning from those mistakes that's the best part in one point I did a scan of this company that or this sorry this attacker that was going after as I was working for a defense contractor in the US and my client was getting hit by this IP over and over again and I just finally got fed up and I hit back I scan them a got in I broke into their system I left a note saying if you want to get in touch with me here's my give him a burner email address and what they needed to fix and then I left a couple days later

I got an email and I was like all feeling all I'm self-important and I was like oh great you know well thank you for the list of things that needed to be remedied we actually didn't know about a couple of these things and oh by the way you got the IP address wrong so when you hear people talk about hacking back this is not a resolved data breach issues I had managed to go after a system and I got one of the octets wrong it was that simple but it doesn't always it happens all the time I had one vendor come in to me where when I was working as a senior security officer at one company and they said

look we'll scan an entire block of addresses for you so you can go through and we'll show you what our tool can do and what can find them like fantastic great so they did a scan and then they printed out a report yes this is a while ago they printed out a report in centimeters about that thick and I was like this is absolutely mental I started scanning through it and I'm like printers there's so many printers why Shanghai Shenzhen Beijing the vendor had managed to get one of the octet strong as well so it can happen to anyone and this is just that we have to make sure that we're doing a far better job of

this because the attackers are going to keep on coming and if we can't get this basics right then this is not going to go well in the future one company I worked at they had an application that was going to go live and they were supposed to come through security and we were both vetted and then it would go into production well it went live without coming to see us so this is one thing I did when I found out about it I didn't lose my cool I actually opened it up in the browser and I said all right I did view source right there that and that so seemed kind of simple basic first step looking at a web

applications like see what's behind it what was the first thing I found commented out in a code I heard so many different answers there yes I heard that it was username of admin and a password of password and this access into a production system thankfully had only been live for a little bit and nobody knew about it yet because they hadn't sent out any of the PR material but guess what when I showed this to the head app developer for that particular team What did he say you hacked my application I reacted sort of like this in my head but I didn't do this because I realized he didn't understand he was really good at running

a team of coders they knew how to code things but I hadn't taken the time to work with them so they could understand what my security concerns were and how to better secure this thankfully this actually transpired well nothing bad happened but this shows that we have to do a far better job of communicating to a wider audience otherwise it'll just keep happening one company I worked at this happened about two weeks after I left I got an email from actually somebody who's here I said you're not gonna believe this something we hit fixed had changed and I could dump the entire password from this particular company and that's just it people are gonna use tools in way you never

imagined and sometimes they'll use tools that don't do anything at all and security blunders can happen to the best of us the first time I ever gave this talk this happened to this particular company I don't want to beat up on them but it was this timing absolute timing their entire database was tarred up and when you have a data breach it feels like vertigo and my friend Bob here he went through this at Yahoo in a giant scale and I really understand his pain now but not at the level that he went through and then we look at things like the Morris worm the unintended consequences are going to continue to happen Melissa worm was another one that

was a fun one but we as security practitioners don't shouldn't look at things as an adversarial fashion we have to be there to help enable the business enable your organization's enable folks to understand the Demma democratization of security so we can get them to be part of the solution don't align against yourself talk to folks like internal audit why because they can test your incident response plan they can go through and look at the risks and benefits of sharing information with other groups and see how it can go and also you want to make sure that you are talking to folks in a concerted a concerted way so they understand what the bad news is and

how to properly take it don't run around with your hair on fire I don't worry about zero days every time I see a zero day in the media I kind of foam at the mouth because I'm more concerned about a hundred a vulnerability how many people in here run Oracle databases in our organization don't put your hand up when's the last time you put a poor little patch bundle on in a while huh I've been in organizations where the Oracle patch models were not applied for three years why because the DBA said it's my database it's fine we have a firewall so what do you do you have to start where you are you have to look at

what you have your organization you have to do a gap analysis of the things that you have in order to avoid these data breaches you have to do the work and you have to look at things like patching I know that sounds overly simplistic it is not it is a it's a painful experience I've been through this far too many times but you want to look at where you can stop the threats earlier because as we move into talking about zero trust in other organizations and make zero trust ostensibly is everything is on fire we have to go back and do everything we should have been doing 20 years ago that's it in a nutshell because

organizations are constantly asking us to innovate so we might make sure that we have that framework in place before anything happens otherwise we're gonna with so many wars and I think that's just cut it was back again and what can be seen as a accurst when it happens data breach it can also be a blessing one organization that I worked at we found a Cisco 1750 under the floorboards or under the floor under the raised floor in the country in the production room whatever and it was actually connected back to a company that used to be part of the same company but that company had been split into smaller companies it was still a live

interconnection nobody knew it was there the only way the reason we knew is because it started airing out that's how crazy things are data breaches are going continue to happen but we have to have a better conversation about it we have to share information we have to make sure that we're having a far better communication path to the folks that don't know and don't assume that they do that being said I would like to thank you very very much for allowing me to sub in and sorry wolf couldn't be here thank you all for listening and that's it for me I don't know if you have any questions but if you do I'm at your

mercy oh oh Jesus okay I'm gonna run my own mic I think there we go who's first there all right there you go no users all right sorry no if the wolf would be here tomorrow no unfortunately wolf can't physically be here and I can't go into why but he'll he'll be back again so arguably a silly question but you know it's easy for us to kind of get caught up and all the exciting tools that come out etc etc and many folks are stuck in environments and stucks the wrong word but are in environments where the conversations you've shared are very real right so it's very difficult to sort of you know push the rope and you

know yada yada yadi in that context would that not make so instead of always trying to kind of fight that seemingly losing battle does that make maybe a backup strategy the best approach and then sort of taking sort of the the stance that yeah it's an inevitable that it's going to happen let's just make sure that we've got as robust a back of a recovery you know part of architecture that we can put in place one that not only looks at duplication but also at an integrity data integrity piece well yeah that's one of the key pieces of it you have to also look at there's so many other elements obviously this was a short talk

but yes there are all these different elements you have to look at before you get into the best one of all is one that no vendor can really sell you is process defined repeatable processes are gonna make you at your life a whole lot easier it just it's painful to generate them because it's a lot of writing it's work exactly Hey so from your experience do you find that all these new regulations that are coming out are changing behaviors or just more knowledge or more more knowledge to people to do the right thing uh unfortunately I see it as a lot of reactionary measure based on whatever the headlines of the day are some of

them are most of them overlap when you look at it like for example I can't remember who was talking about the unified uh compliance framework earlier today that is a perfect example where you can see all the different compliance regimen regimen thank you regimes and how they endure Linc I don't see that a lot of these are actually good some of them are actually very good I like GDP are not for the reasons of the fines or anything like that but is actually forcing organizations to be accountable for the data that they are stewards of good all right cool anybody else are you going to all Stampede for the door oh we got one down here where'd

you go where where she goes oh there we go oh and don't hey hold on hold on hold on don't anybody leave there's a rather as a raffle coming up yes you had a picture earlier of hands stopping a set of dominoes mhm well what's your reaction how do businesses work or like how do they react when they stick the hand there to stop the dominoes from falling but the dominoes are too tall and they still keep going it's going to vary from one organization to the next there is no one-size-fits-all so it really is an iterative process so yeah of course there's gonna be mistakes I've made so many of them that I turned it

into a career but that's just it is you have to make sure that you're constantly iterating constantly improving because no vendor can go out there and go here this is a magic solution so make everything better there are tools that can help you accomplish your mission but you can't just you know take them at their word you have to make sure you do your homework before you ever talk to a vendor and that's coming from a vendor