GF - Zero Days should not be a fire drill

this talk is zero days should not be a fire drill by steve and tony um i do have some announcements before we go ahead and get started first and foremost we'd like to thank our sponsors especially our diamond sponsors lastpass and palo alto networks and our gold sponsors amazon intel and google for their support the support of our sponsors along with donors and volunteers is what makes these events even possible cell phones i do want to give you a quick reminder i'm sure you've heard it many times already these talks are being streamed live so as a courtesy to our speakers please make sure your cell phones are silent so as to not be disruptive um in the event of questions which i believe we're going to have some questions time after uh we're going to ask you to use the crowd mic there's a mic in the middle again these talks are being streamed so for the sake of the audio on the youtube we ask that everyone speak into the mic uh as a reminder besides photo policy strictly prohibits taking pictures of anybody without express permission of everyone who is in the frame so as these talks are being streamed you will have access to the content later so if you uh are concerned and would like to know more about the content you can look at it later and uh unless you're 100 sure that you have the express permission of everyone in the frame please refrain from taking photos please make sure you have your masks on at all times [Music] is it so with that i will kick it over to steve and tony thank you so much so my name is steve winterfeld i have um responded to both data incidents data breached at a loss vulnerability announcement zero days in banking in retail in government and so i want to share some of those lessons with you my co-presenter hello everyone uh so thank you we know you have a lot of choices for talks one thank you for showing up after a happy hour uh and also thank you for coming to uh to our talk this is me um we didn't really do intro slides and then you threw this one together for me so thank you um but uh thank you guys for being here we're gonna talk a little bit about uh zero days uh response to those i've been in infosec since uh the late 90s and whereas i feel like you've kind of been in the management you know senior track i've been the person who's like always been under that i do work i do work do you sometimes sorry i don't actually do but you know when you're the person you know with your boss breathing down your neck like what your next move is is very critical so we're going to talk a little bit about taking the guesswork out of that process and how to make you more successful dealing with zero days so let's get started and so a big fan of the office uh i'm not gonna lie when i went and looked for an example and i found this clip i lost about three hours watching uh clips of the office uh if you go to youtube and you do fire drill in the office i'm not responsible for the next three hours you guys lose but it's awesome and essentially it is saying you know dwight had a fire drill and nobody did anything so then dwight had another fire drill and it went the other direction and so we want to say that you know how often are we going to have a vulnerability coming out and what time does it come out friday at 3 o'clock so log 4 j is going to come out before the holidays on a friday at 3 o'clock how much of our life should we be given up and what can we do to kind of turn that around hey just a quick note the the other most awesome episode of the office is when dwight cuts a face off of the cpr dummy and then puts it on his face and he's like hello clarice remember that one and so when we get to questions if you don't have a question about our talk anybody's willing to willing to can try to stump tony on movies give him a live movie and he should be able to do it so we should have some questions all right so let's talk about some examples here's uh kind of a whole myriad of examples and you'll notice too that these uh arrive in your uh you know in your um life in a number of different ways right solarwinds this is like supply chain like a product that you're using where's that product installed who's the product owner who's the owner of the server that the products are installed on that's a very different story from like a large scale you know like heartbleed or log4j a protocol related risk those require a whole different group of people to be contacted and to respond to this incident and then going down you look at things that are kind of status quo parts of your infrastructure that you're using um you know they're they're obviously you know there was a great talk the other day about devsecops and how it's failing um it was actually in this room um and the funny part is most people always talk about dev and sec the ops part uh is all the other uh all the other things that have to happen for devon sec to work properly because implementation deployment um who's using those systems that's the operational aspect and then obviously stuxnet um you know a meltdown spectre these are things that most people aren't really prepared for in general right because these are these are vulnerabilities or you know exploit attempts will be happening against systems at the very deepest root of your of your environment so it obviously requires a different perspective and you know again this is a team sport so as soon as i hear about what's going on i've got two challenges where is it in my environment and where is it in my critical provider's environment and so you know you need both those process you need to have thought through on you know as soon as i start on my checklist i have a partner in vendor management who's going through so at nordstrom we have had and my wife is very mad that i moved away from the nordstrom discount but the we had 5 000 vendors so then we had a category of which of those vendors had our customer data which of them had credit card data you know some of them were just they were in our system so when we sold a pair of shoes they sent us another pair of shoes some of them knew who bought it what it was for and so you know that's that's the first one the next is where is it in my environment how many have a lot of confidence in their asset management system you're in trouble how many people have an s-bomb you know where all their log4j is and all that that is awesome working on it nice to see i have this going next to me y'all take your mask off and yell you have an s bomb do you know if it's correct if it covers all your stuff you know and then firmware how many had a process when spectre came out to patch firmware i mean for us that was a new process we had to develop and so i and is iot in your asset management system today or is that just a rogue system not considered to have data on it so it's not in your asset management and so the reason we built these out just like when you're doing data breach i think you know third-party data loss i lost data i lost access to ransomware you know there are classic scenarios you should go through in your planning and we think this is a framework yeah the last point too um especially when you're talking about where these things exist in your environment um we you know he just mentioned about s-bombs right a software bill of materials there's a lot of especially on the iot side uh the kind of the antithesis of s-bomb is do you have a problem maybe scanning into your environment because of risk of these fragile tcp stacks being knocked over from your asset identification scanning tools right so there's a there's a big play there are you really catching everything that could be at risk here from these types of vulnerabilities and if you're trying to figure out why the raiders are up there it's because i live in colorado and this is one of the few safe places i can put a raider logo up so while in vegas i'm going to take advantage of that opportunity so i think part of the problem today is we think of a zero day as an incident and these are our two classic processes to deal with an incident i'm saying that a zero day is not an incident it is a systemic problem and it belongs in a different box i think it belongs in crisis management it's not something that's going to be finished out in the sock more often than not it's not a quick remediation with our tools again it's a committee it is a large phone call you know when the ceo hears about log4j on npr on his drive in what's the first phone call you're gonna get that day you know and so it's notification it is having a coordinated response who needs to be notified for what at what level of incident does it um so again a customer's going to hear about log4j and they're going to call the call center and the call center is going to be asked is my data safe from log4j does your call center have any idea what they were just asked so again i don't think this belongs in crisis management because you need a message for people that call in the next person that's going to call when i was a cso for nordstrom bank is the feds the occ they're going to call and say what's a risk to the bank are you guys safe so you have this constant stream which is why i think we should move it out of incident response into crisis management and follow that same thing and so one of the things that is you know which systems impacted by wannacry log4j pick your pick your zero day of of choice solar winds what access was it to data tony was talking about solarwinds who was running it i don't care who was running it what was on it what kind of information was involved is it customer information is it infrastructure information those are different reactions those are different people i have to notify if there's customer information in there now i'm calling privacy they need to be on the phone call if we lost it i have 72 hours to notify somebody and so you see where i'm going um and then again i can't bang on this drum hard enough i need you to be in lockstep with your vendor management or whoever's going to own all that others and so for log4j what's a reasonable amount of and this is manual i haven't seen any program and if somebody has let me know to automate this where you're calling your key vendors and saying are you secure from log4j you have my customer data what's your plan and what are they doing they're doing the same thing you're doing trying to figure out if you even had that protocol anywhere so it's not a quick call and done no no go ahead uh so so this goes back to what we were talking about before uh there are a lot of different people if you look at the racy chart at the bottom this really depicts who is responsible for getting that ring uh into the flames uh to destroy it right um but uh you know when as you kind of map out who in your environment needs to be contacted there's some really kind of practical things you can do so one if you've ever done like a tabletop exercise think of something where you're under duress right so uh so we work a lot with uh you know large-scale web app attacks or ddos attacks and you'd be surprised going through a tabletop exercise uh before we go through the process of developing one you just say hey you're under attack what do you do they're like okay we're gonna call our two contact people in network ops and blah blah blah and then they don't answer the phone because it's lunchtime or it's midnight where they live what do you do next right who's the second person to call on the call tree how do you uh transfer responsibility at what point do you open up a bridge and you say hey we're gonna have a a bridge that's open from now until this this attack is remediated to where anyone who needs to know needs to be notified of what's going on they know this is the bridge to call into right there's a lot of procedures that can be done beforehand to save time because when you're under duress when you're uh you know when you're hard down or uh if you've failed over to your failover data center just say and god forbid your failover data center is not failing over um you know everyone starts to get red in the face they start to make poor decisions uh and you frankly you have you know somebody that looks like this standing over your shoulder just that was that was not cool uh but you have like true you have your boss looking right over your shoulder and you're just like i've got to focus i've got to remain calm the more planning you can put into this whole procedure here for who's involved who owns what who are the secondary contacts all that good stuff that has to be done beforehand um because the longer you wait when when it eventually does happen uh you're not gonna be able to respond uh in time right you're probably gonna have a poor poor experience and that's why i love that when you're building things like this and then you do the exercise you do two things one you discover that you know tony built this and it says create fellowship and there's two people responsible for that is that legitimate can two people own the process i tend to say a racy can only have one r well frodo was a key contributor to the ownership in fact some might say he was not the most equipped to take the ring on that and that's why i say you know i think i think you have it anyway um and so exercises are are critical to both make sure everybody knows who owns it how they own it if it's a partnership how they're going to come to resolution and also to set expectations so for those of you who have been involved in some kind of a major incident the first thing you get from the leadership is what's the status and we want an update every 30 minutes and then you turn to your sock analyst and you say you go do your work and when you have something come tell me and i'll update them every 30 minutes you don't need to come back to me every 30 minutes because guess what how long does it take to troubleshoot something until you figure out what's wrong you know i love those those hypothetical questions when are you going to know how much data was lost when i can tell it's not happening anymore and so building expectations building the team making sure people know their role i can't emphasize how important this is you know it is it is key especially now with the high turnover all of us are experiencing you know how many of us have that that one guy who left was the historical knowledge of all your firewall policies you know so so these exercises are really important so let's take a little journey back in time uh even further than this let's go back to 2014 when there was an ssl vulnerability called heartbleed that was disclosed uh and oh yeah we found out that it had been actively actively exploited in the wild for a previous two years to that right so this was one of my first big experiences when i started awkmy and to the point of you know who gets notified before anything was fixed we had to make a public press announcement right because everyone knows there's tls or ssl certs deployed globally across 350 000 servers uh that are oh yeah in front of big banks big social media companies big commerce platforms so we probably have to give some kind of notification what's going on um that that process what what's really interesting about this is we learned a ton of lessons from this uh and we we've announced this publicly um but the initial response was our implementation of openssl was not vulnerable to this exploit and we started sharing what our implementation was uh and then like three days later a researcher came back to us and said hey you guys are still vulnerable here's here's a poc of of the exploits still working so we had like a three or four day lack of a head start on going out and re-keying all of our our certs globally that's a huge impact and oh yeah when you go through that process you really start to find out man a lot of people need to touch a lot of systems at the same time in order to get this done expediently um so it's like hey can you spell computer congratulations you're on the task force right um so so uh you flash forward to the response to log4j and here we have um what that's i guess eight eight long years after that and we started uh seeing that all the things that we learned from those initial uh you know heartbleed and then later poodle uh we put into action to have a really proper response for log4j one the visibility around you know where it exists in your environment that's a real key issue um but more so you know we saw so much exploitation attempts uh against our customer base at one point i think 60 of our customers were being hit by exploitation attempts as we started to kind of weed out and we started to see patches come out then we could start doing a reduction of that and say okay these are vulnerability scanners that are testing for the exploit based on this vulnerability scanner code that it had and we can start to take that out but this this was a really massive massive problem and so this is where we've kind of diverted from our generic example to an example of a security vendor so how many of you work for a security vendor of some flavor so this is where you're good for you guys never change so are you protecting yourself and simultaneously looking at your product and how your product can protect your customer and so what we're talking about here is what we were seeing hitting our our thousands of customers on our web application firewall and this is where you really kind of break out and there's intricacies to this right so like we have an adversarial resilience team that really looks at how adversaries might uh and do try to bypass or do try to bypass my platform security so securing the platform means making it continue to run properly but then the other side is anything that's deployed on behalf of a customer how are those configurations and that implementation uh secured separately right so again it goes into the the racy chart again who's responsible uh who's gonna be brought into a task force um and uh you know if we go to the next slide we'll see that uh so the the red graph here on the right side is uh percentage times ten so these are six sixty percent at one point sixty percent of our customers were being hit um by these attacks and then you'll notice here as well as new exploit code is being released we saw a huge influx in attacks that test out that new exploit code right because anytime new code is released on a zero day there's a race against the clock like when is this going to get patched if i'm going to exploit it the time is now so we see like a fevered pitch here you'll see around the um a few days after uh christmas time here massive massive uh increase in attack traffic now some of this was attributed to uh scanning tools that got the previous patch uh and then they were testing obviously um but uh but again this is uh the interesting case about this again that i mentioned is uh 92 of the attacks that we saw were being blocked uh far away from where the exploit uh was was actually vulnerable so this is where you come into what what is your model for securing your environment do you have like virtual patching capabilities where you're the exploit may still exist but you're buying time by blocking the visibility of that you know from your uh from the adversaries right so this again it was a great learning experience for us and one of the key lessons is never let a good crisis go to waste and so i want to talk about the three aspects we have here the first is protecting our customer um you know on day one you have to get the message out to whoever your partners are whoever your customers are if it's internal i.t if it's if it's making sure your your third parties are working on it if you're supporting customers if you're a bank managing wealth you're notifying people what you're doing so you know that educating and getting the message out that you're on top of this is critical