Bringing order to the chaos of Security Vulnerabilities

okay great so um yeah this one working too yeah that one's working too okay yeah oh well uh they're getting set up I'm just going to really quickly do an intro so I'm not going to read through the full bio since we have two speakers here um but uh yeah we have both um make sure I have the right names here because I don't want to mess that up yeah so Lisa Bradley and Julia Hopkins uh both with Dell and part of the product security incident response team uh here is going to be about bringing order to the chaos of security vulnerabilities which is a topic that is near and dear to my heart so can't wait

to hear this one so um yeah I'll let you go okay I'm just gonna take this out does it come out hello I like the pace a little okay be good yeah all right some fun stuff so um I spoke earlier a little bit in the panel but Lisa Bradley I'm with Dell in my past um I worked at uh Nvidia and at IBM I've been in security for over 10 years I came from New York down here 23 years ago I went to NC State so we'll pack fans I got my PhD from there and um I uh call my team Versace because I'd like to have a fun name around it uh vulnerability

response customer security and community enablement so uh t-shirts underneath me s-bomb security Champion security training uh lots of other other fun things and I'm Julia Hopkins I'm a technical program manager um on the product security incident response team at Dell prior to that I was with Lenovo in their product Security office and then I also helped to establish their Chief Security office and I have a masters in cyber security from Utica College mm-hmm right so um I want to start off just in general a lot of people when they think cyber security instantly think the sea cert side so peace cert is maybe the red-headed stepchild of it um and actually uh it has um the name incident

but we actually don't deal if there's like an incident that happens in the company uh pcert doesn't handle it um our job is to best protect the brands and our customers by going through a typical process where we're trying to resolve vulnerabilities that are in the product line so P for the P circus is product so very product focused here go to the next one so um any company that has a piece should have a policy a framework that they're following first piece services framework if you haven't heard of first um there's a cert framework a piece framework um a lot of our our endowed what we follow is that framework where we start off with the discovery how did we find

out about the vulnerability was it us through SEL practices is it you know from our customer is it from a vendor uh researcher bug Bounty program whatever we're lucky in in April we started a public bug Bounty program for Dell I'm very happy about that um we go to the triage and Analysis so we found a vulnerability is it valid what's the impact what's a CVSs score so CVSs is the common vulnerability scoring system and um you know the higher the number the typically worse worse the risk uh goes from a low medium high and and critical um then we go into remediation figuring out our plan remediating the vulnerability what is our comms plan

um you know if we can't fix it on time there's an exception procedure that we we fall and then lastly the disclosure uh dell security advisory we call it but security advisory sometimes security bulletin security notice how do we make sure that we tell our customers about the vulnerability the risk that is there with the security update that they're going to get so that they follow their own policies to figure out how quickly they should address uh all along time we through this we have a high profile procedure so uh log J for example Spectrum Belltown harp lead but even sometimes for us even sometimes we could have a um a proprietary code vulnerability that

we could call high profile because um some big researchers are going to disclose it working with PR and all that um governance and reporting a lot of metrics kpis um some of the key kpis that we do are focused on um how are you adhering to our standard how quickly are you addressing your vulnerabilities within our slos how are you doing with your open source component management are the customers finding the issues or are we finding the issues um and with proprietary code are you know how many are finding from researchers versus us how well is our STL process and lastly stakeholder Communications so out to the comms to the rest of the business so this is our formal part of

of our talk here uh where we have it and then and then we'll we'll try to have a little more more fun as we continue on here Okay so so I'm gonna put my uh my researcher costume on here so as a researcher what are my overarching goals here what is what is my motive um as a researcher it's getting payment after finding a valid vulnerability it's uh setting my reputation you know getting my claim to fame uh being able to go to conferences and speak about my Discovery uh it may be helping customers um protecting them helping to keep them more secure just people in general and then there's a consideration of future employment if I find a valid

vulnerability I submit it to a company you know will they bring me on so but what I don't know is what's really going on behind the scenes when I report that vulnerability to a company So Lisa maybe you can yeah the curtain back so so we will walk through this but right now my focus is on my customers and my Brands right how do I make sure that I'm protecting them and making sure I'm adhering to the industry standards to what my customers think my customers think there's a lot of pressure that I have to make sure that I'm protecting the company and when I'm thinking about those goals my end goal is is please

don't get the media involved I don't want to be that label that story I don't want to have to do that I don't want to have to tell the president or the CSO all of the stuff that's going on because really I'm trying my best to handle this um but we do have some common goals I think in general we want to protect our customers and you know help the ecosystem like we we both want to do that

so walking into the first phase of discovery I'm a I'm a researcher I'm submitting the report what do I need to think about and what questions do I need to ask what's the product scope here is it is the vulnerability just in this one product is it in other products does it affect this one company or are other companies going to get involved that I can submit this report to oh crap I just got a vulnerability from a researcher here we go um is it valid let me think well first did they even give me enough information can I even understand what they're trying to tell me do they tell me what products affected that they is the product in scope should

I be paying them you know does it fall into my bug Bounty program am I going to acknowledge them and I'll have a have I worked with you before I hope I worked with you before because maybe we have a good before if not I have no clue what I'm going into I hope they don't zero day me I don't want to be zero day like let's make sure my team really has a raft together and is talking to you have we talked to you did we oh my God device spawns they respond back to that vulnerability that you gave me not yet okay automatic response went out okay I I forgot I had automatic response

this is great um but you but when do I get to talk to a real person Yeah well yeah so we're looking through um I'm figuring it out uh I don't know about this issue okay we're good um is there enough though that I'm going to triage it and the security team or am I going to go bring it to the product team am I ready to bring it to the product team am I ready to go and now scare the crap out of this whole product team that they now have a researcher of vulnerability I think I'm ready where are we going to next next we're going to severity what's the severity I didn't even

confirm the damn vulnerability yet I like I got a little more work here um I just got everybody known I opened up the ticket I put all your information in there I think there's enough so I'm gonna now do my triage and Analysis and I'm gonna figure it out so and when the hell do I get paid yeah not yet not yet I didn't confirm it yet I'm still working on that I think you might have missed something like someone's submission on submission where where can I even submit this there's a bug Bounty program that's an option if the company has one set up uh I do security email I have that too awesome and and if I want to write a

paper on this oh please don't write a paper don't write a paper but I definitely want to tweet about it wait don't don't tweet yet don't tweet don't give everybody away don't nope and how long do I have to wait until I can do all that yeah well let me confirm it still and if you make me wait too long I'm just gonna zero day you yeah yeah so so we're gonna go on to our our triage here I send it out to my product team they're analyzing it and we can't get from Step a to step C there's thinks of B like can you give me a little more information dang now they're asking me all these questions

did I give them enough information no don't feel like I did you didn't um um you give me some more information maybe you could record yourself doing it and give me a little video and here's my proof of concept damn okay I have the proof of concept I can recreate this it's a vulnerability I don't know about it I now need to score it how severe is this does it only affect this product I found that it affects more products so with broader impact do I get a bigger reward now um no uh but wait a minute I gotta look is this my code is it my code is it my odms is my oems is it my vendors where is

this vulnerability who's whose problem is it is it really my problem can I pass you off to my vendor it would really be nice that they had to deal with it uh instead of me and then maybe they could pay you but they might not have a bug value program so then do I pay I don't know if I pay do I pay what did my rules say um maybe I pay I still probably should pay you found something I'd like to be paid yeah yeah I'm sure um so I determine it's my issue so I've got it I it's valid okay I'm gonna now pay you but I'm gonna pay you based on the

severity that I think it is so I think it's a a you know a medium it's a medium here it is we we got this we score things all the time I don't agree they're coming back and they're giving me a lower severity score yeah I think this should be higher I can maybe talk to you do I want to talk to you do I want to have a call do I want to just email I'm certainly not going to let my product team talk to you they will mess this all up um so I'm going to talk about how we know how to score what we think of this talk through the vector do you agree

with me yet I think we're getting close Okay Okay so let's go and talk about this we can have a call do you want to have a call let's have a call okay we'll figure this out so we have a call we're good we're good severity confirmation and agreement yeah I might have moved it up a little bit they were right about one thing so it moved up but it's still a medium but it's like a little higher medium higher medium still important or should we call it a high I don't know we're close we're around there um okay so right now I'm lucky it doesn't affect everybody if it affected all of my product lines I would be calling this

a high profile um also if it was coming from a vendor I'd be working with that vendor uh odm and oh my God let me hope that that odm I actually have a contract of how quickly they should address their vulnerabilities because if not they could take a really long time um and uh you said you're gonna vlog oh yes I intend to blog okay we could cover it they were good um hey if it did though affect a lot of products um are you interested in working with cert or some coordination and it really would help you it would help you certainly help me but it would help you if you were with them because if you try

to coordinate working with all those other vendors it might be difficult yeah what does that entail I think you just tell them and then they they handle all the coordination it's sort of impressive all right yeah yeah okay we're gonna just say right now that it just affects my product we're going to go on but if it we would have a whole nother story of embargo days matching embargo days everything with all these other vendors uh hopefully we would utilize someone like cert uh if we wanted to um and then it would help our coordination ah here we go you ready so now when when do I get to tell people about this yeah this is getting disclosed yeah

um so I just released a security update or my security updates and with my features right now so I just had a big release um so we're just getting started on our journey here or oh crap did we did we not release yet now we're we're about to release we didn't release I got a few more weeks we're on a code freeze I'm in trouble I need now to tell you that it's going to take us a little bit longer you told me it was this date though yeah I'm gonna zero day you if you please suck on this please don't zeroid Amy I'm really trying here but you know I have to get my QA resources aligned

and wait did I even figure out how to fix this yet um okay it's gonna take a little longer it's in this component and this component and you know it's sort of complicated we have to grab a paper around it and it's due at the end of the semester yeah yeah okay uh knock knock exact door um we have a researcher reported issue uh we have a timeline that we typically can't meet right now even though the team's trying their best to follow our slos our standard but um um yeah no uh we're gonna get zero date if you don't fix it by this date um you need to fix it by the state because I

really don't think you should accept the risk here of getting zero day they plan on blogging um okay I got a date you good with the state now what do you think I'm good with the date okay I'm ready to start coming up with my name and brand this thing no please don't fan name it no no no no no you remember like that bad luck sadlock thing like you don't want to be that like is this big enough it's big you of course it gets big I think it's not that big it's only one of my products maybe two that are affected I don't know should we like what do you think I think we're good we're good we're on

it I think we're good I think we got a remediation date or and we're in agreement yeah uh meanwhile I'm talking to PR I'm talking illegal I'm talking to the product team I'm making sure they're on top of things I have a reactive statement then I'm ready in case the media picks up on this but the media shouldn't pick up on this that should be good I should be good I'm good um I have my embargo date coordinating with other vendors if they were affected they don't screw it up um um hey are you gonna blog oh yes oh can I see that okay I'll let you review it okay PR is all over this damn blog what am I going

to do they want to fix like 15 to 20 to 30 things on this blog I'm not going to screw this researcher out I'm not going to do it I'm gonna pick my top five things and I'm gonna go to this researcher with um oh my God they have proof of concept code in this blog I can't proof of content code it's going to kill me my customers don't even have any time to fix this yet um knock knock researcher can you can you not put that proof of concept code in um in your blog can you please wait like even if you give me like two weeks just wait a little bit can I still disclose

the vulnerability without the proof of concept yeah certainly I'm gonna have to put a security advisory out anyways so I'm gonna put that out I'm going to acknowledge I'm gonna acknowledge you yes right um so you're good uh yep I'll have the cve I have the description I have all this if you want to see it you could see the security advisor I would love to see it remember my name in it yeah we already agree with the score just remember that we we agreed on this this is the agreed score that we have in there um but if you could just I'm okay with you blogging just hold off on the approval console code can you do that

for me we'll hold off on the proof of concept code yeah like a couple weeks two weeks yeah perfect great thanks good we're good I got this we're good you feel like we're good you got it I think we're good we're good we got this got it good okay it is disclosure day I get my claimed Fame today I got my cve I contacted miter they're activating it I published my security advisory I'm all ready to go I'm tweeting no no no don't tweet yet Tweety don't tweet like I'm blogging any resource just told me that they came down with covet and I need like one more week one more week delete one more week

yeah we're all good okay got the fix out media looks pretty good calm actually I don't see any media that reactive statement I didn't even need that all that work I did I didn't need that I really need that what are they doing all these articles I Was preparing all my customer support not only my customers aren't even calling they probably aren't paying attention to it I don't have that API thing going on yet so you know they're human readable they're they should know about it I transferred my risk onto them they should pick it up but we're good we're good we're good did you tweet I did oh crap okay I tweeted come up with my name

I've got my conference abstract ready to go it's been accepted now I just need to create my swag that damn sweet that damn tweet got picked up by the media yeah you're doing I'm famous [Music] um I now have 50 customers that just called me today I realized that my security advisory is not clear enough I need to update it because they're confused of what they need to do did the media tell them that we fixed this if the media statement isn't we fixed this we got this we're in trouble your that Media blog post didn't acknowledge it didn't even list it so now it looks like I'm affected I have no fix I got zero Dade but in fact I I

really have this I do media reach out to those resources get them to update their posting please because we already fixed this we got this we fixed it we're good it sucks I had to go tell my CSO I had to go tell all these execs I had all these people emailing me tons like I don't even know these people they're emailing me we have it though we're good I maybe made the stock dip just a little like a little but how you doing over there and now Defcon is coming next week and I am going and I am presenting on my Discovery okay um can you share that with me I can thanks that's awesome because I want to

know what's coming um hey I'm I'm reviewing this and you have a new vulnerability in here that I didn't know about yeah okay media Dr let's get ready for this we fixed this one part we didn't oh wait can you not show the proof of concept for that new thing that you were just showing me but it's the conference um this is the whole purpose of it okay not exact but SVP exec we have a zero day on our hands within a week but wait maybe I can reach out to Defcon um maybe I could tell them like hey I don't have a fix for this yet maybe we could negotiate through the conference to

yeah no okay it's not working okay man conference is coming I'm ready PR is ready legal's ready I got that security note so I have security advisories security notices this is now a high profile right I'm now going to be zero date I only have a week I need to have some statement I fixed one part of this we're working to analyze the rest of them uh the rest of the issues that you put in that presentation but we got it wait I'm going we're good I'm good we're good next time like I do have a bug Bounty program so remember that because the next time if you went through that I would pay you for the other

vulnerabilities you found I intend to get paid for more vulnerabilities yeah but it was very nice working with you it was wonderful working with you yeah yeah okay and then the learning the post-mortem what'd you learn I learned that communication coordination it's going to be to the researchers benefit I'm still working this issue we have a root cause analysis procedure that we put on on determining what SEO controls might have been missed I'm doing a whole Lessons Learned did I have enough guidance why didn't I find this why didn't we find this first what do we need to improve on can I do this better how do I get the media to understand what's going on can we

pre-talk to Media next time and we know it's going to be hot I am working this I'm going to be working this for the next maybe month or two you might not know that I am still working it um I'm documenting I'm updating policies procedures all that they do not take overnight I have to get all my stakeholders to approve it I just change the control that means their maturity assessment is at that level pull it all up because they didn't do this one thing but if they did this one thing you wouldn't have found the issue we're going to find the issue next time I hope I hope if not you know how to find me

make sure you find me please find me I do hope you have more reward for me yeah yeah good okay hope you enjoyed our fun our fun skit

there any questions yes

yeah yeah yeah so um we actually had a private bug Bounty program for uh I believe like a year and a half before we even did a public one um so we did some private programs and then we had our application one I think it's good to start in the private space as you get the teams used to it um to think about funding how are you going to fund it uh are you going to go to each of the teams or the business units are you going to fund it yourself it's a lot to go to the different business units but then if you take that funding you're responsible for keeping funding um executive

make sure the executives are aware you might have to prep media are you going to do a soft launch you're going to do a pro like a big launch um and there's a bug Bounty community of Interest which is um there's a lot of people in the industry but it's you don't act as that industry in it and there's a lot of good advice uh that you could get from learning from other people they posted a few blogs and some things already we could certainly talk offline but um a few of us here uh have a bug Bounty program Intel has a really fancy one um I'm still simple um as I I learn to grow with it because

I didn't know what was going to be coming um for me we have it on the application side like ardell.com and our product side so you know when you have the web apps you will get a lot of activity yep any other yes

yeah it's a the question is is um if it's a uh if the issue is an oem what do you do um and I have to tell you it's sort of not a one-size-fits-all it depends on the maturity um of the odm um is that odm even a naming Authority do they even ever assign cves do they have security advisories all of that um if they're more mature uh we certainly can figure out how to pass it off but there's other things that go in the background like are do we even publicly say that we use that odm for the code or not or do we just you know white label it as our as

our company um so it's not a one-size uh fits-all um in in the past we've had a mixed bag where we would just assign the cves themselves you know for for that odm but with the bug Bounty program it's brought into another complexity because it's like do we pay or do we push it off to the odm um in the end we want that researcher to be happy um so most likely we'll probably still still pay or figure out some way to pay or some kind of recognition um but it's certainly not a one size yep

a lot of researchers in English is not their first language

it's difficult ones you want to take someone um Google translate helps no it uh it can be hard it can be difficult um but uh I mean if if you can get translators in um or bring in the um the technical expertise if it's around the technical language leverage your resources where you can I think email is is a useful resource instead of getting on that phone we we tend to not get on the phone with researchers all that often um because you just never know what's on the other side um and and so that unknown can be really scary in in the call um so email gives it a pace because you can take your time to respond or

understand and then the the receiver if English is in their first language can take their time to re to respond yeah getting on getting on a call with a researcher could actually bring in more hostility especially if they're already in that state um so yeah email yeah it depends like there's some researchers that they're dealing with 50 you know reports all at the same time so they're busy they're not stressing they know you know big companies got this but then there's some researchers where they just think that they found that the the biggest vulnerability that there is and they're only focused on you and and you're in the background we're dealing with I mean we talk about open source we're dealing

with hundreds of vulnerabilities all the time so you know managing that and making sure that your team is staying up with the comms to the researcher even touching base you just say hey you know just touching base we still we still good like everything's still on track um it takes two seconds to do an email and it might save you from a zero day so I can't express how much to to communicate okay oh back there

yeah so the the question is is starting up a bug Bounty program did we see an increase in malicious activity um so through the bug Bounty program we utilize um one of the more top ones you know there's hacker one there's bug crowd um there's what Integrity Integrity there Intel's using um so um there's supposed to be some safeguards where they're watching like if um if we have an incident where where a researcher went too far um we reach out to the program and the program we'll talk to them give a warning half the time it's a lot for Education they didn't realize what they were doing they were so excited into what they're finding that they dropped a

web shell because they wanted to keep finding more but you're not supposed to drop a web shell um in us um so you know I think a lot of that is educating the researchers to say hey you you can't go that far or or you know if you feel like you want to go further talk to us um so I think we saw a little bit of an uptick of it but I do think in general people were having the right intentions attackers it's it's their livelihood they don't want to get kicked out of one of those platforms um you know there are certain people that we did a like a soft launch with our bug Bounty program we didn't go do a

whole media Spiel or anything around it um basically because it was our first time heading into it we didn't know what we were going to get um and I think that slowed us into the process so it's it's a mix I think having a bug Bounty program though I a few years ago you would have said bug Bounty and I said no no way but you know the whole industry is doing it now and these researchers are actually finding really really good things um and if we figure out how much we could what we get out of pen testing versus the bug Bounty program we're realizing per money per value we're getting a lot more out of the bug Bounty

program because we only pay if they find something where in pen testing you pay regardless if they find something or not so not to say that you should not do both you certainly should do both but it should be part of your sdls is extra testing yeah oh there's a question up there

oh that's a hard that's a hard question the question is is how does uh dell compete with nation states who are um for for their finding bugs um I I don't think we can compete uh I think that um you know this is why having a good relationship with your researchers and growing that relationship makes sense so that the researchers actually want to come to you they want to do the right thing um but we can't stop those that don't want to do the right thing unfortunately we could just incentive like try to bring them to the to the side um but uh I mean we can't we just can't pour that money into it you have to

think with the bug Bounty program I love it and I like rewarding researchers but that same money is when we're not maybe taking away from other security practices so we have to have a good balance and we just can't pay the way that that some others can okay thank you [Applause]

I have to say bug Bounty program single best investment I ever made in our security program love all that thank you so much