Survey says…Making progress in the Vulnerability Disclosure Debate - Allan Friedman

um my name's Alan Friedman I'm from the US Department of Commerce and I'm lucky in that I don't actually have to do much talking in this I'm just going to tell you a little bit about what Commerce has been doing and the answer is we've been having really smart people who are dedicated about the security community do a lot of hard work to make progress on a fairly old problem infosec vulnerability disclosure so my corner of the partner Commerce the National Telecommunications and Information Administration is convened in open and consensus driven multi-stakeholder process focusing on vulnerability disclosure it's been about running for about nine months and we've had brilliant folks from the security research community from the vendor

community from the intermediaries who help promote that you've done a lot of work initially and we're going to hear from the three working groups that have been very active over the past nine months one on awareness and adoption another on safety and disclosure and finally on multi-party disclosure so this work that you're about to see is not from the Department of Commerce it is from the stakeholders who believe that this important issue at the end of this talk we're gonna have a fairly brief talk and the goal really is to have some discussion to your feedback so that you can share your perspectives on what we're missing and what we can do better to bring about some positive

change so very briefly Department of Commerce likes it when markets work online Trust is a huge priority for our secretary because without trust and systems that we use there's not going to be innovation there's not going to be adoption markets will fail and sometimes to fix the market failure you need active regulation you need to weigh in with the big stick of government we believe that they're actually often lighter touches so we want to bring together those who care about this issue and say how can we have collaboration around vulnerability disclosure now this isn't a new debate we don't want to reinvent the wheel we want to find some way of saying there are standards out

there people have been thinking about this issue for a long time how can we actually make some progress so rather than trying to write new standards or even produce best practices we're trying to come up with some principles of what we can do to of what we can do so that researchers who want to continue to engage know how to engage organizations who are new to this issue understand what's at stake and what the path forward is and the underlying approach is there is no one-size-fits-all that every organization is going to have something that's unique to them and different researchers each bug is at the end of the day going to be unique and need to

be handled differently so what are the broad issues that we can do so the process has basically involved a lot of talking it has involved some very tedious meetings and even more tedious phone calls so the people have been engaged we really have to thank them for all the hard work you're going to see the working group chairs but behind them as they will say a lot of very smart people were dedicated to this so first we will talk about the awareness and adoption group and what's noteworthy about this work is that I think everyone this process is that at the end of the day raising awareness and driving adoption of existing good practices really is the most important thing and

relates to all the other work that's going on in vulnerability disclosure Jen sorry Jen Ellis from rapid seven and Amanda Craig from Microsoft so the big learning that was the album does all the slides from now on okay so as I'm jealous I run community and public affairs arrived at seven so I had a public engagement and also think about how we can support the security community you guys a little more which is where this falls in Amanda Craig from Microsoft and I work on cyber security policy issues hey rattleballs in this is josh go huh okay so why why this why did we do all of this and and is it just because we wanted to sit around and come

by our I think you know ultimately as Alan said there's been a lot of work that's gone into talking about vulnerability dispersion handling in the past 20 years lots of incredibly smart people have worked on this problem and there are very well-established best practices for both sides of equation for for researchers and for for vendors in fact there are I so there are two of them however there's not very much adoption and so the the reality is that the problem continues and that if we can't solve adoption everything else that we do is just academic it's just a lot of sitting around talking to people to death as I once said so for us that

was why we really wanted to focus on the Adoption Awareness problem is to make the theoretical applied and actually try and see some change occur that was that was a big kind of focus for us so what we're going to talk to you about today's surveys ooh sabes so sexy they be even less of you in the room if you'd know this is all um so we uh it's kind of funny we we were talking in the beginning and there were lots of people in the rooms oh you see people we think that I black um so at the beginning we were talking they were lots of different voices in the room from all different sort of sides of

the conversation and there were lots of people using the kind of dialogue you often hear in this discussion lots of assumptions being made and we had this sort of like very side conversation and I one day about how you know if we're going to get meaningful about coming up with ideas for adoption which is what we were focused on then it would be really good if we could get to the bottom of what was really going on with the surveys and like really understand the truth behind it and so the idea of the surveys initially was something that we were going to just basically put together in a week and get out and it was gonna be like a really quick thing

that's not really what and so they became this like huge thing and it ended up not being one survey it was multiple surveys and talk about the who we surveyed and why yeah so you know the jen has mentioned some one of the reasons why we initially came upon this idea of a survey was to challenge our assumptions and sort of you know challenge everyone that's been part of this conversation for a decade or more to rethink you know have the norms changed but there are also you know new players in this space there the two folks that are newly technology providers automakers aviation companies medical device manufacturers and so on that are newly dealing with this issue

of vulnerability disclosure and so we wanted to also capture you know with what's going on in their world and and we did we were hoping that we would get some real data from these surveys that would help us identify ways to drive greater awareness and adoption and so we thought about different communities that we would serve a very obvious to survey the technology providers and operators and the security researchers as are two very central players to this we also did consider doing it our survey the reasoning behind that was that you know if consumers care about security that of course helps to drive adoption of security practices like having a vulnerability disclosure policy in place for vendors so we were

interested in knowing to what extent this is a consumer issue or a user issue we ultimately decided to not do a user survey jen's going to talk in a few minutes about all the sort of method methodology issues that we faced and disseminating the survey and the bias that we likely captured and surveying the veteran whether we did but all of those issues were really really exacerbated in the case of a user survey because it was just going to be really difficult where this survey the and the destination of the surveys was just done by the awareness and adoption working group as part of this NTA process we didn't really have any funding or any

expertise in and surveying and so you know we were we were just going to be doing what we could to promote the surveys and so we felt that the responses that we would get would just have incredible bias it would either be totally random from users or would be people that were already really really interested in this topic and that would that would be the reason why they would notice this and respond to the survey so we often Utley didn't do that what did we do then we did do the technology Frederator or vendor and the security researchers survey we tried to make these surveys really short really simple I think they were both like nine

questions you can see in general we were just looking to figure out what what do you what are you doing and why what is your general expectation and what is your behavior what is your rationale for that behavior in the process of researching in the process of disclosing or receiving and handling of vulnerability just want to highlight that you know these questions the fact that they were so simple we think helped gather a lot of responses but the flip side of that is that you know there were some limitations and the questions so for instance you know if we were trying to understand that there would be a vast array of experiences researchers would have and we want to

capture all of them a lot of times we enable multiple responses to any single question which then made the data hard to interpret for instance we asked a question about how interaction with a vendor was and we had fifty five percent of researchers say that it they had really frustrating experiences and communicating with a vendor and we had sixty percent say that they had a really productive conversation with a vendor so I can see those numbers don't quite work out and we but we took what we could from from the responses that we received so as one said we surveyed the internet and we have been mocked for a huge amount the nice thing is when you survey

the internet the entire internet mocks you that sir very intimate and they also healthfully provide feedback on your survey after it's already out in the public domain and I mean it's fine because data scientists tell me that there is no validity to the survey anyway so we could have just started changing it but we chose not to do that because it seemed kind of skeezy so we had a lot of learnings in the process you know the reality is like as a man who said we were kind of highly limited you are all playing spot the meme now on you [ __ ] so yeah so we were sort of highly limited in that this is you know

it's a volunteer organization it's not owned by anyone sort of entity and so we were limited in kind of what we could do on the surveying we wanted something that was free and open and easy so we did go the sort of survey monkey routes and I think if I was advising somebody who wanted to do this as more of an academic exercise I would have not to do that but you would have to pour a lot of money into it honestly and we had a huge number of learnings along the way one of the things that we found out is the vendor survey didn't really work for the open source community and that was a

little heartbreaking because we really wanted to capture them we also had used some of serving monkeys kind of standard like yeah demographic splits and for the verticals with a vendor one that was really dumb because one of the categories was technology and a obviously a lot of people identified that way interestingly not everyone did so there were definitely some pretty pretty huge learnings for us in this process but we did try to I would say like push beyond just putting it out there and dumping it and leaving it and we knew that that was never going to be effective as a thing to do anyway so we got really active with trying to get media to promote it trying to get people

we knew that had wide followers to promote it we reached out to I sykes Amanda did a phenomenal job of talking to a lot of vertical alliances a lot of vertical ice axe and getting their members to do it and we could see that coming through as the results were coming through oh is this me again [ __ ] ah so how do we do so on the researcher survey we had 414 responses now again we surveyed the Internet's 414 may not sound like a particularly high number but when we went into it we were very hesitant about what we would really get in terms of response and the stretch goal that we had set ourselves was 250

for each survey so we were really delighted by 414 responses and the data actually did show kind of a pretty decent variety in who we got in terms of whether they were people who were doing research as part of their job and that kind of stuff so the the largest the largest percentage of geographic split for both surveys was definitely the US which was not super surprising since the NCAA is the US government entity however we did see some other governments pay some interest in it and and some people kind of promoting it internationally so for the researchers we got 210 us responses and then you'll see the splits here so half of the people who responded

listed themselves a sort of quote/unquote independent researchers it was nice the thing that I thought was awesome was that we actually got people who were accidental finders responding those were the people we thought would be super hard to reach we didn't think that like the way that we were disseminating it we would easily find those people so it was good that like people who don't consider themselves to be professional researchers took the survey and that data is I think very valuable and then in terms of the tools we had asked this question of how all kinds of tools people use I'm not sure the tools is really the right word for some of these things and this question

again was one that I think kind of became a little crazy and bloated over time when we were preparing it and so ended up covering a sort of weird mismatch of things but you can see I'm not going to talk through all the stats because they're right on the screen in front of me so you can read them so yeah now getting into the actual data that we got from the researchers survey we asked a question about how researchers disclose what's their first action and disclosure and then also a separate question about what their expectations are and disclosing and how that influences their behavior so with the for the first question the response we got back is with captured in the graph

which is like 67% just disclosed to the vendor something like 10 and 13 h % disclosed to a bug bounty provider or to a coordinating organization like CC and then just four percent approximately either just don't disclose or go public and full disclosure so we were excited to see that what we kind of consider coordinated vulnerability disclosure with caption in that circle was that sort of predominant norm but then oh sorry we did look at you know what how you know expectations change behavior and we saw that when a researcher submits of vulnerability to a vendor and then and doing so they also provide a time frame and then that time frame is not met by the vendor then

we'll go public so 24 ish percent of researchers said that they had had done that around seven percent said that they the vendor provided a timeline when they disclosed and then the vendor didn't meet that timeframe and so then they went public but about eight percent said that the vendor provided a time frame when they disclosed the vendor did not meet that time frame and they considered but ultimately did not go public and I think it's just coincidence that it looks as if the researchers were giving us a singer so unsurprisingly well I thought it was unsurprising but then I work in communications people said that communication is valued shockingly people like to know what's going on and

feel like they've been heard so we had ninety-five percent said that they expected notification when the issue was resolved which seems kind of reasonable sixty-eight percent said that they would really kind of like regular updates that seems like a pretty decent reasonable thing so they know that something's happening and fifty-seven percent actually kind of went further and said they'd actually like to be involved in testing the facts to make sure that it really is his Co sharing works and and that was kind of interesting in that there were researchers who were like we're super happy to stay involved in the process we want to partner with the vendor we're not just like hey this shits on fire and then running away

screaming which is cool like we appreciate that about researchers and then eighty-four percent in a similar vein said yeah totally happy to answer question for then from fenders want to stay engaged once stay part of the process and then trying to look at the rest of the old and when I can't get out that so yeah so communication was really important people really value that they expected the vendor to be open and transparent with them and apparently I have now done there we go yes I don't computer thanks Josh I was this me again damn it how do I get some slides um so this is yeah this is a Shiva there's a reason this

one's me because I kind of talk about this issue all the time so we hear a lot about the chilling effect of specifically the CFA and the DMCA and specifically with both of those with that they have civil action in them which means that a lot of vendors use them to threaten researchers when they're afraid of disclosures so we hear a lot about that but it's always a little hard to know whether that's something that is a little bit sort of funny and overhyped it's not sixty-four percent said that fear of legal repercussions is a serious serious issue for them and it it seems as if it is something that makes them question whether or not to disclose to the vendor

the last twenty six percent responded from prison not true by the way for the recording it's not true uh so yeah um and then we also had twenty four percent said that they are afraid of stumbling into confidential information um only twenty four percent the others are like whatever that's great let's see what I can find that's why I do the [ __ ] thirty-one percent said that they are afraid that exploits will be used nefariously I don't know if we had the word nefariously in the survey but I hope we did another finding in asking what does it a researcher expects in return for disclosing a vulnerability was that it's not just about the money

yeah consistent with what Jen was saying about communication we had seventy percent say that they expected to hear from the vendor after they submitted a report fifty-three percent said that they expected to have some recognition only i think twenty percent so they expected nothing and then fifteen percent said they expected a monetary reward and you can kind of see by those numbers and how they add up that this was one of the the answers that you could check many boxes so you weren't limited and saying you know if you wanted one thing you couldn't want another and still only fifteen percent said they wanted they expected a monetary reward when it is a different story probably but expected

yes our interest how many people are surprised by that show of hands like this was one of the biggest surprises for me i was really surprised that because we hear so much dialogue in the community about you know how people are selling vulnerabilities because there's a market for it now and if we don't have bug bounties and we're not going to be comparable and all that kind of stuff and the reality is only fifteen percent care about that which is kind of awesome by the way like that means that this community is doing this stuff for much higher purpose and that is fantastic i love you guys you were amazing fortunately a man is talking through the

next one this we're now moving to the the survey of the technology providers and operators or the vendors so I demographic and we know no it's good okay um demographic information as you can see there was the issue that of the survey monkey self populated a technology being a huge like that percentage of our respondents that may be slightly skewed because we did try to reach these newer kind of technology providers than that but it will say quick anecdote on this one of the meetings that we had I had I asked a question about technology providers in the room and they're asked shield something else for show of hands as i am want to do and there were a whole bunch

of people who didn't put their hands up and I knew that they were from what I would count as technology providers and so doing the thing that Allen tells us not to do I put them on the spot I was like did you know answer that because you don't you can't like publicly answer it or do you just not identify the technology provider and they said we don't identify as technology providers and that was to me super interesting these automakers and they like totally have this this that way car makers that's what we do that's how we've self-identified for a long time so even though now we're dealing with things that have millions of lines of code in them we don't

consider ourselves to be quote unquote technology providers so this this actually he as much as it seems kind of like really obvious we would get this result it was also slightly surprising to us so just like the the researchers survey predominantly our responses were coming from the US we did get some 10 to 15 ish responses from UK Germany Japan Canada I think and then the other thing to highlight here is the divide between large and small organizations we had if you can imagine a thousand employees as the dividing mark there I think does it what 160 respondents were smaller than that 125 were larger but that our respondents that was heavily weighted towards having yeah being really large

or really small so having more than 10,000 employees or less than fewer than 100 employees sure just very briefly on the math side of things one of the ways broke down the respondents using clustering analysis due to the wonderful Brutus is to say hey listen we noticed that roughly half of the participants seem to be mature they've thought about vulnerability disclosure and roughly and have implemented a number of different practices for different reasons and then roughly half were not mature they didn't have many practices so we can learn something about you know what does a mature organization look like versus an immature organization when it comes to vulnerability disclosure Thank You Ellen and one other thing to highlight there

is that we didn't necessarily in the respondent star survey see a split among large or small organizations being over well overwhelmingly mature or immature recognizing that likely outside of the context of our survey and what we captured there might be more of a difference but there were a lot of small organizations that still kind of met this maturity model criteria so one of the the things that we captured is what are mature organizations doing so we were had a kind of a bar of some five percent are higher for for what we were wanting to highlight having a dedicated monitor monitored paths for investigating triaging and resolving vulnerabilities having a process for providing end users with alerts

providing researchers recognition and vulnerable and having vulnerability reports and for mr. kurtz security development lifecycle we're all really have lots of organizations were active consistent with those best practices versus for the immature organizations or the less mature organizations for each of these they were between eight to twelve percent following these behaviors

okay so why are they doing this stuff the ones little mature why so excuse me sorry so basically the biggest thing is always going to be that your customers tell you to you know people people vote with their feet and if you feel that your customers are not going to buy you are going to change your behavior so in the more mature bucket we had seventy-nine percent say that their customers care about this issue whereas in the less mature nine percent we're kind of like our whatevs um and we could you know theorize who falls into which bucket but we weren't that's for in the bar later the other the other big ones were I mean certainly like corporate

social responsibility there was it was a little woolly year on that but not not super low so the more mature was sixty-five percent and I just can't see the number for the lower one and for the further it reduces the cost which we thought would be a really big thing the more mature was just a little over half fifty fifty four percent so we asked them which best practices they were looking at and we gave them three options so one they were deriving their best practices internally to we actually the multiple options but these were the top ones so one they were deriving them internally to they were looking at the ISOs and three they were looking at what

other companies are doing Zen want to guess which stat goes with which thing so other companies anyone want to shout out stat know the one that none of you gets fifty nine percent said they were looking at other companies I so's 49 absolutely which was a little surprising to us honestly and says a lot about the availability in there and the awareness around the ISOs and then seventy-six percent were deriving their practices just looking internally or what made sense for their business which is actually kind of Awesome and I always have an archer slide so we have this it's actually we're going to be doing this bit after you've heard from the others but this is where you

kind of going to help us brainstorm ideas for driving adoption thank you very much sorry this was a little and thank you amanda and jen and now Josh Korman who shouldn't be a stranger to many of you has offered us that slides coming later some the work that's been going on on the safety and disclosure working group and thank you josh for coming and running an entire track on your own but still come up here to share what you're working group has been focusing on a shin of a public coordinated vulnerable exposure multi-stakeholder consensus based process was that we didn't want to relitigate the vulnerability disclosure wars we didn't want to reopen old wounds we didn't want to get into responsible

disclosure fights so there was a lot of polarization coming in but what one thing is that I realized through the work we've been doing with I am the cavalry is that a lot of these safety critical industries there at year 0 of their journey and 15 16 17 years ago Microsoft was sending cease and desist letters to our friends and now they're giving them six-figure cash prizes and celebrating remember and it's a vitally necessary part of their research and development and their product and your customer satisfaction and whatnot so I call that mean time to lighten in about 12 to 15 years and we're going to have a similar journey for these think you could like they are

at here 0 or year one and I don't expect that we can afford 15 years for safety-critical industries but maybe it's going to take one or two so what I tried to put on the table early is that this is where bits invites me flesh and blood and where the consequences of failure will be measured in credit cards or records loss but in your lives in national gdp in crisis of confidence in key markets that are necessary for our way of life and may even lead to a compromise or our civil liberties and our value designation or as a global community so we tried to assert and people seem to go with it that safety

critical industries could be a superset of the requirements and constraints on designing a journey to go from crawl to walk to run if you're playing Josh for my bingo that is the center square but the crawl walk run idea is what we decided to do in the cavalry has worked with Tesla GM with others to get them on their journey towards board need a vulnerability disclosure it's our belief that if you can create a high trust high collaboration so then you'll have better outcomes and essentially the simplest way I put this is do you have it beware of dog sign for the researchers where you have a welcome mat and the crawl idea was what's a minimum viable product that

fits on one page it can be used as a template so you can sell it to your general counsel without a lot of attached surface and red lines and entropy such that maybe you can start it on your journey one of the things we've learned from these folks is if you can never retract you can always expand but you can't ever retract so if you offer cash you can't get rid of it if you have a narrow scope you can't make it more no not really so the safety critical working groups of people seem to agree that maybe it's not a beautiful in the snow flake but if you can solve for forever days an uncatchable

vulnerability a piece of industrial controls equipment you can solve for that then it may also be useful for non safety critical attitude and we've had some consternation over this we've done some duplicate of work with our coop makes us for it but I think that Jen and Amanda's survey work has been very proper ative to some of our assumptions very much so this is a little bit frustrating for me to a collaborative process because I had this burning platform I did not that I can't collaborate but one thing's I tried to remind the group and I will die in this group is whether you have according to vulnerable disclosure program or not thanks to the BMC exception dmca

exceptions coming in october which is two months from now you're going to get a surge or a tidal wave of vulnerabilities and the data the survey shows it was at sixty-four percent are afraid of legal reprisal that fear is reduced you will see submissions and without betraying any confidences the rumor is that in the first 48 hours of GM's program with no port with no vulnerability price or not they got over 100 submissions and I asked them know if you got a lot of submissions won't say how many how many of those do you think were found in the last 14 hours and that was really the key question because you can't find an automotive lodging forty dollars these

were known but people were afraid to share so the same is true for a lot of these folks that simply reduce the barrier so what we wanted to do is show some sort of again minimum viable product and this is a template that we've used with others outside of the collaborative process and I was really trying to add spur to see that even though this process may take a while and will take some time you deserve a back we wanted to cause some sort of action prior to october first because if you want to build capacity of muscles before the flood this is the type of music so we basically need a really really ugly word doc got a lot of fighting but

I essentially said this should fit on one page and there's originally were four blocks but do a brand promise do an initial program scope keyword initial that we will not take legal action in unambiguous terms and then the mechanisms for submission and ongoing communication and then we've had to add this last one where recently because things have changed submission preferences so it doesn't affect your legal posture but kind of bugs you will and well prioritize for accept so really quickly the brand promise idea is safety is super important to us in addition to all the wonderful things we already do with our own staff and with their credit contractors who want to cast as wide an

it is possible and invite the participation of willing allies in a whole of community approach to try to find bugs and be safe or sooner together that kind of thing so this is a marketing thing that can get you a lot of people looking this is a legal thing or an engineering contract this is very much a brand of reinforcement thing people were really pleased to see GM's burger they were very pleased to see Johnson Johnson's burger so this reassures to your customers that you care about this opinion seriously the initial program scope because these are safety critical industries who have no idea how many bugs are gonna get submitted to them they don't know if

they can internally triage them we've encouraged an explicit scope reduction don't ask for bugs and all the things you've ever made maybe pick the most recent model here maybe just take one make your model of your car that way it becomes a throttling mechanism which you can expand after your pocket no one judge the Pentagon for having a 20-day pilot they did a pilot and based on what they learn they're going to do another phase in all likelihood so the idea is start narrow start on something that's newer and more modern perhaps something that you feel the engineering teams to triage and based on the received bugs and the quality of those received bugs you can

always expand scope over time by stating initial you won't be judged for this is a finished product again something like a Microsoft you'll be judged as we're beginning our journey there's also implicit scope which I will skip for now there's the most one of the most important parts is we will not take legal action if and while I don't have Archer slides apparently I have my cousins so if you if you just a pretty good exemplars here but basically we've seen people have is because he was four bullets of as long as you follow these three things these four things these six things having GM have eight things as long as their affirmative unambiguous clear people you're basically having a

covenant with researchers that if I'm willing to admit to these things that I know I won't hear your and these ones what we're finding should be fairly computable and evergreen if they change frequently people won't trust the program researchers one person expectation management is ninety percent of any human do you ever do so this is really how do you submit what are the minimum expectations for submission and one of the expectations for initial acknowledgement of receipt and ongoing communication feedings and rather than prescribing this we've outlined that the negotiation of the ongoing cated should be on a per bug her relationship basis because not all bugs are created equal but you know the actual standard does

with leaves say within seven business days acknowledge receipt and then submissions and preferences and priorities we wanted to separate separate something that's added some confusion some of these programs on day one only had four sections and then as our engineers got flooded they started adding exception after exception after exception and the researchers have stopped dead in their tracks because they didn't know where these exceptions illegal posture or were these things that we just don't want to hear about cross-site scripting and there's a bunch of things we're tackling as well about how to do change management and version control because these things should change and grow over time as you learn more but we also don't want a moving

bowl posts or talking about something like a legal posture and we've had a lot of discussion and we haven't come up with perfect solutions what we have seen how folks like Microsoft and others have tried to dampen the risk so to that and I think many of those things were just alluded to you without slides so scope one of the less obvious let me just do 30 seconds on this we cut it from the document because we had made over it but it's actually proving to be very necessary a survey I basically say white hats have five key motivations and they all start with the peat there's protectors one of those roads safer place just puzzlers do it for challenge

and curiosity there's prestige you do it to win a white jacket ER to give a keynote at Def Con there's profit to do it for money and there's protests do it forward and some political cause or ideological cause if you fail to include a cash reward it's not a failure at all I'm deliberately and overtly encouraging that the first born a disclosure program does not have a bounty attached hood and one of the reasons that is you will only attract the subset of researchers who are protectors and puzzlers they won't give you are maturing conference deadlines it won't fit over how big or small the cash prize is and when you're ready to you can always layer them in so

we've also encouraged the lack of monetary reward in fact one of the coolest rewards that anybody gets as things like this challenge point from Tesla or things like a t-shirt from another one who says I hacked my government and all I got was this lousy t-shirt people prize that t-shirt more than they would prize a small cash award that's well below their day rate the kind of work so at least for these teams critical industries we've considered we strongly encouraged the lack of one and you'll notice that GM was not criticized for not having a bug bounty program but FCA didn't get a little bit of criticism for having too small and we're in this

learning curve together and we're trying to figure it out but i think this working is a very intense very candid and the data you're seeing from jen and Amanda's work is corroborated so with that I'm going to run away right now but if what we really want is more feedback we've had some research and feedback we've had some safety-critical industry feedback and we've actually had people already published there's even though it's not done but for right now I have to run so find me join our work in your calls we have a draft you can always be better and the problems we need the most help on or how did you change management thanks thank you John and now look from

art Manning from search who has literally been at the center of this debate for 15 years now organizationally 28 I think we count it's a hard problem I mean come on it's a the problem is it's a people problem not a technology problem so you can't solve it with the technology or a protocol so art man you insert Coordination Center I'm going to represent the work of a bunch of people who are part of the multi-party disclosure subgroup of the entire process this got merged with first form of incident response and secure whatever it says up here form of instant response in security teams has a special interest group with a very similar bit of work

going on a lot of the same people doing two things people are not happy with that merged reduce duplication increase efficiency the work continues under the under the special interest group within first we get some nice administrative support from first like web XE stuff and we actually have a person taking notes so that's very nice why why look at coordinated vulnerably disclosure for multiparty why coordinated with disclosure at all vulnerabilities exists attackers attack with them against them kind of sort of sure think we're sure that disclosure is a good idea and it is an effective defense that's something we might be challenging coming up there's some general agreement on the model you find a bug you report it you wait and

the vendor fixes it and you disclose and after that people disagree pretty quickly on how long do you wait do you disclose when do you disclose how much information so framework might be four boxes but the details go all over the place so y coordinate disclosure y multi-party disclosure more vendors means more complexity I asked my more math informed colleagues they said it was not exponential growth but it is greater than linear growth when you add people to the to the mix keeping secrets gets very hard after the second person finds out about something when it's the 73rd or a hundred and twenty-fifth vendor that just gets harder and harder managing coms gets very difficult and

expectations and policies get start to construct to get become in conflict after a hope you had a lot of people we're seeing a lot of more shared code format protocols these days different supply chain relationships then you know UNIX and windows used to have years back josh was up talking about safety industries the way cars are made in the US is a supply chain what was new to me and does not fit existing models of how software gets into things so that's interesting this is also covered there are there are vendors as I will refer to them who make things they think their car manufacturers they are technology manufacturers but they don't think of themselves as that yet so GM is not new

to the planet GM is new to disclosure the process we went through to try to come up with something useful was not a survey but sort of a conceptually structured thinking process we are going to try to derive something useful from what what are with the group's real-world experiences are so different types of coordination and disclosure cases variants that come up from those cases because the exception is the rule in lots of places especially here you are always going to have a variant it's very rare you get a coordinating coordination disclosure case that's goes smoothly and perfectly for each variant what caused it how could you prevent it in the future how do you respond to it lists and lists of things

cluster those lists us sort of things rise to the top and that from there you so you get your practices and potentially further from there we're going to get principals out of this thing here's a very simple example so a very clean relatively clean case multi-party coordinated disclosure is going on variant for a vendor leaks early what's the cause they send something in plain text prevention is encryption and responses cats out of the bag everybody run so guess what someone just Josh just said this rated to heat this is a human sort of organizational problem missed expectations huge huge thing publish and follow your policy at least tell other people what's going on doesn't mean you agree it means that

people know what to expect from you or what you expect from them which is at least a starting point some common terms or common reference might help but we don't ask them going to agree necessarily on one CBD policy that works an interesting example coming out here an emerging thing openssl is a good example of this it is so difficult to pick who to tell first they basically van announcement saying there's something coming next week and then on next week on whatever day everyone finds out at the same time I think communication is huge frequent is probably better people feeling they know they're being taught to their part of the thing keep them involved and happy

less likely to leak something if the finder does not reach the vendor you have no coordination going on if the fix information does not reach the users you have no fixing going on and the whole thing is a waste of time if those things don't happen so coms is clear you can get help coordination centers can help sometimes we actually don't want to be involved if we're not needed because we have other things to do but multi-party is a case where often having a neutral someone more objective third party who's actually paid to do all that communications mess can be useful more practices have have relationships know your supply chain up and down know your

peers horizontally no other stakeholders this means reach the researchers your suppliers you want to know when somebody upstream put a new version of lib PNG and the thing that's in your dashboard you want them to tell you that so you know what you need to do or not instance of incentives we've already covered this in previous slides but chilling effects are a thing be cognizant of those I'm not telling you what to do but be aware that it exists they exist you can do you can you can do various intimidation rewards reward structures an idea that groups thrown around is to exclude repeat offenders if you always leak if you always are not playing with the social

group common social creature behavior is your you're out of the group we don't tell you next time keep clicking here whatever you do during a multi-party thing is going to be multiplied be careful with that you do especially disclosing sorts of things provide good information we like CVE or something to tie things together we like machine readable information and lastly I'm going to throw up we have the group has not worked through these principles slides so take them with a they're very very much in draft state but harm reduction being prepared you have some responsibility to inform others if you buy into this whole defensive disclosure is a good thing in the first place you

may not agree in the first place in which case these principles multiply but if you follow that these are the kinds of things we think are sort of over well overarching sort of human goals that you might want to consider feedback is happening now feedback happens is ongoing we expect to have a draft of the MPD multi-party disclosure document out for public comment you can join the first cig you don't be a member of first to join and be part of our I think twice a month calls at this point and contribute that way obviously this is open comment period you can contribute that way that's my email address if you have other questions and thank you

so I want to thank art and you can stay up here because now is the time where there were some questions earlier but we can do some quick takeaways from the survey earlier and the awareness an adoption group and really one of the reasons that we wanted to come out here is because while there has been active researcher participation of people who care about security not just because they make stuff but because they're genuinely interested in security we wanted to make sure that we heard from the security community and so there are the three working groups that you've heard from and here's a refresher of the takeaways on the the awareness and adoption group of what they found it's

behind us and so if you have thoughts and we can revisit some of the specific questions of how can we reach safety organizations organizations that are new to it and build templates and get them aware of the crawl stage how can we coordinate across multiple parties and what are some of the basic frameworks for that and and of course how do we just raise overall awareness so I'm going to stay in here and try to moderate and Jen's gonna swear at the end with an axe when it makes nice as a tweet for you why that uh so please so I know there was a question earlier but I'm going to return the mic out here

okay so first of all I have to do the legal disclaimer my opinions not my companies mostly because my regulator is sitting two seats down for me so I work for a very large pharma company that I am trying to convince to get involved in this so I have some feedback and I have a challenge for Alan and I think for Jan as well so yeah and so I know you guys are doing great work okay so i've been following what you're doing and it's fantastic and amazing however you guys need more of an eco you need to publicize what you're doing yeah yeah no it may not be a problem for some individuals but for as

a working group genuinely of serious feedback you're not selling it enough okay you guys need a website you need to do more engagement I know that NT OA doesn't have money to put behind it but we you need to find some way for me to be able to go to my VPS and my business site people and say I want us as a very large corporation that could support this to get involved in this so so the challenges are you particularly and I do a lot of work with icon as well which is also an FEI you know my boss then yeah I'm good friends Larry and so they do amazing stuff on multi stakeholder

engagement and I think there's really learnings that could come from that side over into what you guys are doing and then just from the community involvement just looking at the slides and on the feedback and the people that you got I think we need more people in my position some more healthcare vendors more automotive manufacturers more people that are part of that whole mix of companies that were manufacturing or were healthcare we are now technology providers which I totally agree with we need to work out as a group how do we get those people involved and that comes back then to the first thing is how do we sell this as something not just as a

this impacts the security and technology people within a company but this is something that the business needs to work on and I want to just call out as well because I didn't get a chance at the am the cavalry thing yesterday what Suzanne has done with the stuff with the FTA has quite literally revolutionized how security is being spoken about him i interfere here so I I have a 30-second anecdotes that I want to say I spend 10 years in many pharma companies trying to push this discussion that we're having here six weeks ago I got an email from my vice president of rag affairs and our director of QA asking me what we were doing in this place not the other

way around that 10 which is changed in the last say six months to a year so this is where we are now and I also think that the interactions between these two groups are how we're going to move forward as a security community and that's really important that we take this and grab it now and set the path for the next particular for my industry we work slowly the next ten years a nice starting now I really appreciate that is fantastic feedback thank you I'm going to let the the experts respond but I'm just going to say that the final step as we draft this process is to say okay how do we take the document that has been

built with the input of those of you who really know what you're talking about and how do we get broader adoption now in some cases we're lucky because we are the Department of Commerce and because it is a voluntary approach we can get industry to wave it at their regulators who aren't as friendly as Suzanne and and the Chamber of Commerce for example hates regulation loves this approach and any advice that you guys have in terms of how we can take this and and build it so that it's something that is not just in the tech community but outside and global because it is always hard for especially in the US government to remember that we are not the entire

world so so how do we take this global variable but it was so I would just say for any of you who were not able to catch Suzanne session they used to check out the video it was really really awesome and she has done incredible at the FDA and and they are certainly to my knowledge the regulatory body that is most ahead with adopting site security norms for their their vertical and it is super inspiring to see honestly so you should definitely check out her talk in terms of what you're saying one of the things that a manner and I have talked about fair amount is at some point ntia has to move on you know that for them to

continue forever is not the best use of taxpayers dollars and as the example of representative taxation with no representation I want Alan to be doing other things so I i think you know at some point there has to be a moment where we decide that this can stand on its own two feet without ntia and so you know something that a matter and I have talked about is for adoption you know as I said at the beginning when we first start talking about surveys our whole thing was about coming up with ways to drive adoption it wasn't about coming up with surveys the surveys were means to an end and they ended up taking so much

time there now at this point where we're kind of like we hope that this doesn't lose momentum we hope that people don't feel like the surveys were it and it's done and then they just like our well someone else will figure out option and move on like this is what it's all for so now we have to solve that problem and we have to think about how do the people who are interested in this topic continue to work together and I agree with you that there needs to be some kind of centralizing point that people focus on I think the challenge is when you have shared ownership how you do that and how you can let's get a

particular when you don't have a lot of resources behind it so it's definitely something that we have to think that I agree with you completely on that front but as kapadia is going to be a challenge I'm a question back in a way I'm guessing that it's probably all of the above but you mentioned that your VP of regulatory affairs is someone who is coming to you but I ideally it would be like the different business perspective coming to you and saying for our reputation or for the value of our product or so on we are also concerned in engage so my question back is what is the way to reach that maybe is it going

and talking about others in your industry like peers that are doing this well what they're getting from it is it just more publicity like what are one of the mechanisms that are going to be meaningful for those bp's we're an eye we're gonna Mike you for if those watching at home thank you so two points from my industry we are pretty much all year zero companies on this and so we're probably not the best example of that I will say one thing though my VP of regulatory affairs would not be talking about this if our CEO and board and everything we're not asking questions about it it's just not on her radar it's just not there so it is being talked

about at that level and for us that's because of FDA but from across the industry and this comes back to I think we're still in the security researchers and technologists talking about this and the comms piece that we need to move on be it when we take ownership of it ourselves is how are we showing the business value in this and not in you know return on investments or anything else it's we need to keep really simple messaging to you know sea level people that here is the impact that this will have and like a company like Microsoft is probably a great example of that that of it is really been ingrained into company culture and you know you guys

probably have experiences from 10 15 years ago when you started that process and how you solve that and that's something that we need to then take over and say okay this community thing that started with the NTIA but is now evolving into a community-based set of resources or working that's going on how do you sell that to your company how do i as a security person within a large corporation say no longer do I want to try and get involved in this as just me I want my company to support this I want my company to send me to the meetings I want them to put ten fifteen percent of my time to that and to that I have to sell a business

case first not a technologists case and that's the messaging that would need to change the work and stay the same but the messaging would have to tweak a little it just got thrown real quickly since the car guys not around in the USDOT and it's are there signs of progress there as well and not as far along I'd say as the FDA if I had to somehow measure them but there's some you know the the vehicle safety regulators are looking into it in the US so that's a good sign as well and I will also add that you know art has been doing this for a very long time and if we are busy at the Department of

Commerce on cyber security cert is really busy right now and the fact that they're engaging this I think is for me a signal that says that at very least they want to make sure that we don't ruin it but I think that it's going to be something that they can then take to the broader community together with universities like the chamber of commerce and and we're also talking to you know the auto trade groups who like to communicate and be able to speak as one voice to their regulator to say hey we're moving forward on this so think about what we're already doing before you go and impose something that may be quite dangerous because early discussion

that we saw coming out of the auto sector you know there was never anything very official but there were some signs that said that they were proposing you know treating security flaws the same way you treat some of the existing safety flaws which are huge events for a company and there they make a company not want to learn about it and not want to affirmative lee engage

so there was discussion about the disagreement between the industry on when you report something to the vendor or company and then when you disclose it and we've seen a lot of back and forth between Google and Microsoft on this very matter so Google has that policy of you have exactly 90 days and then Microsoft has the policy of we only patch on patch Tuesdays do you think there should be more give between one or the other such as Microsoft should be so specific on patching or Google shouldn't be so stringent on 90 days or do you think that they need to find some sort of better middle ground that's policy on myself so it's a fun debate certainly in

a sense I don't know how he super useful it is so so actually so here's because actually to take your example I'm pretty sure if I remember after that whole thing happened who will modify their policy so they now have an extra to get a two-week kick at the end so that's an example of policy we have a policy you mutually what it was we disagreed a premature disclosure happened and then somebody adjusted their policy so things things shifted to work a little better together there are so many people and organizations to so many ideas different business practices different release cycles we're not going to get agreement I'm not even going to try to get you

know what's the exact number of days we surveyed a little bit and the average was ins in the 60 department but that's that's just what's actually happening and it's not you know it's not being a good number um yeah right we write we write 45 days we almost never almost never do 45 days that's our we couldn't get back we didn't hear from the vendor in 45 so we'll drop 45 on you instead of 0 so it's a fun debate I'm not sure how super useful it is to try to figure out what that embargo period should be I think you're not going to get agreement there's a lot a lot a lot of phoner

abilities there are too many to have to really almost care about individual ones it's very fun too don't get me wrong I love it sort of but things there are so many we need to just it needs to be all this in a process not that specific thing that Tavis was yelling at my foul on this day it was like everybody stop and pay attention to that it's a there's more more scale than that yeah I'll just add that I think also that I think that the reality is that a disclosure is a collaboration and in any collaboration there has to be some give-and-take there has to be willingness to find common ground and so

that works pervert provided that both sides are collaborating right so I think the reason that you set a timeline is exactly as I said for the situations where people are not collaborating when you're not getting a response and nothing's happening but I think if you are getting response if you are finding that there is engagement then the reality is to hold on to a deadline for just like because the salon deadline is and we're going to do it is actually not putting the users first it's not really sort of thinking about how you it's not really thinking about the risk model in the big picture sets and it also is not going to create the the trust and the

benefit of the doubt between the parties that is going to lead to the best possible outcome so we like just obviously two hours and we work very closely with that we push all our disclosures reset we do a lot a disclosure either for our own research or four members of the metasploit community so our published policy is we go to the vendor we give them 15 days and then we go to cert and we follow certs 45 days they get 60 days in total but what typically happens is if there is engagement then we talk to Sir we talk to the vendor and we agree on a time line that makes sense and it's normally less than 90 days depending on

the sector and how complex it is but we are all agreed that will be flexible because at the end of the day that's the best thing for the users and that's what we care about would you say the specific complication there was just then Google and Microsoft not collaborating well absolutely it was TV companies not going to bow to each other sorry I think the yeah if you start with the framing that we all care about security then you can have the productive discussion where you know what about the shadow of the future will if you say it takes a long time to fix it now what about the next time it helps unpack why people have the

policies they do and it lends itself to compromise as long as you started saying hey this this is about protecting people can I ask you do differentiate privacy and security somehow I can refer you to the fifth chapter of my dissertation which was called privacy securing the dynamics of networked information sharing probably a little dated by now you the it is a flashbacks of trying to convince Harvard to laminate Creative Commons in pagosa so the the challenge is there a couple of tensions there which sometimes there really are concerns especially because oddly enough we often have more privacy regulations than we have security regulations and so you might have a vendor in fact when we

were talking to companies initially as proposed miss uh we had vendors a listen I have data and I really want someone to tell me when that data is not secure but if in the process of finding out that that data is not secure they access that data that triggers a compliance event that could cost me hundreds of thousands if not millions of dollars and so again if our goal is to incentivize organizations to take security seriously and to work with those outside that's where it gets to what can a researcher know about how they should engage the best companies and that's something that we've tried to work with of bringing the company perspective in of how do you what why

don't they just engage and why do they lawyer up so often and often it's because there are real risks that they're worried about I think a lot of the work that we've done and the stakeholder have done is like to say stand down let's let's get the lawyers out of this and focus on security you guys have thoughts on privacy just that I think if you're talking about in terms of disclosure then it does change the game a little bit because if there has been a violation of privacy standards then it's imposes a timeline there are state notification requirements and they have specific timeline some of them and so if you as a researcher have

accidentally accessed PII or ph I then then that does change the game somewhat in terms of your disclosure and you need to be aware of what that is and the vendor will be aware of what it is and the chances are if you've done that it will also change the vendors response vendors tend to be much more sensitive to disclosures when it's going to trigger an action from them that has regulatory consequences so that's just something to be grateful yeah the reason why I'm asking because I'm from Europe we are getting the new EU regulation in 2018 and it's a 72 hours to report it so it's a quite tough thing yeah the gentleman from Fitz ok again i'm going

to do my disclaimer no not my company's opinion mind and so as somebody who deals with insane amounts of PII I actually think that it's one of the reasons that this is so important because if you have a defined process that researchers so researchers are going to try to try and break into my [ __ ] anyway if we have a process out there that at least set some boundaries and some guidelines and bright lines that I can point to and say okay if you want to try and break into my stuff for good reasons then look at this because i can say then ok this set of stuff has PII in it and if you're going to break

into it well at least give me a heads up that you're going to try so that we can meet things like a 72-hour disclosure deadline because if you're breaking in and then you send me something and it's going to take me tell me two hours to evaluate at them it's too late for me whereas if we set that expectation for security researchers upfront as part of a whole framework then suddenly it makes my life on the pii side a lot easier so it's not that the two are in competition but particularly for those of us who store insane amounts of PII they're actually intrinsically linked practices for vendors but there are also best practices for researchers and we as

a community are allergic to the idea we hate the idea of conformity of some kind particularly the idea that that could come from vendors that seems like the worst possible thing I mean God were not generally what's going on and like that's kind of a crazy thing for us today and so you know actually sending a baseline of like you know we were talking yesterday in the cfa session that we and i made a comment that ended up being like more contentious than my readership was which is that whilst Weavin and i'm not going to go into if you guys in off me with the case you can google it but he well it shouldn't set

case law that what he did is a violation of the survey at the same time the number of times that he access or not him but his buddy accessed the if the creds they got that they collected that is not a proof of concept a proof of concept is like 23 times you don't need to do a hundred thousand times two hundred thousand times and so I think it's completely fine for a vendor to say hey you know what like we're going to make it so that you have to do this on a minimal level if you do more than that i'm not going to be as friendly as I would be four colleagues in Europe the

Dutch have been incredibly forward-leaning and thinking in they still use the term responsible disclosure which might be unfortunate i don't think we're going to lose that term on the European side but on the other hand what they've done has actually they have national policy all the way down to prosecutorial discretion saying if a researcher you know has engages in this behavior avoids these things that that are clearly bad and is doing it with good intent then prosecutors have to stand down and I think that's very powerful yeah and I'd like to just to throw in here a bit the site I also believe there are you know they're responsible ethical researcher guidelines I mean they're they're things

researchers you know the quote should follow more I think than just being told to conform and I sorry I'm speaking to the choir up here but if there were such a guideline written down somewhere and published that can be used against a researcher to say you fell outside of the guidelines guess what the safe harbors out here comes every every law we can throw at you so the concern is you know how how how bright those lines are and if that's out there you're your it can be flipped around on basically system policy judo versus researchers as it gets done right yeah nonetheless we advocate for like yeah first time you get the pii you got

it stop right and tell someone that indicates your intent of not downloading a thousand people stuff just the one compact other questions comments feedback what do we do yes non vertical space I could say like the FTC has a few cases that have said that companies that fail to have accurate vulnerability disclosure processes possibly fall in the realm of unfairness counts again on FTC act so the recent azuz little clutter good ok so the recent like a Zeus case was the router security case and part of that unfairness count included their failure to have an adequate vulnerability disclosure process so outside of the kind of regulated verticals I think the FTC is trying to kind of make statements

about vulnerability disclosure yeah I completely agree with that i think that the FTC's like sort of enforcement capabilities will will will over time drive a shift in culture it's just going to take a long time whereas I think the vertical sector regulatory authorities will have more direct short-term impact for those specific verticals sorry but right exactly and the thing and the problem is awareness that's the challenge is like there isn't all good enough awareness you get the letter from them you become aware but is that how we want to do this so I don't see the time I mean that is he got very least it's a bit it's at least shows that there's a

starting base line for sure I mean I'm I'm 13 you might you might get some traction out of the example right so every other crappy router vendor with no security now it looks hey ASA scalded up for this yeah maybe we should take the steps before we also get lit up because we do just poor job as needed yeah I mean I hope so for sure yessir other suggestions how do we save the world safe chilly no such an old reference that was surprisingly dated it didn't didn't know that was the liner always little dated references

if you'd left it on the artists life may be much more talkative so who's here oh good no I don't actually particularly have a question I wanted to get Jen to expand a little bit more on the because it does Archer obviously that goes without saying but the the sort of change that the researchers side needs to be cognizant of because when you're talking about automobiles and installations in oil refineries those are not things that fit a hard-and-fast 90-day completely great and even once you've gotten past the disclosure you look at cart I mean I go buy a car today that has not had its airbag replaced or does not have all the modern safety

stuff so from the researcher side we're in a totally new yeah area where we can inadvertently put people at risk yeah while following all the ethical rules I agree like I think it's the Wild West all over again and I think what's interesting is that we've entered a sort of new point in the culture of the community I kind of think of it as like sort of the security community tudo which is terrible but it's almost like you know we had our generation alpha and now we're in a in the sort of next generation and we've we've evolved or we've matured out from being from skinnies who were just curious and wanted to know how it worked and test

limits suddenly being like people who are buying cars and putting our kids in them and going holy [ __ ] like there's actual implications with this stuff and so I think you know a lot of the researchers I talked to feel the weight of that they actually you know going back to that fifteen percent caring about the money a lot of the researchers I talk to you are doing this because they want to save the world like when it comes down to that you really care about the impact to consumers yes and so Krieger is not a good good example of this not one sec um and so I think I think that you know that is

difficult like we are sort of making up as we go as Josh said like we have to find the new paradigms of what that looks like and even though rapid7 that's the process we're going through right now is as I said we have this disclosure policy it's been a disclosure policy for the five years I've work to the company and right now we are internally looking about going we need to now have a category of this is this except when it comes to things that affect human life and then it's going to be like what does that look like how do we set time lines when you're dealing with cars right it's a totally different paradigm we have no

clue what that looks like but I think the only way to get there is to work together and when I say work together I don't just mean within the research community although that for sure but we have to work with water makers we have to get to understand what their process is we know what software development looks like a lot of us have worked in software development we don't know what automaking looks like no clue so Anna like insane to medical devices you know we need to understand what that involves and the only way to do that is to work with them and have empathy for what they're going through and if you approach that as you know you're the bad

guys mind a good guy never going to achieve anything I set that the right way around right yeah sometimes i'm gonna sell high taking said they were yeah so we like we really need to find that common ground I think that's the first step

do I get an award for that so one thing that that art did not mention that that some of the the smart people in his group have been thinking about is there will be instances when you will disagree on how to handle things and that doesn't that still doesn't mean that it go back goes back to the Wild West and we should have anarchy there still are some things that we can do to still have good practice in that case or do can I put you on the spot to share some of the thoughts that you guys have had so this is this is disagree the agree to disagree how do you make that constructive well you may not I mean it

it may come down to not agree and me simply not agreeing so the Google Microsoft example Google Google win right that happens sometimes one of the parties involves pulled to the ground business tired of waiting values it puts different value on the impact of the vulnerability or the the safety impact of the world over it might be you know chances of it being actually attacked in the wild chances of an attack being successful time to deploy fixes our attacker is going to bother with a very complicated attack when phishing emails are still working what kind of attack are you looking at broad opportunistic attackers or a nation-state targeted attacks people have different opinions all those things and that can come down

to how bad is that they think something is which causes them to say it's fine to deflate disclose now or in less weight to it to some extent the dumb answer is you have to accept you're going to lose out sometimes your opinions not going to carry the day and be able to deal with that you know it's it's it's largely if you have the information about the vulnerability you generally legally have the ability to publish it at least in the US Machar that's quite the same you but you know publishing details of a bug is free speech in the u.s. attacking someone with it is not is typically illegal so I don't know what to say

really except I mean it's certain i'm trying to think of our cases we accept that we will ask people will ask nicely and repeatedly and make good arguments to do certain things delay or extend or hurry up and publish or ebony to fix out and we try to not surprised people you know if we're gonna if we're going to be the ones pulling the trigger we will say look we're publishing in 24 hours you know we're not done negotiating that's that's going to be our date so you can not surprised people and you can accept that you're not going to win the negotiation every time you can negotiate it for us and try to achieve consensus I

don't know that I've got any any magic that was there was the not surprising in the trying to negotiate the things that I was sorry about yeah

along the lines of how to speed adoption or make it more companies adopt especially those companies outside of the technology industry you need to actually take a step back and take one of those companies and have them look at this this is terrifying to them they've been making you know widgets for all of these years and all of a sudden you've got a punch of crazy black headers running at them regulation coming everywhere and actually making them understand that the security research community we're not coming to get them and obviously as your survey shows we're really not coming to get them we're really trying to make a better a better world they're seen the guy hacking an

airplane they're seen you know these crazy people you know trying to free the world's information they think making them under skates and yeah yeah making them understand that this this room this is actually who's out there trying to help them not her to thank you any media story with the picture of the scary looking you know it's just gone past well yes i think i saw a mask recently yeah i mean how many people like worrying about a claw i always do I and she can't see the keyboard properly without one yeah I think that this fear on both sides and the problem is fear makes people behave in irrational and defensive and overlays right and MP just come here X down I

agree so who thought that what we said today was fairly common sense so that's actually this is this is the sororities I think that it means that this is not a contentious issue who is worked with a vendor before to to actually say hey I found something got a couple hands has anyone been on the receiving side mooreville I won't put anyone on the spot but is there anyone who's been on the receiving side who wants to share some experiences or react to what we said today yes and it's hard without a process at the simplest level because there are people out there that want to do things and there are so many industries not just my own but others as

well that if there's not a process there when you get a revealer ability and reporting to your customer service email address or something and you have no idea what to do with it at the company has no idea what to do with it I worked in a company in number of years ago where I got an email about it because in my internal company profile thing I had interested in vulnerability disclosure on and somebody had literally googled on her internal portal and it came to me so that's where so many companies are so without the work that you guys are doing it's hard and we need to make it easier because then it becomes easier for

everybody like people are super busy they have processes of how they to do things like generally people are not bad people they just they have things that they've been told these your priorities these your goals and your happily working away as an engineer and you've won a delivery deadline and you know that if that delivery deadline slips then that has a high impact for the business and that's a problem and as going back for the customer as well and so when somebody comes to you and says hey we're going to blow up your schedule we're going to divert you on to this other task on this thing that you thought you were dealt with ages ago and

it's going to completely impact all your other work like that's frustrating ab fast and at worst it's like it it makes people in secure it makes an anxious it's difficult there are no like people doing the stuff or typically there are not a lot of people doing the stuff who would just mare about it I'm when the same like we get vulnerabilities flourishes and you know there are times when I have to have a conversation with pizza fly geyser I'm like okay you've done the patch have we put it out yet and they're kind of like oh we have a release going out on a week and I'm like you don't get there were a security

company right and it's not like they don't care they do it's just that people have this whole thing of like this is the process that we go through we always do a release on a specific date microsoft has tuesday like it's the same mentality and so you you just that's how you think and people don't like change very much so when you're like hey we're going to do this completely different thing that you've not done they kind of sit there scratching their heads so for those of you who would like to engage further there's still a lot of work to be done please talk with art talk with Amanda and Jen talk to me tomorrow the eff is hosting an event to

continue this discussion I'm particularly proud of the fact that the eff in the Department of Commerce I said listen this is something that we need to actually work together on and and and you know find some solutions that work well for all involved and I'm going to give a very quick plug if you really like the idea of you know talking with other security people about processes our assistant secretary just announced our newest multi-stakeholder process on IOT security and it's closely related to this so the challenge is there's all sorts of smart consumer stuff out there and not all of it is built for security so how do we promote consumer awareness of security and how to reward

manufacturer attention to security we're gonna be starting with aftermarket security taking a page from our colleagues at the FDA and saying listen patch ability is important but there's no real understanding of what it means for something to be patchable so let's have a discussion around the technical definitions of patch ability and then figure out a way to condense that into a way that it can be easily communicated to the consumer the consumer is something quickly to look for here are some small words that I can understand and look for in a package but they are supported by a well-defined technical description that can be used to build standards or built to spec on so trying

to derive it from the demand side and the supply side and if you're interested in that I'm very happy to talk further so please engage thank you for your time and attention I'm guessing that you have a last word because you always do in this case but I just want to thank everyone who's participation thank you very much and I want to thank the three people up here who have given a lot of work they have well their day jobs are far beyond what they already have time for and this is something that is on top of that and it works because people who care passionate about these issues get engaged so please i invite you to also

get engaged thank you